Use Clarifai and LangChain to create text embeddings

Embeddings create a vector representation of textual content. This is beneficial because it implies we can conceptualize text within a vector space, and facilitate tasks such as semantic search where we look for pieces of text that exhibit the highest similarity within that vector space.

Let’s illustrate how you can use LangChain to interact with Clarifai models and create text embeddings.

Prerequisites

• Python development environment
• Get a PAT (Personal Access Token) from the Clarifai’s portal under the Settings/Security section
• Get the ID of the user owning the model you want to use
• Get the ID of the app where the model is found
• Get the ID of the model you want to use. Text embedding models can be found here
• Install the Clarifai Python SDK by running pip install clarifai
• Install LangChain by running pip install langchain

You can learn how to authenticate with the Clarifai platform here.

Here is how you can create text embeddings.

• Python

#################################################################################################
# In this section, we set the user authentication, user and app ID, model ID, and the text
# we want to use to create embeddings. Change these strings to run your own example.
#################################################################################################

# Your PAT (Personal Access Token) can be found in the Account's Security section
PAT = 'YOUR_PAT_HERE'
# Specify the correct user_id/app_id pairings
# Since you're making inferences outside your app's scope
USER_ID = 'cohere'
APP_ID = 'embed'
# Change these to whatever model and text you want to use
MODEL_ID = 'cohere-text-to-embeddings'
# Optionally, you can provide a specific model version as the model_version_id arg
# MODEL_VERSION_ID = 'MODEL_VERSION_ID'
RAW_TEXT = 'This is a test document.'

############################################################################
# YOU DO NOT NEED TO CHANGE ANYTHING BELOW THIS LINE TO RUN THIS EXAMPLE
############################################################################

# Import the required modules
from langchain.embeddings import ClarifaiEmbeddings
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain

# Create a prompt template to be used with the LLM Chain
template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate(template=template, input_variables=["question"])

# Initialize a Clarifai embedding model
embeddings = ClarifaiEmbeddings(pat=PAT, user_id=USER_ID, app_id=APP_ID, model_id=MODEL_ID)
text = RAW_TEXT

# Call out to Clarifai’s embedding models
query_result = embeddings.embed_query(text)
doc_result = embeddings.embed_documents([text])

print(doc_result)

Output Example

[[0.011901138350367546, -0.0018139687599614263, 0.04467465728521347, 0.002624698681756854, -0.013935787603259087, ...
... -0.010991679180920601, 0.011240349747776985, 0.024154152169704437]]

You can explore the LangChain documentation to learn more on how to use the framework with Clarifai for text embedding tasks.