# Use Clarifai and LangChain to create a vector store and perform searches --- Clarifai offers a powerful, built-in [vector database](/content/blog/using-clarifais-native-vector-database/index.html) within its AI platform. It is designed to store and retrieve large amounts of unstructured data quickly and accurately. The Clarifai platform uses an embedding model to automatically index any piece of data uploaded to it. It creates high-dimensional vectors that are kept in the vector store, from where you can efficiently search for and retrieve data based on these vectors. Let’s illustrate how you can use Clarifai and LangChain to [retrieve text data](https://docs.clarifai.com/portal-guide/psearch/text-search) based on their content and semantic similarity. ## Prerequisites - Python development environment - Get a PAT (Personal Access Token) from the Clarifai’s portal under the [Settings/Security](/content/settings/security/index.html) section - Get your Clarifai’s user ID - Get the ID of the text app where the text data will be uploaded. _Ensure your text app has the appropriate base workflow for indexing your text documents, such as the Language-Understanding workflow_ - Install the [Clarifai Python SDK](https://docs.clarifai.com/python-sdk/sdk-overview) by running `pip install clarifai` - Install LangChain by running `pip install langchain` You can learn how to authenticate with the Clarifai platform [here](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens). ## Similarity Search on Uploaded Texts You can create a Clarifai vector store from a list of texts. You can upload each text with its respective metadata to a Clarifai application. You can then use your Clarifai application to perform a semantic search and find contextually relevant texts. Here is an example. - Python ```python ###################################################################################################### # In this section, we set the user authentication, user and app ID, and the text data we want to # upload to the specified app. Change these strings to run your own example. ####################################################################################################### # Your PAT (Personal Access Token) can be found in the Account's Security section PAT = 'YOUR_PAT_HERE' # Specify the correct user_id/app_id pairings USER_ID = 'YOUR_USER_ID_HERE' APP_ID = 'YOUR_APP_ID_HERE' # Specify the texts to upload texts = [ "I really enjoy spending time with you", "I hate spending time with my dog", "I want to go for a run", "I went to the movies yesterday", "I love playing soccer with my friends", ] # Optionally, specify the number of documents to return NUMBER_OF_DOCS = 4 ############################################################################ # YOU DO NOT NEED TO CHANGE ANYTHING BELOW THIS LINE TO RUN THIS EXAMPLE ############################################################################ # Import the required modules from langchain.text_splitter import CharacterTextSplitter from langchain.document_loaders import TextLoader from langchain.vectorstores import Clarifai # Specify list of metadatas metadatas = [{"id": i, "text": text, "source": "book 1", "category": ["books", "modern"]} for i, text in enumerate(texts)] # Create a Clarifai vector store from a list of texts clarifai_vector_db = Clarifai.from_texts( user_id=USER_ID, app_id=APP_ID, texts=texts, pat=PAT, number_of_docs=NUMBER_OF_DOCS, metadatas=metadatas ) # Run similarity search using Clarifai docs = clarifai_vector_db.similarity_search("I would love to see you") print(docs) ``` Output Example ```text [Document(page_content='I really enjoy spending time with you', metadata={'text': 'I really enjoy spending time with you', 'id': 0.0, 'category': ['books', 'modern'], 'source': 'book 1'}), Document(page_content='I want to go for a run', metadata={'text': 'I want to go for a run', 'source': 'book 1', 'category': ['books', 'modern'], 'id': 2.0}), Document(page_content='I went to the movies yesterday', metadata={'text': 'I went to the movies yesterday', 'category': ['books', 'modern'], 'id': 3.0, 'source': 'book 1'}), Document(page_content='I love playing soccer with my friends', metadata={'category': ['books', 'modern'], 'id': 4.0, 'text': 'I love playing soccer with my friends', 'source': 'book 1'})] ``` You can also leverage metadata filters to enhance the filtering capabilities within an app. ```python # This one will limit the similarity query to only the texts that have key of "source" matching value of "book 1" book1_similar_docs = clarifai_vector_db.similarity_search("I would love to see you", filter={"source": "book 1"}) # You can also use lists in the input's metadata and then select things that match an item in the list. This is useful for categories like below: book_category_similar_docs = clarifai_vector_db.similarity_search("I would love to see you", filter={"category": ["books"]}) ``` ## Similarity Search on Uploaded Web Documents You can create a Clarifai vector store from a list of web documents. You can upload each document with its respective metadata to a Clarifai application. You can then use your Clarifai application to perform a semantic search and find contextually relevant documents. Here is an example. - Python ```python ########################################################################################################## # In this section, we set the user authentication, user and app ID, model ID, and the text # documents we want to upload to the specified app. Change these strings to run your own example. ########################################################################################################## # Your PAT (Personal Access Token) can be found in the Account's Security section PAT = 'YOUR_PAT_HERE' # Specify the correct user_id/app_id pairings USER_ID = 'YOUR_USER_ID_HERE' APP_ID = 'YOUR_APP_ID_HERE' # Specify the URLs of the documents to upload URLs = ["https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023", "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023"] # Specify the text to search for TEXT_SEARCH = "Texts related to Russia attacks" # Optionally, specify the number of documents to return NUMBER_OF_DOCS = 4 # Import the required modules from langchain.text_splitter import CharacterTextSplitter from langchain.document_loaders import UnstructuredURLLoader from langchain.vectorstores import Clarifai # Load files from remote URLs using the unstructured library loader = UnstructuredURLLoader(urls=URLs) documents = loader.load() # Split text into chunks text_splitter = CharacterTextSplitter(chunk_size=2000, chunk_overlap=0) docs = text_splitter.split_documents(documents) # Create a Clarifai vector store from the documents clarifai_vector_db = Clarifai.from_documents( user_id=USER_ID, app_id=APP_ID, documents=docs, pat=PAT, number_of_docs=NUMBER_OF_DOCS ) # Run similarity search using Clarifai docs = clarifai_vector_db.similarity_search(TEXT_SEARCH) print(docs) ``` Output Example ```text [Document(page_content='https://t.me/zoda_gov_ua/16505\xa0\xa0;\n\nhttps://t.me/vilkul/2680\xa0\xa0;\n\nhttps://t.me/Yevtushenko_E/2419\n\n[48]\xa0https://t.me/mykolaivskaODA/4236\xa0;\xa0\xa0https://www.facebook.com/GeneralStaff.ua/posts/pfbid0FuH223o7wLNSiJSNdCX...\xa0;\xa0\xa0https://www.facebook.com/GeneralStaff.ua/posts/pfbid02kL8XZwXNsSUphpcF5S...\xa0;\xa0https://t.me/khersonskaODA/3607\n\n[49]\xa0https://podolyaka dot ru/2023/02/08/zayavleniya-premer-ministra-rf-mihaila-mishustina-o-podderzhke-uchastnikov-svo-i-voennoy-promyshlennosti/; https://stolica-s dot su/archives/366231; https://t.me/rybar/43402\n\n[50]\xa0https://isw.pub/UkrWar020723; https://t.me/concordgroup_official/425; https://t.me/Prigozhin_hat/2612\n\n[51]\xa0https://www.understandingwar.org/backgrounder/russian-offensive-campaign...\n\n[52]\xa0https://t.me/agentstvonews/2482\n\n[53]\xa0https://t.me/agentstvonews/2482\xa0;\xa0https://isw.pub/UkrWar020123\n\n[54]\xa0https://t.me/turchak_andrey/1032\xa0;\xa0\xa0https://isw.pub/UkrWar122922\n\n[55]\xa0https://t.me/turchak_andrey/1032\n\n[56]\xa0https://t.me/turchak_andrey/1032\n\n[57]\xa0https://t.me/SALDO_VGA/363;\xa0https://t.me/SALDO_VGA/365\n\n[58]\xa0https://t.me/SALDO_VGA/363;\xa0https://t.me/SALDO_VGA/365\n\n[59]\xa0https://www.understandingwar.org/backgrounder/russian-offensive-campaign...\n\n[60]\xa0https://t.me/pushilindenis/3165\n\n[61]\xa0https://t.me/glava_lnr_info/745;\xa0https://t.me/glava_lnr_info/748\n\n[62]\xa0https://t.me/glava_lnr_info/745;\xa0https://t.me/glava_lnr_info/748\n\n[63]\xa0https://t.me/modmilby/22455; https://t.me/modmilby/22419\n\n[64]\xa0https://www.osw.waw.pl/sites/default/files/OSW-Report_Russia%E2%80%99s-B...\n\n[65]\xa0https://t.me/modmilby/22470\n\nTags\n\nUkraine Project\n\nFile Attachments:\n\nZaporizhia Battle Map Draft February 08,2023.png\n\nKherson-Mykolaiv Battle Map Draft February 08,2023.png\n\nDonetsk Battle Map Draft February 08,2023.png\n\nDraftUkraineCoTFebruary08,2023.png\n\nKharkiv Battle Map Draft February 08,2023.png\n\n1400 16th Street NW, Suite 515 Washington, DC 20036\n\nph (202) 293-5550\n\n©2007 – 2023 THE INSTITUTE FOR THE STUDY OF WAR', metadata={'source': 'https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023'}), Document(page_content='[37]\xa0https://www.facebook.com/GeneralStaff.ua/posts/pfbid02opH8aKBK88janDxszu... https://twitter.com/IntelCrab/status/1623561968626941953?s=20&t=sp6Y-hSy... ; https://twitter.com/IntelCrab/status/1623562915331690500?s=20&t=sp6Y-hSy... ; https://twitter.com/IntelCrab/status/1623563711859290113?s=20&t=sp6Y-hSy... ;\xa0https://t.me/zoda_gov_ua/16537\xa0;\xa0https://t.me/skarlatop/976\xa0;\xa0https:...\n\n[38]\xa0https://t.me/Yevtushenko_E/2427\xa0;\xa0https://www.facebook.com/GeneralStaf...\n\n[39]\xa0https://t.me/grey_zone/17129\n\n[40]\xa0https://www.understandingwar.org/backgrounder/russian-offensive-campaign...\n\n[41]\xa0https://59 dot ru /text/gorod/2023/02/07/72040580/?utm_source=telegram&utm_medium=messenger&utm_campaign=59\n\n[42]\xa0https://59 dot ru /text/gorod/2023/02/07/72040580/?utm_source=telegram&utm_medium=messenger&utm_campaign=59\n\n[43]\xa0https://www.politnavigator dot net/my-vse-budem-udivleny-kogda-nachnjotsya-nastuplenie-rogov.html\n\n[44]\xa0https://isw.pub/UkrWar020223\n\n[45]\xa0http://kremlin\xa0dot ru/events/president/news/70482\n\n[46]\xa0http://kremlin\xa0dot ru/events/president/news/70482\n\n[47]\xa0https://t.me/pushilindenis/3166\n\n[48]\xa0https://t.me/BalitskyEV/778\xa0;\n\xa0https://t.me/BalitskyEV/778\n\n[50]\xa0https://isw.pub/UkrWar011323\n\n[51]\xa0http://kremlin\xa0dot ru/events/president/news/70482\n\n[52]\xa0http://kremlin\xa0dot ru/events/president/news/70482\n\n[53]\xa0https://t.me/vrogov/7593;\xa0https://t.me/vrogov/7582;\xa0https://t.me/rea...\n\n[54]\xa0https://t.me/vrogov/7593\xa0;\xa0https://t.me/rybar/43417\n\n[55]\xa0https://www.understandingwar.org/backgrounder/russian-offensive-campaign...\n\n[56]\xa0https://t.me/modmilby/22518\n\n[57]\xa0https://t.me/modmilby/22515\n\n[58]\xa0https://t.me/grey_zone/17123\n\nFile Attachments:\n\nZaporizhia Battle Map Draft February 09,2023.png\n\nKherson-Mykolaiv Battle Map Draft February 09,2023.png\n\nDonetsk Battle Map Draft February 09,2023.png\n\nKharkiv Battle Map Draft February 09,2023.png\n\nDraftUkraineCoTFebruary09,2023.png\n\n1400 16th Street NW, Suite 515 Washington, DC 20036\n\nph (202) 293-5550', metadata={'source': 'https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023'})] ``` ## Similarity Search on Existing Applications Most users are likely to have already added data to their Clarifai applications before engaging with LangChain. In this example, we will leverage the existing data within an application to perform a semantic search and find relevant documents. - Python ```python ###################################################################################################### # In this section, we set the user authentication, user, and ID of the app already existing on # Clarifai. Change these strings to run your own example. ####################################################################################################### # Your PAT (Personal Access Token) can be found in the Account's Security section PAT = 'YOUR_PAT_HERE' # Specify the correct user_id/app_id pairings USER_ID = 'YOUR_USER_ID_HERE' APP_ID = 'YOUR_APP_ID_HERE' # Optionally, specify the number of documents to return NUMBER_OF_DOCS = 4 # Specify the text to search for TEXT_SEARCH = "Texts related to criminals and violence" # Import the required modules from langchain.vectorstores import Clarifai clarifai_vector_db = Clarifai( user_id=USER_ID, app_id=APP_ID, pat=PAT, number_of_docs=NUMBER_OF_DOCS, ) # Perform the search docs = clarifai_vector_db.similarity_search(TEXT_SEARCH) print(docs) ``` Output Example ```text [Document(page_content='[37]\xa0https://www.facebook.com/GeneralStaff.ua/posts/pfbid02opH8aKBK88janDxszu... https://twitter.com/IntelCrab/status/1623561968626941953?s=20&t=sp6Y-hSy... ; https://twitter.com/IntelCrab/status/1623562915331690500?s=20&t=sp6Y-hSy... ; https://twitter.com/IntelCrab/status/1623563711859290113?s=20&t=sp6Y-hSy... ;\xa0https://t.me/zoda_gov_ua/16537\xa0;\xa0https://t.me/skarlatop/976\xa0;\xa0https:...\n\n[38]\xa0https://t.me/Yevtushenko_E/2427\xa0;\xa0https://www.facebook.com/GeneralStaf...\n\n[39]\xa0https://t.me/grey_zone/17129\n\n[40]\xa0https://www.understandingwar.org/backgrounder/russian-offensive-campaign...\n\n[41]\xa0https://59 dot ru /text/gorod/2023/02/07/72040580/?utm_source=telegram&utm_medium=messenger&utm_campaign=59\n\n[42]\xa0https://59 dot ru /text/gorod/2023/02/07/72040580/?utm_source=telegram&utm_medium=messenger&utm_campaign=59\n\n[43]\xa0https://www.politnavigator dot net/my-vse-budem-udivleny-kogda-nachnjotsya-nastuplenie-rogov.html\n\n[44]\xa0https://isw.pub/UkrWar020223\n\n[45]\xa0http://kremlin\xa0dot ru/events/president/news/70482\n\n[46]\xa0http://kremlin\xa0dot ru/events/president/news/70482\n\n[47]\xa0https://t.me/pushilindenis/3166\n\n[48]\xa0https://t.me/BalitskyEV/778\xa0;\n\xa0https://t.me/BalitskyEV/778\n\n[50]\xa0https://isw.pub/UkrWar011323\n\xa0\n\xa0https://t.me/modmilby/22518\n\xa0\n\xa0https://t.me/modmilby/22515\n\xa0\n\xa0https://t.me/grey_zone/17123\n\xa0\nFile Attachments:\n\nZaporizhia Battle Map Draft February 09,2023.png\n\nKherson-Mykolaiv Battle Map Draft February 09,2023.png\n\nDonetsk Battle Map Draft February 09,2023.png\n\nKharkiv Battle Map Draft February 09,2023.png\n\nDraftUkraineCoTFebruary09,2023.png\n\n1400 16th Street NW, Suite 515 Washington, DC 20036\n\nph (202) 293-5550', metadata={'source': 'https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023'})] ``` You can explore the [LangChain documentation](https://api.python.langchain.com/en/latest/vectorstores/langchain.vectorstores.clarifai.Clarifai.html#langchain.vectorstores.clarifai.Clarifai) to learn more on how to use the framework with the Clarifai vector store. - [Prerequisites](https://docs.clarifai.com/integrations/langchain/vector-store/#prerequisites) - [Similarity Search on Uploaded Texts](https://docs.clarifai.com/integrations/langchain/vector-store/#similarity-search-on-uploaded-texts) - [Similarity Search on Uploaded Web Documents](https://docs.clarifai.com/integrations/langchain/vector-store/#similarity-search-on-uploaded-web-documents) - [Similarity Search on Existing Applications](https://docs.clarifai.com/integrations/langchain/vector-store/#similarity-search-on-existing-applications) Build your next AI app, test and tune popular LLMs models, and much more. [Get started for free](/content/explore/index.html)