[Skip to main content](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#__docusaurus_skipToContent_fallback) GPU Shortage?Make the most out of your GPUs with Compute Orchestration. [Learn More](/content/ebook-clarifai-executive-guide-2025/index.html) On this page **Pre-process and ingest diverse data formats, including images and text-based documents** * * * The [Data Ingestion Pipelines](https://github.com/Clarifai/examples/tree/main/Data_Utils/Ingestion%20pipelines) framework, part of the Data Utils library, offers a comprehensive suite of robust functions — commonly referred to as pipelines — designed to pre-process, transform, and prepare images and text documents for seamless ingestion into the Clarifai platform. These ready-to-use pipelines enable efficient processing of unstructured data, including partitioning, chunking, cleaning, and extracting valuable information, ensuring the data is optimized for downstream use cases such as [Retrieval Augmented Generation](https://docs.clarifai.com/portal-guide/agent-system-operators/rag-prompter) (RAG). Leveraging the capabilities of the open-source [Unstructured](https://docs.unstructured.io/welcome) library, this framework is designed to streamline data processing workflows, making it an essential tool for working with Large Language Models (LLMs) and other AI-driven applications. It supports these file formats: - PDF - Text (.txt) - Docx - Markdown ## Prerequisites [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#prerequisites "Direct link to Prerequisites") ### Install Python SDK and Data Utils [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#install-python-sdk-and-data-utils "Direct link to Install Python SDK and Data Utils") Install the latest version of the `clarifai` Python SDK package. Also, install the Data Utils library. - Bash ```text pip install --upgrade clarifai pip install clarifai-datautils ``` ### Install Extra Dependencies [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#install-extra-dependencies "Direct link to Install Extra Dependencies") The Data Ingestion Pipelines framework requires additional libraries to function properly. First, create a `requirements-dev.txt` file and add the following dependencies: - Text ```text unstructured[pdf] @ git+https://github.com/clarifai/unstructured.git@support_clarifai_model llama-index-core==0.10.33 llama-index-llms-clarifai==0.1.2 pi_heif==0.18.0 markdown==3.7 python-docx==1.1.2 schema==0.7.5 ``` > _Note that this command `pip install unstructured[pdf] @ git+https://github.com/clarifai/unstructured.git@support_clarifai_model` installs the `support_clarifai_model` branch from the [Clarifai fork of `unstructured`library](https://github.com/Clarifai/unstructured/tree/support_clarifai_model)._ Then, run the following command to install the required dependencies: - Bash ```text pip install -r requirements-dev.txt ``` You can also install the following system dependencies if they are not already available on your system. Based on the document types you're handling, you may not need all of them. - [`opencv-python-headless`](https://pypi.org/project/opencv-python-headless/) — A lightweight version of OpenCV (Open Source Computer Vision Library) designed for environments where GUI functionalities (such as image or video display) are not needed. You can install it by running: `pip install opencv-python-headless`. - [`poppler-utils`](https://pypi.org/project/poppler-utils/) — Essential for processing and extracting data from PDF files. You can install it by running: `sudo apt update && sudo apt install poppler-utils`. - [`tesseract-ocr`](https://github.com/tesseract-ocr/tessdoc) — Required for performing OCR on images or scanned documents to extract text. You can install it by running: `sudo apt update && sudo apt install tesseract-ocr`. - [`libgl1-mesa-glx`](https://mesa3d.org/) — Ensures compatibility with graphical operations, which may be required by certain libraries (e.g., OpenCV) even in headless environments. You can install it by running: `sudo apt update && sudo apt install libgl1-mesa-glx`. - [`punkt_tab`](https://www.nltk.org/api/nltk.tokenize.html) — Enables tokenization of text data with tab-separated values; it's part of the NLTK library. You can install it by running: `nltk.download('punkt_tab')`. - [`averaged_perceptron_tagger_eng`](https://www.nltk.org/api/nltk.tag.html) — Provides a pre-trained model for accurate part-of-speech tagging in English; it's part of the NLTK library. You can install it by running: `nltk.download('averaged_perceptron_tagger_eng')`. ### Get a PAT [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#get-a-pat "Direct link to Get a PAT") You need a PAT (Personal Access Token) key to authenticate your connection to the Clarifai platform. You can generate it in your Personal Settings page by navigating to the [Security section](/content/settings/security/index.html). Then, set it as an environment variable in your script. - Python ```python import os os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # replace with your own PAT key ``` ### Create a Dataset [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#create-a-dataset "Direct link to Create a Dataset") [Create a dataset](https://docs.clarifai.com/sdk/managing-datasets#creating-datasets) on the Clarifai platform to use for uploading your pre-processed data. - Python ```python from clarifai.client.app import App app = App(app_id="YOUR_APP_ID_HERE", user_id="YOUR_USER_ID_HERE",pat="YOUR_PAT_HERE") # Provide the dataset name as parameter in the create_dataset function dataset = app.create_dataset(dataset_id="annotations_dataset") ``` ## Building Pipelines [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#building-pipelines "Direct link to Building Pipelines") When working with unstructured documents like PDFs, building pipelines is a crucial step to automate the processing and transformation of data. Here is an example of a basic pipeline for PDF partitioning. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition # Define the processing pipeline pipeline = Pipeline( name="basic_pdf", transformations=[\ PDFPartition()\ ] ) # Load predefined pipeline pipeline = Pipeline.load(name="basic_pdf") # View the pipeline pipeline.info() ``` Output Example ```text Pipeline: basic_pdf ``` Note that: - `Pipeline` and `PDFPartition` classes are imported from `clarifai_datautils.multimodal`. These are used to define and execute processing pipelines for PDF documents. - A `Pipeline` object is created with the name `"basic_pdf"`. You can provide any arbitrary name for the pipeline. The name can be used to identify or call the pipeline. - `PDFPartition()` uses [default parameters](https://github.com/Clarifai/clarifai-python-datautils/blob/main/clarifai_datautils/constants/pipeline.py) (such as `max_characters=500`) for ingesting PDFs. - After loading a predefined pipeline, you can view its details. ## Partitioning & Chunking [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#partitioning--chunking "Direct link to Partitioning & Chunking") [Partitioning](https://docs.unstructured.io/open-source/core-functionality/partitioning) is the first step in document processing. It breaks down a raw, unstructured document into smaller, meaningful units called document elements, while preserving the document’s semantic structure. These elements — such as paragraphs, titles, tables, and images — help maintain the original context. The process involves reading the document, segmenting it into sections, categorizing those sections, and extracting the relevant text. Chunking follows partitioning and involves grouping or rearranging document elements generated by partitioning into "chunks" based on specific size constraints or criteria. This step ensures that the resulting segments are optimized for use cases like search, summarization, or content retrieval. info Once a chunk of text or image data is uploaded to the Clarifai platform, [metadata fields](https://docs.clarifai.com/portal-guide/input-viewer/) — such as `filename`, `page_number`, `orig_elements`, and `type` — are automatically added to provide detailed information about the uploaded input. Example ### PDF Partitioning [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#pdf-partitioning "Direct link to PDF Partitioning") PDF partitioning helps transform PDFs into a structured format that can be used for further processing. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024)\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Run the pipeline on a PDF file # Set `loader=False` to return the transformed chunks as elements instead of loading them into a dataset elements = pipeline.run(files=LOCAL_PDF_PATH, loader=False) # Print the resulting chunks (document elements) print(elements) # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` Output Example ```text Transforming Files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:06<00:00, 6.82s/it] [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] Transforming Files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:06<00:00, 6.73s/it] Uploading Dataset: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:16<00:00, 2.42s/it] ``` Note that: - The transformation step uses the `PDFPartition` object to partition the PDF into smaller chunks. - `chunking_strategy` is set to `"by_title"`, meaning the document is split based on its title sections. - `max_characters` limits each chunk to 1024 characters for better processing and retrieval efficiency. The default behavior is 500 characters. - The `loader=False`, which is the default, argument ensures the transformed chunks are returned as Python objects (`elements`), allowing for local inspection or further processing. Conversely, setting `loader=True` directly ingests the transformed chunks into a Clarifai dataset instead of just returning them locally. - The partitioned and chunked PDF elements are uploaded to a Clarifai dataset. The uploaded data is automatically [annotated](https://docs.clarifai.com/portal-guide/input-viewer/) with the pipeline name on the Clarifai platform. This makes it easy to identify and distinguish between data processed through different pipelines. tips You can also configure the following arguments for the `PDFPartition` object: - Set `chunking_strategy="basic"` for the document to be chunked purely based on character length and sequential order rather than structural elements like section titles or page boundaries. It's useful when you simply want to group text into evenly sized chunks without preserving the document’s logical structure. - Set `ocr=True` to enable OCR for extracting text from scanned or image-based PDFs. Set it to `False`, which is the default, to disable OCR. - By default, `overlap=None` or `overlap=0` ensures no overlap between chunks; that is, chunks are created without any shared text between them. To enable overlap, provide an integer value (e.g., `overlap=100`) to specify the number of overlapping characters between consecutive chunks. - Set `overlap_all=True` to enable overlapping across all chunks. Set it to `False`, which is the default, to disable this behavior. - Set `strategy="ocr_only"` to force the document to be processed using the Tesseract OCR strategy. If Tesseract is unavailable and the document contains extractable text, it falls back to the `"fast"` strategy. Set `strategy="fast"` to extract text using `pdfminer`, which is faster and suitable for text-based PDFs. Otherwise, `strategy="auto"` is the default that selects the partitioning strategy based on document characteristics and the function kwargs. - Use `clarifai_ocr_model` to set the URL of a Clarifai OCR model for processing the document. The default is `None`. ### PDF Partitioning Multimodal [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#pdf-partitioning-multimodal "Direct link to PDF Partitioning Multimodal") The `PDFPartitionMultimodal` ingestion pipeline supports multimodal scenarios, where files containing a mix of text, images, and other elements are to be processed and ingested into the Clarifai platform. We use the Clarifai-hosted YOLOX object detection model to process the PDFs containing embedded images. - Python ```python from clarifai_datautils.multimodal import Pipeline from clarifai_datautils.multimodal.pipeline.PDF import PDFPartitionMultimodal from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartitionMultimodal(chunking_strategy="by_title", max_characters=1024)\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` Note that: - The`PDFPartitionMultimodal` supports the following arguments for configuration: `chunking_strategy`, `max_characters`, `overlap`, and `overlap_all`, which have been explained earlier. tips You can also configure the following arguments for the `PDFPartitionMultimodal` object: - By default, `extract_images_in_pdf=True` extracts images from a PDF file. In that case, the partitioning strategy is set as `strategy="hi_res"`, which is intended to identify the layout of the document and gain additional information about the document elements. Otherwise, set `extract_images_in_pdf=False` to disable this behavior. - Set `extract_image_block_types=["Image"]` to specify that you want to extract a list of image block types. - Set `extract_image_block_to_payload=True` to allow for the conversion of extracted images from a PDF into base64 format (return images as bytes). Note that to use this feature, you must set the `strategy` parameter to `hi_res` and `extract_images_in_pdf` to `True`. Otherwise, set `extract_image_block_to_payload=False` to disable this behavior. ### Text Partitioning [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#text-partitioning "Direct link to Text Partitioning") Text partitioning transforms unstructured `.txt` documents into text elements, making them easier to process, analyze, and utilize in downstream applications. - Python ```python from clarifai_datautils.multimodal import Pipeline, TextPartition from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ TextPartition(chunking_strategy="by_title", max_characters=1024)\ ] ) LOCAL_TEXT_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.txt" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed text chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_TEXT_PATH, loader=True)) ``` Note that: - The `TextPartition` object supports the following arguments for configuration: `chunking_strategy`, `max_characters`, `overlap`, and `overlap_all`, which have been explained earlier. ### Docx Partitioning [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#docx-partitioning "Direct link to Docx Partitioning") Docx partitioning processes `.docx` files, extracting and partitioning their contents into structured text elements. - Python ```python from clarifai_datautils.multimodal import Pipeline, DocxPartition from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ DocxPartition(chunking_strategy="by_title", max_characters=1024)\ ] ) LOCAL_DOCX_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.docx" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed text chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_DOCX_PATH, loader=True)) ``` Note that: - The `DocxPartition` object supports the following arguments for configuration: `chunking_strategy`, `max_characters`, `overlap`, and `overlap_all`, which have been explained earlier. ### Markdown Partitioning [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#markdown-partitioning "Direct link to Markdown Partitioning") Markdown partitioning processes `.md` files, breaking them down into structured text elements for improved usability in downstream applications. - Python ```python from clarifai_datautils.multimodal import Pipeline, MarkdownPartition from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ MarkdownPartition(chunking_strategy="by_title", max_characters=1024)\ ] ) LOCAL_MD_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.md" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed text chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_MD_PATH, loader=True)) ``` Note that: - The `MarkdownPartition` object supports the following arguments for configuration: `chunking_strategy`, `max_characters`, `overlap`, and `overlap_all`, which have been explained earlier. ## Image Summarization [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#image-summarization "Direct link to Image Summarization") The Image Summarizer pipeline enables you to utilize a Clarifai's multimodal-to-text model to generate text summaries for the uploaded image data. Each summary is stored as an individual input on the Clarifai platform, and you can view its metadata field to see the source image it’s associated with. The generated summaries are concise, optimized for retrieval, and enriched with relevant keywords, making them highly effective for search and indexing. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartitionMultimodal from clarifai_datautils.multimodal.pipeline.summarizer import ImageSummarizer from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartitionMultimodal(chunking_strategy="by_title", max_characters=1024),\ ImageSummarizer(model_url="https://clarifai.com/openai/chat-completion/models/gpt-4o") # You can use any other multimodal-to-text model available on the Clarifai platform\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` Example ## Text Cleaning [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#text-cleaning "Direct link to Text Cleaning") The Data Ingestion Pipelines framework allows you to prepare and refine raw text data by removing or correcting unwanted elements to improve readability, consistency, and usability for downstream applications. note The following examples use the `PDFPartition` object, but they can also be applied to any other supported partitioning objects. ### Clean Extra Whitespaces [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#clean-extra-whitespaces "Direct link to Clean Extra Whitespaces") You can remove unnecessary spaces, tabs, or newlines from documents. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Clean_extra_whitespace from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Clean_extra_whitespace()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Replace Unicode Quotes [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#replace-unicode-quotes "Direct link to Replace Unicode Quotes") You can replace Unicode quotes with ASCII quotes for standardization. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Replace_unicode_quotes from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Replace_unicode_quotes()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Clean Dashes [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#clean-dashes "Direct link to Clean Dashes") You can remove unnecessary dashes from texts. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Clean_dashes from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Clean_dashes()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Clean Bullets [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#clean-bullets "Direct link to Clean Bullets") You can remove unnecessary bullets from texts. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Clean_bullets from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Clean_bullets()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Group Broken Paragraphs [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#group-broken-paragraphs "Direct link to Group Broken Paragraphs") You can merge fragmented paragraphs that were unintentionally split, restoring proper text flow and improving readability. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Group_broken_paragraphs from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Group_broken_paragraphs()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Remove Punctuations [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#remove-punctuations "Direct link to Remove Punctuations") You can remove unnecessary punctuations from texts. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Remove_punctuation from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Remove_punctuation()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Convert Byte Strings [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#convert-byte-strings "Direct link to Convert Byte Strings") You can convert a byte string (such as `b'hello'`) into a regular string (`'hello'`), ensuring proper text formatting and usability. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Bytes_string_to_string from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Bytes_string_to_string()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Clean Non-ASCII Characters [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#clean-non-ascii-characters "Direct link to Clean Non-ASCII Characters") You can remove non-ASCII characters from text, ensuring compatibility with systems that only support standard ASCII encoding. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Clean_non_ascii_chars from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Clean_non_ascii_chars()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Clean Ordered Bullets [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#clean-ordered-bullets "Direct link to Clean Ordered Bullets") You can remove ordered bullet points(such as `1.`, `2)`, or `III.`) from text. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Clean_ordered_bullets from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Clean_ordered_bullets()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Clean Prefix [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#clean-prefix "Direct link to Clean Prefix") You can remove a specified prefix from a document. The `Clean_prefix` object supports the following arguments: - `pattern` — Defines the prefix to remove. The pattern must be provided, and it can be a simple string or a regex pattern. - `ignore_case` (optional, default is `False`) — Determines whether to ignore case. If `True`, ensures case-insensitive matching. - `strip` (optional, default is `True`) — If `True`, removes any leading whitespace after the prefix is removed. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Clean_prefix from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Clean_prefix(pattern="Example", ignore_case=True, strip=True)\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Clean Postfix [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#clean-postfix "Direct link to Clean Postfix") You can remove a specified postfix from a document using the `Clean_postfix` object, which supports the same arguments as `Clean_prefix`. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.cleaners import Clean_postfix from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ Clean_postfix(pattern="Example", ignore_case=True, strip=True)\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ## Text Extraction [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#text-extraction "Direct link to Text Extraction") The Data Ingestion Pipelines framework allows you to identify and retrieve meaningful texts from documents. ### Extract Email Addresses [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#extract-email-addresses "Direct link to Extract Email Addresses") You can extract email addresses from texts. Note that if a chunk contains the addresses, they will be extracted and stored in the `email_address` metadata field of the uploaded input on the Clarifai platform, [as previously mentioned](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#partitioning--chunking). - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.extractors import ExtractEmailAddress from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ ExtractEmailAddress()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Datetime With Time Zones [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#datetime-with-time-zones "Direct link to Datetime With Time Zones") You can extract datetime values with time zones from texts, ensuring accurate timestamp retrieval. Note that if a chunk contains the values, they will be extracted and stored in the `date_time` metadata field of the uploaded input on the Clarifai platform. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.extractors import ExtractDateTimeTz from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ ExtractDateTimeTz()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Extract IP Addresses [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#extract-ip-addresses "Direct link to Extract IP Addresses") You can extract IP addresses from texts. Note that if a chunk contains the addresses, they will be extracted and stored in the `ip_address` metadata field of the uploaded input on the Clarifai platform. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.extractors import ExtractIpAddress from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ ExtractIpAddress()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Extract IP Addresses Names [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#extract-ip-addresses-names "Direct link to Extract IP Addresses Names") You can extract IP addresses along with associated names from texts. Note that if a chunk contains the names, they will be extracted and stored in the `ip_address_name` metadata field of the uploaded input on the Clarifai platform. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.extractors import ExtractIpAddressName from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ ExtractIpAddressName()\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Extract Text After [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#extract-text-after "Direct link to Extract Text After") You can extract text appearing after a specified string in a given text input. The `ExtractTextAfter` object supports the following string arguments: - `key` — Key to store the extracted text in the metadata field of the uploaded input on the Clarifai platform. - `string` — The reference string after which the text will be extracted. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.extractors import ExtractTextAfter from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ ExtractTextAfter(key="example", string="Example:")\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` ### Extract Text Before [​](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/\#extract-text-before "Direct link to Extract Text Before") You can extract text appearing before a specified string in a given text input. The `ExtractTextBefore` object supports the following string arguments: - `key` — Key to store the extracted text in the metadata field of the uploaded input on the Clarifai platform. - `string` — The reference string before which the text will be extracted. - Python ```python from clarifai_datautils.multimodal import Pipeline, PDFPartition from clarifai_datautils.multimodal.pipeline.extractors import ExtractTextBefore from clarifai.client import Dataset import os # Set the Clarifai Personal Access Token (PAT) for authentication os.environ["CLARIFAI_PAT"] = "YOUR_PAT_HERE" # Define the processing pipeline pipeline = Pipeline( name="pipeline-1", transformations=[\ PDFPartition(chunking_strategy="by_title", max_characters=1024),\ ExtractTextBefore(key="example", string="Example:")\ ] ) LOCAL_PDF_PATH = "YOUR_LOCAL_PDF_PATH_HERE" # Example: "./assets/multimodal/DA-1p.pdf" # Initialize dataset object for ingesting data dataset = Dataset(user_id="YOUR_USER_ID_HERE", app_id="YOUR_APP_ID_HERE", dataset_id="YOUR_DATASET_ID_HERE") # Alternative: Initialize the dataset using a dataset URL (commented out) # dataset = Dataset("DATASET_URL_HERE") # Use Python SDK to upload the processed PDF chunks to Clarifai dataset.upload_dataset(pipeline.run(files=LOCAL_PDF_PATH, loader=True)) ``` - [Prerequisites](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#prerequisites) - [Install Python SDK and Data Utils](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#install-python-sdk-and-data-utils) - [Install Extra Dependencies](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#install-extra-dependencies) - [Get a PAT](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#get-a-pat) - [Create a Dataset](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#create-a-dataset) - [Building Pipelines](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#building-pipelines) - [Partitioning & Chunking](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#partitioning--chunking) - [PDF Partitioning](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#pdf-partitioning) - [PDF Partitioning Multimodal](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#pdf-partitioning-multimodal) - [Text Partitioning](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#text-partitioning) - [Docx Partitioning](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#docx-partitioning) - [Markdown Partitioning](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#markdown-partitioning) - [Image Summarization](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#image-summarization) - [Text Cleaning](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#text-cleaning) - [Clean Extra Whitespaces](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#clean-extra-whitespaces) - [Replace Unicode Quotes](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#replace-unicode-quotes) - [Clean Dashes](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#clean-dashes) - [Clean Bullets](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#clean-bullets) - [Group Broken Paragraphs](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#group-broken-paragraphs) - [Remove Punctuations](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#remove-punctuations) - [Convert Byte Strings](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#convert-byte-strings) - [Clean Non-ASCII Characters](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#clean-non-ascii-characters) - [Clean Ordered Bullets](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#clean-ordered-bullets) - [Clean Prefix](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#clean-prefix) - [Clean Postfix](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#clean-postfix) - [Text Extraction](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#text-extraction) - [Extract Email Addresses](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#extract-email-addresses) - [Datetime With Time Zones](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#datetime-with-time-zones) - [Extract IP Addresses](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#extract-ip-addresses) - [Extract IP Addresses Names](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#extract-ip-addresses-names) - [Extract Text After](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#extract-text-after) - [Extract Text Before](https://docs.clarifai.com/create/datasets/data-utils/ingestion-pipelines/#extract-text-before) Build your next AI app, test and tune popular LLMs models, and much more. [Get started for free](/content/explore/index.html)