## Learn how to upload data to a dataset via the API

Uploading data to a dataset in Clarifai is essential for training and evaluating your machine learning models.

Whether you're working with images, videos, text, audio, or other data types, we provide flexible and efficient methods to upload data from various sources.

Before using the [Python SDK](https://docs.clarifai.com/additional-resources/api-overview/python-sdk), [Node.js SDK](https://docs.clarifai.com/additional-resources/api-overview/nodejs-sdk), or any of our [gRPC clients](https://docs.clarifai.com/additional-resources/api-overview/grpc-clients), ensure they are properly installed on your machine. Refer to their respective installation guides for instructions on how to install and initialize them.

[Click here](https://docs.clarifai.com/additional-resources/api-references/api-reference/#dataset) to learn more about working with the `Dataset` class.

## Customize Batch Size

When uploading inputs to the Clarifai platform, there are limits on the size and number of inputs per upload, as detailed [here](https://docs.clarifai.com/create-manage/inputs/upload/#upload-limits). However, by using methods from the `Dataset` class — such as `Dataset.upload_from_folder()` or `Dataset.upload_from_csv()` — you can bypass these restrictions and efficiently upload larger volumes of inputs.

For example, when uploading images in bulk, such methods incrementally process and upload them in multiple batches, ensuring that each batch contains a maximum of 128 images and does not exceed 128MB in size – which ensures adherence to the upload restrictions.

You can also customize the `batch_size` variable, which allows for concurrent upload of inputs and annotations. For example, if your images folder exceeds 128MB, you can set the variable to ensure that each batch contains an appropriate number of images while staying within the 128MB per batch limit.

The default `batch_size` is set to 32, but you can customize it to any value between 1 (minimum) and 128 (maximum).

Here is an example:

```text
dataset.upload_from_folder(folder_path='/path/to/your/folder', input_type='image', labels=True, batch_size=50)
```

## Add Inputs to a Dataset

After uploading inputs to the Clarifai platform, you can add them to a dataset by specifying their input IDs.

- cURL

```bash
curl --location --request POST "https://api.clarifai.com/v2/users/YOUR_USER_ID_HERE/apps/YOUR_APP_ID_HERE/datasets/YOUR_DATASET_ID_HERE/inputs" \
  --header "Authorization: Key YOUR_PAT_HERE" \
  --header "Content-Type: application/json" \
  --data-raw '{
    "dataset_inputs": [\
      {\
        "input": {\
          "id": "YOUR_EXISTING_INPUT_ID_HERE"\
        }\
      }\
    ]
  }'
```

## Upload From Folder

The `upload_from_folder` method lets you bulk-upload images or text files from a local folder directly into a Clarifai dataset.

- Python SDK
- Node.js SDK

```python
from clarifai.client.dataset import Dataset

# Set PAT as an environment variable before running this script
#   export CLARIFAI_PAT=YOUR_PAT_HERE   # Unix-like systems
#   set CLARIFAI_PAT=YOUR_PAT_HERE      # Windows

# Create a dataset object
dataset = Dataset(
    user_id="YOUR_USER_ID",
    app_id="YOUR_APP_ID",
    dataset_id="YOUR_DATASET_ID"
)

# Upload data from a folder
dataset.upload_from_folder(
    folder_path="/path/to/your/folder",
    input_type="image", # or "text" for text files
    labels=True   # Set to False to upload without concepts
)
```

```typescript
const { Dataset } = require("clarifai-nodejs");
const path = require("path");

(async () => {
  const dataset = new Dataset({
    datasetId: "YOUR_DATASET_ID",
    authConfig: {
      pat: process.env.CLARIFAI_PAT,
      userId: "YOUR_USER_ID",
      appId: "YOUR_APP_ID",
    },
  });

await dataset.uploadFromFolder({
    folderPath: path.resolve(__dirname, "/path/to/your/folder"),
    inputType: "image",
    labels: true,
  });
})();
```

Note that:

- The `upload_from_folder` method only supports "image" and "text" input types.
- Ensure your dataset (`dataset_id`) already exists before calling this method.
- Large datasets should be uploaded with an appropriate [`batch_size`](https://docs.clarifai.com/create/datasets/upload/#customize-batch-size) (default 128).
- If `labels=True`, the folder name is assigned as the input’s concept label.
- The filename (without extension) is used as the `input_id` in Clarifai.

## Upload From CSV

The `upload_from_csv` method lets you bulk-upload data into a Clarifai dataset using a CSV file. This method is useful when your data is already structured in tabular form with URLs, local file paths, or raw text.

- Python SDK
- Node.js SDK

```python
from clarifai.client.dataset import Dataset

# Create a dataset object
dataset = Dataset(
    user_id="YOUR_USER_ID",
    app_id="YOUR_APP_ID",
    dataset_id="YOUR_DATASET_ID"
)

# Upload local image files
dataset.upload_from_csv(
    csv_path="path_to_your_csv_file.csv",
    input_type="image",
    csv_type="file_path",
    labels=True # Set to False to upload without concepts
)
```

Example CSV Files

File-path based (for local files)

```text
inputid,input,concepts,metadata
img1,"data/metro-north.jpg","train","{'source': 'local'}"
img2,"data/puppy.jpeg","dog","{'source': 'local'}"
```

Raw text dataset (only valid with input_type="text")

```text
inputid,input,concepts,metadata
TXT1,"The sky is clear and blue","weather","{'lang': 'en'}"
TXT2,"The puppy is playing in the garden","dog","{'lang': 'en'}"
```

With geopoints

```text
inputid,input,concepts,metadata,geopoints
img1,"data/metro-north.jpg","train","{'source': 'clarifai-samples'}","-73.935242,40.730610"
img2,"data/puppy.jpeg","dog","{'source': 'clarifai-samples'}","-118.243683,34.052235"
```

Note that:

- The `upload_from_csv` method supports "image", "text", "video", and "audio" file types.
- The `csv_type` parameter defines how the CSV file will be interpreted. It can be:
  - "url" — Inputs are hosted online, and the CSV provides URLs.
  - "file_path" — Inputs are stored locally, and the CSV provides file paths.
  - "raw" — Only valid for text datasets; the CSV provides raw text strings.
- If `labels=True` (default), the CSV must include a `concepts` column with labels. If `False`, inputs are uploaded without labels.
- The [`batch_size`](https://docs.clarifai.com/create/datasets/upload/#customize-batch-size) (default = 128) parameter defines the maximum number of inputs to upload concurrently in one batch.
- The CSV file must include column headers. These are the supported headers:
  - `inputid` — Unique identifier for the input.
  - `input` — URL, file path, or raw text depending on `csv_type`.
  - `concepts` — Concept labels (if `labels=True`).
  - `metadata` — JSON metadata, formatted with single quotes inside. Example: `"{'source': 'web'}"`.
  - `geopoints` — Geolocation in "longitude,latitude" format.
- When uploading text data, ensure the target app is configured to accept text inputs. Set the primary input type to **Text/Document** when creating the app.

## Upload Image Data With Annotations

You can [upload image data](https://docs.clarifai.com/create/inputs/upload/api#upload-image-data) together with bounding box annotations into a Clarifai dataset, adding richer context and detail to your visual data.

- Python SDK
- Node.js SDK

```python
from clarifai.client.input import Inputs
import time

# Initialize the Inputs client
input_client = Inputs(
    user_id="YOUR_USER_ID",
    app_id="YOUR_APP_ID"
)

# Upload image data from a specified URL with a unique input ID
input_id = "bbox_example"
input_client.upload_from_url(
    input_id=input_id,
    dataset_id="YOUR_DATASET_ID", # Optional: specify dataset ID to add the input to a dataset
    image_url="https://samples.clarifai.com/BarackObama.jpg"
)

# Poll until input is processed successfully
status = None
for _ in range(10):  # max retries
    inp = input_client.get_input(input_id)
    status = inp.status.code
    if status == 30000:  # SUCCESS
        break
    time.sleep(2)

if status != 30000:
    raise RuntimeError("Input not processed, cannot add annotations yet.")

# Define bounding box coordinates (format: [left, top, right, bottom])
bbox_points = [0.1, 0.1, 0.8, 0.9]

# Generate a bounding box annotation with specified label ("face") and bounding box coordinates
annotation = input_client.get_bbox_proto(
    input_id=input_id,
    label="face",
    bbox=bbox_points
)

# Upload the generated annotation to associate with the previously uploaded image
input_client.upload_annotations([annotation])
```

## Upload Image Data With Mask Annotations

You can add masks to image data in a Clarifai dataset by providing polygon coordinates with the image, enabling precise region-based annotations.

- Python SDK
- Node.js SDK

```python
from clarifai.client.input import Inputs
import time

# Initialize the Inputs client
input_client = Inputs(
    user_id="YOUR_USER_ID",
    app_id="YOUR_APP_ID"
)

# Upload image data from a specified URL with a unique input ID
input_id = "mask_example"
input_client.upload_from_url(
    input_id=input_id,
    dataset_id="YOUR_DATASET_ID", # Optional: specify dataset ID to add the input to a dataset
    image_url="https://samples.clarifai.com/BarackObama.jpg"
)

if status != 30000:
    raise RuntimeError("Input not processed, cannot add annotations yet.")

# Define polygon points for the mask
# Coordinates are normalized (0.0 to 1.0) relative to image width and height
mask_points = [\
    [0.30, 0.20],  # top-left forehead\
    [0.70, 0.20],  # top-right forehead\
    [0.85, 0.45],  # right cheek\
    [0.70, 0.80],  # right jaw\
    [0.30, 0.80],  # left jaw\
    [0.15, 0.45]   # left cheek\
]

# Create a mask annotation with label "obama"
annotation = input_client.get_mask_proto(
    input_id=input_id,
    label="obama",
    polygons=mask_points
)

# Upload the generated annotation to associate with the previously uploaded image
input_client.upload_annotations([annotation])
```

## Upload Video Data With Annotations

You can [upload videos](https://docs.clarifai.com/create/inputs/upload/api#upload-video-data) in a Clarifai dataset with enriched annotations by including bounding box coordinates that define regions of interest within individual frames, adding valuable context to your video content.

- Python SDK
- Node.js SDK

```python
from clarifai.client.input import Inputs
import time
from clarifai_grpc.grpc.api import resources_pb2 as r

# Initialize the Inputs client
input_client = Inputs(
    user_id="YOUR_USER_ID",
    app_id="YOUR_APP_ID",
)

# Upload video data from a specified URL with a unique input ID
input_id = "video_bbox_example" # change per test if re-running
input_client.upload_from_url(
    input_id=input_id,
    dataset_id="YOUR_DATASET_ID", # Optional: specify dataset ID to add the input to a dataset
    video_url="https://samples.clarifai.com/beer.mp4"
)

if status != 30000:
    raise RuntimeError("Input not processed, cannot add annotations yet.")

# Bounding box coordinates for annotation, (top_row, left_col, bottom_row, right_col), relative 0–1
bbox_points = [0.1, 0.1, 0.8, 0.9]

# Build a video frame annotation with one region (Clarifai requires frame -> region -> bbox)
bbox_pb = r.BoundingBox(
    top_row=bbox_points[0],
    left_col=bbox_points[1],
    bottom_row=bbox_points[2],
    right_col=bbox_points[3],
)
region = r.Region(
    region_info=r.RegionInfo(bounding_box=bbox_pb),
    data=r.Data(concepts=[r.Concept(id="glass", name="glass", value=1.0)])
)
frame = r.Frame(
    frame_info=r.FrameInfo(time=0, index=0),   # first frame (time in ms)
    data=r.Data(regions=[region])
)
annotation = r.Annotation(
    input_id=input_id,
    data=r.Data(frames=[frame])
)

# Upload the annotation associated with the video
input_client.upload_annotations([annotation])
```

## Upload Text Data With Annotations

You can [upload text data](https://docs.clarifai.com/create/inputs/upload/api#upload-text-data) in a Clarifai dataset and enrich it by attaching metadata, categorizing the content, or adding detailed annotations to enhance structure and context.

- Python SDK
- Node.js SDK

```python
from clarifai.client.input import Inputs

# Initialize the Inputs client
input_client = Inputs(
    user_id="YOUR_USER_ID",
    app_id="YOUR_APP_ID",
)

# Define input details
input_id = "text_example"
concepts = ["mobile", "camera"]

# Upload data from URL and annotate with concepts
input_client.upload_from_url(
    input_id=input_id,
    dataset_id="YOUR_DATASET_ID",  # Optional: specify dataset ID to add the input to a dataset
    text_url="https://samples.clarifai.com/featured-models/Llama2_Conversational-agent.txt",
    labels=concepts
)
```

## Batch Upload Image Data While Tracking Status

You can actively monitor the status of your dataset upload, giving you clear visibility into the progress and making it easy to track and analyze the data transfer process.

- Python SDK

```python
from clarifai.client.dataset import Dataset
from clarifai.datasets.upload.utils import load_module_dataloader

#replace your "user_id", "app_id", "dataset_id".
dataset = Dataset(user_id="user_id", app_id="test_app", dataset_id="first_dataset")
#create dataloader object
cifar_dataloader = load_module_dataloader('./image_classification/cifar10')
#set get_upload_status=True for showing upload status
dataset.upload_dataset(dataloader=cifar_dataloader,get_upload_status=True)
```

## Retry Upload From Log File

You can retry uploads for failed inputs directly from the logs. When using the `upload_dataset` function, any failed inputs are automatically logged to a file, which can later be used to resume and retry the upload process seamlessly.

Set `retry_duplicates` to `True` if you want to retry duplicate with new `input_id` in current dataset.

- Python SDK

```python
#importing load_module_dataloader for calling the dataloader object in dataset.py in the local data folder
from clarifai.datasets.upload.utils import load_module_dataloader
from clarifai.client.dataset import Dataset

#replace your "user_id", "app_id", "dataset_id".
dataset = Dataset(user_id="user_id", app_id="app_id", dataset_id="dataset_id")

cifar_dataloader = load_module_dataloader('./image_classification/cifar10')

dataset.retry_upload_from_logs(dataloader=cifar_dataloader, log_file_path='path to log file', retry_duplicates=True, log_warnings=True)
```