Ollama | Clarifai Docs

Download and run Ollama models locally and expose them via a public API

Ollama is an open-source tool that lets you run LLMs directly on your local machine. Combined with Clarifai's Local Runners, you can serve Ollama models from your machine, expose them via a public API, and access them through the Clarifai platform — all while keeping the speed, privacy, and control of local inference.

Step 1: Install Prerequisites

Install Ollama

Go to the Ollama website and install the appropriate version for your system (macOS, Windows, or Linux).

Note: On Windows, restart your machine after installing to ensure environment variables are applied.

Install Clarifai

Bash

pip install --upgrade clarifai

Note: Python 3.11 or 3.12 is required. The openai package is included with clarifai.

Step 2: Log In

clarifai login

You'll be prompted for your user ID and PAT. This saves your credentials locally so you don't need to set environment variables manually.

Step 3: Initialize a Model

Scaffold a model project using any model from the Ollama library:

clarifai model init --toolkit ollama --model-name gpt-oss:20b

Example Output

clarifai model init --toolkit ollama --model-name gpt-oss:20b
[INFO] Initializing model with ollama toolkit...

Model initialized in "./gpt-oss:20b"

Test locally:
    clarifai model serve "./gpt-oss:20b"
    clarifai model serve "./gpt-oss:20b" --mode env       # auto-create venv and install deps
    clarifai model serve "./gpt-oss:20b" --mode container # run inside Docker

Deploy to Clarifai:
    clarifai model deploy "./gpt-oss:20b" --instance a10g
    clarifai list-instances                               # list available instances

This creates a ./gpt-oss:20b/ directory with three files:

gpt-oss:20b/
├── 1/
│   └── model.py       # Ollama inference logic
├── requirements.txt   # Python dependencies
└── config.yaml        # Model config (user_id/app_id auto-filled from login)

Omitting --model-name will install the default model (llama3.2):

clarifai model init --toolkit ollama

Use ollama list to see downloaded models and ollama rm to remove one.

Note: Some models are very large and may require significant memory or GPU resources. Check your machine's capacity before initializing.

Step 4: Serve Locally

Start the model as a local runner:

clarifai model serve "./gpt-oss:20b"

Note: The first run pulls the model weights from Ollama, which may take a few minutes. Add -v for verbose logs.

Example Output

clarifai model local-runner
[INFO] 16:01:28.904230 > Checking local runner requirements... |  thread=8800297152
[INFO] 16:01:28.928129 Checking 2 dependencies... |  thread=8800297152
[INFO] 16:01:28.928672 ✅ All 2 dependencies are installed! |  thread=8800297152
[INFO] 16:01:28.928886 Verifying Ollama installation... |  thread=8800297152
[INFO] 16:01:29.004234 > Verifying local runner setup... |  thread=8800297152
[INFO] 16:01:29.004427 Current context: default |  thread=8800297152
[INFO] 16:01:29.004463 Current user_id: alfrick |  thread=8800297152
[INFO] 16:01:29.004490 Current PAT: d6570**** |  thread=8800297152
[INFO] 16:01:29.005945 Current compute_cluster_id: local-runner-compute-cluster |  thread=8800297152
[WARNING] 16:01:35.936440 Failed to get compute cluster with ID 'local-runner-compute-cluster':
code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "ComputeCluster with ID 'local-runner-compute-cluster' not found. Check your request fields."
req_id: "sdk-python-11.8.2-75ca9226003a4b34a770885b119d5814"
 |  thread=8800297152
Compute cluster not found. Do you want to create a new compute cluster alfrick/local-runner-compute-cluster? (y/n): y
[INFO] 16:01:58.382096 Compute Cluster with ID 'local-runner-compute-cluster' is created:
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.8.2-a0474b8ba93c4e069804500a188694db"
 |  thread=8800297152
[INFO] 16:01:58.391571 Current nodepool_id: local-runner-nodepool |  thread=8800297152
[WARNING] 16:02:00.633687 Failed to get nodepool with ID 'local-runner-nodepool':
code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "Nodepool not found. Check your request fields."
req_id: "sdk-python-11.8.2-62149d46e7104d35bcb2a36546710329"
 |  thread=8800297152
Nodepool not found. Do you want to create a new nodepool alfrick/local-runner-compute-cluster/local-runner-nodepool? (y/n): y
[INFO] 16:02:03.909005 Nodepool with ID 'local-runner-nodepool' is created:
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.8.2-05836a431940495b94b9c3691f6c6d4d"
 |  thread=8800297152
[INFO] 16:02:03.921694 Current app_id: local-runner-app |  thread=8800297152
[INFO] 16:02:04.203774 Current model_id: local-runner-model |  thread=8800297152
[WARNING] 16:02:10.933734 Attempting to patch latest version: 9d38bb9398944de4bdef699835f17ec9 |  thread=8800297152
[INFO] 16:02:14.195999 Successfully patched version 9d38bb9398944de4bdef699835f17ec9 |  thread=8800297152
[INFO] 16:02:14.197924 Current model version 9d38bb9398944de4bdef699835f17ec9 |  thread=8800297152
[WARNING] 16:02:18.679567 Failed to get runner with ID 'f3c46913186449ba99dedd38123d47a3':
code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "Runner not found. Check your request fields."
req_id: "sdk-python-11.8.2-0b3b241c76ef429290c8c54a318f2f21"
 |  thread=8800297152
[INFO] 16:02:18.679913 Creating the local runner tying this 'alfrick/local-runner-app/models/local-runner-model' model (version: 9d38bb9398944de4bdef699835f17ec9) to the 'alfrick/local-runner-compute-cluster/local-runner-nodepool' nodepool. |  thread=8800297152
[INFO] 16:02:19.757117 Runner with ID '2f84d7194ee8464fad485fd058663fe5' is created:
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.8.2-8ac525dc13ec47629213f0283e89c6a7"
 |  thread=8800297152
[INFO] 16:02:19.765198 Current runner_id: 2f84d7194ee8464fad485fd058663fe5 |  thread=8800297152
[WARNING] 16:02:20.331980 Failed to get deployment with ID local-runner-deployment:
code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "Deployment with ID 'local-runner-deployment' not found. Check your request fields."
req_id: "sdk-python-11.8.2-864f55e6bc554614894ee031cc30cdb9"
 |  thread=8800297152
Deployment not found. Do you want to create a new deployment alfrick/local-runner-compute-cluster/local-runner-nodepool/local-runner-deployment? (y/n): y
[INFO] 16:02:25.016935 Deployment with ID 'local-runner-deployment' is created:
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.8.2-b6bf9aad5aa545f8a041113608fc9365"
 |  thread=8800297152
[INFO] 16:02:25.024579 Current deployment_id: local-runner-deployment |  thread=8800297152
[INFO] 16:02:25.027108 Current model section of config.yaml: {'app_id': 'local-dev-runner-app', 'id': 'local-dev-model', 'model_type_id': 'text-to-text', 'user_id': 'clarifai-user-id'} |  thread=8800297152
Do you want to backup config.yaml to config.yaml.bk then update the config.yaml with the new model information? (y/n): y
[INFO] 16:02:27.407724 Checking 2 dependencies... |  thread=8800297152
[INFO] 16:02:27.408555 ✅ All 2 dependencies are installed! |  thread=8800297152
[INFO] 16:02:27.451117 Customizing Ollama model with provided parameters... |  thread=8800297152
[INFO] 16:02:27.451785 ✅ Starting local runner... |  thread=8800297152
[INFO] 16:02:27.451852 No secrets path configured, running without secrets |  thread=8800297152
[INFO] 16:02:30.020253 Detected OpenAI chat completions for Clarifai model streaming - validating stream_options... |  thread=8800297152
[INFO] 16:02:30.027464 Starting Ollama server in the host: 127.0.0.1:23333 |  thread=8800297152
[INFO] 16:02:30.040882 Model llama3.2 pulled successfully. |  thread=8800297152
[INFO] 16:02:30.041191 Ollama server started successfully on 127.0.0.1:23333 |  thread=8800297152
[INFO] 16:02:30.096053 Ollama model loaded successfully: llama3.2 |  thread=8800297152
[INFO] 16:02:30.096133 ModelServer initialized successfully |  thread=8800297152
[INFO] 16:02:30.100802 ✅ Your model is running locally and is ready for requests from the API...
 |  thread=8800297152
[INFO] 16:02:30.100873 > Code Snippet: To call your model via the API, use this code snippet:

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.clarifai.com/v2/ext/openai/v1",
    api_key=os.environ['CLARIFAI_PAT'],
)

response = client.chat.completions.create(
    model="https://clarifai.com/alfrick/local-runner-app/models/local-runner-model",
    messages=[\
        {"role": "system", "content": "Talk like a pirate."},\
        {\
            "role": "user",\
            "content": "How do I check if a Python object is an instance of a class?",\
        },\
    ],
    temperature=1.0,
    stream=False,  # stream=True also works, just iterator over the response
)
print(response)
 |  thread=8800297152
[INFO] 16:02:30.100944 > Playground:   To chat with your model, visit: https://clarifai.com/playground?model=local-runner-model__9d38bb9398944de4bdef699835f17ec9&user_id=alfrick&app_id=local-runner-app
 |  thread=8800297152
[INFO] 16:02:30.101006 > API URL:      To call your model via the API, use this model URL: https://clarifai.com/alfrick/local-runner-app/models/local-runner-model
 |  thread=8800297152
[INFO] 16:02:30.101070 Press CTRL+C to stop the runner.
 |  thread=8800297152

When ready, the CLI prints:

A model URL for API calls
A Playground link for browser-based testing
A sample code snippet

Press Ctrl+C to stop the runner.

Step 5: Run Inference

While the local runner is active, test it using the OpenAI-compatible client:

Python

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.clarifai.com/v2/ext/openai/v1",
    api_key=os.environ['CLARIFAI_PAT'],
)

response = client.chat.completions.create(
    model="https://clarifai.com/<user-id>/local-runner-app/models/local-runner-model",
    messages=[\
        {"role": "system", "content": "Talk like a pirate."},\
        {"role": "user", "content": "How do I check if a Python object is an instance of a class?"},\
    ],
    stream=False,  # stream=True also works
)
print(response)

Or use the Clarifai CLI:

clarifai model predict https://clarifai.com/<user-id>/local-runner-app/models/local-runner-model "Explain AI in one sentence"

You can also open the Runners dashboard, find your runner, and select Open in Playground from the three-dot menu.

When you're done, close the terminal running the local runner to shut it down.

Additional Resources

Build your next AI app, test and tune popular LLMs models, and much more.