# Download and run Ollama models locally and expose them via a public API

Ollama is an open-source tool that lets you run LLMs directly on your local machine. Combined with Clarifai's Local Runners, you can serve Ollama models from your machine, expose them via a public API, and access them through the Clarifai platform — all while keeping the speed, privacy, and control of local inference.

## Step 1: Install Prerequisites

### Install Ollama

Go to the [Ollama website](https://ollama.com/download) and install the appropriate version for your system (macOS, Windows, or Linux).

> **Note:** On Windows, restart your machine after installing to ensure environment variables are applied.

### Install Clarifai

- Bash

```bash
pip install --upgrade clarifai
```

> **Note:** Python 3.11 or 3.12 is required. The `openai` package is included with `clarifai`.

## Step 2: Log In

- CLI

```bash
clarifai login
```

You'll be prompted for your user ID and [PAT](https://docs.clarifai.com/control/authentication/pat). This saves your credentials locally so you don't need to set environment variables manually.

## Step 3: Initialize a Model

Scaffold a model project using any model from the [Ollama library](https://ollama.com/library):

- CLI

```bash
clarifai model init --toolkit ollama --model-name gpt-oss:20b
```

Example Output

```text
clarifai model init --toolkit ollama --model-name gpt-oss:20b
[INFO] Initializing model with ollama toolkit...

Model initialized in "./gpt-oss:20b"

Test locally:
    clarifai model serve "./gpt-oss:20b"
    clarifai model serve "./gpt-oss:20b" --mode env       # auto-create venv and install deps
    clarifai model serve "./gpt-oss:20b" --mode container # run inside Docker

Deploy to Clarifai:
    clarifai model deploy "./gpt-oss:20b" --instance a10g
    clarifai list-instances                               # list available instances
```

This creates a `./gpt-oss:20b/` directory with three files:

```text
gpt-oss:20b/
├── 1/
│   └── model.py       # Ollama inference logic
├── requirements.txt   # Python dependencies
└── config.yaml        # Model config (user_id/app_id auto-filled from login)
```

Omitting `--model-name` will install the default model (`llama3.2`):

```sh
clarifai model init --toolkit ollama
```

Use `ollama list` to see downloaded models and `ollama rm` to remove one.

> **Note:** Some models are very large and may require significant memory or GPU resources. Check your machine's capacity before initializing.

## Step 4: Serve Locally

Start the model as a local runner:

- CLI

```bash
clarifai model serve "./gpt-oss:20b"
```

> **Note:** The first run pulls the model weights from Ollama, which may take a few minutes. Add `-v` for verbose logs.

Example Output

```text
clarifai model local-runner
[INFO] 16:01:28.904230 > Checking local runner requirements... |  thread=8800297152
[INFO] 16:01:28.928129 Checking 2 dependencies... |  thread=8800297152
[INFO] 16:01:28.928672 ✅ All 2 dependencies are installed! |  thread=8800297152
[INFO] 16:01:28.928886 Verifying Ollama installation... |  thread=8800297152
[INFO] 16:01:29.004234 > Verifying local runner setup... |  thread=8800297152
[INFO] 16:01:29.004427 Current context: default |  thread=8800297152
[INFO] 16:01:29.004463 Current user_id: alfrick |  thread=8800297152
[INFO] 16:01:29.004490 Current PAT: d6570**** |  thread=8800297152
[INFO] 16:01:29.005945 Current compute_cluster_id: local-runner-compute-cluster |  thread=8800297152
[WARNING] 16:01:35.936440 Failed to get compute cluster with ID 'local-runner-compute-cluster':
code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "ComputeCluster with ID 'local-runner-compute-cluster' not found. Check your request fields."
req_id: "sdk-python-11.8.2-75ca9226003a4b34a770885b119d5814"
 |  thread=8800297152
Compute cluster not found. Do you want to create a new compute cluster alfrick/local-runner-compute-cluster? (y/n): y
[INFO] 16:01:58.382096 Compute Cluster with ID 'local-runner-compute-cluster' is created:
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.8.2-a0474b8ba93c4e069804500a188694db"
 |  thread=8800297152
[INFO] 16:01:58.391571 Current nodepool_id: local-runner-nodepool |  thread=8800297152
[WARNING] 16:02:00.633687 Failed to get nodepool with ID 'local-runner-nodepool':
code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "Nodepool not found. Check your request fields."
req_id: "sdk-python-11.8.2-62149d46e7104d35bcb2a36546710329"
 |  thread=8800297152
Nodepool not found. Do you want to create a new nodepool alfrick/local-runner-compute-cluster/local-runner-nodepool? (y/n): y
[INFO] 16:02:03.909005 Nodepool with ID 'local-runner-nodepool' is created:
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.8.2-05836a431940495b94b9c3691f6c6d4d"
 |  thread=8800297152
[INFO] 16:02:03.921694 Current app_id: local-runner-app |  thread=8800297152
[INFO] 16:02:04.203774 Current model_id: local-runner-model |  thread=8800297152
[WARNING] 16:02:10.933734 Attempting to patch latest version: 9d38bb9398944de4bdef699835f17ec9 |  thread=8800297152
[INFO] 16:02:14.195999 Successfully patched version 9d38bb9398944de4bdef699835f17ec9 |  thread=8800297152
[INFO] 16:02:14.197924 Current model version 9d38bb9398944de4bdef699835f17ec9 |  thread=8800297152
[WARNING] 16:02:18.679567 Failed to get runner with ID 'f3c46913186449ba99dedd38123d47a3':
code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "Runner not found. Check your request fields."
req_id: "sdk-python-11.8.2-0b3b241c76ef429290c8c54a318f2f21"
 |  thread=8800297152
[INFO] 16:02:18.679913 Creating the local runner tying this 'alfrick/local-runner-app/models/local-runner-model' model (version: 9d38bb9398944de4bdef699835f17ec9) to the 'alfrick/local-runner-compute-cluster/local-runner-nodepool' nodepool. |  thread=8800297152
[INFO] 16:02:19.757117 Runner with ID '2f84d7194ee8464fad485fd058663fe5' is created:
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.8.2-8ac525dc13ec47629213f0283e89c6a7"
 |  thread=8800297152
[INFO] 16:02:19.765198 Current runner_id: 2f84d7194ee8464fad485fd058663fe5 |  thread=8800297152
[WARNING] 16:02:20.331980 Failed to get deployment with ID local-runner-deployment:
code: CONN_DOES_NOT_EXIST
description: "Resource does not exist"
details: "Deployment with ID 'local-runner-deployment' not found. Check your request fields."
req_id: "sdk-python-11.8.2-864f55e6bc554614894ee031cc30cdb9"
 |  thread=8800297152
Deployment not found. Do you want to create a new deployment alfrick/local-runner-compute-cluster/local-runner-nodepool/local-runner-deployment? (y/n): y
[INFO] 16:02:25.016935 Deployment with ID 'local-runner-deployment' is created:
code: SUCCESS
description: "Ok"
req_id: "sdk-python-11.8.2-b6bf9aad5aa545f8a041113608fc9365"
 |  thread=8800297152
[INFO] 16:02:25.024579 Current deployment_id: local-runner-deployment |  thread=8800297152
[INFO] 16:02:25.027108 Current model section of config.yaml: {'app_id': 'local-dev-runner-app', 'id': 'local-dev-model', 'model_type_id': 'text-to-text', 'user_id': 'clarifai-user-id'} |  thread=8800297152
Do you want to backup config.yaml to config.yaml.bk then update the config.yaml with the new model information? (y/n): y
[INFO] 16:02:27.407724 Checking 2 dependencies... |  thread=8800297152
[INFO] 16:02:27.408555 ✅ All 2 dependencies are installed! |  thread=8800297152
[INFO] 16:02:27.451117 Customizing Ollama model with provided parameters... |  thread=8800297152
[INFO] 16:02:27.451785 ✅ Starting local runner... |  thread=8800297152
[INFO] 16:02:27.451852 No secrets path configured, running without secrets |  thread=8800297152
[INFO] 16:02:30.020253 Detected OpenAI chat completions for Clarifai model streaming - validating stream_options... |  thread=8800297152
[INFO] 16:02:30.027464 Starting Ollama server in the host: 127.0.0.1:23333 |  thread=8800297152
[INFO] 16:02:30.040882 Model llama3.2 pulled successfully. |  thread=8800297152
[INFO] 16:02:30.041191 Ollama server started successfully on 127.0.0.1:23333 |  thread=8800297152
[INFO] 16:02:30.096053 Ollama model loaded successfully: llama3.2 |  thread=8800297152
[INFO] 16:02:30.096133 ModelServer initialized successfully |  thread=8800297152
[INFO] 16:02:30.100802 ✅ Your model is running locally and is ready for requests from the API...
 |  thread=8800297152
[INFO] 16:02:30.100873 > Code Snippet: To call your model via the API, use this code snippet:

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.clarifai.com/v2/ext/openai/v1",
    api_key=os.environ['CLARIFAI_PAT'],
)

response = client.chat.completions.create(
    model="https://clarifai.com/alfrick/local-runner-app/models/local-runner-model",
    messages=[\
        {"role": "system", "content": "Talk like a pirate."},\
        {\
            "role": "user",\
            "content": "How do I check if a Python object is an instance of a class?",\
        },\
    ],
    temperature=1.0,
    stream=False,  # stream=True also works, just iterator over the response
)
print(response)
 |  thread=8800297152
[INFO] 16:02:30.100944 > Playground:   To chat with your model, visit: https://clarifai.com/playground?model=local-runner-model__9d38bb9398944de4bdef699835f17ec9&user_id=alfrick&app_id=local-runner-app
 |  thread=8800297152
[INFO] 16:02:30.101006 > API URL:      To call your model via the API, use this model URL: https://clarifai.com/alfrick/local-runner-app/models/local-runner-model
 |  thread=8800297152
[INFO] 16:02:30.101070 Press CTRL+C to stop the runner.
 |  thread=8800297152
```

When ready, the CLI prints:
- A model URL for API calls
- A Playground link for browser-based testing
- A sample code snippet

Press `Ctrl+C` to stop the runner.

## Step 5: Run Inference

While the local runner is active, test it using the OpenAI-compatible client:

- Python

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.clarifai.com/v2/ext/openai/v1",
    api_key=os.environ['CLARIFAI_PAT'],
)

response = client.chat.completions.create(
    model="https://clarifai.com/<user-id>/local-runner-app/models/local-runner-model",
    messages=[\
        {"role": "system", "content": "Talk like a pirate."},\
        {"role": "user", "content": "How do I check if a Python object is an instance of a class?"},\
    ],
    stream=False,  # stream=True also works
)
print(response)
```

Or use the Clarifai CLI:

```sh
clarifai model predict https://clarifai.com/<user-id>/local-runner-app/models/local-runner-model "Explain AI in one sentence"
```

You can also open the **[Runners](/content/compute/runners/index.html)** dashboard, find your runner, and select **Open in Playground** from the three-dot menu.

When you're done, close the terminal running the local runner to shut it down.

## Additional Resources

- [More Ollama Python examples](https://github.com/ollama/ollama-python/tree/main/examples)
- [Clarifai inference examples with Ollama](https://docs.clarifai.com/compute/inference/clarifai/api)
- [Ollama runner example on GitHub](https://github.com/Clarifai/runners-examples/tree/main/local-runners/ollama-model-upload)
- [YouTube: Running OpenAI's GPT-OSS-20B with Ollama](https://www.youtube.com/watch?v=TfS2p8LZYBE)

Build your next AI app, test and tune popular LLMs models, and much more.