# Serve any HuggingFace LLM locally or deploy to cloud compute

---

## SGLang

[SGLang](https://github.com/sgl-project/sglang) is a high-performance open-source inference engine for LLMs with support for structured generation and multimodal reasoning. With Clarifai, you can deploy SGLang models to cloud GPUs with a single command.

The workflow is: **init → deploy**

> **Important:** SGLang requires **Linux with an NVIDIA GPU** (Ampere or newer, compute capability >= 8.0). It does not run on macOS or Windows. For local testing on non-Linux machines, use [vLLM](https://docs.clarifai.com/compute/toolkits/vllm) or [Ollama](https://docs.clarifai.com/compute/toolkits/ollama) instead.

## Step 1: Install Clarifai

- Bash

```bash
pip install --upgrade clarifai
```

> **Note:** Python 3.11 or 3.12 is required. The `openai` package is included with `clarifai`.

## Step 2: Log In

- CLI

```bash
clarifai login
```

You'll be prompted for your user ID and [PAT](https://docs.clarifai.com/control/authentication/pat). This saves your credentials locally so you don't need to set environment variables manually.

### Example Output

```text
clarifai login
Enter your Clarifai user ID: user-id
> To authenticate, you'll need a Personal Access Token (PAT).
> You can create one from your account settings: https://clarifai.com/user-id/settings/security

Enter your Personal Access Token (PAT) value (or type "ENVVAR" to use an environment variable): XXXXXXXXXX

> Verifying token...
[INFO] 12:10:55.558733 Validating the Context Credentials... |  thread=8729403584
[INFO] 12:10:56.693295 ✅ Context is valid |  thread=8729403584

> Let's save these credentials to a new context.
> You can have multiple contexts to easily switch between accounts or projects.

Enter a name for this context [default]:
✅ Success! You are now logged in.
Credentials saved to the 'default' context.

💡 To switch contexts later, use `clarifai config use-context <name>`.
[INFO] 12:10:59.177368 Login successful for user 'alfrick' in context 'default' |  thread=8729403584
```

## Step 3: Initialize a Model

Scaffold a model project using a HuggingFace model name:

- CLI

```bash
clarifai model init --toolkit sglang --model-name Qwen/Qwen3-4B
```

The CLI auto-selects an Ampere+ GPU instance based on the model's VRAM requirements. You can initialize any model [supported by SGLang](https://docs.sglang.ai/references/supported_models.html) — just change `--model-name` to a different HuggingFace repo ID.

### Example Output

```text
clarifai model init --toolkit sglang --model-name Qwen/Qwen3-4B
[INFO] Initializing model with sglang toolkit...
[INFO] Updated Hugging Face model repo_id to: Qwen/Qwen3-4B
  Instance: g5.xlarge (Estimated 15.9 GiB VRAM (7.5 GiB weights + 5.6 GiB KV cache for 40960 ctx), fits g5.xlarge (22 GiB))

Model initialized in ./Qwen3-4B

Test locally:
    clarifai model serve ./Qwen3-4B
    clarifai model serve ./Qwen3-4B --mode env       # auto-create venv and install deps
    clarifai model serve ./Qwen3-4B --mode container # run inside Docker

Deploy to Clarifai:
    clarifai model deploy ./Qwen3-4B
    clarifai list-instances                             # list available instances
```

This creates a `./Qwen3-4B/` directory:

```text
Qwen3-4B/
├── 1/
│   └── model.py       # SGLang inference logic
├── requirements.txt   # Lightweight deps (SGLang is pre-installed in the base image)
└── config.yaml        # Model config (user_id/app_id auto-filled from login)
```

> **Note:** For private or gated models (Llama, Gemma, etc.), set `HF_TOKEN` in your environment before initializing:
> 
> ```sh
> export HF_TOKEN=your_token_here
> ```

### model.py

```text
import os
import sys

sys.path.append(os.path.dirname(__file__))
from typing import Iterator, List

from clarifai.runners.models.model_builder import ModelBuilder
from clarifai.runners.models.openai_class import OpenAIModelClass
from clarifai.runners.utils.data_utils import Param
from clarifai.runners.utils.openai_convertor import build_openai_messages
from clarifai.utils.logging import logger
from openai import OpenAI
from openai_server_starter import OpenAI_APIServer
##################

class SglangModel(OpenAIModelClass):
    """
    A custom runner that integrates with the Clarifai platform and uses Server inference
    to process inputs, including text.
    """

client = True  # This will be set in load_model method
    model = True  # This will be set in load_model method

def load_model(self):
        """Load the model here and start the server."""
        os.path.join(os.path.dirname(__file__))

# Use downloaded checkpoints.
        # Or if you intend to download checkpoint at runtime, set hf id instead. For example:
        # checkpoints = "Qwen/Qwen2-7B-Instruct"

# server args were generated by `upload` module
        server_args = {
            'dtype': 'auto',
            'kv_cache_dtype': 'auto',
            'tp_size': 1,
            'load_format': 'auto',
            'context_length': None,
            'device': 'cuda',
            'port': 23333,
            'host': '0.0.0.0',
            'mem_fraction_static': 0.9,
            'max_total_tokens': '8192',
            'max_prefill_tokens': None,
            'schedule_policy': 'fcfs',
            'schedule_conservativeness': 1.0,
            'checkpoints': 'runtime',
        }

# if checkpoints == "checkpoints" => assign to checkpoints var aka local checkpoints path
        stage = server_args.get("checkpoints")
        if stage in ["build", "runtime"]:
            # checkpoints = os.path.join(os.path.dirname(__file__), "checkpoints")
            config_path = os.path.dirname(os.path.dirname(__file__))
            builder = ModelBuilder(config_path, download_validation_only=True)
            checkpoints = builder.download_checkpoints(stage=stage)
            server_args.update({"checkpoints": checkpoints})

if server_args.get("additional_list_args") == ['']:
            server_args.pop("additional_list_args")

# Start server
        # This line were generated by `upload` module
        self.server = OpenAI_APIServer.from_sglang_backend(**server_args)

# Create client
        self.client = OpenAI(
            api_key="notset", base_url=SglangModel.make_api_url(self.server.host, self.server.port)
        )
        self.model = self._get_model()

logger.info(f"OpenAI {self.model} model loaded successfully!")

def _get_model(self):
        try:
            return self.client.models.list().data[0].id
        except Exception as e:
            raise ConnectionError("Failed to retrieve model ID from API") from e

@staticmethod
    def make_api_url(host: str, port: int, version: str = "v1") -> str:
        return f"http://{host}:{port}/{version}"

@OpenAIModelClass.method
    def predict(
        self,
        prompt: str,
        chat_history: List[dict] = None,
        max_tokens: int = Param(
            default=512,
            description="The maximum number of tokens to generate. Shorter token lengths will provide faster performance.",
        ),
        temperature: float = Param(
            default=0.7,
            description="A decimal number that determines the degree of randomness in the response",
        ),
        top_p: float = Param(
            default=0.8,
            description="An alternative to sampling with temperature, where the model considers the results of the tokens with top_p probability mass.",
        ),
    ) -> str:
        """This is the method that will be called when the runner is run. It takes in an input and
        returns an output."""
        openai_messages = build_openai_messages(prompt=prompt, messages=chat_history)
        response = self.client.chat.completions.create(
            model=self.model,
            messages=openai_messages,
            max_completion_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
        )
        if response.usage and response.usage.prompt_tokens and response.usage.completion_tokens:
            self.set_output_context(
                prompt_tokens=response.usage.prompt_tokens,
                completion_tokens=response.usage.completion_tokens,
            )
        return response.choices[0].message.content

@OpenAIModelClass.method
    def generate(
        self,
        prompt: str,
        chat_history: List[dict] = None,
        max_tokens: int = Param(
            default=512,
            description="The maximum number of tokens to generate. Shorter token lengths will provide faster performance.",
        ),
        temperature: float = Param(
            default=0.7,
            description="A decimal number that determines the degree of randomness in the response",
        ),
        top_p: float = Param(
            default=0.8,
            description="An alternative to sampling with temperature, where the model considers the results of the tokens with top_p probability mass.",
        ),
    ) -> Iterator[str]:
        """Example yielding a whole batch of streamed stuff back."""
        openai_messages = build_openai_messages(prompt=prompt, messages=chat_history)
        for chunk in self.client.chat.completions.create(
            model=self.model,
            messages=openai_messages,
            max_completion_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
            stream=True,
        ):
            if chunk.choices:
                text = (
                    chunk.choices[0].delta.content
                    if (chunk and chunk.choices[0].delta.content) is not None
                    else ''
                )
                yield text

# This method is needed to test the model with the test-locally CLI command.
    def test(self):
        """Test the model here."""
        try:
            print("Testing predict...")
            # Test predict
            print(
                self.predict(
                    prompt="Hello, how are you?",
                )
            )
        except Exception as e:
            print("Error in predict", e)

try:
            print("Testing generate...")
            # Test generate
            for each in self.generate(
                prompt="Hello, how are you?",
            ):
                print(each, end=" ")
        except Exception as e:
            print("Error in generate", e)
```

### config.yaml

```text
build_info:
  image: lmsysorg/sglang:latest
checkpoints:
  repo_id: Qwen/Qwen3-4B
  type: huggingface
  when: runtime
compute:
  instance: g5.xlarge
model:
  id: qwen3-4b
```

### requirements.txt

```text
clarifai
openai
```

## Step 4: Deploy to Cloud

Deploy to dedicated cloud compute. Clarifai automatically provisions all required infrastructure.

Since `config.yaml` already has a `compute.instance` value (auto-selected during `init`), you can deploy directly:

- CLI

```bash
clarifai model deploy ./Qwen3-4B
```

To override the instance type:

- CLI

```bash
clarifai model deploy ./Qwen3-4B --instance g5.xlarge
```

To see all available instance types and pricing:

- CLI

```bash
clarifai list-instances
```

> **Tip:** If you have a local Linux GPU and want to test before deploying, run `clarifai model serve ./Qwen3-4B --mode env` first.

> **Note:** Clarifai automatically optimizes replica routing using KV cache affinity, improving throughput for shared system prompts, RAG pipelines, and multi-turn conversations. See [Request Routing](https://docs.clarifai.com/compute/inference/routing) for details.

## Step 5: Run Inference

- Python

```python
import os
from openai import OpenAI

# Initialize the OpenAI client, pointing to Clarifai's API
client = OpenAI(
    base_url="https://api.clarifai.com/v2/ext/openai/v1",  # Clarifai's OpenAI-compatible API endpoint
    api_key=os.environ["CLARIFAI_PAT"]  # Ensure CLARIFAI_PAT is set as an environment variable
)

# Make a chat completion request to a Clarifai-hosted model
response = client.chat.completions.create(
    model="https://clarifai.com/<user-id>/local-runner-app/models/local-runner-model",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the future of AI?"}
    ],
)

# Print the model's response
print(response.choices[0].message.content)
```

Or use the Clarifai CLI:

```sh
clarifai model predict https://clarifai.com/<user-id>/main/models/Qwen3-4B "Explain AI in one sentence"
```

### Manage Your Deployment

```bash
# Stream live logs
clarifai model logs --deployment <deployment-id>

# Check status
clarifai model status --deployment <deployment-id>

# Remove deployment when done (stops billing)
clarifai model undeploy --deployment <deployment-id>
```

For the full CLI reference, see [CLI Reference](https://docs.clarifai.com/resources/api-overview/cli#clarifai-model-deploy).

Build your next AI app, test and tune popular LLMs models, and much more.