How Clarifai routes prediction requests for optimal performance

When you send a prediction request to a deployed model, Clarifai's routing system determines where that request is processed.

Routing happens at two levels:

• First selecting the right nodepool (which hardware);
• Then, selecting the right replica (which pod within that hardware).

This page covers both levels, what Clarifai optimizes automatically, and what you can configure.

Nodepool Selection

When a request arrives, Clarifai picks a nodepool in this order:

1. Explicit routing — If you specify a deployment_id (or compute_cluster_id + nodepool_id) in the request, the request goes directly to that nodepool.
2. Deployment lookup — If you don't specify routing, Clarifai finds active deployments for the model version and picks the best nodepool based on queue depth and latency.
3. Shared compute fallback — For Clarifai-owned models with no user deployment, the request falls back to Clarifai's shared infrastructure.

Multi-Nodepool Deployments

A single deployment can span multiple nodepools with priority ordering. When you add nodepools to a deployment, you assign each a priority rank (1 being highest).

Requests are routed to the highest-priority nodepool that has available capacity, with overflow spilling to the next.

Note: You can configure multi-nodepool deployments in the UI by dragging nodepools into your preferred priority order, or via the API using deployment_nodepools with sequence values.

This enables several powerful patterns:

• Cross-cloud failover — Place nodepools in different cloud providers (AWS, GCP) so traffic fails over automatically if one region has issues.
• Cross-hardware routing — Since Clarifai's Compute Orchestration supports any hardware, you can route between NVIDIA and AMD GPUs, or even between GPU and CPU nodepools, within a single deployment. This means you can mix x86 and ARM architectures, or combine spot and on-demand instances for cost optimization.
• Tiered performance — Use a high-performance GPU nodepool as primary and a cheaper instance as overflow.

Explicit Request-Level Routing

You can override the default routing behavior for individual requests by specifying a particular deployment ID or nodepool.

• Python SDK

# Route to a specific deployment
response = model.predict(inputs, deployment_id="my-deployment")

# Route to a specific nodepool
response = model.predict(inputs, compute_cluster_id="my-cluster", nodepool_id="my-nodepool")

Automatic Replica Optimization

Once a nodepool is selected, Clarifai intelligently routes the request to the replica most likely to deliver the fastest response.

This is fully automatic — you don't configure it, but understanding what it does helps you design better deployments.

KV Cache Routing (Prefix Cache Routing)

For models backed by inference engines with KV caching (such as vLLM and SGLang), Clarifai detects shared prompt prefixes and routes requests to replicas that already hold those prefixes in their KV cache.

This avoids redundant recomputation of attention keys and values, resulting in higher throughput and lower time-to-first-token (TTFT).

Note: This is fully automatic — no configuration or code changes required.

This is especially effective for:

• Shared system prompts — Multiple users hitting the same model with the same system instruction share the same prefix. Rather than recomputing it on every replica, requests are routed to replicas that already have that prefix cached.
• RAG pipelines — Users querying the same knowledge base receive similar retrieved-document prefixes, which are automatically reused across requests.
• Multi-turn conversations — Follow-up messages share the prior conversation as a prefix, so subsequent turns reuse cached state from earlier turns.

When a matching prefix is found, the request routes to the best-matched replica. When no prefix match exists, the system falls back gracefully — there is no performance penalty compared to routing without this feature.

Autoscaling

Autoscaling controls how many replicas are running within a nodepool. You configure this per-nodepool within a deployment:

Note: You can configure autoscaling in the UI during deployment, via the Python SDK, or via the CLI.

Prediction Caching

Prediction caching stores and reuses previously computed model outputs for identical requests, eliminating redundant inference and significantly reducing latency and compute costs.

For repeated requests with the same input, model, and model version, you can bypass inference entirely by enabling prediction caching with the use_predict_cache parameter.

• cURL

curl -X POST "https://api.clarifai.com/v2/users/moonshotai/apps/chat-completion/models/Kimi-K2_6/outputs" \
  -H "Authorization: Key YOUR_PAT_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {
        "data": {
          "text": {
            "raw": "How do I check if a Python object is an instance of a class?"
          }
        }
      }
    ],
    "use_predict_cache": true
  }'

When enabled, Clarifai caches the full prediction response for identical input + model + version combinations. Subsequent matching requests are served directly from cache without invoking model compute.

Inference Cost Savings and Cache Hits

For requests using KV cache routing, if a request is routed to a replica that already contains cached prompt state, the cached portion of the prompt is billed at a 90% discount (10% of standard prompt token pricing).

The Clarifai Python SDK automatically extracts cached_tokens from the model response metadata and forwards it to the output proto.