LLM Providers

Overview

clawft uses a pluggable provider system for LLM access. All providers use the OpenAI-compatible chat completions API format (POST /v1/chat/completions). The provider layer lives in the clawft-llm crate and has no dependencies on other clawft crates.

Key design principles:

• Single protocol -- every provider speaks the same OpenAI-compatible request/response format
• Prefix-based routing -- model identifiers like openai/gpt-4o encode both the target provider and the model name
• Environment-driven keys -- API keys are resolved from environment variables at request time, never persisted in configuration files
• Zero boilerplate -- nine providers are pre-configured and ready to use

Provider Routing

When clawft receives a model identifier (e.g., anthropic/claude-sonnet-4-20250514), the ProviderRouter performs these steps:

1. Scan registered prefixes using longest-prefix-first matching
2. Match the prefix (e.g., "anthropic/") to a provider
3. Strip the prefix to get the model name (e.g., "claude-sonnet-4-20250514")
4. Call provider.complete(ChatRequest { model: "claude-sonnet-4-20250514", ... })

If no prefix matches, the router falls back to the default provider (OpenAI) and passes the model string unchanged.

Built-in Providers

ProviderRouter::with_builtins() registers nine providers. Each requires only its corresponding environment variable.

Anthropic requests automatically include the anthropic-version: 2023-06-01 header.

Model Identifier Format

Model identifiers use provider/model-name. Only the first slash separates the prefix from the model name.

Minimal Configuration

Set an environment variable, then reference the model:

export ANTHROPIC_API_KEY="sk-ant-..."

{
  "agents": {
    "defaults": {
      "model": "anthropic/claude-sonnet-4-20250514"
    }
  }
}

No provider-specific configuration is needed for built-in providers.

Provider Overrides

Override base URLs and add custom headers per provider:

{
  "providers": {
    "anthropic": {
      "api_base": "https://my-proxy.example.com/v1",
      "extra_headers": {
        "X-Org-Id": "my-org-123"
      }
    }
  }
}

When api_base is set, it replaces the built-in base URL. When extra_headers are provided, they merge with existing provider headers.

Custom Providers

Via Configuration

{
  "providers": {
    "custom": {
      "api_base": "http://localhost:11434/v1",
      "api_key": "not-needed"
    }
  }
}

Programmatic

use clawft_llm::{ProviderConfig, ProviderRouter};
use std::collections::HashMap;

let mut configs = clawft_llm::config::builtin_providers();

configs.push(ProviderConfig {
    name: "my-service".into(),
    base_url: "https://my-llm-service.example.com/v1".into(),
    api_key_env: "MY_SERVICE_API_KEY".into(),
    model_prefix: Some("my-service/".into()),
    default_model: Some("my-model-v2".into()),
    headers: HashMap::new(),
});

let router = ProviderRouter::from_configs(configs);

Implementing the Provider Trait

For services that do not follow the OpenAI format:

#[async_trait]
impl Provider for MyCustomProvider {
    fn name(&self) -> &str { "my-custom" }

    async fn complete(&self, request: &ChatRequest) -> Result<ChatResponse> {
        // Transform request, call API, transform response
        todo!()
    }
}

Using Local LLMs

Any local server with an OpenAI-compatible /v1/chat/completions endpoint works.

Ollama

ollama pull llama3

{
  "providers": {
    "openai": {
      "api_base": "http://localhost:11434/v1"
    }
  },
  "agents": {
    "defaults": {
      "model": "openai/llama3"
    }
  }
}

export OPENAI_API_KEY="not-needed"

llama.cpp Server

./llama-server -m model.gguf --port 8080

Point api_base to http://localhost:8080/v1.

vLLM

python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3-8B

Point api_base to http://localhost:8000/v1.

Automatic Retry

Every provider is wrapped in a RetryPolicy with exponential backoff:

Retryable errors: HTTP 429 (rate limited), 500-599 (server errors), timeouts, and network failures. Non-retryable: authentication failures, model not found, invalid responses, and permanent billing issues.

Error Handling

All provider operations return Result<T, ProviderError>:

API Key Management

Keys are resolved at request time:

1. Explicit key (if OpenAiCompatProvider::with_api_key() was used)
2. Environment variable named in ProviderConfig.api_key_env

If neither provides a key, ProviderError::NotConfigured is returned before any network request.

Best practices:

• Never commit API keys to source control
• Use environment variables or a secrets manager
• Rotate keys regularly
• The Debug implementation redacts keys, displaying ***

Troubleshooting

"provider not configured" -- The environment variable is not set. The error message names the specific variable.

"authentication failed" (401/403) -- Verify the key is correct and not expired.

"model not found" (404) -- Check for typos. Remember the prefix is stripped: openai/gpt-4o sends gpt-4o to the API.

Requests go to the wrong provider -- Verify your model string starts with a registered prefix. Use RUST_LOG=clawft_llm=debug to see routing details.

Local LLM parse errors -- Ensure the server returns the exact OpenAI chat completion JSON format with all required fields.