Skip to content

LiteLLM proxy for the harness (backlog B32, DR-018 POOL-1)

A LiteLLM proxy puts every model we use behind one OpenAI-compatible endpoint: the Ollama machines on the LAN, Gemini, OpenRouter (with ZDR) and Claude. It adds per-key budgets and one place to see spend. The harness talks to it as an ordinary openai backend, so no new Python dependency is needed in MMDEP; the proxy runs separately (on the tools server or one LAN machine).

1. A harness model that points at the proxy

In mmdep/harness_config.json under models:

"pool-coder": {
  "backend": "openai",
  "litellm": true,
  "base_url": "http://192.168.1.20:4000/v1",
  "key_env": "LITELLM_API_KEY",
  "model": "coder-local",
  "allowed_classes": ["public", "internal", "restricted"],
  "usd_in": 0,
  "usd_out": 0
}
  • model is the LiteLLM model name (model_name below), not a provider model ID.
  • allowed_classes follows the most restrictive model behind that name (B17): a name that can route to a cloud model must not allow restricted.
  • Put it in fallback_order where it should be tried. mmdep harness check reports the proxy's health (/health/liveliness) and whether LITELLM_API_KEY is set.

2. Example litellm_config.yaml (on the proxy machine)

model_list:
  # Local Ollama machines on the LAN (DR-010): any data class, $0
  - model_name: coder-local
    litellm_params:
      model: ollama_chat/qwen2.5-coder:7b
      api_base: http://192.168.1.21:11434
  - model_name: coder-local            # same name twice = a pool; LiteLLM balances and fails over
    litellm_params:
      model: ollama_chat/qwen2.5-coder:7b
      api_base: http://192.168.1.22:11434

  # Gemini, paid tier (internal data allowed, DR-014 §3.1)
  - model_name: gemini-flash-lite
    litellm_params:
      model: gemini/gemini-3.5-flash-lite
      api_key: os.environ/GEMINI_API_KEY

  # OpenRouter free router, always with zero data retention (DR-015 §7.1, B22)
  - model_name: openrouter-free
    litellm_params:
      model: openrouter/openrouter/free
      api_key: os.environ/OPENROUTER_API_KEY
      extra_body:
        provider:
          zdr: true
          data_collection: deny

  # Claude (its own API account)
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 1
  fallbacks:
    - coder-local: [gemini-flash-lite]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY      # admin only; never given to agents
  database_url: os.environ/LITELLM_DATABASE_URL  # needed for keys and budgets (Postgres or SQLite)

Start it: litellm --config litellm_config.yaml --port 4000 (or the Docker image ghcr.io/berriai/litellm).

3. A budget per key

Create one key per consumer with the master key, each with its own spend cap and allowed models:

curl -s http://192.168.1.20:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"key_alias": "mmdep-harness", "max_budget": 10, "budget_duration": "30d",
       "models": ["coder-local", "gemini-flash-lite", "openrouter-free"]}'
  • mmdep-harness: the harness (store the returned key as LITELLM_API_KEY on the laptop or server).
  • One key per other agent or project (for example pe-agent, mcrm-agent), with smaller budgets.
  • The harness's own monthly cap (monthly_cap_usd, harness/spend.csv) still applies on top.

Notes

  • Keys live in environment variables only, never in this repository.
  • Data classes still apply (B17): the harness filters models by allowed_classes before it calls the proxy.