llm-router/docs/ROUTING.md

Routing

How model: "auto" decides

choose_model() in router.py runs cheap→expensive, first match wins:

1. Image in the request → vision model (vision_heavy if the text mentions OCR/document/table, else vision).
2. Request carries tools → agentic model (code_heavy if the text looks code-ish, else agentic).
3. Text heuristics → code regex → coding model; reasoning words → reasoning model; very short (< SHORT_CHARS) → the small fast model.
4. Tiny classifier — anything left over is labeled by a small always-warm model (CLASSIFIER) into code / reason / vision / general.

The chosen model, plus any fallback, is reported in the response headers x-router-initial-model (the decision) and x-router-model (what actually served).

The model map — customize this for your machine

Two dicts at the top of router.py:

M = {
    "fast":   "qwen3:8b",       "general": "qwen3:14b",
    "code":   "glm-4.7-flash",  "code_heavy": "qwen3-coder:30b",
    "reason": "qwen3.6:35b-a3b", "agentic": "glm-4.7-flash",
    "vision": "qwen3-vl:8b",    "vision_heavy": "qwen3-vl:30b-a3b-instruct",
}
CLASSIFIER = "qwen3:8b"

Point these at models you've actually pulled (ollama list). Names must match exactly, including any :tag. Restart the router after editing.

FALLBACK maps each model to a backup used when it errors or isn't pulled yet — so the router degrades gracefully instead of failing.

Explicit model override

Any request that names a real model (not auto) is passed straight through: {"model": "glm-4.7-flash", ...}. Unknown names are handled by LiteLLM's catch-all (model_name: "*" in litellm.config.yaml), which forwards any name to Ollama — so :tag variants and models you pull later work without config changes.

Policy: forcing a client onto a fleet (the "uncensored" example)

Put a token in .uncensored_key. Any request whose Authorization: Bearer <token> matches is forced through choose_uncensored() — the same content heuristics, but only ever selecting from the UNCENSORED map — regardless of the model asked for. This pins a specific client (identified by its key) to a specific fleet.

Clients can also opt in per-request with model: "auto-uncensored" (no key needed).

Use this pattern for any policy split (a "coding-only" client, a "cheap models" client, etc.): add a map + a choose_* function + a trigger in do_POST.

Adding a model from Hugging Face

Ollama pulls GGUF repos directly:

ollama pull hf.co/USER/REPO:Q4_K_M          # tag must match the quant in the filename
ollama cp   hf.co/USER/REPO:Q4_K_M  friendly-name   # optional: rename

Then add friendly-name to the relevant map in router.py. Multi-file repos: Ollama auto-pulls a vision mmproj if present; mtp/draft files are ignored. Big prompts truncate at Ollama's default context — make a bigger-context copy with ./scripts/make-context-variant.sh <model> <num_ctx>.