Joseph CostaandClaude Opus 4.8
9938d46a67
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
69 lines
3.0 KiB
Markdown
69 lines
3.0 KiB
Markdown
# Routing
|
|
|
|
## How `model: "auto"` decides
|
|
|
|
`choose_model()` in [`router.py`](../router.py) runs cheap→expensive, first match wins:
|
|
|
|
1. **Image in the request** → vision model (`vision_heavy` if the text mentions
|
|
OCR/document/table, else `vision`).
|
|
2. **Request carries `tools`** → agentic model (`code_heavy` if the text looks
|
|
code-ish, else `agentic`).
|
|
3. **Text heuristics** → code regex → coding model; reasoning words → reasoning
|
|
model; very short (< `SHORT_CHARS`) → the small fast model.
|
|
4. **Tiny classifier** — anything left over is labeled by a small always-warm
|
|
model (`CLASSIFIER`) into code / reason / vision / general.
|
|
|
|
The chosen model, plus any fallback, is reported in the response headers
|
|
`x-router-initial-model` (the decision) and `x-router-model` (what actually served).
|
|
|
|
## The model map — customize this for your machine
|
|
|
|
Two dicts at the top of `router.py`:
|
|
|
|
```python
|
|
M = {
|
|
"fast": "qwen3:8b", "general": "qwen3:14b",
|
|
"code": "glm-4.7-flash", "code_heavy": "qwen3-coder:30b",
|
|
"reason": "qwen3.6:35b-a3b", "agentic": "glm-4.7-flash",
|
|
"vision": "qwen3-vl:8b", "vision_heavy": "qwen3-vl:30b-a3b-instruct",
|
|
}
|
|
CLASSIFIER = "qwen3:8b"
|
|
```
|
|
|
|
Point these at models you've actually pulled (`ollama list`). Names must match
|
|
exactly, including any `:tag`. Restart the router after editing.
|
|
|
|
`FALLBACK` maps each model to a backup used when it errors or isn't pulled yet —
|
|
so the router degrades gracefully instead of failing.
|
|
|
|
## Explicit model override
|
|
|
|
Any request that names a real model (not `auto`) is passed straight through:
|
|
`{"model": "glm-4.7-flash", ...}`. Unknown names are handled by LiteLLM's catch-all
|
|
(`model_name: "*"` in `litellm.config.yaml`), which forwards *any* name to Ollama —
|
|
so `:tag` variants and models you pull later work without config changes.
|
|
|
|
## Policy: forcing a client onto a fleet (the "uncensored" example)
|
|
|
|
Put a token in `.uncensored_key`. Any request whose `Authorization: Bearer <token>`
|
|
matches is forced through `choose_uncensored()` — the same content heuristics, but
|
|
only ever selecting from the `UNCENSORED` map — **regardless of the model asked for**.
|
|
This pins a specific client (identified by its key) to a specific fleet.
|
|
|
|
Clients can also opt in per-request with `model: "auto-uncensored"` (no key needed).
|
|
|
|
Use this pattern for any policy split (a "coding-only" client, a "cheap models"
|
|
client, etc.): add a map + a `choose_*` function + a trigger in `do_POST`.
|
|
|
|
## Adding a model from Hugging Face
|
|
|
|
Ollama pulls GGUF repos directly:
|
|
```bash
|
|
ollama pull hf.co/USER/REPO:Q4_K_M # tag must match the quant in the filename
|
|
ollama cp hf.co/USER/REPO:Q4_K_M friendly-name # optional: rename
|
|
```
|
|
Then add `friendly-name` to the relevant map in `router.py`. Multi-file repos:
|
|
Ollama auto-pulls a vision `mmproj` if present; `mtp`/draft files are ignored.
|
|
Big prompts truncate at Ollama's default context — make a bigger-context copy with
|
|
`./scripts/make-context-variant.sh <model> <num_ctx>`.
|