Initial commit: llm-router — smart OpenAI/Anthropic/Ollama router
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Joseph Costa
@@ -0,0 +1,45 @@
|
||||
# Audio / voice
|
||||
|
||||
Ollama can *understand* audio with speech-capable models (e.g. Gemma 4 `e4b`/`e2b`/
|
||||
`12b`), but only via its **native `/api/chat`** with the audio base64 in the
|
||||
`images` field — the OpenAI `input_audio` format is silently dropped, and there's
|
||||
no native speech-to-text endpoint. This router bridges that gap.
|
||||
|
||||
Set the model with `ROUTER_AUDIO_MODEL` (default `gemma4:e4b`). It must be a model
|
||||
whose `ollama show` capabilities include `audio`. **Note:** some fine-tunes/quants
|
||||
have broken audio even if the flag is set — test before relying on one.
|
||||
|
||||
## Speech-to-text (Whisper-shaped)
|
||||
|
||||
```bash
|
||||
curl http://<host>:8080/v1/audio/transcriptions \
|
||||
-F file=@recording.wav \
|
||||
-F model=whisper-1
|
||||
# -> {"text": "the transcription"}
|
||||
```
|
||||
- `response_format=text` returns plain text instead of JSON.
|
||||
- `/v1/audio/translations` does the same but translates to English.
|
||||
- Works with the OpenAI SDK: set `base_url` to the router and call
|
||||
`audio.transcriptions.create(...)`. The `model` field is ignored — the router
|
||||
always uses `ROUTER_AUDIO_MODEL`.
|
||||
|
||||
## Audio inside a chat
|
||||
|
||||
Send OpenAI `input_audio` content to `/v1/chat/completions` and ask about it:
|
||||
```json
|
||||
{"model":"auto","messages":[{"role":"user","content":[
|
||||
{"type":"text","text":"answer the question in this audio"},
|
||||
{"type":"input_audio","input_audio":{"data":"<base64-wav>","format":"wav"}}
|
||||
]}]}
|
||||
```
|
||||
The router detects the audio, translates it to Ollama's native call, runs it on
|
||||
`ROUTER_AUDIO_MODEL`, and returns a normal OpenAI response (streaming supported).
|
||||
|
||||
## Limitations
|
||||
|
||||
- **WAV is verified.** Other formats (mp3/m4a/ogg) depend on Ollama's decoding —
|
||||
transcode to WAV if they fail.
|
||||
- **Not streaming STT** — it transcribes a complete clip per request (like
|
||||
Whisper's file API), not a live mic stream. Perfect for record-then-send.
|
||||
- Audio only works through the router's native bridge, so audio requests always
|
||||
target `ROUTER_AUDIO_MODEL` (they bypass the normal model-selection map).
|
||||
@@ -0,0 +1,40 @@
|
||||
# Benchmarks
|
||||
|
||||
Four self-contained harnesses in [`benchmarks/`](../benchmarks/) score your local
|
||||
fleet on speed, quality, and tool use. They talk to Ollama (`:11434`) and the
|
||||
router (`:8080`) directly. Run them with the venv Python, from the `benchmarks/`
|
||||
directory (they import each other):
|
||||
|
||||
```bash
|
||||
cd benchmarks
|
||||
../.venv/bin/python benchmark.py # speed + OCR + routing correctness
|
||||
../.venv/bin/python quality_bench.py # objective quality (code exec, math, MC, instr)
|
||||
../.venv/bin/python agent_bench.py # basic tool-calling
|
||||
../.venv/bin/python hard_agent_bench.py # multi-turn agentic (seq/parallel/recover/abstain)
|
||||
```
|
||||
|
||||
Each writes a `*_report.md` (and some a `*_results.json`). Sample outputs from the
|
||||
reference machine (Apple M4 Max, 36 GB) are checked in under `benchmarks/results/`.
|
||||
|
||||
## What each measures
|
||||
|
||||
- **benchmark.py** — generation & prompt tok/s and GPU placement (from Ollama's
|
||||
own eval metrics), an OCR pass on vision models, and live routing-decision checks.
|
||||
- **quality_bench.py** — objective auto-grading: executed unit tests (coding),
|
||||
exact-match math (GSM8K-style), multiple-choice, and programmatic
|
||||
instruction-following. Runs with thinking disabled for a fast, comparable baseline.
|
||||
- **agent_bench.py** — tool-calling basics via Ollama's tools API: invoke, args,
|
||||
selection, use-result, and *abstain* (not calling a tool when none is needed).
|
||||
- **hard_agent_bench.py** — a real multi-turn tool loop (tools actually execute and
|
||||
results feed back): sequential dependency, parallel/multi-entity, 3-hop chains,
|
||||
tool selection under distractors, error recovery, and no-fabrication honesty.
|
||||
Pass model names as args to limit the set: `... hard_agent_bench.py glm-4.7-flash gemma4:26b`.
|
||||
|
||||
## Notes
|
||||
|
||||
- These assume the same model names as the reference fleet — edit the model lists
|
||||
at the top of each script to match yours.
|
||||
- `quality_bench.py` / `benchmark.py` generate a test image with Pillow (in the
|
||||
venv) for the OCR/vision checks.
|
||||
- Objective grading (running generated code, exact-match answers) is deliberately
|
||||
used over LLM-as-judge to avoid judge bias.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Deploying llm-router
|
||||
|
||||
## Run manually
|
||||
```bash
|
||||
./scripts/setup.sh # once: venv + LiteLLM
|
||||
./run.sh # foreground: LiteLLM (:4000) + router (:8080)
|
||||
```
|
||||
`run.sh` starts LiteLLM in the background, waits for it, then runs the router in
|
||||
the foreground. Ctrl-C stops both (it tracks PIDs in `.litellm.pid` / `.router.pid`).
|
||||
|
||||
## Run as a boot service
|
||||
```bash
|
||||
./deploy/install-service.sh
|
||||
```
|
||||
- **macOS** → a launchd user agent (`~/Library/LaunchAgents/com.llm-router.plist`),
|
||||
`KeepAlive` (auto-restart), starts at login.
|
||||
- **Linux** → a systemd user service (`~/.config/systemd/user/llm-router.service`),
|
||||
`Restart=always`. Use `loginctl enable-linger $USER` to run without an active login.
|
||||
|
||||
The installer prints status / logs / removal commands for your platform.
|
||||
|
||||
Restart after editing `router.py` or config:
|
||||
```bash
|
||||
# macOS
|
||||
launchctl kickstart -k gui/$(id -u)/com.llm-router
|
||||
# Linux
|
||||
systemctl --user restart llm-router
|
||||
```
|
||||
|
||||
## Networking
|
||||
|
||||
- The router binds `ROUTER_HOST` (default `0.0.0.0` = reachable on the LAN).
|
||||
- Reach it from other machines at `http://<this-host-ip>:8080/v1` or, on the same
|
||||
LAN, `http://<hostname>.local:8080/v1` (mDNS; more stable than a DHCP IP).
|
||||
- Set `ROUTER_HOST=127.0.0.1` to make it local-only.
|
||||
|
||||
### Exposing beyond the LAN
|
||||
|
||||
Turn on auth first — put a token in `.apikey` (see `apikey.example`) and restart.
|
||||
Then front it with one of:
|
||||
|
||||
- **Reverse proxy (recommended for a fixed setup)** — e.g. Caddy:
|
||||
```
|
||||
router.example.com {
|
||||
reverse_proxy localhost:8080
|
||||
}
|
||||
```
|
||||
Caddy handles TLS; the router handles auth. Pair with `ROUTER_HOST=127.0.0.1`
|
||||
so the only way in is through the proxy.
|
||||
- **Quick tunnel (ephemeral)** — `cloudflared tunnel --url http://localhost:8080`
|
||||
gives a temporary `https://…trycloudflare.com` URL. Good for a quick test; the
|
||||
URL changes on restart and it's internet-facing, so keep `.apikey` set.
|
||||
|
||||
### A note on Ollama itself
|
||||
The router connects to Ollama at `ROUTER_OLLAMA` (default `http://127.0.0.1:11434`).
|
||||
If you want *other machines* to reach Ollama directly (not just via the router),
|
||||
set Ollama's own `OLLAMA_HOST=0.0.0.0:11434` and restart Ollama — but that is the
|
||||
**server bind** var and is unrelated to `ROUTER_OLLAMA` (the router's client URL).
|
||||
Don't set `ROUTER_OLLAMA` to `0.0.0.0` — it's a connect address.
|
||||
|
||||
## Health & logs
|
||||
```bash
|
||||
curl -s http://localhost:8080/healthz # {"ok":true}
|
||||
curl -s http://localhost:8080/v1/models # route targets
|
||||
tail -f launchd.out.log # macOS service logs (router access + decisions)
|
||||
tail -f litellm.log # LiteLLM backend
|
||||
```
|
||||
Each routed request logs a line like `[router] router -> glm-4.7-flash`. Responses
|
||||
carry `x-router-model`, `x-router-initial-model`, and `x-router-decided-by` headers.
|
||||
@@ -0,0 +1,68 @@
|
||||
# Routing
|
||||
|
||||
## How `model: "auto"` decides
|
||||
|
||||
`choose_model()` in [`router.py`](../router.py) runs cheap→expensive, first match wins:
|
||||
|
||||
1. **Image in the request** → vision model (`vision_heavy` if the text mentions
|
||||
OCR/document/table, else `vision`).
|
||||
2. **Request carries `tools`** → agentic model (`code_heavy` if the text looks
|
||||
code-ish, else `agentic`).
|
||||
3. **Text heuristics** → code regex → coding model; reasoning words → reasoning
|
||||
model; very short (< `SHORT_CHARS`) → the small fast model.
|
||||
4. **Tiny classifier** — anything left over is labeled by a small always-warm
|
||||
model (`CLASSIFIER`) into code / reason / vision / general.
|
||||
|
||||
The chosen model, plus any fallback, is reported in the response headers
|
||||
`x-router-initial-model` (the decision) and `x-router-model` (what actually served).
|
||||
|
||||
## The model map — customize this for your machine
|
||||
|
||||
Two dicts at the top of `router.py`:
|
||||
|
||||
```python
|
||||
M = {
|
||||
"fast": "qwen3:8b", "general": "qwen3:14b",
|
||||
"code": "glm-4.7-flash", "code_heavy": "qwen3-coder:30b",
|
||||
"reason": "qwen3.6:35b-a3b", "agentic": "glm-4.7-flash",
|
||||
"vision": "qwen3-vl:8b", "vision_heavy": "qwen3-vl:30b-a3b-instruct",
|
||||
}
|
||||
CLASSIFIER = "qwen3:8b"
|
||||
```
|
||||
|
||||
Point these at models you've actually pulled (`ollama list`). Names must match
|
||||
exactly, including any `:tag`. Restart the router after editing.
|
||||
|
||||
`FALLBACK` maps each model to a backup used when it errors or isn't pulled yet —
|
||||
so the router degrades gracefully instead of failing.
|
||||
|
||||
## Explicit model override
|
||||
|
||||
Any request that names a real model (not `auto`) is passed straight through:
|
||||
`{"model": "glm-4.7-flash", ...}`. Unknown names are handled by LiteLLM's catch-all
|
||||
(`model_name: "*"` in `litellm.config.yaml`), which forwards *any* name to Ollama —
|
||||
so `:tag` variants and models you pull later work without config changes.
|
||||
|
||||
## Policy: forcing a client onto a fleet (the "uncensored" example)
|
||||
|
||||
Put a token in `.uncensored_key`. Any request whose `Authorization: Bearer <token>`
|
||||
matches is forced through `choose_uncensored()` — the same content heuristics, but
|
||||
only ever selecting from the `UNCENSORED` map — **regardless of the model asked for**.
|
||||
This pins a specific client (identified by its key) to a specific fleet.
|
||||
|
||||
Clients can also opt in per-request with `model: "auto-uncensored"` (no key needed).
|
||||
|
||||
Use this pattern for any policy split (a "coding-only" client, a "cheap models"
|
||||
client, etc.): add a map + a `choose_*` function + a trigger in `do_POST`.
|
||||
|
||||
## Adding a model from Hugging Face
|
||||
|
||||
Ollama pulls GGUF repos directly:
|
||||
```bash
|
||||
ollama pull hf.co/USER/REPO:Q4_K_M # tag must match the quant in the filename
|
||||
ollama cp hf.co/USER/REPO:Q4_K_M friendly-name # optional: rename
|
||||
```
|
||||
Then add `friendly-name` to the relevant map in `router.py`. Multi-file repos:
|
||||
Ollama auto-pulls a vision `mmproj` if present; `mtp`/draft files are ignored.
|
||||
Big prompts truncate at Ollama's default context — make a bigger-context copy with
|
||||
`./scripts/make-context-variant.sh <model> <num_ctx>`.
|
||||
Reference in New Issue
Block a user