Initial commit: llm-router — smart OpenAI/Anthropic/Ollama router

A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 02:05:16 -05:00
commit 9938d46a67
32 changed files with 2419 additions and 0 deletions
@@ -0,0 +1,126 @@
+# llm-router
+
+A small, dependency-light **smart router** in front of [Ollama](https://ollama.com).
+It speaks the **OpenAI** and **Anthropic** APIs, picks the best local model per
+request (by content, tools, modality), adds fallbacks, and bolts on the pieces
+Ollama lacks — **audio transcription** and **identity-based model policies**.
+
+```
+ OpenAI / Anthropic / Ollama clients
+              │
+              ▼
+     pre-router  (:8080, this repo)        ← smart model selection, auth, audio, policy
+      │                 │
+      │ OpenAI /v1      │ native /api/*
+      ▼                 ▼
+   LiteLLM (:4000)   Ollama (:11434)        ← LiteLLM = fallbacks/logging; Ollama = the models
+      └──────┬──────────┘
+             ▼
+          Ollama models
+```
+
+The pre-router is **pure Python standard library** (no pip deps) — it runs on any
+Python 3.9+. LiteLLM (the backend proxy) needs a small venv.
+
+## What it does
+
+- **Auto model selection** — `model: "auto"` routes by content: code → a coding
+  model, images → a vision model, tool calls → an agentic model, short chats → a
+  small fast model, everything else → a classifier or general model.
+- **Multiple wire protocols** — OpenAI `/v1/chat/completions`, Anthropic
+  `/v1/messages` (so Claude Code works), and Ollama-native `/api/*` (so
+  `ollama launch` sees it as a real Ollama).
+- **Audio** — an OpenAI-compatible `/v1/audio/transcriptions` (Whisper-shaped)
+  endpoint plus in-chat `input_audio`, backed by a speech-capable local model.
+  See [docs/AUDIO.md](docs/AUDIO.md).
+- **Fallbacks** — if a model errors (or isn't pulled yet) the request falls back
+  to a related model automatically.
+- **Policies** — a dedicated API key can force a client onto a specific fleet
+  (e.g. uncensored models), regardless of what model it requests.
+- **Optional auth** — Bearer-token gate for when you expose it beyond localhost.
+
+## Requirements
+
+- [Ollama](https://ollama.com) running locally with some models pulled
+- **Python 3.12** (for the LiteLLM venv). macOS: `brew install python@3.12`.
+  Linux: your distro's `python3.12`.
+- The pre-router itself only needs any Python 3.9+.
+
+## Quickstart
+
+```bash
+git clone <your-fork-url> llm-router && cd llm-router
+
+# 1. one-time: create the LiteLLM venv
+./scripts/setup.sh
+
+# 2. pull a model fleet (edit the list first — see the script)
+./scripts/pull-models.sh
+
+# 3. run it (LiteLLM on :4000, router on :8080)
+./run.sh
+```
+
+Then point any OpenAI client at `http://localhost:8080/v1` with `model: "auto"`:
+
+```bash
+curl http://localhost:8080/v1/chat/completions -H 'content-type: application/json' \
+  -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'
+```
+
+Run it as a boot service (launchd on macOS, systemd on Linux):
+see [docs/DEPLOY.md](docs/DEPLOY.md).
+
+## Configuration
+
+Everything is environment variables (read by `run.sh` / the router):
+
+| Var | Default | Purpose |
+|---|---|---|
+| `ROUTER_HOST` | `0.0.0.0` | bind address (use `127.0.0.1` to keep it local-only) |
+| `ROUTER_PORT` | `8080` | router port |
+| `ROUTER_OLLAMA` | `http://127.0.0.1:11434` | how the router reaches Ollama (client URL) |
+| `ROUTER_UPSTREAM` | `http://127.0.0.1:4000/v1` | LiteLLM proxy URL |
+| `ROUTER_AUDIO_MODEL` | `gemma4:e4b` | model used for audio/transcription |
+| `ROUTER_API_KEY` | *(unset)* | if set, require this Bearer token on /v1 |
+| `ROUTER_UNCENSORED_KEY` | *(unset)* | Bearer token that forces the uncensored fleet |
+
+> ⚠️ `ROUTER_OLLAMA` is the router's **client** target — keep it `127.0.0.1`.
+> Do **not** confuse it with Ollama's own `OLLAMA_HOST` server-bind variable.
+
+**Keys** are loaded from files so they stay out of git: put a token in `.apikey`
+and/or `.uncensored_key` (see the `*.example` files). Both are `.gitignore`d.
+
+**The model map** lives at the top of [`router.py`](router.py) (the `M` and
+`UNCENSORED` dicts). Edit those to match the models you've pulled — this is the
+main thing to customize for your machine. See [docs/ROUTING.md](docs/ROUTING.md).
+
+## Endpoints
+
+| Method | Path | Notes |
+|---|---|---|
+| POST | `/v1/chat/completions` | OpenAI chat; `model:"auto"` = smart routing |
+| POST | `/v1/messages` | Anthropic Messages (Claude Code) → LiteLLM |
+| POST | `/v1/audio/transcriptions` | Whisper-shaped speech-to-text |
+| POST | `/v1/audio/translations` | speech → English text |
+| GET | `/v1/models` | lists route targets |
+| GET | `/`, `/api/version`, `/api/tags`, `/api/ps`, POST `/api/show` | Ollama-native probes (so `ollama launch` accepts the router) |
+| GET | `/healthz` | liveness |
+
+## Docs
+
+- [docs/DEPLOY.md](docs/DEPLOY.md) — run as a service, expose it, tunnels, Caddy
+- [docs/ROUTING.md](docs/ROUTING.md) — how routing decides + customizing the map
+- [docs/AUDIO.md](docs/AUDIO.md) — voice input & transcription
+- [docs/BENCHMARKS.md](docs/BENCHMARKS.md) — the benchmark harnesses & sample results
+
+## Security
+
+- Default bind is `0.0.0.0` (LAN-reachable). Set `ROUTER_HOST=127.0.0.1` for
+  local-only, or set `ROUTER_API_KEY` before exposing it anywhere.
+- The Ollama-native `/api/*` endpoints the router serves are **read-only**
+  (version/tags/show) — there is no unauthenticated `/api/chat` inference path.
+
+## License
+
+MIT — see [LICENSE](LICENSE).