# llm-router A small, dependency-light **smart router** in front of [Ollama](https://ollama.com). It speaks the **OpenAI** and **Anthropic** APIs, picks the best local model per request (by content, tools, modality), adds fallbacks, and bolts on the pieces Ollama lacks — **audio transcription** and **identity-based model policies**. ``` OpenAI / Anthropic / Ollama clients │ ▼ pre-router (:8080, this repo) ← smart model selection, auth, audio, policy │ │ │ OpenAI /v1 │ native /api/* ▼ ▼ LiteLLM (:4000) Ollama (:11434) ← LiteLLM = fallbacks/logging; Ollama = the models └──────┬──────────┘ ▼ Ollama models ``` The pre-router is **pure Python standard library** (no pip deps) — it runs on any Python 3.9+. LiteLLM (the backend proxy) needs a small venv. ## What it does - **Auto model selection** — `model: "auto"` routes by content: code → a coding model, images → a vision model, tool calls → an agentic model, short chats → a small fast model, everything else → a classifier or general model. - **Multiple wire protocols** — OpenAI `/v1/chat/completions`, Anthropic `/v1/messages` (so Claude Code works), and Ollama-native `/api/*` (so `ollama launch` sees it as a real Ollama). - **Audio** — an OpenAI-compatible `/v1/audio/transcriptions` (Whisper-shaped) endpoint plus in-chat `input_audio`, backed by a speech-capable local model. See [docs/AUDIO.md](docs/AUDIO.md). - **Fallbacks** — if a model errors (or isn't pulled yet) the request falls back to a related model automatically. - **Policies** — a dedicated API key can force a client onto a specific fleet (e.g. uncensored models), regardless of what model it requests. - **Optional auth** — Bearer-token gate for when you expose it beyond localhost. ## Requirements - **[Ollama](https://ollama.com)** — the model backend. Install: - macOS: `brew install ollama` (or the app from [ollama.com/download](https://ollama.com/download)) - Linux: `curl -fsSL https://ollama.com/install.sh | sh` - Windows: `winget install Ollama.Ollama` - **Python 3.12** — for the LiteLLM venv: - macOS: `brew install python@3.12` · Linux: distro `python3.12` · Windows: `winget install Python.Python.3.12` - The pre-router itself needs only any Python 3.9+. - **curl** — used by `run.sh` for health checks (standard on macOS/Linux). `./scripts/setup.*` checks Python **and** Ollama for you and installs the Python deps; `./scripts/pull-models.*` fetches the model fleet. ## Quickstart ```bash git clone llm-router && cd llm-router # 1. one-time: create the LiteLLM venv ./scripts/setup.sh # 2. pull a model fleet (edit the list first — see the script) ./scripts/pull-models.sh # 3. run it (LiteLLM on :4000, router on :8080) ./run.sh ``` On **Windows** (PowerShell), use the `.ps1` equivalents: ```powershell powershell -ExecutionPolicy Bypass -File .\scripts\setup.ps1 powershell -ExecutionPolicy Bypass -File .\scripts\pull-models.ps1 powershell -ExecutionPolicy Bypass -File .\run.ps1 ``` Then point any OpenAI client at `http://localhost:8080/v1` with `model: "auto"`: ```bash curl http://localhost:8080/v1/chat/completions -H 'content-type: application/json' \ -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}' ``` Run it as a boot service (launchd on macOS, systemd on Linux): see [docs/DEPLOY.md](docs/DEPLOY.md). ## Configuration Everything is environment variables (read by `run.sh` / the router): | Var | Default | Purpose | |---|---|---| | `ROUTER_HOST` | `0.0.0.0` | bind address (use `127.0.0.1` to keep it local-only) | | `ROUTER_PORT` | `8080` | router port | | `ROUTER_OLLAMA` | `http://127.0.0.1:11434` | how the router reaches Ollama (client URL) | | `ROUTER_UPSTREAM` | `http://127.0.0.1:4000/v1` | LiteLLM proxy URL | | `ROUTER_AUDIO_MODEL` | `gemma4:e4b` | model used for audio/transcription | | `ROUTER_API_KEY` | *(unset)* | if set, require this Bearer token on /v1 | | `ROUTER_UNCENSORED_KEY` | *(unset)* | Bearer token that forces the uncensored fleet | | `ROUTER_COMFYUI` | `http://127.0.0.1:8188` | ComfyUI server for image generation | | `ROUTER_COMFY_CHECKPOINT` | *(unset)* | checkpoint filename for the default image workflow | > ⚠️ `ROUTER_OLLAMA` is the router's **client** target — keep it `127.0.0.1`. > Do **not** confuse it with Ollama's own `OLLAMA_HOST` server-bind variable. **Keys** are loaded from files so they stay out of git: put a token in `.apikey` and/or `.uncensored_key` (see the `*.example` files). Both are `.gitignore`d. **The model map** lives at the top of [`router.py`](router.py) (the `M` and `UNCENSORED` dicts). Edit those to match the models you've pulled — this is the main thing to customize for your machine. See [docs/ROUTING.md](docs/ROUTING.md). ## Endpoints | Method | Path | Notes | |---|---|---| | POST | `/v1/chat/completions` | OpenAI chat; `model:"auto"` = smart routing | | POST | `/v1/messages` | Anthropic Messages (Claude Code) → LiteLLM | | POST | `/v1/audio/transcriptions` | Whisper-shaped speech-to-text | | POST | `/v1/audio/translations` | speech → English text | | POST | `/v1/images/generations` | OpenAI-shaped image gen → ComfyUI | | GET | `/v1/models` | lists route targets | | GET | `/`, `/api/version`, `/api/tags`, `/api/ps`, POST `/api/show` | Ollama-native probes (so `ollama launch` accepts the router) | | GET | `/healthz` | liveness | ## Docs - [docs/DEPLOY.md](docs/DEPLOY.md) — run as a service, expose it, tunnels, Caddy - [docs/ROUTING.md](docs/ROUTING.md) — how routing decides + customizing the map - [docs/AUDIO.md](docs/AUDIO.md) — voice input & transcription - [docs/IMAGES.md](docs/IMAGES.md) — image generation via ComfyUI - [docs/BENCHMARKS.md](docs/BENCHMARKS.md) — the benchmark harnesses & sample results ## Security - Default bind is `0.0.0.0` (LAN-reachable). Set `ROUTER_HOST=127.0.0.1` for local-only, or set `ROUTER_API_KEY` before exposing it anywhere. - The Ollama-native `/api/*` endpoints the router serves are **read-only** (version/tags/show) — there is no unauthenticated `/api/chat` inference path. ## License MIT — see [LICENSE](LICENSE).