llm-router/README.md

llm-router

A small, dependency-light smart router in front of Ollama. It speaks the OpenAI and Anthropic APIs, picks the best local model per request (by content, tools, modality), adds fallbacks, and bolts on the pieces Ollama lacks — audio transcription and identity-based model policies.

 OpenAI / Anthropic / Ollama clients
              │
              ▼
     pre-router  (:8080, this repo)        ← smart model selection, auth, audio, policy
      │                 │
      │ OpenAI /v1      │ native /api/*
      ▼                 ▼
   LiteLLM (:4000)   Ollama (:11434)        ← LiteLLM = fallbacks/logging; Ollama = the models
      └──────┬──────────┘
             ▼
          Ollama models

The pre-router is pure Python standard library (no pip deps) — it runs on any Python 3.9+. LiteLLM (the backend proxy) needs a small venv.

What it does

• Auto model selection — model: "auto" routes by content: code → a coding model, images → a vision model, tool calls → an agentic model, short chats → a small fast model, everything else → a classifier or general model.
• Multiple wire protocols — OpenAI /v1/chat/completions, Anthropic /v1/messages (so Claude Code works), and Ollama-native /api/* (so ollama launch sees it as a real Ollama).
• Audio — an OpenAI-compatible /v1/audio/transcriptions (Whisper-shaped) endpoint plus in-chat input_audio, backed by a speech-capable local model. See docs/AUDIO.md.
• Fallbacks — if a model errors (or isn't pulled yet) the request falls back to a related model automatically.
• Policies — a dedicated API key can force a client onto a specific fleet (e.g. uncensored models), regardless of what model it requests.
• Optional auth — Bearer-token gate for when you expose it beyond localhost.

Requirements

• Ollama — the model backend. Install:
  • macOS: brew install ollama (or the app from ollama.com/download)
  • Linux: curl -fsSL https://ollama.com/install.sh | sh
  • Windows: winget install Ollama.Ollama
• Python 3.12 — for the LiteLLM venv:
  • macOS: brew install python@3.12 · Linux: distro python3.12 · Windows: winget install Python.Python.3.12
  • The pre-router itself needs only any Python 3.9+.
• curl — used by run.sh for health checks (standard on macOS/Linux).

./scripts/setup.* checks Python and Ollama for you and installs the Python deps; ./scripts/pull-models.* fetches the model fleet.

Quickstart

git clone <your-fork-url> llm-router && cd llm-router

# 1. one-time: create the LiteLLM venv
./scripts/setup.sh

# 2. pull a model fleet (edit the list first — see the script)
./scripts/pull-models.sh

# 3. run it (LiteLLM on :4000, router on :8080)
./run.sh

On Windows (PowerShell), use the .ps1 equivalents:

powershell -ExecutionPolicy Bypass -File .\scripts\setup.ps1
powershell -ExecutionPolicy Bypass -File .\scripts\pull-models.ps1
powershell -ExecutionPolicy Bypass -File .\run.ps1

Then point any OpenAI client at http://localhost:8080/v1 with model: "auto":

curl http://localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'

Run it as a boot service (launchd on macOS, systemd on Linux): see docs/DEPLOY.md.

Configuration

Everything is environment variables (read by run.sh / the router):

Var	Default	Purpose
ROUTER_HOST	0.0.0.0	bind address (use 127.0.0.1 to keep it local-only)
ROUTER_PORT	8080	router port
ROUTER_OLLAMA	http://127.0.0.1:11434	how the router reaches Ollama (client URL)
ROUTER_UPSTREAM	http://127.0.0.1:4000/v1	LiteLLM proxy URL
ROUTER_AUDIO_MODEL	gemma4:e4b	model used for audio/transcription
ROUTER_API_KEY	(unset)	if set, require this Bearer token on /v1
ROUTER_UNCENSORED_KEY	(unset)	Bearer token that forces the uncensored fleet

⚠️ ROUTER_OLLAMA is the router's client target — keep it 127.0.0.1. Do not confuse it with Ollama's own OLLAMA_HOST server-bind variable.

Keys are loaded from files so they stay out of git: put a token in .apikey and/or .uncensored_key (see the *.example files). Both are .gitignored.

The model map lives at the top of router.py (the M and UNCENSORED dicts). Edit those to match the models you've pulled — this is the main thing to customize for your machine. See docs/ROUTING.md.

Endpoints

Method	Path	Notes
POST	/v1/chat/completions	OpenAI chat; model:"auto" = smart routing
POST	/v1/messages	Anthropic Messages (Claude Code) → LiteLLM
POST	/v1/audio/transcriptions	Whisper-shaped speech-to-text
POST	/v1/audio/translations	speech → English text
GET	/v1/models	lists route targets
GET	/, /api/version, /api/tags, /api/ps, POST /api/show	Ollama-native probes (so ollama launch accepts the router)
GET	/healthz	liveness

Docs

• docs/DEPLOY.md — run as a service, expose it, tunnels, Caddy
• docs/ROUTING.md — how routing decides + customizing the map
• docs/AUDIO.md — voice input & transcription
• docs/BENCHMARKS.md — the benchmark harnesses & sample results

Security

• Default bind is 0.0.0.0 (LAN-reachable). Set ROUTER_HOST=127.0.0.1 for local-only, or set ROUTER_API_KEY before exposing it anywhere.
• The Ollama-native /api/* endpoints the router serves are read-only (version/tags/show) — there is no unauthenticated /api/chat inference path.

License

MIT — see LICENSE.