Initial commit: llm-router — smart OpenAI/Anthropic/Ollama router

A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 02:05:16 -05:00
commit 9938d46a67
32 changed files with 2419 additions and 0 deletions
@@ -0,0 +1,69 @@
+# Deploying llm-router
+
+## Run manually
+```bash
+./scripts/setup.sh      # once: venv + LiteLLM
+./run.sh                # foreground: LiteLLM (:4000) + router (:8080)
+```
+`run.sh` starts LiteLLM in the background, waits for it, then runs the router in
+the foreground. Ctrl-C stops both (it tracks PIDs in `.litellm.pid` / `.router.pid`).
+
+## Run as a boot service
+```bash
+./deploy/install-service.sh
+```
+- **macOS** → a launchd user agent (`~/Library/LaunchAgents/com.llm-router.plist`),
+  `KeepAlive` (auto-restart), starts at login.
+- **Linux** → a systemd user service (`~/.config/systemd/user/llm-router.service`),
+  `Restart=always`. Use `loginctl enable-linger $USER` to run without an active login.
+
+The installer prints status / logs / removal commands for your platform.
+
+Restart after editing `router.py` or config:
+```bash
+# macOS
+launchctl kickstart -k gui/$(id -u)/com.llm-router
+# Linux
+systemctl --user restart llm-router
+```
+
+## Networking
+
+- The router binds `ROUTER_HOST` (default `0.0.0.0` = reachable on the LAN).
+- Reach it from other machines at `http://<this-host-ip>:8080/v1` or, on the same
+  LAN, `http://<hostname>.local:8080/v1` (mDNS; more stable than a DHCP IP).
+- Set `ROUTER_HOST=127.0.0.1` to make it local-only.
+
+### Exposing beyond the LAN
+
+Turn on auth first — put a token in `.apikey` (see `apikey.example`) and restart.
+Then front it with one of:
+
+- **Reverse proxy (recommended for a fixed setup)** — e.g. Caddy:
+  ```
+  router.example.com {
+      reverse_proxy localhost:8080
+  }
+  ```
+  Caddy handles TLS; the router handles auth. Pair with `ROUTER_HOST=127.0.0.1`
+  so the only way in is through the proxy.
+- **Quick tunnel (ephemeral)** — `cloudflared tunnel --url http://localhost:8080`
+  gives a temporary `https://…trycloudflare.com` URL. Good for a quick test; the
+  URL changes on restart and it's internet-facing, so keep `.apikey` set.
+
+### A note on Ollama itself
+The router connects to Ollama at `ROUTER_OLLAMA` (default `http://127.0.0.1:11434`).
+If you want *other machines* to reach Ollama directly (not just via the router),
+set Ollama's own `OLLAMA_HOST=0.0.0.0:11434` and restart Ollama — but that is the
+**server bind** var and is unrelated to `ROUTER_OLLAMA` (the router's client URL).
+Don't set `ROUTER_OLLAMA` to `0.0.0.0` — it's a connect address.
+
+## Health & logs
+```bash
+curl -s http://localhost:8080/healthz          # {"ok":true}
+curl -s http://localhost:8080/v1/models        # route targets
+tail -f launchd.out.log                        # macOS service logs (router access + decisions)
+tail -f litellm.log                            # LiteLLM backend
+```
+Each routed request logs a line like `[router] router -> glm-4.7-flash`. Responses
+carry `x-router-model`, `x-router-initial-model`, and `x-router-decided-by` headers.