llm-router/docs/DEPLOY.md

Deploying llm-router

Run manually

./scripts/setup.sh      # once: venv + LiteLLM
./run.sh                # foreground: LiteLLM (:4000) + router (:8080)

run.sh starts LiteLLM in the background, waits for it, then runs the router in the foreground. Ctrl-C stops both (it tracks PIDs in .litellm.pid / .router.pid).

Run as a boot service

./deploy/install-service.sh

• macOS → a launchd user agent (~/Library/LaunchAgents/com.llm-router.plist), KeepAlive (auto-restart), starts at login.
• Linux → a systemd user service (~/.config/systemd/user/llm-router.service), Restart=always. Use loginctl enable-linger $USER to run without an active login.

The installer prints status / logs / removal commands for your platform.

Restart after editing router.py or config:

# macOS
launchctl kickstart -k gui/$(id -u)/com.llm-router
# Linux
systemctl --user restart llm-router

Windows

powershell -ExecutionPolicy Bypass -File .\scripts\setup.ps1        # venv + LiteLLM
powershell -ExecutionPolicy Bypass -File .\scripts\pull-models.ps1  # pull models
powershell -ExecutionPolicy Bypass -File .\run.ps1                  # LiteLLM + router

run.ps1 starts LiteLLM in the background, waits for it, then runs the router in the foreground; closing it stops both.

Run at login (Task Scheduler):

$action  = New-ScheduledTaskAction -Execute "powershell.exe" `
  -Argument "-ExecutionPolicy Bypass -WindowStyle Hidden -File `"$PWD\run.ps1`""
$trigger = New-ScheduledTaskTrigger -AtLogOn
Register-ScheduledTask -TaskName "llm-router" -Action $action -Trigger $trigger `
  -Settings (New-ScheduledTaskSettingsSet -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1))

Requires Python 3.12 (winget install Python.Python.3.12) and Ollama for Windows. The router (stdlib Python) and LiteLLM run the same as on macOS/Linux.

Networking

• The router binds ROUTER_HOST (default 0.0.0.0 = reachable on the LAN).
• Reach it from other machines at http://<this-host-ip>:8080/v1 or, on the same LAN, http://<hostname>.local:8080/v1 (mDNS; more stable than a DHCP IP).
• Set ROUTER_HOST=127.0.0.1 to make it local-only.

Exposing beyond the LAN

Turn on auth first — put a token in .apikey (see apikey.example) and restart. Then front it with one of:

• Reverse proxy (recommended for a fixed setup) — e.g. Caddy:

  router.example.com {
      reverse_proxy localhost:8080
  }
  

  Caddy handles TLS; the router handles auth. Pair with ROUTER_HOST=127.0.0.1 so the only way in is through the proxy.
• Quick tunnel (ephemeral) — cloudflared tunnel --url http://localhost:8080 gives a temporary https://…trycloudflare.com URL. Good for a quick test; the URL changes on restart and it's internet-facing, so keep .apikey set.

A note on Ollama itself

The router connects to Ollama at ROUTER_OLLAMA (default http://127.0.0.1:11434). If you want other machines to reach Ollama directly (not just via the router), set Ollama's own OLLAMA_HOST=0.0.0.0:11434 and restart Ollama — but that is the server bind var and is unrelated to ROUTER_OLLAMA (the router's client URL). Don't set ROUTER_OLLAMA to 0.0.0.0 — it's a connect address.

Health & logs

curl -s http://localhost:8080/healthz          # {"ok":true}
curl -s http://localhost:8080/v1/models        # route targets
tail -f launchd.out.log                        # macOS service logs (router access + decisions)
tail -f litellm.log                            # LiteLLM backend

Each routed request logs a line like [router] router -> glm-4.7-flash. Responses carry x-router-model, x-router-initial-model, and x-router-decided-by headers.