Joseph CostaandClaude Opus 4.8
95da8fb38d
- scripts/setup.ps1, scripts/pull-models.ps1, run.ps1 — PowerShell equivalents of the bash setup/pull/run scripts - README + docs/DEPLOY.md: Windows quickstart and Task Scheduler autostart Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
91 lines
3.7 KiB
Markdown
91 lines
3.7 KiB
Markdown
# Deploying llm-router
|
|
|
|
## Run manually
|
|
```bash
|
|
./scripts/setup.sh # once: venv + LiteLLM
|
|
./run.sh # foreground: LiteLLM (:4000) + router (:8080)
|
|
```
|
|
`run.sh` starts LiteLLM in the background, waits for it, then runs the router in
|
|
the foreground. Ctrl-C stops both (it tracks PIDs in `.litellm.pid` / `.router.pid`).
|
|
|
|
## Run as a boot service
|
|
```bash
|
|
./deploy/install-service.sh
|
|
```
|
|
- **macOS** → a launchd user agent (`~/Library/LaunchAgents/com.llm-router.plist`),
|
|
`KeepAlive` (auto-restart), starts at login.
|
|
- **Linux** → a systemd user service (`~/.config/systemd/user/llm-router.service`),
|
|
`Restart=always`. Use `loginctl enable-linger $USER` to run without an active login.
|
|
|
|
The installer prints status / logs / removal commands for your platform.
|
|
|
|
Restart after editing `router.py` or config:
|
|
```bash
|
|
# macOS
|
|
launchctl kickstart -k gui/$(id -u)/com.llm-router
|
|
# Linux
|
|
systemctl --user restart llm-router
|
|
```
|
|
|
|
## Windows
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File .\scripts\setup.ps1 # venv + LiteLLM
|
|
powershell -ExecutionPolicy Bypass -File .\scripts\pull-models.ps1 # pull models
|
|
powershell -ExecutionPolicy Bypass -File .\run.ps1 # LiteLLM + router
|
|
```
|
|
`run.ps1` starts LiteLLM in the background, waits for it, then runs the router in
|
|
the foreground; closing it stops both.
|
|
|
|
**Run at login (Task Scheduler):**
|
|
```powershell
|
|
$action = New-ScheduledTaskAction -Execute "powershell.exe" `
|
|
-Argument "-ExecutionPolicy Bypass -WindowStyle Hidden -File `"$PWD\run.ps1`""
|
|
$trigger = New-ScheduledTaskTrigger -AtLogOn
|
|
Register-ScheduledTask -TaskName "llm-router" -Action $action -Trigger $trigger `
|
|
-Settings (New-ScheduledTaskSettingsSet -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1))
|
|
```
|
|
Requires **Python 3.12** (`winget install Python.Python.3.12`) and Ollama for
|
|
Windows. The router (stdlib Python) and LiteLLM run the same as on macOS/Linux.
|
|
|
|
## Networking
|
|
|
|
- The router binds `ROUTER_HOST` (default `0.0.0.0` = reachable on the LAN).
|
|
- Reach it from other machines at `http://<this-host-ip>:8080/v1` or, on the same
|
|
LAN, `http://<hostname>.local:8080/v1` (mDNS; more stable than a DHCP IP).
|
|
- Set `ROUTER_HOST=127.0.0.1` to make it local-only.
|
|
|
|
### Exposing beyond the LAN
|
|
|
|
Turn on auth first — put a token in `.apikey` (see `apikey.example`) and restart.
|
|
Then front it with one of:
|
|
|
|
- **Reverse proxy (recommended for a fixed setup)** — e.g. Caddy:
|
|
```
|
|
router.example.com {
|
|
reverse_proxy localhost:8080
|
|
}
|
|
```
|
|
Caddy handles TLS; the router handles auth. Pair with `ROUTER_HOST=127.0.0.1`
|
|
so the only way in is through the proxy.
|
|
- **Quick tunnel (ephemeral)** — `cloudflared tunnel --url http://localhost:8080`
|
|
gives a temporary `https://…trycloudflare.com` URL. Good for a quick test; the
|
|
URL changes on restart and it's internet-facing, so keep `.apikey` set.
|
|
|
|
### A note on Ollama itself
|
|
The router connects to Ollama at `ROUTER_OLLAMA` (default `http://127.0.0.1:11434`).
|
|
If you want *other machines* to reach Ollama directly (not just via the router),
|
|
set Ollama's own `OLLAMA_HOST=0.0.0.0:11434` and restart Ollama — but that is the
|
|
**server bind** var and is unrelated to `ROUTER_OLLAMA` (the router's client URL).
|
|
Don't set `ROUTER_OLLAMA` to `0.0.0.0` — it's a connect address.
|
|
|
|
## Health & logs
|
|
```bash
|
|
curl -s http://localhost:8080/healthz # {"ok":true}
|
|
curl -s http://localhost:8080/v1/models # route targets
|
|
tail -f launchd.out.log # macOS service logs (router access + decisions)
|
|
tail -f litellm.log # LiteLLM backend
|
|
```
|
|
Each routed request logs a line like `[router] router -> glm-4.7-flash`. Responses
|
|
carry `x-router-model`, `x-router-initial-model`, and `x-router-decided-by` headers.
|