Joseph CostaandClaude Opus 4.8
2f95e6be96
OpenAI-compatible image endpoint that proxies to a ComfyUI server: fills a
workflow template with per-request params (prompt/size/n/seed/steps/checkpoint),
submits to ComfyUI, polls history, fetches images, returns OpenAI image shape.
- router.py: _handle_image + fill_workflow + route; env ROUTER_COMFYUI /
ROUTER_COMFY_CHECKPOINT / ROUTER_COMFY_WORKFLOW
- comfyui-workflow.json: default SD/SDXL text2img template with {{SENTINELS}}
- docs/IMAGES.md + README: config, usage, custom workflows
- verified: templating + clean error when ComfyUI is down; end-to-end pending
a live ComfyUI
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
145 lines
6.2 KiB
Markdown
145 lines
6.2 KiB
Markdown
# llm-router
|
|
|
|
A small, dependency-light **smart router** in front of [Ollama](https://ollama.com).
|
|
It speaks the **OpenAI** and **Anthropic** APIs, picks the best local model per
|
|
request (by content, tools, modality), adds fallbacks, and bolts on the pieces
|
|
Ollama lacks — **audio transcription** and **identity-based model policies**.
|
|
|
|
```
|
|
OpenAI / Anthropic / Ollama clients
|
|
│
|
|
▼
|
|
pre-router (:8080, this repo) ← smart model selection, auth, audio, policy
|
|
│ │
|
|
│ OpenAI /v1 │ native /api/*
|
|
▼ ▼
|
|
LiteLLM (:4000) Ollama (:11434) ← LiteLLM = fallbacks/logging; Ollama = the models
|
|
└──────┬──────────┘
|
|
▼
|
|
Ollama models
|
|
```
|
|
|
|
The pre-router is **pure Python standard library** (no pip deps) — it runs on any
|
|
Python 3.9+. LiteLLM (the backend proxy) needs a small venv.
|
|
|
|
## What it does
|
|
|
|
- **Auto model selection** — `model: "auto"` routes by content: code → a coding
|
|
model, images → a vision model, tool calls → an agentic model, short chats → a
|
|
small fast model, everything else → a classifier or general model.
|
|
- **Multiple wire protocols** — OpenAI `/v1/chat/completions`, Anthropic
|
|
`/v1/messages` (so Claude Code works), and Ollama-native `/api/*` (so
|
|
`ollama launch` sees it as a real Ollama).
|
|
- **Audio** — an OpenAI-compatible `/v1/audio/transcriptions` (Whisper-shaped)
|
|
endpoint plus in-chat `input_audio`, backed by a speech-capable local model.
|
|
See [docs/AUDIO.md](docs/AUDIO.md).
|
|
- **Fallbacks** — if a model errors (or isn't pulled yet) the request falls back
|
|
to a related model automatically.
|
|
- **Policies** — a dedicated API key can force a client onto a specific fleet
|
|
(e.g. uncensored models), regardless of what model it requests.
|
|
- **Optional auth** — Bearer-token gate for when you expose it beyond localhost.
|
|
|
|
## Requirements
|
|
|
|
- **[Ollama](https://ollama.com)** — the model backend. Install:
|
|
- macOS: `brew install ollama` (or the app from [ollama.com/download](https://ollama.com/download))
|
|
- Linux: `curl -fsSL https://ollama.com/install.sh | sh`
|
|
- Windows: `winget install Ollama.Ollama`
|
|
- **Python 3.12** — for the LiteLLM venv:
|
|
- macOS: `brew install python@3.12` · Linux: distro `python3.12` · Windows: `winget install Python.Python.3.12`
|
|
- The pre-router itself needs only any Python 3.9+.
|
|
- **curl** — used by `run.sh` for health checks (standard on macOS/Linux).
|
|
|
|
`./scripts/setup.*` checks Python **and** Ollama for you and installs the Python
|
|
deps; `./scripts/pull-models.*` fetches the model fleet.
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
git clone <your-fork-url> llm-router && cd llm-router
|
|
|
|
# 1. one-time: create the LiteLLM venv
|
|
./scripts/setup.sh
|
|
|
|
# 2. pull a model fleet (edit the list first — see the script)
|
|
./scripts/pull-models.sh
|
|
|
|
# 3. run it (LiteLLM on :4000, router on :8080)
|
|
./run.sh
|
|
```
|
|
|
|
On **Windows** (PowerShell), use the `.ps1` equivalents:
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File .\scripts\setup.ps1
|
|
powershell -ExecutionPolicy Bypass -File .\scripts\pull-models.ps1
|
|
powershell -ExecutionPolicy Bypass -File .\run.ps1
|
|
```
|
|
|
|
Then point any OpenAI client at `http://localhost:8080/v1` with `model: "auto"`:
|
|
|
|
```bash
|
|
curl http://localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
|
-d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'
|
|
```
|
|
|
|
Run it as a boot service (launchd on macOS, systemd on Linux):
|
|
see [docs/DEPLOY.md](docs/DEPLOY.md).
|
|
|
|
## Configuration
|
|
|
|
Everything is environment variables (read by `run.sh` / the router):
|
|
|
|
| Var | Default | Purpose |
|
|
|---|---|---|
|
|
| `ROUTER_HOST` | `0.0.0.0` | bind address (use `127.0.0.1` to keep it local-only) |
|
|
| `ROUTER_PORT` | `8080` | router port |
|
|
| `ROUTER_OLLAMA` | `http://127.0.0.1:11434` | how the router reaches Ollama (client URL) |
|
|
| `ROUTER_UPSTREAM` | `http://127.0.0.1:4000/v1` | LiteLLM proxy URL |
|
|
| `ROUTER_AUDIO_MODEL` | `gemma4:e4b` | model used for audio/transcription |
|
|
| `ROUTER_API_KEY` | *(unset)* | if set, require this Bearer token on /v1 |
|
|
| `ROUTER_UNCENSORED_KEY` | *(unset)* | Bearer token that forces the uncensored fleet |
|
|
| `ROUTER_COMFYUI` | `http://127.0.0.1:8188` | ComfyUI server for image generation |
|
|
| `ROUTER_COMFY_CHECKPOINT` | *(unset)* | checkpoint filename for the default image workflow |
|
|
|
|
> ⚠️ `ROUTER_OLLAMA` is the router's **client** target — keep it `127.0.0.1`.
|
|
> Do **not** confuse it with Ollama's own `OLLAMA_HOST` server-bind variable.
|
|
|
|
**Keys** are loaded from files so they stay out of git: put a token in `.apikey`
|
|
and/or `.uncensored_key` (see the `*.example` files). Both are `.gitignore`d.
|
|
|
|
**The model map** lives at the top of [`router.py`](router.py) (the `M` and
|
|
`UNCENSORED` dicts). Edit those to match the models you've pulled — this is the
|
|
main thing to customize for your machine. See [docs/ROUTING.md](docs/ROUTING.md).
|
|
|
|
## Endpoints
|
|
|
|
| Method | Path | Notes |
|
|
|---|---|---|
|
|
| POST | `/v1/chat/completions` | OpenAI chat; `model:"auto"` = smart routing |
|
|
| POST | `/v1/messages` | Anthropic Messages (Claude Code) → LiteLLM |
|
|
| POST | `/v1/audio/transcriptions` | Whisper-shaped speech-to-text |
|
|
| POST | `/v1/audio/translations` | speech → English text |
|
|
| POST | `/v1/images/generations` | OpenAI-shaped image gen → ComfyUI |
|
|
| GET | `/v1/models` | lists route targets |
|
|
| GET | `/`, `/api/version`, `/api/tags`, `/api/ps`, POST `/api/show` | Ollama-native probes (so `ollama launch` accepts the router) |
|
|
| GET | `/healthz` | liveness |
|
|
|
|
## Docs
|
|
|
|
- [docs/DEPLOY.md](docs/DEPLOY.md) — run as a service, expose it, tunnels, Caddy
|
|
- [docs/ROUTING.md](docs/ROUTING.md) — how routing decides + customizing the map
|
|
- [docs/AUDIO.md](docs/AUDIO.md) — voice input & transcription
|
|
- [docs/IMAGES.md](docs/IMAGES.md) — image generation via ComfyUI
|
|
- [docs/BENCHMARKS.md](docs/BENCHMARKS.md) — the benchmark harnesses & sample results
|
|
|
|
## Security
|
|
|
|
- Default bind is `0.0.0.0` (LAN-reachable). Set `ROUTER_HOST=127.0.0.1` for
|
|
local-only, or set `ROUTER_API_KEY` before exposing it anywhere.
|
|
- The Ollama-native `/api/*` endpoints the router serves are **read-only**
|
|
(version/tags/show) — there is no unauthenticated `/api/chat` inference path.
|
|
|
|
## License
|
|
|
|
MIT — see [LICENSE](LICENSE).
|