Joseph CostaandClaude Opus 4.8
9938d46a67
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1.6 KiB
1.6 KiB
Local fleet benchmark — 2026-07-03 17:43
Performance (sorted by generation tok/s)
| Model | Gen tok/s | Prompt tok/s | Cold load | TTFT | Placement | OCR | Status |
|---|---|---|---|---|---|---|---|
gemma4:e2b |
100.0 | 610 | 0.2s | 0.3s | 100% GPU | PASS | ok |
qwen3-coder:30b |
85.7 | 297 | 0.1s | 0.2s | 100% GPU | - | ok |
qwen3-vl:30b-a3b-instruct |
85.4 | 129 | 0.1s | 0.4s | 100% GPU | PASS | ok |
qwen3.6:35b-a3b |
74.5 | 250 | 0.2s | 0.3s | 100% GPU | - | ok |
gpt-oss:20b |
72.9 | 516 | 0.2s | 0.4s | 100% GPU | - | ok |
gemma4:26b |
70.2 | 212 | 0.2s | 0.4s | 100% GPU | PASS | ok |
gemma4:e4b |
66.0 | 369 | 0.2s | 0.3s | 100% GPU | PASS | ok |
glm-4.7-flash:latest |
65.5 | 150 | 0.1s | 0.3s | 100% GPU | - | ok |
qwen3-vl:8b |
60.4 | 363 | 0.1s | 0.2s | 100% GPU | PASS | ok |
qwen3:8b |
59.0 | 244 | 0.1s | 0.2s | 100% GPU | - | ok |
gemma4:12b |
35.9 | 167 | 0.2s | 0.5s | 100% GPU | PASS | ok |
qwen3:14b |
35.7 | 200 | 0.1s | 0.3s | 100% GPU | - | ok |
qwen3.6:27b |
18.3 | 71 | 0.2s | 0.7s | 100% GPU | - | ok |
gemma4:31b |
16.6 | 66 | 0.2s | 0.8s | 100% GPU | PASS | ok |
Routing correctness
| Case | Expected | Routed to | HTTP | Verdict |
|---|---|---|---|---|
| short chat | qwen3:8b |
qwen3:8b |
200 | PASS |
| code | glm-4.7-flash |
glm-4.7-flash |
200 | PASS |
| repo code | qwen3-coder:30b |
qwen3-coder:30b |
200 | PASS |
| reasoning | qwen3.6:35b-a3b |
qwen3.6:35b-a3b |
200 | PASS |
| vision | qwen3-vl:8b |
qwen3-vl:8b |
200 | PASS |
| ocr/doc | qwen3-vl:30b-a3b-instruct |
qwen3-vl:30b-a3b-instruct |
200 | PASS |