Joseph CostaandClaude Opus 4.8
9938d46a67
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
32 lines
1.6 KiB
Markdown
32 lines
1.6 KiB
Markdown
# Local fleet benchmark — 2026-07-03 17:43
|
|
|
|
## Performance (sorted by generation tok/s)
|
|
|
|
| Model | Gen tok/s | Prompt tok/s | Cold load | TTFT | Placement | OCR | Status |
|
|
|---|--:|--:|--:|--:|:--|:--:|:--|
|
|
| `gemma4:e2b` | 100.0 | 610 | 0.2s | 0.3s | 100% GPU | PASS | ok |
|
|
| `qwen3-coder:30b` | 85.7 | 297 | 0.1s | 0.2s | 100% GPU | - | ok |
|
|
| `qwen3-vl:30b-a3b-instruct` | 85.4 | 129 | 0.1s | 0.4s | 100% GPU | PASS | ok |
|
|
| `qwen3.6:35b-a3b` | 74.5 | 250 | 0.2s | 0.3s | 100% GPU | - | ok |
|
|
| `gpt-oss:20b` | 72.9 | 516 | 0.2s | 0.4s | 100% GPU | - | ok |
|
|
| `gemma4:26b` | 70.2 | 212 | 0.2s | 0.4s | 100% GPU | PASS | ok |
|
|
| `gemma4:e4b` | 66.0 | 369 | 0.2s | 0.3s | 100% GPU | PASS | ok |
|
|
| `glm-4.7-flash:latest` | 65.5 | 150 | 0.1s | 0.3s | 100% GPU | - | ok |
|
|
| `qwen3-vl:8b` | 60.4 | 363 | 0.1s | 0.2s | 100% GPU | PASS | ok |
|
|
| `qwen3:8b` | 59.0 | 244 | 0.1s | 0.2s | 100% GPU | - | ok |
|
|
| `gemma4:12b` | 35.9 | 167 | 0.2s | 0.5s | 100% GPU | PASS | ok |
|
|
| `qwen3:14b` | 35.7 | 200 | 0.1s | 0.3s | 100% GPU | - | ok |
|
|
| `qwen3.6:27b` | 18.3 | 71 | 0.2s | 0.7s | 100% GPU | - | ok |
|
|
| `gemma4:31b` | 16.6 | 66 | 0.2s | 0.8s | 100% GPU | PASS | ok |
|
|
|
|
## Routing correctness
|
|
|
|
| Case | Expected | Routed to | HTTP | Verdict |
|
|
|---|---|---|--:|:--:|
|
|
| short chat | `qwen3:8b` | `qwen3:8b` | 200 | PASS |
|
|
| code | `glm-4.7-flash` | `glm-4.7-flash` | 200 | PASS |
|
|
| repo code | `qwen3-coder:30b` | `qwen3-coder:30b` | 200 | PASS |
|
|
| reasoning | `qwen3.6:35b-a3b` | `qwen3.6:35b-a3b` | 200 | PASS |
|
|
| vision | `qwen3-vl:8b` | `qwen3-vl:8b` | 200 | PASS |
|
|
| ocr/doc | `qwen3-vl:30b-a3b-instruct` | `qwen3-vl:30b-a3b-instruct` | 200 | PASS |
|