Joseph CostaandClaude Opus 4.8
9938d46a67
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
39 lines
1.6 KiB
Markdown
39 lines
1.6 KiB
Markdown
# Local fleet — combined benchmark (quality × agentic × speed)
|
||
|
||
Date: 2026-07-03. Hardware: Apple M4 Max, 36 GB. All models 100% GPU.
|
||
|
||
- **quality** = auto-graded mean of code (executed unit tests), math (GSM8K-style),
|
||
MC (knowledge/reasoning), instruction-following. think disabled for a fast baseline.
|
||
- **agentic** = tool-use via Ollama tools API: invoke / args / select / multi-arg /
|
||
use-result / abstain (6 checks).
|
||
- **tok/s** = generation speed from Ollama eval metrics.
|
||
- \* = corrected after a think/budget artifact in the first quality pass.
|
||
|
||
| Model | Quality | Agentic | tok/s |
|
||
|---|--:|--:|--:|
|
||
| gemma4:26b | 100% | 100% | 71.6 |
|
||
| gemma4:31b | 100% | 100% | 16.6 |
|
||
| qwen3.6:27b | 96% | 100% | 18.3 |
|
||
| gemma4:e4b | 96% | 100% | 66.9 |
|
||
| gemma4:e2b | 92% | 100% | 101.3 |
|
||
| qwen3-vl:30b-a3b-instruct | 92% | 100% | 86.1 |
|
||
| gpt-oss:20b | 92%* | 100% | 70.9 |
|
||
| qwen3:14b | 92% | 100% | 36.2 |
|
||
| qwen3.6:35b-a3b | 91% | 100% | 74.5 |
|
||
| qwen3-coder:30b | 88% | 100% | 85.5 |
|
||
| qwen3:8b | 88% | 100% | 59.9 |
|
||
| gemma4:12b | 87% | 100% | 36.3 |
|
||
| glm-4.7-flash | 83% | 100% | 65.6 |
|
||
| qwen3-vl:8b | 62%* | 100% | 58.1 |
|
||
|
||
## Takeaways
|
||
- gemma4:26b — best all-round (top quality + fast, MoE). Strong router default.
|
||
- MoE >> dense on this Mac: gemma4:31b / qwen3.6:27b match on quality but ~4x slower.
|
||
- gemma4:e2b — best speed/size/quality ratio (92% @ 101 tok/s, 7 GB).
|
||
- Agentic basics are universal (all 6/6); needs harder tasks to rank.
|
||
|
||
## Harnesses
|
||
- benchmark.py — speed + OCR + routing
|
||
- quality_bench.py — objective quality auto-grading
|
||
- agent_bench.py — tool-use scenarios
|