Joseph CostaandClaude Opus 4.8
9938d46a67
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1.2 KiB
1.2 KiB
Quality benchmark — 2026-07-03 18:15
Objective auto-grading. code=executed unit tests, math=GSM8K-style exact, mc=multiple-choice, ifollow=programmatic checks, ocr=multi-image text.
| Model | Overall | Code | Math | MC | Instr | OCR | tok/s |
|---|---|---|---|---|---|---|---|
gemma4:31b |
100% | 100 | 100 | 100 | 100 | 100 | 16.6 |
gemma4:26b |
100% | 100 | 100 | 100 | 100 | 100 | 71.6 |
qwen3.6:27b |
96% | 100 | 100 | 83 | 100 | — | 18.3 |
gemma4:e4b |
96% | 100 | 100 | 83 | 100 | 100 | 66.9 |
qwen3-vl:30b-a3b-instruct |
92% | 100 | 100 | 100 | 67 | 100 | 86.1 |
qwen3:14b |
92% | 100 | 100 | 100 | 67 | — | 36.2 |
gemma4:e2b |
92% | 100 | 100 | 67 | 100 | 100 | 101.3 |
qwen3.6:35b-a3b |
91% | 100 | 80 | 83 | 100 | — | 74.5 |
qwen3-coder:30b |
88% | 100 | 100 | 83 | 67 | — | 85.5 |
qwen3:8b |
88% | 100 | 100 | 83 | 67 | — | 59.9 |
gemma4:12b |
87% | 100 | 80 | 67 | 100 | 100 | 36.3 |
glm-4.7-flash:latest |
83% | 100 | 100 | 67 | 67 | — | 65.6 |
gpt-oss:20b |
62% | 100 | 80 | 0 | 67 | — | 70.9 |
qwen3-vl:8b |
22% | 33 | 20 | 0 | 33 | 100 | 58.1 |