ginnoir/llm-router
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: llm-router — smart OpenAI/Anthropic/Ollama router

Browse Source 
A stdlib pre-router in front of Ollama with LiteLLM backend:
- auto model selection by content/tools/modality, with fallbacks
- OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints
- Whisper-shaped /v1/audio/transcriptions + in-chat audio
- key-based fleet policies (e.g. force a client onto uncensored models)
- optional Bearer auth; launchd/systemd service install
- benchmark harnesses (speed, quality, agentic tool use) with sample results

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Joseph Costa
2026-07-05 02:05:16 -05:00
co-authored by Claude Opus 4.8
commit 9938d46a67
32 changed files with 2419 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/results/agent_report.md
+20
View File
@@ -0,0 +1,20 @@
# Agentic tool-use benchmark — 2026-07-03 18:18
Via Ollama tools API. Each column 0/1 (or %). abstain = correctly did NOT call a tool when none was needed.
| Model | Overall | Invoke | Args | Select | MultiArg | UseResult | Abstain |
|---|--:|:--:|:--:|:--:|:--:|:--:|:--:|
| `qwen3-vl:30b-a3b-instruct` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `qwen3-vl:8b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `qwen3-coder:30b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `glm-4.7-flash:latest` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `gpt-oss:20b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `qwen3.6:35b-a3b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `gemma4:31b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `gemma4:26b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `qwen3.6:27b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `gemma4:e4b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `qwen3:14b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `gemma4:12b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `gemma4:e2b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
| `qwen3:8b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
benchmarks/results/agent_results.json
+128
View File
@@ -0,0 +1,128 @@
{
"qwen3-vl:30b-a3b-instruct": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"qwen3-vl:8b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"qwen3-coder:30b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"glm-4.7-flash:latest": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"gpt-oss:20b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"qwen3.6:35b-a3b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"gemma4:31b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"gemma4:26b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"qwen3.6:27b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"gemma4:e4b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"qwen3:14b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"gemma4:12b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"gemma4:e2b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
},
"qwen3:8b": {
"invoke": 1,
"args": 1,
"select": 1,
"multiarg": 1,
"useresult": 1,
"abstain": 1,
"overall": 1.0
}
}
benchmarks/results/bench_report.md
+31
View File
@@ -0,0 +1,31 @@
# Local fleet benchmark — 2026-07-03 17:43
## Performance (sorted by generation tok/s)
| Model | Gen tok/s | Prompt tok/s | Cold load | TTFT | Placement | OCR | Status |
|---|--:|--:|--:|--:|:--|:--:|:--|
| `gemma4:e2b` | 100.0 | 610 | 0.2s | 0.3s | 100% GPU | PASS | ok |
| `qwen3-coder:30b` | 85.7 | 297 | 0.1s | 0.2s | 100% GPU | - | ok |
| `qwen3-vl:30b-a3b-instruct` | 85.4 | 129 | 0.1s | 0.4s | 100% GPU | PASS | ok |
| `qwen3.6:35b-a3b` | 74.5 | 250 | 0.2s | 0.3s | 100% GPU | - | ok |
| `gpt-oss:20b` | 72.9 | 516 | 0.2s | 0.4s | 100% GPU | - | ok |
| `gemma4:26b` | 70.2 | 212 | 0.2s | 0.4s | 100% GPU | PASS | ok |
| `gemma4:e4b` | 66.0 | 369 | 0.2s | 0.3s | 100% GPU | PASS | ok |
| `glm-4.7-flash:latest` | 65.5 | 150 | 0.1s | 0.3s | 100% GPU | - | ok |
| `qwen3-vl:8b` | 60.4 | 363 | 0.1s | 0.2s | 100% GPU | PASS | ok |
| `qwen3:8b` | 59.0 | 244 | 0.1s | 0.2s | 100% GPU | - | ok |
| `gemma4:12b` | 35.9 | 167 | 0.2s | 0.5s | 100% GPU | PASS | ok |
| `qwen3:14b` | 35.7 | 200 | 0.1s | 0.3s | 100% GPU | - | ok |
| `qwen3.6:27b` | 18.3 | 71 | 0.2s | 0.7s | 100% GPU | - | ok |
| `gemma4:31b` | 16.6 | 66 | 0.2s | 0.8s | 100% GPU | PASS | ok |
## Routing correctness
| Case | Expected | Routed to | HTTP | Verdict |
|---|---|---|--:|:--:|
| short chat | `qwen3:8b` | `qwen3:8b` | 200 | PASS |
| code | `glm-4.7-flash` | `glm-4.7-flash` | 200 | PASS |
| repo code | `qwen3-coder:30b` | `qwen3-coder:30b` | 200 | PASS |
| reasoning | `qwen3.6:35b-a3b` | `qwen3.6:35b-a3b` | 200 | PASS |
| vision | `qwen3-vl:8b` | `qwen3-vl:8b` | 200 | PASS |
| ocr/doc | `qwen3-vl:30b-a3b-instruct` | `qwen3-vl:30b-a3b-instruct` | 200 | PASS |
benchmarks/results/combined_report.md
+38
View File
@@ -0,0 +1,38 @@
# Local fleet — combined benchmark (quality × agentic × speed)
Date: 2026-07-03. Hardware: Apple M4 Max, 36 GB. All models 100% GPU.
- **quality** = auto-graded mean of code (executed unit tests), math (GSM8K-style),
MC (knowledge/reasoning), instruction-following. think disabled for a fast baseline.
- **agentic** = tool-use via Ollama tools API: invoke / args / select / multi-arg /
use-result / abstain (6 checks).
- **tok/s** = generation speed from Ollama eval metrics.
- \* = corrected after a think/budget artifact in the first quality pass.
| Model | Quality | Agentic | tok/s |
|---|--:|--:|--:|
| gemma4:26b | 100% | 100% | 71.6 |
| gemma4:31b | 100% | 100% | 16.6 |
| qwen3.6:27b | 96% | 100% | 18.3 |
| gemma4:e4b | 96% | 100% | 66.9 |
| gemma4:e2b | 92% | 100% | 101.3 |
| qwen3-vl:30b-a3b-instruct | 92% | 100% | 86.1 |
| gpt-oss:20b | 92%* | 100% | 70.9 |
| qwen3:14b | 92% | 100% | 36.2 |
| qwen3.6:35b-a3b | 91% | 100% | 74.5 |
| qwen3-coder:30b | 88% | 100% | 85.5 |
| qwen3:8b | 88% | 100% | 59.9 |
| gemma4:12b | 87% | 100% | 36.3 |
| glm-4.7-flash | 83% | 100% | 65.6 |
| qwen3-vl:8b | 62%* | 100% | 58.1 |
## Takeaways
- gemma4:26b — best all-round (top quality + fast, MoE). Strong router default.
- MoE >> dense on this Mac: gemma4:31b / qwen3.6:27b match on quality but ~4x slower.
- gemma4:e2b — best speed/size/quality ratio (92% @ 101 tok/s, 7 GB).
- Agentic basics are universal (all 6/6); needs harder tasks to rank.
## Harnesses
- benchmark.py — speed + OCR + routing
- quality_bench.py — objective quality auto-grading
- agent_bench.py — tool-use scenarios
benchmarks/results/hard_agent_report.md
+11
View File
@@ -0,0 +1,11 @@
# Hard agentic benchmark (multi-turn tool loop) — 2026-07-03 19:09
Real tool execution with results fed back; thinking enabled.
| Model | Overall | Seq | Parallel | Chain | Select | Recover | Abstain | Honesty |
|---|--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| `glm-4.7-flash` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| `gemma4:26b` | **86%** | 1 | 1 | 1 | 1 | 0 | 1 | 1 |
| `qwen3.6:35b-a3b` | **86%** | 1 | 1 | 1 | 1 | 0 | 1 | 1 |
| `qwen3-coder:30b` | **86%** | 1 | 1 | 0 | 1 | 1 | 1 | 1 |
| `gpt-oss:20b` | **71%** | 1 | 1 | 1 | 1 | 0 | 1 | 0 |
benchmarks/results/hard_agent_results.json
+52
View File
@@ -0,0 +1,52 @@
{
"gemma4:26b": {
"seq": 1,
"parallel": 1,
"chain": 1,
"select": 1,
"recover": 0,
"abstain": 1,
"honesty": 1,
"overall": 0.8571428571428571
},
"qwen3.6:35b-a3b": {
"seq": 1,
"parallel": 1,
"chain": 1,
"select": 1,
"recover": 0,
"abstain": 1,
"honesty": 1,
"overall": 0.8571428571428571
},
"glm-4.7-flash": {
"seq": 1,
"parallel": 1,
"chain": 1,
"select": 1,
"recover": 1,
"abstain": 1,
"honesty": 1,
"overall": 1.0
},
"qwen3-coder:30b": {
"seq": 1,
"parallel": 1,
"chain": 0,
"select": 1,
"recover": 1,
"abstain": 1,
"honesty": 1,
"overall": 0.8571428571428571
},
"gpt-oss:20b": {
"seq": 1,
"parallel": 1,
"chain": 1,
"select": 1,
"recover": 0,
"abstain": 1,
"honesty": 0,
"overall": 0.7142857142857143
}
}
benchmarks/results/quality_report.md
+20
View File
@@ -0,0 +1,20 @@
# Quality benchmark — 2026-07-03 18:15
Objective auto-grading. code=executed unit tests, math=GSM8K-style exact, mc=multiple-choice, ifollow=programmatic checks, ocr=multi-image text.
| Model | Overall | Code | Math | MC | Instr | OCR | tok/s |
|---|--:|--:|--:|--:|--:|--:|--:|
| `gemma4:31b` | **100%** | 100 | 100 | 100 | 100 | 100 | 16.6 |
| `gemma4:26b` | **100%** | 100 | 100 | 100 | 100 | 100 | 71.6 |
| `qwen3.6:27b` | **96%** | 100 | 100 | 83 | 100 | — | 18.3 |
| `gemma4:e4b` | **96%** | 100 | 100 | 83 | 100 | 100 | 66.9 |
| `qwen3-vl:30b-a3b-instruct` | **92%** | 100 | 100 | 100 | 67 | 100 | 86.1 |
| `qwen3:14b` | **92%** | 100 | 100 | 100 | 67 | — | 36.2 |
| `gemma4:e2b` | **92%** | 100 | 100 | 67 | 100 | 100 | 101.3 |
| `qwen3.6:35b-a3b` | **91%** | 100 | 80 | 83 | 100 | — | 74.5 |
| `qwen3-coder:30b` | **88%** | 100 | 100 | 83 | 67 | — | 85.5 |
| `qwen3:8b` | **88%** | 100 | 100 | 83 | 67 | — | 59.9 |
| `gemma4:12b` | **87%** | 100 | 80 | 67 | 100 | 100 | 36.3 |
| `glm-4.7-flash:latest` | **83%** | 100 | 100 | 67 | 67 | — | 65.6 |
| `gpt-oss:20b` | **62%** | 100 | 80 | 0 | 67 | — | 70.9 |
| `qwen3-vl:8b` | **22%** | 33 | 20 | 0 | 33 | 100 | 58.1 |
benchmarks/results/quality_results.json
+121
View File
@@ -0,0 +1,121 @@
{
"qwen3-vl:30b-a3b-instruct": {
"math": 1.0,
"mc": 1.0,
"code": 1.0,
"ifollow": 0.6666666666666666,
"ocr": 1.0,
"overall": 0.9166666666666666,
"tps": 86.08918225170864
},
"qwen3-vl:8b": {
"math": 0.2,
"mc": 0.0,
"code": 0.3333333333333333,
"ifollow": 0.3333333333333333,
"ocr": 1.0,
"overall": 0.21666666666666667,
"tps": 58.06267489952323
},
"qwen3-coder:30b": {
"math": 1.0,
"mc": 0.8333333333333334,
"code": 1.0,
"ifollow": 0.6666666666666666,
"overall": 0.875,
"tps": 85.51818292912147
},
"glm-4.7-flash:latest": {
"math": 1.0,
"mc": 0.6666666666666666,
"code": 1.0,
"ifollow": 0.6666666666666666,
"overall": 0.8333333333333333,
"tps": 65.60893213491771
},
"gpt-oss:20b": {
"math": 0.8,
"mc": 0.0,
"code": 1.0,
"ifollow": 0.6666666666666666,
"overall": 0.6166666666666667,
"tps": 70.93934599819382
},
"qwen3.6:35b-a3b": {
"math": 0.8,
"mc": 0.8333333333333334,
"code": 1.0,
"ifollow": 1.0,
"overall": 0.9083333333333333,
"tps": 74.50862085584875
},
"gemma4:31b": {
"math": 1.0,
"mc": 1.0,
"code": 1.0,
"ifollow": 1.0,
"ocr": 1.0,
"overall": 1.0,
"tps": 16.5738240176472
},
"gemma4:26b": {
"math": 1.0,
"mc": 1.0,
"code": 1.0,
"ifollow": 1.0,
"ocr": 1.0,
"overall": 1.0,
"tps": 71.55240664310993
},
"qwen3.6:27b": {
"math": 1.0,
"mc": 0.8333333333333334,
"code": 1.0,
"ifollow": 1.0,
"overall": 0.9583333333333334,
"tps": 18.287797403043964
},
"gemma4:e4b": {
"math": 1.0,
"mc": 0.8333333333333334,
"code": 1.0,
"ifollow": 1.0,
"ocr": 1.0,
"overall": 0.9583333333333334,
"tps": 66.86943765975911
},
"qwen3:14b": {
"math": 1.0,
"mc": 1.0,
"code": 1.0,
"ifollow": 0.6666666666666666,
"overall": 0.9166666666666666,
"tps": 36.22066333837863
},
"gemma4:12b": {
"math": 0.8,
"mc": 0.6666666666666666,
"code": 1.0,
"ifollow": 1.0,
"ocr": 1.0,
"overall": 0.8666666666666667,
"tps": 36.30471836404827
},
"gemma4:e2b": {
"math": 1.0,
"mc": 0.6666666666666666,
"code": 1.0,
"ifollow": 1.0,
"ocr": 1.0,
"overall": 0.9166666666666666,
"tps": 101.34789160325633
},
"qwen3:8b": {
"math": 1.0,
"mc": 0.8333333333333334,
"code": 1.0,
"ifollow": 0.6666666666666666,
"overall": 0.875,
"tps": 59.89123849272082
}
}