Joseph CostaandClaude Opus 4.8
9938d46a67
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
21 lines
1.0 KiB
Markdown
21 lines
1.0 KiB
Markdown
# Agentic tool-use benchmark — 2026-07-03 18:18
|
|
|
|
Via Ollama tools API. Each column 0/1 (or %). abstain = correctly did NOT call a tool when none was needed.
|
|
|
|
| Model | Overall | Invoke | Args | Select | MultiArg | UseResult | Abstain |
|
|
|---|--:|:--:|:--:|:--:|:--:|:--:|:--:|
|
|
| `qwen3-vl:30b-a3b-instruct` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `qwen3-vl:8b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `qwen3-coder:30b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `glm-4.7-flash:latest` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `gpt-oss:20b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `qwen3.6:35b-a3b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `gemma4:31b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `gemma4:26b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `qwen3.6:27b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `gemma4:e4b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `qwen3:14b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `gemma4:12b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `gemma4:e2b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|
|
| `qwen3:8b` | **100%** | 1 | 1 | 1 | 1 | 1 | 1 |
|