# `.73` Ollama fleet benchmark — all 9 models (2026-06-28)

**TL;DR:** Throughput across every model on `192.168.1.73`. Generation speed spans an
**~5× range** — from `gemma4:e4b` at **~93 tok/s** down to the big `qwen3:30b-a3b` at
**~18 tok/s**. **All 9 models emit valid tool-calls.** For an interactive agent backend the
sweet spot is **`gpt-oss:20b` (~29 tok/s)** or **`gemma4:12b` (~48 tok/s)** if 12B quality
suffices; the 30B-class Qwen MoEs are the slowest here (heavy CPU offload at 64K on this
VRAM-limited box). **Caveat:** code-correctness for the heavy *thinking* models is
indeterminate — they used the whole 768-token gen cap reasoning and never emitted code (see
Caveats); re-run with a bigger budget to judge quality.

## Setup
- Endpoint `http://192.168.1.73:11434`, Ollama 0.30.11. Each model served at **`num_ctx
  65536`** (Hermes' ≥64K requirement), sampling `temp 0.6 / top_p 0.95 / top_k 20`.
- Native `/api/chat` timings. 4 tasks: tool-call, codegen (`merge_intervals`), debug
  (`second_largest`), deep-prefill (~16K-token filler). Gen capped: 256 / 768 / 768 / 128.
- Same `.73` box as the gpt-oss head-to-head in
  `docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md` (GPU still unidentified — no SSH).

## Generation speed (tok/s) — the headline

Average of the three real tasks (tool-call / codegen / debug), sorted fastest first:

| Model | avg gen t/s | tool | codegen | debug | deep-ctx gen | cold load s | tool-call? |
|---|--:|--:|--:|--:|--:|--:|:--:|
| **gemma4:e4b** | **92.7** | 91.1 | 93.3 | 93.6 | 82.6 | 24.8 | ✅ |
| **gemma4:12b** | **48.2** | 46.8 | 49.0 | 48.8 | 45.7 | 10.6 | ✅ |
| **gpt-oss:20b** | **29.4** | 32.7 | 27.9 | 27.7 | 29.9 | 0.4¹ | ✅ |
| **qwen3.6:35b-a3b** | **27.3** | 29.1 | 26.4 | 26.4 | 28.2 | 35.9 | ✅ |
| **gemma4:26b** | **25.7** | 27.6 | 25.0 | 24.5 | 26.5 | 53.8 | ✅ |
| **glm-4.7-flash** | **21.2** | 24.2 | 19.7 | 19.7 | 21.4 | 34.8 | ✅ |
| **qwen3-vl:30b-a3b** | **19.1** | 22.7 | 16.5 | 18.2 | 20.6 | 29.8 | ✅ |
| **qwen3-coder:30b** | **19.0** | 22.8 | 17.1 | 17.2 | 20.4 | 25.3 | ✅ |
| **qwen3:30b-a3b** | **17.6** | 19.5 | 16.6 | 16.6 | 18.5 | 25.0 | ✅ |

¹ gpt-oss was already resident from the prior run; real cold load is ~65 s.

## Prefill speed (tok/s)

| Model | shallow (~100 tok) | deep (~16K tok) |
|---|--:|--:|
| gemma4:e4b | 1193–1799 | 7283 |
| gemma4:12b | 667–1078 | 3318 |
| gpt-oss:20b | 281–379 | 2655 |
| qwen3:30b-a3b | 63–149 | 1030 |
| qwen3-coder:30b | 76–270 | 979 |
| gemma4:26b | 110–127² | 977 |
| qwen3.6:35b-a3b | 78–209 | 620 |
| glm-4.7-flash | 70–163 | 649 |
| qwen3-vl:30b-a3b | 68–127 | 588 |

² gemma4:26b's first request after load measured 11.7 t/s (cold-cache artifact); ignore.

## What stands out
- **The two small gemmas are in a different league.** `gemma4:e4b` (~93 t/s) and
  `gemma4:12b` (~48 t/s) are dense but small, so they sit fully on GPU and fly. If a 4B/12B
  is smart enough for the job, they're the most responsive options by far.
- **gpt-oss:20b is the best "big-brain, still-fast" pick** (~29 t/s) — MoE ~3.6B active keeps
  it quick despite 20B total. `qwen3.6:35b-a3b` nearly matches it (~27 t/s) and may be
  stronger; worth A/B-ing on real tasks.
- **The 30B-a3b Qwen trio is the slowest** (~17–19 t/s). Same "3B-active" MoE label, but
  larger total weights → more spills to CPU RAM at 64K on this VRAM-limited box, dragging
  generation below gpt-oss. `qwen3-coder` being this slow undercuts it as a *fast* coding
  model here.
- **Every model tool-calls.** All 9 emitted a valid `get_weather({"city":"Tokyo"})`, so any
  of them can drive Hermes' tool loop.

## Correctness (partial — see caveat)
- **Confirmed correct** code on the non-/light-thinking models that finished within the cap:
  `gpt-oss:20b`, `gemma4:12b`, `qwen3-vl:30b-a3b`, `qwen3-coder:30b` (clean `def`,
  `reason=stop` or code present), plus `gemma4:26b` & `qwen3:30b-a3b` on the task each
  finished.
- **Indeterminate** (truncated mid-reasoning, `content=0`, `reason=length`): `glm-4.7-flash`
  (both), `gemma4:e4b` (both), `qwen3.6:35b-a3b` (both), `gemma4:26b` (codegen),
  `qwen3:30b-a3b` (debug). These spent all 768 gen tokens in the `thinking` channel — **not
  wrong, just unfinished.** A re-run at `num_predict ~3072` is needed to grade their output.

## Caveats
- The 768-token gen cap was too low for heavy chain-of-thought models — it bounds runtime but
  truncates their answers. Speed (tok/s) is unaffected and valid; code *quality* for the
  truncated set is not measured here.
- Per-model VRAM split not captured (models unload after 2 min `keep_alive`); only
  gpt-oss-20b is known (8.82 GB on-GPU / 14.16 GB total at 64K → partial CPU offload). The
  slow 30B-class numbers are consistent with heavier offload.
- Small hand-written suite, not SWE-bench. Quants are each model's Ollama default.
- Raw per-task responses on valhalla at `/tmp/ollama-bench-all/` (`summary.json` + one JSON
  per model/task); progress log `/tmp/ollama-bench-all/progress.txt`.