homelabstack/docs/2026-06-28-ollama-73-fleet-benchmark.md

.73 Ollama fleet benchmark — all 9 models (2026-06-28)

TL;DR: Throughput across every model on 192.168.1.73. Generation speed spans an ~5× range — from gemma4:e4b at ~93 tok/s down to the big qwen3:30b-a3b at ~18 tok/s. All 9 models emit valid tool-calls. For an interactive agent backend the sweet spot is gpt-oss:20b (~29 tok/s) or gemma4:12b (~48 tok/s) if 12B quality suffices; the 30B-class Qwen MoEs are the slowest here (heavy CPU offload at 64K on this VRAM-limited box). Caveat: code-correctness for the heavy thinking models is indeterminate — they used the whole 768-token gen cap reasoning and never emitted code (see Caveats); re-run with a bigger budget to judge quality.

Setup

• Endpoint http://192.168.1.73:11434, Ollama 0.30.11. Each model served at num_ctx 65536 (Hermes' ≥64K requirement), sampling temp 0.6 / top_p 0.95 / top_k 20.
• Native /api/chat timings. 4 tasks: tool-call, codegen (merge_intervals), debug (second_largest), deep-prefill (~16K-token filler). Gen capped: 256 / 768 / 768 / 128.
• Same .73 box as the gpt-oss head-to-head in docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md (GPU still unidentified — no SSH).

Generation speed (tok/s) — the headline

Average of the three real tasks (tool-call / codegen / debug), sorted fastest first:

Model	avg gen t/s	tool	codegen	debug	deep-ctx gen	cold load s	tool-call?
gemma4:e4b	92.7	91.1	93.3	93.6	82.6	24.8	✅
gemma4:12b	48.2	46.8	49.0	48.8	45.7	10.6	✅
gpt-oss:20b	29.4	32.7	27.9	27.7	29.9	0.4¹	✅
qwen3.6:35b-a3b	27.3	29.1	26.4	26.4	28.2	35.9	✅
gemma4:26b	25.7	27.6	25.0	24.5	26.5	53.8	✅
glm-4.7-flash	21.2	24.2	19.7	19.7	21.4	34.8	✅
qwen3-vl:30b-a3b	19.1	22.7	16.5	18.2	20.6	29.8	✅
qwen3-coder:30b	19.0	22.8	17.1	17.2	20.4	25.3	✅
qwen3:30b-a3b	17.6	19.5	16.6	16.6	18.5	25.0	✅

¹ gpt-oss was already resident from the prior run; real cold load is ~65 s.

Prefill speed (tok/s)

Model	shallow (~100 tok)	deep (~16K tok)
gemma4:e4b	1193–1799	7283
gemma4:12b	667–1078	3318
gpt-oss:20b	281–379	2655
qwen3:30b-a3b	63–149	1030
qwen3-coder:30b	76–270	979
gemma4:26b	110–127²	977
qwen3.6:35b-a3b	78–209	620
glm-4.7-flash	70–163	649
qwen3-vl:30b-a3b	68–127	588

² gemma4:26b's first request after load measured 11.7 t/s (cold-cache artifact); ignore.

What stands out

• The two small gemmas are in a different league. gemma4:e4b (~93 t/s) and gemma4:12b (~48 t/s) are dense but small, so they sit fully on GPU and fly. If a 4B/12B is smart enough for the job, they're the most responsive options by far.
• gpt-oss:20b is the best "big-brain, still-fast" pick (~29 t/s) — MoE ~3.6B active keeps it quick despite 20B total. qwen3.6:35b-a3b nearly matches it (~27 t/s) and may be stronger; worth A/B-ing on real tasks.
• The 30B-a3b Qwen trio is the slowest (~17–19 t/s). Same "3B-active" MoE label, but larger total weights → more spills to CPU RAM at 64K on this VRAM-limited box, dragging generation below gpt-oss. qwen3-coder being this slow undercuts it as a fast coding model here.
• Every model tool-calls. All 9 emitted a valid get_weather({"city":"Tokyo"}), so any of them can drive Hermes' tool loop.

Correctness (partial — see caveat)

• Confirmed correct code on the non-/light-thinking models that finished within the cap: gpt-oss:20b, gemma4:12b, qwen3-vl:30b-a3b, qwen3-coder:30b (clean def, reason=stop or code present), plus gemma4:26b & qwen3:30b-a3b on the task each finished.
• Indeterminate (truncated mid-reasoning, content=0, reason=length): glm-4.7-flash (both), gemma4:e4b (both), qwen3.6:35b-a3b (both), gemma4:26b (codegen), qwen3:30b-a3b (debug). These spent all 768 gen tokens in the thinking channel — not wrong, just unfinished. A re-run at num_predict ~3072 is needed to grade their output.

Caveats

• The 768-token gen cap was too low for heavy chain-of-thought models — it bounds runtime but truncates their answers. Speed (tok/s) is unaffected and valid; code quality for the truncated set is not measured here.
• Per-model VRAM split not captured (models unload after 2 min keep_alive); only gpt-oss-20b is known (8.82 GB on-GPU / 14.16 GB total at 64K → partial CPU offload). The slow 30B-class numbers are consistent with heavier offload.
• Small hand-written suite, not SWE-bench. Quants are each model's Ollama default.
• Raw per-task responses on valhalla at /tmp/ollama-bench-all/ (summary.json + one JSON per model/task); progress log /tmp/ollama-bench-all/progress.txt.