homelabstack/docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md

gpt-oss-20b — .73 Ollama box vs valhalla P100 (2026-06-28)

TL;DR: The same model (gpt-oss-20b) runs ~1.3–1.5× faster on generation and ~1.6–2× faster on prefill on ginnoir's 192.168.1.73 Ollama box than on valhalla's Tesla P100 llama-swap backend — despite the .73 box partially offloading to CPU at 64K context. Tool-calls and code outputs are correct on both. For interactive Hermes use the .73 box is the better backend whenever it's powered on; the P100 remains the always-on fallback. Caveat: at very deep context (~43K tokens) .73 generation drops to ~8 tok/s (KV no longer fully GPU-resident).

Setup

• Endpoint: http://192.168.1.73:11434 — Ollama 0.30.11, model gpt-oss:20b, quant MXFP4 (OpenAI's native gpt-oss 4-bit format), reported 20.9B params.
• Serving config: num_ctx 65536 (matches the P100's 64K and Hermes' hard ≥64K requirement), sampling temperature 0.6 / top_p 0.95 / top_k 20 (identical to the 2026-06-27 Ornith/P100 run).
• Measured via Ollama's native /api/chat (exact prompt_eval/eval token counts + durations). Cold model load at 64K ctx took ~65 s (one-time).
• P100 baseline = the gpt-oss-20b column from docs/2026-06-27-ornith-9b-benchmark.md (llama-swap, q8/q8 KV, --parallel 1, --jinja, 64K).
• GPU on .73 could not be identified — no SSH (port 22 filtered). Inferred from VRAM behavior (below): a fast but VRAM-limited card (or unified-memory host).

Results

Test	Metric	.73 Ollama (MXFP4)	valhalla P100 (GGUF)	.73 advantage
Tool call	valid tool_calls?	✅ get_weather({"city":"Tokyo"})	✅ identical	tie (both correct)
Codegen (merge_intervals)	correct?	✅ correct, non-mutating, type-hinted	✅ correct (mutates input)	.73 slightly cleaner
Debug (second_largest)	correct?	✅ correct single-pass, handles dup max	✅ correct	tie
Deep-prefill	answered?	✅ "quick brown fox"	✅ (capped probe)	tie
Gen speed	tok/s — tool_call	33.1	23.0	1.44×
Gen speed	tok/s — codegen	27.4	21.7	1.26×
Gen speed	tok/s — debug	27.6	18.9	1.46×
Prefill	tok/s — shallow (~100–135 tok)	240–287	133–188	~1.5–1.8×
Prefill	tok/s — deep	2542 (n=43.8k)	155 (n=23.4k)	far faster (diff depths)
Gen @ deep ctx	tok/s	8.4 (n=43.8k ctx)	12.7 (n=23.4k ctx)	P100 wins at depth
Reasoning verbosity	think chars (codegen/debug)	1760 / 1317	1671 / 2118	comparable
VRAM @ 64K	on-GPU / total	8.82 / 14.16 GB (partial CPU offload)	~12.6 GB (100% GPU)	—

Reading the numbers

• Shallow/typical depth is where .73 wins decisively. All three real tasks (tool-call, codegen, debug) run at shallow context, and .73 generates at 27–33 tok/s vs the P100's ~19–23 — roughly the difference between "comfortable" and "sluggish" for an interactive agent loop. Prefill is also ~1.5–1.8× faster, so first-token latency improves too.
• The .73 box is VRAM-limited, not compute-limited. At 64K ctx only 8.82 GB of the 14.16 GB working set sits in VRAM — the rest (weights tail + deep KV) spills to system RAM. It still beats the full-GPU P100, which means the card itself is much faster than the P100; with more VRAM (or a smaller num_ctx) it would pull further ahead.
• The one place the P100 wins: very deep context. At ~43K resident tokens, .73 generation falls to 8.4 tok/s because the KV cache is partly in CPU RAM (memory- bandwidth-bound attention). The P100 holds its whole 64K KV in VRAM and degrades more gracefully (12.7 tok/s at 23K). In practice Hermes' steady-state prompt is ~16K, so this rarely bites — but long sessions on .73 will slow down more than on the P100.
• Quant differs, so this isn't a pure hardware A/B: .73 runs MXFP4 (gpt-oss's native, near-lossless 4-bit) while the P100 GGUF quant is whatever llama-swap pulled. Both are genuine gpt-oss-20b and both produced correct outputs; no quality regression observed.

Verdict for the Hermes backend

• Prefer .73 when it's up. It's the faster daily driver for gpt-oss-20b at the depths Hermes actually runs at. Switch in-session with /model --provider ollama --model gpt-oss:20b.
• Keep the P100 (valhalla-p100) as the always-on default. It's a container on the 24/7 server; the .73 box may be a desktop/workstation that isn't always powered. The P100 also degrades more gracefully at very deep context.
• If you want .73 to be strictly better, drop its num_ctx toward what Hermes needs (it hard-requires ≥64K, so you can't go below that for Hermes) or put gpt-oss on a bigger-VRAM card there — eliminating the CPU spill would lift both prefill and deep-context generation.

Caveats

• Small hand-written suite (4 tasks), not SWE-bench — measures latency/throughput and basic correctness, not deep code quality.
• The codegen prompt lost its back-ticked tokens to shell quoting during the run (prompt_n 95 vs the P100's 113); the model still produced a correct merge_intervals, and gen tok/s is prompt-content-independent, so the speed comparison stands.
• Deep-prefill rows use different context depths (43.8K on .73 vs 23.4K on P100), so the prefill-tok/s cells aren't directly comparable — read them as "each box's deep-prefill rate at that depth," not a head-to-head ratio.
• Raw responses saved on valhalla at /tmp/ollama-bench/ (one .json per task); P100 baselines at /tmp/ornith-bench/.