# gpt-oss-20b — `.73` Ollama box vs valhalla P100 (2026-06-28)

**TL;DR:** The same model (`gpt-oss-20b`) runs **~1.3–1.5× faster on generation** and
**~1.6–2× faster on prefill** on ginnoir's `192.168.1.73` Ollama box than on valhalla's
Tesla P100 llama-swap backend — *despite* the `.73` box partially offloading to CPU at 64K
context. Tool-calls and code outputs are correct on both. For interactive Hermes use the
`.73` box is the better backend whenever it's powered on; the P100 remains the always-on
fallback. **Caveat:** at very deep context (~43K tokens) `.73` generation drops to ~8 tok/s
(KV no longer fully GPU-resident).

## Setup

- **Endpoint:** `http://192.168.1.73:11434` — Ollama **0.30.11**, model `gpt-oss:20b`,
  quant **MXFP4** (OpenAI's native gpt-oss 4-bit format), reported 20.9B params.
- **Serving config:** `num_ctx 65536` (matches the P100's 64K and Hermes' hard ≥64K
  requirement), sampling `temperature 0.6 / top_p 0.95 / top_k 20` (identical to the
  2026-06-27 Ornith/P100 run).
- **Measured via** Ollama's native `/api/chat` (exact `prompt_eval`/`eval` token counts +
  durations). Cold model load at 64K ctx took **~65 s** (one-time).
- **P100 baseline** = the gpt-oss-20b column from `docs/2026-06-27-ornith-9b-benchmark.md`
  (llama-swap, q8/q8 KV, `--parallel 1`, `--jinja`, 64K).
- GPU on `.73` **could not be identified** — no SSH (port 22 filtered). Inferred from VRAM
  behavior (below): a fast but VRAM-limited card (or unified-memory host).

## Results

| Test | Metric | **.73 Ollama (MXFP4)** | **valhalla P100 (GGUF)** | `.73` advantage |
|---|---|---|---|---|
| Tool call | valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie (both correct) |
| Codegen (`merge_intervals`) | correct? | ✅ correct, non-mutating, type-hinted | ✅ correct (mutates input) | `.73` slightly cleaner |
| Debug (`second_largest`) | correct? | ✅ correct single-pass, handles dup max | ✅ correct | tie |
| Deep-prefill | answered? | ✅ "quick brown fox" | ✅ (capped probe) | tie |
| **Gen speed** | tok/s — tool_call | **33.1** | 23.0 | **1.44×** |
| **Gen speed** | tok/s — codegen | **27.4** | 21.7 | **1.26×** |
| **Gen speed** | tok/s — debug | **27.6** | 18.9 | **1.46×** |
| **Prefill** | tok/s — shallow (~100–135 tok) | **240–287** | 133–188 | ~1.5–1.8× |
| **Prefill** | tok/s — deep | **2542** (n=43.8k) | 155 (n=23.4k) | far faster (diff depths) |
| Gen @ deep ctx | tok/s | 8.4 (n=43.8k ctx) | 12.7 (n=23.4k ctx) | **P100 wins at depth** |
| Reasoning verbosity | think chars (codegen/debug) | 1760 / 1317 | 1671 / 2118 | comparable |
| VRAM @ 64K | on-GPU / total | **8.82 / 14.16 GB** (partial CPU offload) | ~12.6 GB (100% GPU) | — |

## Reading the numbers

- **Shallow/typical depth is where `.73` wins decisively.** All three real tasks (tool-call,
  codegen, debug) run at shallow context, and `.73` generates at **27–33 tok/s vs the P100's
  ~19–23** — roughly the difference between "comfortable" and "sluggish" for an interactive
  agent loop. Prefill is also ~1.5–1.8× faster, so first-token latency improves too.
- **The `.73` box is VRAM-limited, not compute-limited.** At 64K ctx only **8.82 GB of the
  14.16 GB** working set sits in VRAM — the rest (weights tail + deep KV) spills to system
  RAM. It still beats the full-GPU P100, which means the card itself is much faster than the
  P100; with more VRAM (or a smaller `num_ctx`) it would pull further ahead.
- **The one place the P100 wins: very deep context.** At ~43K resident tokens, `.73`
  generation falls to **8.4 tok/s** because the KV cache is partly in CPU RAM (memory-
  bandwidth-bound attention). The P100 holds its whole 64K KV in VRAM and degrades more
  gracefully (12.7 tok/s at 23K). In practice Hermes' steady-state prompt is ~16K, so this
  rarely bites — but long sessions on `.73` will slow down more than on the P100.
- **Quant differs**, so this isn't a pure hardware A/B: `.73` runs MXFP4 (gpt-oss's native,
  near-lossless 4-bit) while the P100 GGUF quant is whatever llama-swap pulled. Both are
  genuine gpt-oss-20b and both produced correct outputs; no quality regression observed.

## Verdict for the Hermes backend

- **Prefer `.73` when it's up.** It's the faster daily driver for gpt-oss-20b at the depths
  Hermes actually runs at. Switch in-session with `/model --provider ollama --model gpt-oss:20b`.
- **Keep the P100 (`valhalla-p100`) as the always-on default.** It's a container on the
  24/7 server; the `.73` box may be a desktop/workstation that isn't always powered. The
  P100 also degrades more gracefully at very deep context.
- **If you want `.73` to be strictly better,** drop its `num_ctx` toward what Hermes needs
  (it hard-requires ≥64K, so you can't go below that for Hermes) **or** put gpt-oss on a
  bigger-VRAM card there — eliminating the CPU spill would lift both prefill and deep-context
  generation.

## Caveats

- Small hand-written suite (4 tasks), not SWE-bench — measures latency/throughput and basic
  correctness, not deep code quality.
- The codegen prompt lost its back-ticked tokens to shell quoting during the run (prompt_n 95
  vs the P100's 113); the model still produced a correct `merge_intervals`, and gen tok/s is
  prompt-content-independent, so the speed comparison stands.
- Deep-prefill rows use different context depths (43.8K on `.73` vs 23.4K on P100), so the
  prefill-tok/s cells aren't directly comparable — read them as "each box's deep-prefill rate
  at that depth," not a head-to-head ratio.
- Raw responses saved on valhalla at `/tmp/ollama-bench/` (one `.json` per task); P100
  baselines at `/tmp/ornith-bench/`.