ginnoirandClaude Sonnet 5
10997e4b16
check_uptime.js gets a fetchWithRetry wrapper (3 attempts, 2s backoff) for transient failures against the status page/heartbeat API. check_uptime_to_obsidian.js is a variant that logs results into the Obsidian vault instead of stdout. Also adds two benchmark writeups (gpt-oss-20b on Ollama, 73-node Ollama fleet). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
85 lines
5.6 KiB
Markdown
85 lines
5.6 KiB
Markdown
# gpt-oss-20b — `.73` Ollama box vs valhalla P100 (2026-06-28)
|
||
|
||
**TL;DR:** The same model (`gpt-oss-20b`) runs **~1.3–1.5× faster on generation** and
|
||
**~1.6–2× faster on prefill** on ginnoir's `192.168.1.73` Ollama box than on valhalla's
|
||
Tesla P100 llama-swap backend — *despite* the `.73` box partially offloading to CPU at 64K
|
||
context. Tool-calls and code outputs are correct on both. For interactive Hermes use the
|
||
`.73` box is the better backend whenever it's powered on; the P100 remains the always-on
|
||
fallback. **Caveat:** at very deep context (~43K tokens) `.73` generation drops to ~8 tok/s
|
||
(KV no longer fully GPU-resident).
|
||
|
||
## Setup
|
||
|
||
- **Endpoint:** `http://192.168.1.73:11434` — Ollama **0.30.11**, model `gpt-oss:20b`,
|
||
quant **MXFP4** (OpenAI's native gpt-oss 4-bit format), reported 20.9B params.
|
||
- **Serving config:** `num_ctx 65536` (matches the P100's 64K and Hermes' hard ≥64K
|
||
requirement), sampling `temperature 0.6 / top_p 0.95 / top_k 20` (identical to the
|
||
2026-06-27 Ornith/P100 run).
|
||
- **Measured via** Ollama's native `/api/chat` (exact `prompt_eval`/`eval` token counts +
|
||
durations). Cold model load at 64K ctx took **~65 s** (one-time).
|
||
- **P100 baseline** = the gpt-oss-20b column from `docs/2026-06-27-ornith-9b-benchmark.md`
|
||
(llama-swap, q8/q8 KV, `--parallel 1`, `--jinja`, 64K).
|
||
- GPU on `.73` **could not be identified** — no SSH (port 22 filtered). Inferred from VRAM
|
||
behavior (below): a fast but VRAM-limited card (or unified-memory host).
|
||
|
||
## Results
|
||
|
||
| Test | Metric | **.73 Ollama (MXFP4)** | **valhalla P100 (GGUF)** | `.73` advantage |
|
||
|---|---|---|---|---|
|
||
| Tool call | valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie (both correct) |
|
||
| Codegen (`merge_intervals`) | correct? | ✅ correct, non-mutating, type-hinted | ✅ correct (mutates input) | `.73` slightly cleaner |
|
||
| Debug (`second_largest`) | correct? | ✅ correct single-pass, handles dup max | ✅ correct | tie |
|
||
| Deep-prefill | answered? | ✅ "quick brown fox" | ✅ (capped probe) | tie |
|
||
| **Gen speed** | tok/s — tool_call | **33.1** | 23.0 | **1.44×** |
|
||
| **Gen speed** | tok/s — codegen | **27.4** | 21.7 | **1.26×** |
|
||
| **Gen speed** | tok/s — debug | **27.6** | 18.9 | **1.46×** |
|
||
| **Prefill** | tok/s — shallow (~100–135 tok) | **240–287** | 133–188 | ~1.5–1.8× |
|
||
| **Prefill** | tok/s — deep | **2542** (n=43.8k) | 155 (n=23.4k) | far faster (diff depths) |
|
||
| Gen @ deep ctx | tok/s | 8.4 (n=43.8k ctx) | 12.7 (n=23.4k ctx) | **P100 wins at depth** |
|
||
| Reasoning verbosity | think chars (codegen/debug) | 1760 / 1317 | 1671 / 2118 | comparable |
|
||
| VRAM @ 64K | on-GPU / total | **8.82 / 14.16 GB** (partial CPU offload) | ~12.6 GB (100% GPU) | — |
|
||
|
||
## Reading the numbers
|
||
|
||
- **Shallow/typical depth is where `.73` wins decisively.** All three real tasks (tool-call,
|
||
codegen, debug) run at shallow context, and `.73` generates at **27–33 tok/s vs the P100's
|
||
~19–23** — roughly the difference between "comfortable" and "sluggish" for an interactive
|
||
agent loop. Prefill is also ~1.5–1.8× faster, so first-token latency improves too.
|
||
- **The `.73` box is VRAM-limited, not compute-limited.** At 64K ctx only **8.82 GB of the
|
||
14.16 GB** working set sits in VRAM — the rest (weights tail + deep KV) spills to system
|
||
RAM. It still beats the full-GPU P100, which means the card itself is much faster than the
|
||
P100; with more VRAM (or a smaller `num_ctx`) it would pull further ahead.
|
||
- **The one place the P100 wins: very deep context.** At ~43K resident tokens, `.73`
|
||
generation falls to **8.4 tok/s** because the KV cache is partly in CPU RAM (memory-
|
||
bandwidth-bound attention). The P100 holds its whole 64K KV in VRAM and degrades more
|
||
gracefully (12.7 tok/s at 23K). In practice Hermes' steady-state prompt is ~16K, so this
|
||
rarely bites — but long sessions on `.73` will slow down more than on the P100.
|
||
- **Quant differs**, so this isn't a pure hardware A/B: `.73` runs MXFP4 (gpt-oss's native,
|
||
near-lossless 4-bit) while the P100 GGUF quant is whatever llama-swap pulled. Both are
|
||
genuine gpt-oss-20b and both produced correct outputs; no quality regression observed.
|
||
|
||
## Verdict for the Hermes backend
|
||
|
||
- **Prefer `.73` when it's up.** It's the faster daily driver for gpt-oss-20b at the depths
|
||
Hermes actually runs at. Switch in-session with `/model --provider ollama --model gpt-oss:20b`.
|
||
- **Keep the P100 (`valhalla-p100`) as the always-on default.** It's a container on the
|
||
24/7 server; the `.73` box may be a desktop/workstation that isn't always powered. The
|
||
P100 also degrades more gracefully at very deep context.
|
||
- **If you want `.73` to be strictly better,** drop its `num_ctx` toward what Hermes needs
|
||
(it hard-requires ≥64K, so you can't go below that for Hermes) **or** put gpt-oss on a
|
||
bigger-VRAM card there — eliminating the CPU spill would lift both prefill and deep-context
|
||
generation.
|
||
|
||
## Caveats
|
||
|
||
- Small hand-written suite (4 tasks), not SWE-bench — measures latency/throughput and basic
|
||
correctness, not deep code quality.
|
||
- The codegen prompt lost its back-ticked tokens to shell quoting during the run (prompt_n 95
|
||
vs the P100's 113); the model still produced a correct `merge_intervals`, and gen tok/s is
|
||
prompt-content-independent, so the speed comparison stands.
|
||
- Deep-prefill rows use different context depths (43.8K on `.73` vs 23.4K on P100), so the
|
||
prefill-tok/s cells aren't directly comparable — read them as "each box's deep-prefill rate
|
||
at that depth," not a head-to-head ratio.
|
||
- Raw responses saved on valhalla at `/tmp/ollama-bench/` (one `.json` per task); P100
|
||
baselines at `/tmp/ornith-bench/`.
|