feat(scripts): add uptime-check retry + Obsidian variant, add LLM benchmark docs

check_uptime.js gets a fetchWithRetry wrapper (3 attempts, 2s backoff)
for transient failures against the status page/heartbeat API.
check_uptime_to_obsidian.js is a variant that logs results into the
Obsidian vault instead of stdout. Also adds two benchmark writeups
(gpt-oss-20b on Ollama, 73-node Ollama fleet).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ginnoir
2026-07-01 02:17:53 -05:00
co-authored by Claude Sonnet 5
parent 1792dd964b
commit 10997e4b16
4 changed files with 369 additions and 2 deletions
@@ -0,0 +1,84 @@
# gpt-oss-20b — `.73` Ollama box vs valhalla P100 (2026-06-28)
**TL;DR:** The same model (`gpt-oss-20b`) runs **~1.31.5× faster on generation** and
**~1.62× faster on prefill** on ginnoir's `192.168.1.73` Ollama box than on valhalla's
Tesla P100 llama-swap backend — *despite* the `.73` box partially offloading to CPU at 64K
context. Tool-calls and code outputs are correct on both. For interactive Hermes use the
`.73` box is the better backend whenever it's powered on; the P100 remains the always-on
fallback. **Caveat:** at very deep context (~43K tokens) `.73` generation drops to ~8 tok/s
(KV no longer fully GPU-resident).
## Setup
- **Endpoint:** `http://192.168.1.73:11434` — Ollama **0.30.11**, model `gpt-oss:20b`,
quant **MXFP4** (OpenAI's native gpt-oss 4-bit format), reported 20.9B params.
- **Serving config:** `num_ctx 65536` (matches the P100's 64K and Hermes' hard ≥64K
requirement), sampling `temperature 0.6 / top_p 0.95 / top_k 20` (identical to the
2026-06-27 Ornith/P100 run).
- **Measured via** Ollama's native `/api/chat` (exact `prompt_eval`/`eval` token counts +
durations). Cold model load at 64K ctx took **~65 s** (one-time).
- **P100 baseline** = the gpt-oss-20b column from `docs/2026-06-27-ornith-9b-benchmark.md`
(llama-swap, q8/q8 KV, `--parallel 1`, `--jinja`, 64K).
- GPU on `.73` **could not be identified** — no SSH (port 22 filtered). Inferred from VRAM
behavior (below): a fast but VRAM-limited card (or unified-memory host).
## Results
| Test | Metric | **.73 Ollama (MXFP4)** | **valhalla P100 (GGUF)** | `.73` advantage |
|---|---|---|---|---|
| Tool call | valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie (both correct) |
| Codegen (`merge_intervals`) | correct? | ✅ correct, non-mutating, type-hinted | ✅ correct (mutates input) | `.73` slightly cleaner |
| Debug (`second_largest`) | correct? | ✅ correct single-pass, handles dup max | ✅ correct | tie |
| Deep-prefill | answered? | ✅ "quick brown fox" | ✅ (capped probe) | tie |
| **Gen speed** | tok/s — tool_call | **33.1** | 23.0 | **1.44×** |
| **Gen speed** | tok/s — codegen | **27.4** | 21.7 | **1.26×** |
| **Gen speed** | tok/s — debug | **27.6** | 18.9 | **1.46×** |
| **Prefill** | tok/s — shallow (~100135 tok) | **240287** | 133188 | ~1.51.8× |
| **Prefill** | tok/s — deep | **2542** (n=43.8k) | 155 (n=23.4k) | far faster (diff depths) |
| Gen @ deep ctx | tok/s | 8.4 (n=43.8k ctx) | 12.7 (n=23.4k ctx) | **P100 wins at depth** |
| Reasoning verbosity | think chars (codegen/debug) | 1760 / 1317 | 1671 / 2118 | comparable |
| VRAM @ 64K | on-GPU / total | **8.82 / 14.16 GB** (partial CPU offload) | ~12.6 GB (100% GPU) | — |
## Reading the numbers
- **Shallow/typical depth is where `.73` wins decisively.** All three real tasks (tool-call,
codegen, debug) run at shallow context, and `.73` generates at **2733 tok/s vs the P100's
~1923** — roughly the difference between "comfortable" and "sluggish" for an interactive
agent loop. Prefill is also ~1.51.8× faster, so first-token latency improves too.
- **The `.73` box is VRAM-limited, not compute-limited.** At 64K ctx only **8.82 GB of the
14.16 GB** working set sits in VRAM — the rest (weights tail + deep KV) spills to system
RAM. It still beats the full-GPU P100, which means the card itself is much faster than the
P100; with more VRAM (or a smaller `num_ctx`) it would pull further ahead.
- **The one place the P100 wins: very deep context.** At ~43K resident tokens, `.73`
generation falls to **8.4 tok/s** because the KV cache is partly in CPU RAM (memory-
bandwidth-bound attention). The P100 holds its whole 64K KV in VRAM and degrades more
gracefully (12.7 tok/s at 23K). In practice Hermes' steady-state prompt is ~16K, so this
rarely bites — but long sessions on `.73` will slow down more than on the P100.
- **Quant differs**, so this isn't a pure hardware A/B: `.73` runs MXFP4 (gpt-oss's native,
near-lossless 4-bit) while the P100 GGUF quant is whatever llama-swap pulled. Both are
genuine gpt-oss-20b and both produced correct outputs; no quality regression observed.
## Verdict for the Hermes backend
- **Prefer `.73` when it's up.** It's the faster daily driver for gpt-oss-20b at the depths
Hermes actually runs at. Switch in-session with `/model --provider ollama --model gpt-oss:20b`.
- **Keep the P100 (`valhalla-p100`) as the always-on default.** It's a container on the
24/7 server; the `.73` box may be a desktop/workstation that isn't always powered. The
P100 also degrades more gracefully at very deep context.
- **If you want `.73` to be strictly better,** drop its `num_ctx` toward what Hermes needs
(it hard-requires ≥64K, so you can't go below that for Hermes) **or** put gpt-oss on a
bigger-VRAM card there — eliminating the CPU spill would lift both prefill and deep-context
generation.
## Caveats
- Small hand-written suite (4 tasks), not SWE-bench — measures latency/throughput and basic
correctness, not deep code quality.
- The codegen prompt lost its back-ticked tokens to shell quoting during the run (prompt_n 95
vs the P100's 113); the model still produced a correct `merge_intervals`, and gen tok/s is
prompt-content-independent, so the speed comparison stands.
- Deep-prefill rows use different context depths (43.8K on `.73` vs 23.4K on P100), so the
prefill-tok/s cells aren't directly comparable — read them as "each box's deep-prefill rate
at that depth," not a head-to-head ratio.
- Raw responses saved on valhalla at `/tmp/ollama-bench/` (one `.json` per task); P100
baselines at `/tmp/ornith-bench/`.