feat(scripts): add uptime-check retry + Obsidian variant, add LLM benchmark docs
check_uptime.js gets a fetchWithRetry wrapper (3 attempts, 2s backoff) for transient failures against the status page/heartbeat API. check_uptime_to_obsidian.js is a variant that logs results into the Obsidian vault instead of stdout. Also adds two benchmark writeups (gpt-oss-20b on Ollama, 73-node Ollama fleet). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
1792dd964b
commit
10997e4b16
@@ -0,0 +1,84 @@
|
||||
# gpt-oss-20b — `.73` Ollama box vs valhalla P100 (2026-06-28)
|
||||
|
||||
**TL;DR:** The same model (`gpt-oss-20b`) runs **~1.3–1.5× faster on generation** and
|
||||
**~1.6–2× faster on prefill** on ginnoir's `192.168.1.73` Ollama box than on valhalla's
|
||||
Tesla P100 llama-swap backend — *despite* the `.73` box partially offloading to CPU at 64K
|
||||
context. Tool-calls and code outputs are correct on both. For interactive Hermes use the
|
||||
`.73` box is the better backend whenever it's powered on; the P100 remains the always-on
|
||||
fallback. **Caveat:** at very deep context (~43K tokens) `.73` generation drops to ~8 tok/s
|
||||
(KV no longer fully GPU-resident).
|
||||
|
||||
## Setup
|
||||
|
||||
- **Endpoint:** `http://192.168.1.73:11434` — Ollama **0.30.11**, model `gpt-oss:20b`,
|
||||
quant **MXFP4** (OpenAI's native gpt-oss 4-bit format), reported 20.9B params.
|
||||
- **Serving config:** `num_ctx 65536` (matches the P100's 64K and Hermes' hard ≥64K
|
||||
requirement), sampling `temperature 0.6 / top_p 0.95 / top_k 20` (identical to the
|
||||
2026-06-27 Ornith/P100 run).
|
||||
- **Measured via** Ollama's native `/api/chat` (exact `prompt_eval`/`eval` token counts +
|
||||
durations). Cold model load at 64K ctx took **~65 s** (one-time).
|
||||
- **P100 baseline** = the gpt-oss-20b column from `docs/2026-06-27-ornith-9b-benchmark.md`
|
||||
(llama-swap, q8/q8 KV, `--parallel 1`, `--jinja`, 64K).
|
||||
- GPU on `.73` **could not be identified** — no SSH (port 22 filtered). Inferred from VRAM
|
||||
behavior (below): a fast but VRAM-limited card (or unified-memory host).
|
||||
|
||||
## Results
|
||||
|
||||
| Test | Metric | **.73 Ollama (MXFP4)** | **valhalla P100 (GGUF)** | `.73` advantage |
|
||||
|---|---|---|---|---|
|
||||
| Tool call | valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie (both correct) |
|
||||
| Codegen (`merge_intervals`) | correct? | ✅ correct, non-mutating, type-hinted | ✅ correct (mutates input) | `.73` slightly cleaner |
|
||||
| Debug (`second_largest`) | correct? | ✅ correct single-pass, handles dup max | ✅ correct | tie |
|
||||
| Deep-prefill | answered? | ✅ "quick brown fox" | ✅ (capped probe) | tie |
|
||||
| **Gen speed** | tok/s — tool_call | **33.1** | 23.0 | **1.44×** |
|
||||
| **Gen speed** | tok/s — codegen | **27.4** | 21.7 | **1.26×** |
|
||||
| **Gen speed** | tok/s — debug | **27.6** | 18.9 | **1.46×** |
|
||||
| **Prefill** | tok/s — shallow (~100–135 tok) | **240–287** | 133–188 | ~1.5–1.8× |
|
||||
| **Prefill** | tok/s — deep | **2542** (n=43.8k) | 155 (n=23.4k) | far faster (diff depths) |
|
||||
| Gen @ deep ctx | tok/s | 8.4 (n=43.8k ctx) | 12.7 (n=23.4k ctx) | **P100 wins at depth** |
|
||||
| Reasoning verbosity | think chars (codegen/debug) | 1760 / 1317 | 1671 / 2118 | comparable |
|
||||
| VRAM @ 64K | on-GPU / total | **8.82 / 14.16 GB** (partial CPU offload) | ~12.6 GB (100% GPU) | — |
|
||||
|
||||
## Reading the numbers
|
||||
|
||||
- **Shallow/typical depth is where `.73` wins decisively.** All three real tasks (tool-call,
|
||||
codegen, debug) run at shallow context, and `.73` generates at **27–33 tok/s vs the P100's
|
||||
~19–23** — roughly the difference between "comfortable" and "sluggish" for an interactive
|
||||
agent loop. Prefill is also ~1.5–1.8× faster, so first-token latency improves too.
|
||||
- **The `.73` box is VRAM-limited, not compute-limited.** At 64K ctx only **8.82 GB of the
|
||||
14.16 GB** working set sits in VRAM — the rest (weights tail + deep KV) spills to system
|
||||
RAM. It still beats the full-GPU P100, which means the card itself is much faster than the
|
||||
P100; with more VRAM (or a smaller `num_ctx`) it would pull further ahead.
|
||||
- **The one place the P100 wins: very deep context.** At ~43K resident tokens, `.73`
|
||||
generation falls to **8.4 tok/s** because the KV cache is partly in CPU RAM (memory-
|
||||
bandwidth-bound attention). The P100 holds its whole 64K KV in VRAM and degrades more
|
||||
gracefully (12.7 tok/s at 23K). In practice Hermes' steady-state prompt is ~16K, so this
|
||||
rarely bites — but long sessions on `.73` will slow down more than on the P100.
|
||||
- **Quant differs**, so this isn't a pure hardware A/B: `.73` runs MXFP4 (gpt-oss's native,
|
||||
near-lossless 4-bit) while the P100 GGUF quant is whatever llama-swap pulled. Both are
|
||||
genuine gpt-oss-20b and both produced correct outputs; no quality regression observed.
|
||||
|
||||
## Verdict for the Hermes backend
|
||||
|
||||
- **Prefer `.73` when it's up.** It's the faster daily driver for gpt-oss-20b at the depths
|
||||
Hermes actually runs at. Switch in-session with `/model --provider ollama --model gpt-oss:20b`.
|
||||
- **Keep the P100 (`valhalla-p100`) as the always-on default.** It's a container on the
|
||||
24/7 server; the `.73` box may be a desktop/workstation that isn't always powered. The
|
||||
P100 also degrades more gracefully at very deep context.
|
||||
- **If you want `.73` to be strictly better,** drop its `num_ctx` toward what Hermes needs
|
||||
(it hard-requires ≥64K, so you can't go below that for Hermes) **or** put gpt-oss on a
|
||||
bigger-VRAM card there — eliminating the CPU spill would lift both prefill and deep-context
|
||||
generation.
|
||||
|
||||
## Caveats
|
||||
|
||||
- Small hand-written suite (4 tasks), not SWE-bench — measures latency/throughput and basic
|
||||
correctness, not deep code quality.
|
||||
- The codegen prompt lost its back-ticked tokens to shell quoting during the run (prompt_n 95
|
||||
vs the P100's 113); the model still produced a correct `merge_intervals`, and gen tok/s is
|
||||
prompt-content-independent, so the speed comparison stands.
|
||||
- Deep-prefill rows use different context depths (43.8K on `.73` vs 23.4K on P100), so the
|
||||
prefill-tok/s cells aren't directly comparable — read them as "each box's deep-prefill rate
|
||||
at that depth," not a head-to-head ratio.
|
||||
- Raw responses saved on valhalla at `/tmp/ollama-bench/` (one `.json` per task); P100
|
||||
baselines at `/tmp/ornith-bench/`.
|
||||
Reference in New Issue
Block a user