ginnoirandClaude Sonnet 5
10997e4b16
check_uptime.js gets a fetchWithRetry wrapper (3 attempts, 2s backoff) for transient failures against the status page/heartbeat API. check_uptime_to_obsidian.js is a variant that logs results into the Obsidian vault instead of stdout. Also adds two benchmark writeups (gpt-oss-20b on Ollama, 73-node Ollama fleet). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
88 lines
4.9 KiB
Markdown
88 lines
4.9 KiB
Markdown
# `.73` Ollama fleet benchmark — all 9 models (2026-06-28)
|
||
|
||
**TL;DR:** Throughput across every model on `192.168.1.73`. Generation speed spans an
|
||
**~5× range** — from `gemma4:e4b` at **~93 tok/s** down to the big `qwen3:30b-a3b` at
|
||
**~18 tok/s**. **All 9 models emit valid tool-calls.** For an interactive agent backend the
|
||
sweet spot is **`gpt-oss:20b` (~29 tok/s)** or **`gemma4:12b` (~48 tok/s)** if 12B quality
|
||
suffices; the 30B-class Qwen MoEs are the slowest here (heavy CPU offload at 64K on this
|
||
VRAM-limited box). **Caveat:** code-correctness for the heavy *thinking* models is
|
||
indeterminate — they used the whole 768-token gen cap reasoning and never emitted code (see
|
||
Caveats); re-run with a bigger budget to judge quality.
|
||
|
||
## Setup
|
||
- Endpoint `http://192.168.1.73:11434`, Ollama 0.30.11. Each model served at **`num_ctx
|
||
65536`** (Hermes' ≥64K requirement), sampling `temp 0.6 / top_p 0.95 / top_k 20`.
|
||
- Native `/api/chat` timings. 4 tasks: tool-call, codegen (`merge_intervals`), debug
|
||
(`second_largest`), deep-prefill (~16K-token filler). Gen capped: 256 / 768 / 768 / 128.
|
||
- Same `.73` box as the gpt-oss head-to-head in
|
||
`docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md` (GPU still unidentified — no SSH).
|
||
|
||
## Generation speed (tok/s) — the headline
|
||
|
||
Average of the three real tasks (tool-call / codegen / debug), sorted fastest first:
|
||
|
||
| Model | avg gen t/s | tool | codegen | debug | deep-ctx gen | cold load s | tool-call? |
|
||
|---|--:|--:|--:|--:|--:|--:|:--:|
|
||
| **gemma4:e4b** | **92.7** | 91.1 | 93.3 | 93.6 | 82.6 | 24.8 | ✅ |
|
||
| **gemma4:12b** | **48.2** | 46.8 | 49.0 | 48.8 | 45.7 | 10.6 | ✅ |
|
||
| **gpt-oss:20b** | **29.4** | 32.7 | 27.9 | 27.7 | 29.9 | 0.4¹ | ✅ |
|
||
| **qwen3.6:35b-a3b** | **27.3** | 29.1 | 26.4 | 26.4 | 28.2 | 35.9 | ✅ |
|
||
| **gemma4:26b** | **25.7** | 27.6 | 25.0 | 24.5 | 26.5 | 53.8 | ✅ |
|
||
| **glm-4.7-flash** | **21.2** | 24.2 | 19.7 | 19.7 | 21.4 | 34.8 | ✅ |
|
||
| **qwen3-vl:30b-a3b** | **19.1** | 22.7 | 16.5 | 18.2 | 20.6 | 29.8 | ✅ |
|
||
| **qwen3-coder:30b** | **19.0** | 22.8 | 17.1 | 17.2 | 20.4 | 25.3 | ✅ |
|
||
| **qwen3:30b-a3b** | **17.6** | 19.5 | 16.6 | 16.6 | 18.5 | 25.0 | ✅ |
|
||
|
||
¹ gpt-oss was already resident from the prior run; real cold load is ~65 s.
|
||
|
||
## Prefill speed (tok/s)
|
||
|
||
| Model | shallow (~100 tok) | deep (~16K tok) |
|
||
|---|--:|--:|
|
||
| gemma4:e4b | 1193–1799 | 7283 |
|
||
| gemma4:12b | 667–1078 | 3318 |
|
||
| gpt-oss:20b | 281–379 | 2655 |
|
||
| qwen3:30b-a3b | 63–149 | 1030 |
|
||
| qwen3-coder:30b | 76–270 | 979 |
|
||
| gemma4:26b | 110–127² | 977 |
|
||
| qwen3.6:35b-a3b | 78–209 | 620 |
|
||
| glm-4.7-flash | 70–163 | 649 |
|
||
| qwen3-vl:30b-a3b | 68–127 | 588 |
|
||
|
||
² gemma4:26b's first request after load measured 11.7 t/s (cold-cache artifact); ignore.
|
||
|
||
## What stands out
|
||
- **The two small gemmas are in a different league.** `gemma4:e4b` (~93 t/s) and
|
||
`gemma4:12b` (~48 t/s) are dense but small, so they sit fully on GPU and fly. If a 4B/12B
|
||
is smart enough for the job, they're the most responsive options by far.
|
||
- **gpt-oss:20b is the best "big-brain, still-fast" pick** (~29 t/s) — MoE ~3.6B active keeps
|
||
it quick despite 20B total. `qwen3.6:35b-a3b` nearly matches it (~27 t/s) and may be
|
||
stronger; worth A/B-ing on real tasks.
|
||
- **The 30B-a3b Qwen trio is the slowest** (~17–19 t/s). Same "3B-active" MoE label, but
|
||
larger total weights → more spills to CPU RAM at 64K on this VRAM-limited box, dragging
|
||
generation below gpt-oss. `qwen3-coder` being this slow undercuts it as a *fast* coding
|
||
model here.
|
||
- **Every model tool-calls.** All 9 emitted a valid `get_weather({"city":"Tokyo"})`, so any
|
||
of them can drive Hermes' tool loop.
|
||
|
||
## Correctness (partial — see caveat)
|
||
- **Confirmed correct** code on the non-/light-thinking models that finished within the cap:
|
||
`gpt-oss:20b`, `gemma4:12b`, `qwen3-vl:30b-a3b`, `qwen3-coder:30b` (clean `def`,
|
||
`reason=stop` or code present), plus `gemma4:26b` & `qwen3:30b-a3b` on the task each
|
||
finished.
|
||
- **Indeterminate** (truncated mid-reasoning, `content=0`, `reason=length`): `glm-4.7-flash`
|
||
(both), `gemma4:e4b` (both), `qwen3.6:35b-a3b` (both), `gemma4:26b` (codegen),
|
||
`qwen3:30b-a3b` (debug). These spent all 768 gen tokens in the `thinking` channel — **not
|
||
wrong, just unfinished.** A re-run at `num_predict ~3072` is needed to grade their output.
|
||
|
||
## Caveats
|
||
- The 768-token gen cap was too low for heavy chain-of-thought models — it bounds runtime but
|
||
truncates their answers. Speed (tok/s) is unaffected and valid; code *quality* for the
|
||
truncated set is not measured here.
|
||
- Per-model VRAM split not captured (models unload after 2 min `keep_alive`); only
|
||
gpt-oss-20b is known (8.82 GB on-GPU / 14.16 GB total at 64K → partial CPU offload). The
|
||
slow 30B-class numbers are consistent with heavier offload.
|
||
- Small hand-written suite, not SWE-bench. Quants are each model's Ollama default.
|
||
- Raw per-task responses on valhalla at `/tmp/ollama-bench-all/` (`summary.json` + one JSON
|
||
per model/task); progress log `/tmp/ollama-bench-all/progress.txt`.
|