diff --git a/docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md b/docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md new file mode 100644 index 0000000..2fe67b0 --- /dev/null +++ b/docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md @@ -0,0 +1,84 @@ +# gpt-oss-20b — `.73` Ollama box vs valhalla P100 (2026-06-28) + +**TL;DR:** The same model (`gpt-oss-20b`) runs **~1.3–1.5× faster on generation** and +**~1.6–2× faster on prefill** on ginnoir's `192.168.1.73` Ollama box than on valhalla's +Tesla P100 llama-swap backend — *despite* the `.73` box partially offloading to CPU at 64K +context. Tool-calls and code outputs are correct on both. For interactive Hermes use the +`.73` box is the better backend whenever it's powered on; the P100 remains the always-on +fallback. **Caveat:** at very deep context (~43K tokens) `.73` generation drops to ~8 tok/s +(KV no longer fully GPU-resident). + +## Setup + +- **Endpoint:** `http://192.168.1.73:11434` — Ollama **0.30.11**, model `gpt-oss:20b`, + quant **MXFP4** (OpenAI's native gpt-oss 4-bit format), reported 20.9B params. +- **Serving config:** `num_ctx 65536` (matches the P100's 64K and Hermes' hard ≥64K + requirement), sampling `temperature 0.6 / top_p 0.95 / top_k 20` (identical to the + 2026-06-27 Ornith/P100 run). +- **Measured via** Ollama's native `/api/chat` (exact `prompt_eval`/`eval` token counts + + durations). Cold model load at 64K ctx took **~65 s** (one-time). +- **P100 baseline** = the gpt-oss-20b column from `docs/2026-06-27-ornith-9b-benchmark.md` + (llama-swap, q8/q8 KV, `--parallel 1`, `--jinja`, 64K). +- GPU on `.73` **could not be identified** — no SSH (port 22 filtered). Inferred from VRAM + behavior (below): a fast but VRAM-limited card (or unified-memory host). + +## Results + +| Test | Metric | **.73 Ollama (MXFP4)** | **valhalla P100 (GGUF)** | `.73` advantage | +|---|---|---|---|---| +| Tool call | valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie (both correct) | +| Codegen (`merge_intervals`) | correct? | ✅ correct, non-mutating, type-hinted | ✅ correct (mutates input) | `.73` slightly cleaner | +| Debug (`second_largest`) | correct? | ✅ correct single-pass, handles dup max | ✅ correct | tie | +| Deep-prefill | answered? | ✅ "quick brown fox" | ✅ (capped probe) | tie | +| **Gen speed** | tok/s — tool_call | **33.1** | 23.0 | **1.44×** | +| **Gen speed** | tok/s — codegen | **27.4** | 21.7 | **1.26×** | +| **Gen speed** | tok/s — debug | **27.6** | 18.9 | **1.46×** | +| **Prefill** | tok/s — shallow (~100–135 tok) | **240–287** | 133–188 | ~1.5–1.8× | +| **Prefill** | tok/s — deep | **2542** (n=43.8k) | 155 (n=23.4k) | far faster (diff depths) | +| Gen @ deep ctx | tok/s | 8.4 (n=43.8k ctx) | 12.7 (n=23.4k ctx) | **P100 wins at depth** | +| Reasoning verbosity | think chars (codegen/debug) | 1760 / 1317 | 1671 / 2118 | comparable | +| VRAM @ 64K | on-GPU / total | **8.82 / 14.16 GB** (partial CPU offload) | ~12.6 GB (100% GPU) | — | + +## Reading the numbers + +- **Shallow/typical depth is where `.73` wins decisively.** All three real tasks (tool-call, + codegen, debug) run at shallow context, and `.73` generates at **27–33 tok/s vs the P100's + ~19–23** — roughly the difference between "comfortable" and "sluggish" for an interactive + agent loop. Prefill is also ~1.5–1.8× faster, so first-token latency improves too. +- **The `.73` box is VRAM-limited, not compute-limited.** At 64K ctx only **8.82 GB of the + 14.16 GB** working set sits in VRAM — the rest (weights tail + deep KV) spills to system + RAM. It still beats the full-GPU P100, which means the card itself is much faster than the + P100; with more VRAM (or a smaller `num_ctx`) it would pull further ahead. +- **The one place the P100 wins: very deep context.** At ~43K resident tokens, `.73` + generation falls to **8.4 tok/s** because the KV cache is partly in CPU RAM (memory- + bandwidth-bound attention). The P100 holds its whole 64K KV in VRAM and degrades more + gracefully (12.7 tok/s at 23K). In practice Hermes' steady-state prompt is ~16K, so this + rarely bites — but long sessions on `.73` will slow down more than on the P100. +- **Quant differs**, so this isn't a pure hardware A/B: `.73` runs MXFP4 (gpt-oss's native, + near-lossless 4-bit) while the P100 GGUF quant is whatever llama-swap pulled. Both are + genuine gpt-oss-20b and both produced correct outputs; no quality regression observed. + +## Verdict for the Hermes backend + +- **Prefer `.73` when it's up.** It's the faster daily driver for gpt-oss-20b at the depths + Hermes actually runs at. Switch in-session with `/model --provider ollama --model gpt-oss:20b`. +- **Keep the P100 (`valhalla-p100`) as the always-on default.** It's a container on the + 24/7 server; the `.73` box may be a desktop/workstation that isn't always powered. The + P100 also degrades more gracefully at very deep context. +- **If you want `.73` to be strictly better,** drop its `num_ctx` toward what Hermes needs + (it hard-requires ≥64K, so you can't go below that for Hermes) **or** put gpt-oss on a + bigger-VRAM card there — eliminating the CPU spill would lift both prefill and deep-context + generation. + +## Caveats + +- Small hand-written suite (4 tasks), not SWE-bench — measures latency/throughput and basic + correctness, not deep code quality. +- The codegen prompt lost its back-ticked tokens to shell quoting during the run (prompt_n 95 + vs the P100's 113); the model still produced a correct `merge_intervals`, and gen tok/s is + prompt-content-independent, so the speed comparison stands. +- Deep-prefill rows use different context depths (43.8K on `.73` vs 23.4K on P100), so the + prefill-tok/s cells aren't directly comparable — read them as "each box's deep-prefill rate + at that depth," not a head-to-head ratio. +- Raw responses saved on valhalla at `/tmp/ollama-bench/` (one `.json` per task); P100 + baselines at `/tmp/ornith-bench/`. diff --git a/docs/2026-06-28-ollama-73-fleet-benchmark.md b/docs/2026-06-28-ollama-73-fleet-benchmark.md new file mode 100644 index 0000000..554b9b7 --- /dev/null +++ b/docs/2026-06-28-ollama-73-fleet-benchmark.md @@ -0,0 +1,87 @@ +# `.73` Ollama fleet benchmark — all 9 models (2026-06-28) + +**TL;DR:** Throughput across every model on `192.168.1.73`. Generation speed spans an +**~5× range** — from `gemma4:e4b` at **~93 tok/s** down to the big `qwen3:30b-a3b` at +**~18 tok/s**. **All 9 models emit valid tool-calls.** For an interactive agent backend the +sweet spot is **`gpt-oss:20b` (~29 tok/s)** or **`gemma4:12b` (~48 tok/s)** if 12B quality +suffices; the 30B-class Qwen MoEs are the slowest here (heavy CPU offload at 64K on this +VRAM-limited box). **Caveat:** code-correctness for the heavy *thinking* models is +indeterminate — they used the whole 768-token gen cap reasoning and never emitted code (see +Caveats); re-run with a bigger budget to judge quality. + +## Setup +- Endpoint `http://192.168.1.73:11434`, Ollama 0.30.11. Each model served at **`num_ctx + 65536`** (Hermes' ≥64K requirement), sampling `temp 0.6 / top_p 0.95 / top_k 20`. +- Native `/api/chat` timings. 4 tasks: tool-call, codegen (`merge_intervals`), debug + (`second_largest`), deep-prefill (~16K-token filler). Gen capped: 256 / 768 / 768 / 128. +- Same `.73` box as the gpt-oss head-to-head in + `docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md` (GPU still unidentified — no SSH). + +## Generation speed (tok/s) — the headline + +Average of the three real tasks (tool-call / codegen / debug), sorted fastest first: + +| Model | avg gen t/s | tool | codegen | debug | deep-ctx gen | cold load s | tool-call? | +|---|--:|--:|--:|--:|--:|--:|:--:| +| **gemma4:e4b** | **92.7** | 91.1 | 93.3 | 93.6 | 82.6 | 24.8 | ✅ | +| **gemma4:12b** | **48.2** | 46.8 | 49.0 | 48.8 | 45.7 | 10.6 | ✅ | +| **gpt-oss:20b** | **29.4** | 32.7 | 27.9 | 27.7 | 29.9 | 0.4¹ | ✅ | +| **qwen3.6:35b-a3b** | **27.3** | 29.1 | 26.4 | 26.4 | 28.2 | 35.9 | ✅ | +| **gemma4:26b** | **25.7** | 27.6 | 25.0 | 24.5 | 26.5 | 53.8 | ✅ | +| **glm-4.7-flash** | **21.2** | 24.2 | 19.7 | 19.7 | 21.4 | 34.8 | ✅ | +| **qwen3-vl:30b-a3b** | **19.1** | 22.7 | 16.5 | 18.2 | 20.6 | 29.8 | ✅ | +| **qwen3-coder:30b** | **19.0** | 22.8 | 17.1 | 17.2 | 20.4 | 25.3 | ✅ | +| **qwen3:30b-a3b** | **17.6** | 19.5 | 16.6 | 16.6 | 18.5 | 25.0 | ✅ | + +¹ gpt-oss was already resident from the prior run; real cold load is ~65 s. + +## Prefill speed (tok/s) + +| Model | shallow (~100 tok) | deep (~16K tok) | +|---|--:|--:| +| gemma4:e4b | 1193–1799 | 7283 | +| gemma4:12b | 667–1078 | 3318 | +| gpt-oss:20b | 281–379 | 2655 | +| qwen3:30b-a3b | 63–149 | 1030 | +| qwen3-coder:30b | 76–270 | 979 | +| gemma4:26b | 110–127² | 977 | +| qwen3.6:35b-a3b | 78–209 | 620 | +| glm-4.7-flash | 70–163 | 649 | +| qwen3-vl:30b-a3b | 68–127 | 588 | + +² gemma4:26b's first request after load measured 11.7 t/s (cold-cache artifact); ignore. + +## What stands out +- **The two small gemmas are in a different league.** `gemma4:e4b` (~93 t/s) and + `gemma4:12b` (~48 t/s) are dense but small, so they sit fully on GPU and fly. If a 4B/12B + is smart enough for the job, they're the most responsive options by far. +- **gpt-oss:20b is the best "big-brain, still-fast" pick** (~29 t/s) — MoE ~3.6B active keeps + it quick despite 20B total. `qwen3.6:35b-a3b` nearly matches it (~27 t/s) and may be + stronger; worth A/B-ing on real tasks. +- **The 30B-a3b Qwen trio is the slowest** (~17–19 t/s). Same "3B-active" MoE label, but + larger total weights → more spills to CPU RAM at 64K on this VRAM-limited box, dragging + generation below gpt-oss. `qwen3-coder` being this slow undercuts it as a *fast* coding + model here. +- **Every model tool-calls.** All 9 emitted a valid `get_weather({"city":"Tokyo"})`, so any + of them can drive Hermes' tool loop. + +## Correctness (partial — see caveat) +- **Confirmed correct** code on the non-/light-thinking models that finished within the cap: + `gpt-oss:20b`, `gemma4:12b`, `qwen3-vl:30b-a3b`, `qwen3-coder:30b` (clean `def`, + `reason=stop` or code present), plus `gemma4:26b` & `qwen3:30b-a3b` on the task each + finished. +- **Indeterminate** (truncated mid-reasoning, `content=0`, `reason=length`): `glm-4.7-flash` + (both), `gemma4:e4b` (both), `qwen3.6:35b-a3b` (both), `gemma4:26b` (codegen), + `qwen3:30b-a3b` (debug). These spent all 768 gen tokens in the `thinking` channel — **not + wrong, just unfinished.** A re-run at `num_predict ~3072` is needed to grade their output. + +## Caveats +- The 768-token gen cap was too low for heavy chain-of-thought models — it bounds runtime but + truncates their answers. Speed (tok/s) is unaffected and valid; code *quality* for the + truncated set is not measured here. +- Per-model VRAM split not captured (models unload after 2 min `keep_alive`); only + gpt-oss-20b is known (8.82 GB on-GPU / 14.16 GB total at 64K → partial CPU offload). The + slow 30B-class numbers are consistent with heavier offload. +- Small hand-written suite, not SWE-bench. Quants are each model's Ollama default. +- Raw per-task responses on valhalla at `/tmp/ollama-bench-all/` (`summary.json` + one JSON + per model/task); progress log `/tmp/ollama-bench-all/progress.txt`. diff --git a/scripts/check_uptime.js b/scripts/check_uptime.js index cbafc46..72b4e10 100644 --- a/scripts/check_uptime.js +++ b/scripts/check_uptime.js @@ -1,10 +1,27 @@ const fs = require('fs'); const path = require('path'); +async function fetchWithRetry(url, options = {}, retries = 3, backoff = 2000) { + for (let i = 0; i < retries; i++) { + try { + const response = await fetch(url, options); + if (response.ok) return response; + if (response.status >= 500) { + throw new Error(`HTTP ${response.status}: ${response.statusText}`); + } + return response; + } catch (err) { + if (i === retries - 1) throw err; + console.warn(`Fetch to ${url} failed (attempt ${i + 1}/${retries}): ${err.message}. Retrying in ${backoff}ms...`); + await new Promise(resolve => setTimeout(resolve, backoff)); + } + } +} + async function checkStatus() { try { // 1. Fetch status page HTML - const htmlResponse = await fetch('https://uptime.ginnoir.com/status/default'); + const htmlResponse = await fetchWithRetry('https://uptime.ginnoir.com/status/default'); if (!htmlResponse.ok) { throw new Error(`Failed to fetch status page: ${htmlResponse.statusText}`); } @@ -34,7 +51,7 @@ async function checkStatus() { } // 2. Fetch heartbeat JSON - const heartbeatResponse = await fetch('https://uptime.ginnoir.com/api/status-page/heartbeat/default'); + const heartbeatResponse = await fetchWithRetry('https://uptime.ginnoir.com/api/status-page/heartbeat/default'); if (!heartbeatResponse.ok) { throw new Error(`Failed to fetch heartbeat: ${heartbeatResponse.statusText}`); } diff --git a/scripts/check_uptime_to_obsidian.js b/scripts/check_uptime_to_obsidian.js new file mode 100644 index 0000000..eb731de --- /dev/null +++ b/scripts/check_uptime_to_obsidian.js @@ -0,0 +1,179 @@ +const fs = require('fs'); +const path = require('path'); + +async function fetchWithRetry(url, options = {}, retries = 3, backoff = 2000) { + for (let i = 0; i < retries; i++) { + try { + const response = await fetch(url, options); + if (response.ok) return response; + if (response.status >= 500) { + throw new Error(`HTTP ${response.status}: ${response.statusText}`); + } + return response; + } catch (err) { + if (i === retries - 1) throw err; + console.warn(`Fetch to ${url} failed (attempt ${i + 1}/${retries}): ${err.message}. Retrying in ${backoff}ms...`); + await new Promise(resolve => setTimeout(resolve, backoff)); + } + } +} + +async function checkStatus() { + try { + // 1. Fetch status page HTML + const htmlResponse = await fetchWithRetry('https://uptime.ginnoir.com/status/default'); + if (!htmlResponse.ok) { + throw new Error(`Failed to fetch status page: ${htmlResponse.statusText}`); + } + const html = await htmlResponse.text(); + + // Extract window.preloadData = ...; + const preloadRegex = /window\.preloadData\s*=\s*({.*?});/s; + const match = html.match(preloadRegex); + if (!match) { + throw new Error("Could not find window.preloadData in HTML"); + } + + let preloadData; + try { + preloadData = new Function(`return ${match[1]};`)(); + } catch (e) { + throw new Error(`Failed to parse preloadData: ${e.message}`); + } + + const monitors = []; + if (preloadData && preloadData.publicGroupList) { + for (const group of preloadData.publicGroupList) { + if (group.monitorList) { + monitors.push(...group.monitorList); + } + } + } + + // 2. Fetch heartbeat JSON + const heartbeatResponse = await fetchWithRetry('https://uptime.ginnoir.com/api/status-page/heartbeat/default'); + if (!heartbeatResponse.ok) { + throw new Error(`Failed to fetch heartbeat: ${heartbeatResponse.statusText}`); + } + const heartbeats = await heartbeatResponse.json(); + + // 3. Map status and print summary + const monitorMap = {}; + for (const monitor of monitors) { + monitorMap[monitor.id] = { + name: monitor.name, + type: monitor.type, + status: 'UNKNOWN', + ping: null, + lastCheck: null, + msg: '' + }; + } + + const heartbeatList = heartbeats.heartbeatList || {}; + for (const id in heartbeatList) { + const list = heartbeatList[id]; + if (list && list.length > 0) { + const latest = list[list.length - 1]; + if (monitorMap[id]) { + monitorMap[id].status = latest.status === 1 ? 'UP' : 'DOWN'; + monitorMap[id].ping = latest.ping; + monitorMap[id].lastCheck = latest.time; + monitorMap[id].msg = latest.msg || ''; + } + } + } + + // Generate summary + const monitorValues = Object.values(monitorMap); + const total = monitorValues.length; + const up = monitorValues.filter(m => m.status === 'UP').length; + const down = monitorValues.filter(m => m.status === 'DOWN').length; + const unknown = monitorValues.filter(m => m.status === 'UNKNOWN').length; + + let summaryMd = `### Uptime Status Summary (Checked at ${new Date().toLocaleString()})\n\n`; + + if (down > 0) { + summaryMd += `⚠️ **Status: Degraded (${down}/${total} services DOWN)**\n\n`; + } else if (up === total) { + summaryMd += `✅ **Status: Healthy (All ${total} services UP)**\n\n`; + } else { + summaryMd += `ℹ️ **Status: Mixed (UP: ${up}, DOWN: ${down}, UNKNOWN: ${unknown})**\n\n`; + } + + if (down > 0) { + summaryMd += `#### 🚨 DOWN Services:\n`; + monitorValues.filter(m => m.status === 'DOWN').forEach(m => { + summaryMd += `- **${m.name}** (${m.type}) - ${m.msg || 'No message'} (Last checked: ${m.lastCheck})\n`; + }); + summaryMd += `\n`; + } + + summaryMd += `#### 📋 Service Statuses:\n`; + summaryMd += `| Service | Status | Latency (ms) | Last Check |\n`; + summaryMd += `| :--- | :---: | :---: | :--- |\n`; + + // Sort services by name + monitorValues.sort((a, b) => a.name.localeCompare(b.name)); + + for (const m of monitorValues) { + const statusIcon = m.status === 'UP' ? '🟢 UP' : m.status === 'DOWN' ? '🔴 DOWN' : '⚪ UNKNOWN'; + const pingText = m.ping !== null ? `${m.ping} ms` : 'N/A'; + summaryMd += `| ${m.name} | ${statusIcon} | ${pingText} | ${m.lastCheck || 'N/A'} |\n`; + } + + // Output to stdout + console.log(summaryMd); + + // 1. Write the latest report to Uptime Status Report.md + const reportPath = 'c:/Users/MattC/Documents/Obsidian Vault/Homelab/Uptime Status Report.md'; + const reportContent = `--- +project: Homelab +type: status +status: current +tags: [monitoring, uptime] +updated: ${new Date().toISOString().split('T')[0]} +--- + +# Live Uptime Status Report + +${summaryMd} +`; + fs.writeFileSync(reportPath, reportContent, 'utf8'); + console.log(`Wrote status report to ${reportPath}`); + + // 2. Append to today's session log + const todayStr = new Date().toISOString().split('T')[0]; + const sessionLogDir = 'c:/Users/MattC/Documents/Obsidian Vault/Homelab/Sessions'; + if (!fs.existsSync(sessionLogDir)) { + fs.mkdirSync(sessionLogDir, { recursive: true }); + } + const sessionLogPath = path.join(sessionLogDir, `${todayStr}-uptime-status-monitoring-schedule.md`); + + let sessionContent = ''; + if (!fs.existsSync(sessionLogPath)) { + sessionContent = `# Session Log — ${todayStr} — Uptime Status Monitoring Schedule + +## What was done +- Automatically checked uptime status page and updated reports. + +## Daily Log of Checks +`; + } else { + sessionContent = fs.readFileSync(sessionLogPath, 'utf8'); + } + + const timeStr = new Date().toLocaleTimeString(); + const statusText = down > 0 ? `🚨 DEGRADED (${down}/${total} services DOWN)` : `✅ Healthy (${total}/${total} services UP)`; + const logEntry = `\n### Check at ${timeStr}\n- **Status**: ${statusText}\n`; + + fs.writeFileSync(sessionLogPath, sessionContent + logEntry, 'utf8'); + console.log(`Appended check entry to ${sessionLogPath}`); + + } catch (error) { + console.error(`Error checking status: ${error.message}`); + process.exit(1); + } +} + +checkStatus();