diff --git a/docs/2026-06-27-ornith-9b-benchmark.md b/docs/2026-06-27-ornith-9b-benchmark.md new file mode 100644 index 0000000..66f8c99 --- /dev/null +++ b/docs/2026-06-27-ornith-9b-benchmark.md @@ -0,0 +1,61 @@ +# Ornith-1.0-9B vs gpt-oss-20b — P100 benchmark (2026-06-27) + +**TL;DR:** Ornith-1.0-9B works cleanly as a Hermes backend (tool-calls, ``, +content all correct) and matches gpt-oss-20b on **code quality** — but it is a **dense +9B**, so on the compute-bound P100 it generates **~2–3× slower** than the MoE gpt-oss-20b +(~6.3 vs ~13–23 tok/s). **Keep gpt-oss-20b as the default daily driver.** Ornith earns a +spot in the menu as a coding *specialist* to A/B on real tasks, but doesn't displace it. + +## What was done +- Downloaded `ornith-1.0-9b-Q5_K_M.gguf` (6.47 GB, exact byte match) → `/storage1/labdata/llm/models/`. +- Added `ornith-1.0-9b` to `stacks/llm/llama-swap-config.yaml` (same Pascal macro: q8/q8 KV, + `--parallel 1`, `--flash-attn on`, `--jinja`, 64k ctx; native 256k so no YaRN). Deployed to + `/config/llm/` + restarted llama-swap. **Live in the menu now.** +- Quant choice: **Q5_K_M** (safe overnight, no OOM). Loads using only **7.7 GB VRAM** at 64k — + huge headroom, so Q6_K or Q8_0 (9.5 GB, near-lossless) would also fit easily if you want more quality. + +## Results (identical prompts, temp 0.6 / top_p 0.95 / top_k 20) + +| Test | Metric | **ornith-1.0-9b** | **gpt-oss-20b** | Winner | +|---|---|---|---|---| +| Tool call | emits valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie | +| Codegen (merge_intervals) | correct? | ✅ correct, non-mutating, tests pass | ✅ correct (mutates input list) | ornith (slightly cleaner) | +| Debug (second_largest) | correct? | ✅ correct + edge-case guard | ✅ correct + O(n) alt | tie (gpt-oss more thorough) | +| **Gen speed** | tok/s (codegen) | **6.4** | **21.7** | **gpt-oss 3.4×** | +| **Gen speed** | tok/s (debug) | **6.4** | **18.9** | **gpt-oss 2.9×** | +| Prefill (small) | tok/s | 49–104 | 133–188 | gpt-oss | +| Prefill (deep ~25–30k ctx) | tok/s | 101.9 (n=30295) | 154.9 (n=23422) | gpt-oss 1.5× | +| Reasoning verbosity | think chars (codegen/debug) | 433 / 452 (concise) | 1671 / 2118 (verbose) | — | +| VRAM @ 64k | MiB | **7684** | ~12600 | ornith (more headroom) | + +## Why the speed gap (the important takeaway) +gpt-oss-20b is **MoE with ~3.6B active params/token**; Ornith-9B is **dense (9B active/token)**. +The P100 (GP100, cc 6.0) is compute-bound, so per-token cost scales with *active* params — +2.5× more active params ≈ the ~2.5–3× slower generation we measured. This is architectural, +not a config problem; no amount of tuning closes it. Your memory's "~5 tok/s ceiling" held for +the MoE models; a dense 9B genuinely sits below that line on this GPU. + +## Verdict for your Hermes backend +- **Default stays gpt-oss-20b.** Gen speed is already the UX pain point (~6 vs ~20 tok/s is the + difference between usable and painful for interactive agent loops), and Ornith doesn't win on + quality to justify being 3× slower. +- **Keep Ornith as a menu specialist.** It's MIT, purpose-built for agentic coding, refreshingly + concise (less token waste on overthinking), and tool-calls cleanly. Good for one-shot coding + asks where you'll wait for quality. +- **The real coding upgrade you already have on disk:** `Qwen3-Coder-30B-A3B-Instruct-UD-Q2_K_XL.gguf` + is a **coder-tuned MoE (3B active)** — it would be both *faster* than Ornith (MoE) *and* + code-specialized. If you want a faster coding model than gpt-oss, that's the one to benchmark + next, not Ornith. + +## Caveats +- Benchmarks are a small hand-written suite (4 tasks), not SWE-bench. Ornith's headline 69.4 + SWE-Bench Verified is **self-reported/unverified** by DeepReinforce. +- The `deep_prefill` row measures prefill speed only — both models hit the 256-token cap mid-think + (`finish=length`, empty `content`), which is expected for that probe. +- Raw responses saved on valhalla at `/tmp/ornith-bench/` (one `.json` per model+test). + +## Open decision for you +The repo config change (`stacks/llm/llama-swap-config.yaml`) is **deployed to the host but NOT +committed/pushed** — I don't push without your say-so, and pushing would trigger a Portainer +redeploy of stack 34. Tell me to commit+push if you want the repo (canonical source) to match the +live host, or to revert the menu entry if you'd rather not keep Ornith around. diff --git a/stacks/llm/llama-swap-config.yaml b/stacks/llm/llama-swap-config.yaml index 1b1daec..a2f9b1f 100644 --- a/stacks/llm/llama-swap-config.yaml +++ b/stacks/llm/llama-swap-config.yaml @@ -3,10 +3,10 @@ # only one fits in VRAM at a time, so selecting a different model triggers a # brief reload. Default is chosen by Hermes (model.default = gpt-oss-20b). # -# All four serve 64k context (Hermes' minimum). Args are the validated Pascal +# All serve 64k context (Hermes' minimum). Args are the validated Pascal # config: q8_0 KV (q4_0 V-cache is pathological on GP100), flash-attn on, -# --parallel 1 so one sequence gets the full 64k. gpt-oss/gemma4 are natively -# >=128k so no YaRN/override-kv needed. +# --parallel 1 so one sequence gets the full 64k. gpt-oss/gemma4/ornith are +# natively >=128k so no YaRN/override-kv needed. # # Excluded: qwen3-30b-a3b-2507 (Q3) — OOMs at 64k in 16GB, so it can't meet # Hermes' 64k minimum on this GPU. @@ -27,6 +27,15 @@ models: -m /models/gpt-oss-20b-mxfp4.gguf --alias gpt-oss-20b ${common} + "ornith-1.0-9b": + # DeepReinforce Ornith-1.0, dense ~9B on Qwen 3.5 (Q5_K_M). MIT. Agentic-coding + # tuned: block (-> reasoning_content under --jinja) + Qwen3 XML tool calls. + # Native 256k so no YaRN. Recommended sampling: temp 0.6 / top_p 0.95 / top_k 20. + cmd: > + /app/llama-server --port ${PORT} --host 0.0.0.0 + -m /models/ornith-1.0-9b-Q5_K_M.gguf --alias ornith-1.0-9b + ${common} + "gemma-4-26b-a4b": # gemma4 MoE, 4B active / 26B total (UD-Q3_K_M). Quality-leaning; ~147 tok/s prefill. cmd: >