homelabstack/docs/2026-06-27-ornith-9b-benchmark.md

Ornith-1.0-9B vs gpt-oss-20b — P100 benchmark (2026-06-27)

TL;DR: Ornith-1.0-9B works cleanly as a Hermes backend (tool-calls, <think>, content all correct) and matches gpt-oss-20b on code quality — but it is a dense 9B, so on the compute-bound P100 it generates ~2–3× slower than the MoE gpt-oss-20b (~6.3 vs ~13–23 tok/s). Keep gpt-oss-20b as the default daily driver. Ornith earns a spot in the menu as a coding specialist to A/B on real tasks, but doesn't displace it.

What was done

• Downloaded ornith-1.0-9b-Q5_K_M.gguf (6.47 GB, exact byte match) → /storage1/labdata/llm/models/.
• Added ornith-1.0-9b to stacks/llm/llama-swap-config.yaml (same Pascal macro: q8/q8 KV, --parallel 1, --flash-attn on, --jinja, 64k ctx; native 256k so no YaRN). Deployed to /config/llm/ + restarted llama-swap. Live in the menu now.
• Quant choice: Q5_K_M (safe overnight, no OOM). Loads using only 7.7 GB VRAM at 64k — huge headroom, so Q6_K or Q8_0 (9.5 GB, near-lossless) would also fit easily if you want more quality.

Results (identical prompts, temp 0.6 / top_p 0.95 / top_k 20)

Test	Metric	ornith-1.0-9b	gpt-oss-20b	Winner
Tool call	emits valid tool_calls?	✅ get_weather({"city":"Tokyo"})	✅ identical	tie
Codegen (merge_intervals)	correct?	✅ correct, non-mutating, tests pass	✅ correct (mutates input list)	ornith (slightly cleaner)
Debug (second_largest)	correct?	✅ correct + edge-case guard	✅ correct + O(n) alt	tie (gpt-oss more thorough)
Gen speed	tok/s (codegen)	6.4	21.7	gpt-oss 3.4×
Gen speed	tok/s (debug)	6.4	18.9	gpt-oss 2.9×
Prefill (small)	tok/s	49–104	133–188	gpt-oss
Prefill (deep ~25–30k ctx)	tok/s	101.9 (n=30295)	154.9 (n=23422)	gpt-oss 1.5×
Reasoning verbosity	think chars (codegen/debug)	433 / 452 (concise)	1671 / 2118 (verbose)	—
VRAM @ 64k	MiB	7684	~12600	ornith (more headroom)

Why the speed gap (the important takeaway)

gpt-oss-20b is MoE with ~3.6B active params/token; Ornith-9B is dense (9B active/token). The P100 (GP100, cc 6.0) is compute-bound, so per-token cost scales with active params — 2.5× more active params ≈ the ~2.5–3× slower generation we measured. This is architectural, not a config problem; no amount of tuning closes it. Your memory's "~5 tok/s ceiling" held for the MoE models; a dense 9B genuinely sits below that line on this GPU.

Verdict for your Hermes backend

• Default stays gpt-oss-20b. Gen speed is already the UX pain point (~6 vs ~20 tok/s is the difference between usable and painful for interactive agent loops), and Ornith doesn't win on quality to justify being 3× slower.
• Keep Ornith as a menu specialist. It's MIT, purpose-built for agentic coding, refreshingly concise (less token waste on overthinking), and tool-calls cleanly. Good for one-shot coding asks where you'll wait for quality.
• The real coding upgrade you already have on disk: Qwen3-Coder-30B-A3B-Instruct-UD-Q2_K_XL.gguf is a coder-tuned MoE (3B active) — it would be both faster than Ornith (MoE) and code-specialized. If you want a faster coding model than gpt-oss, that's the one to benchmark next, not Ornith.

Caveats

• Benchmarks are a small hand-written suite (4 tasks), not SWE-bench. Ornith's headline 69.4 SWE-Bench Verified is self-reported/unverified by DeepReinforce.
• The deep_prefill row measures prefill speed only — both models hit the 256-token cap mid-think (finish=length, empty content), which is expected for that probe.
• Raw responses saved on valhalla at /tmp/ornith-bench/ (one .json per model+test).

Open decision for you

The repo config change (stacks/llm/llama-swap-config.yaml) is deployed to the host but NOT committed/pushed — I don't push without your say-so, and pushing would trigger a Portainer redeploy of stack 34. Tell me to commit+push if you want the repo (canonical source) to match the live host, or to revert the menu entry if you'd rather not keep Ornith around.