feat(llm): add ornith-1.0-9b coding model to llama-swap menu

DeepReinforce Ornith-1.0 (dense 9B on Qwen 3.5, Q5_K_M, MIT), an
agentic-coding model. Tool-calls + <think> work under --jinja; native
256k so no YaRN. Loads at ~7.7GB VRAM @ 64k.

Benchmark (docs/2026-06-27-ornith-9b-benchmark.md): quality ties
gpt-oss-20b but gen is ~2.5-3x slower (dense 9B active vs gpt-oss MoE
3.6B active on the compute-bound P100). Default stays gpt-oss-20b;
ornith kept as a coding specialist in the menu.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ginnoir
2026-06-27 13:51:19 -05:00
co-authored by Claude Opus 4.8
parent dc2225d384
commit 009a474e90
2 changed files with 73 additions and 3 deletions
+61
View File
@@ -0,0 +1,61 @@
# Ornith-1.0-9B vs gpt-oss-20b — P100 benchmark (2026-06-27)
**TL;DR:** Ornith-1.0-9B works cleanly as a Hermes backend (tool-calls, `<think>`,
content all correct) and matches gpt-oss-20b on **code quality** — but it is a **dense
9B**, so on the compute-bound P100 it generates **~23× slower** than the MoE gpt-oss-20b
(~6.3 vs ~1323 tok/s). **Keep gpt-oss-20b as the default daily driver.** Ornith earns a
spot in the menu as a coding *specialist* to A/B on real tasks, but doesn't displace it.
## What was done
- Downloaded `ornith-1.0-9b-Q5_K_M.gguf` (6.47 GB, exact byte match) → `/storage1/labdata/llm/models/`.
- Added `ornith-1.0-9b` to `stacks/llm/llama-swap-config.yaml` (same Pascal macro: q8/q8 KV,
`--parallel 1`, `--flash-attn on`, `--jinja`, 64k ctx; native 256k so no YaRN). Deployed to
`/config/llm/` + restarted llama-swap. **Live in the menu now.**
- Quant choice: **Q5_K_M** (safe overnight, no OOM). Loads using only **7.7 GB VRAM** at 64k —
huge headroom, so Q6_K or Q8_0 (9.5 GB, near-lossless) would also fit easily if you want more quality.
## Results (identical prompts, temp 0.6 / top_p 0.95 / top_k 20)
| Test | Metric | **ornith-1.0-9b** | **gpt-oss-20b** | Winner |
|---|---|---|---|---|
| Tool call | emits valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie |
| Codegen (merge_intervals) | correct? | ✅ correct, non-mutating, tests pass | ✅ correct (mutates input list) | ornith (slightly cleaner) |
| Debug (second_largest) | correct? | ✅ correct + edge-case guard | ✅ correct + O(n) alt | tie (gpt-oss more thorough) |
| **Gen speed** | tok/s (codegen) | **6.4** | **21.7** | **gpt-oss 3.4×** |
| **Gen speed** | tok/s (debug) | **6.4** | **18.9** | **gpt-oss 2.9×** |
| Prefill (small) | tok/s | 49104 | 133188 | gpt-oss |
| Prefill (deep ~2530k ctx) | tok/s | 101.9 (n=30295) | 154.9 (n=23422) | gpt-oss 1.5× |
| Reasoning verbosity | think chars (codegen/debug) | 433 / 452 (concise) | 1671 / 2118 (verbose) | — |
| VRAM @ 64k | MiB | **7684** | ~12600 | ornith (more headroom) |
## Why the speed gap (the important takeaway)
gpt-oss-20b is **MoE with ~3.6B active params/token**; Ornith-9B is **dense (9B active/token)**.
The P100 (GP100, cc 6.0) is compute-bound, so per-token cost scales with *active* params —
2.5× more active params ≈ the ~2.53× slower generation we measured. This is architectural,
not a config problem; no amount of tuning closes it. Your memory's "~5 tok/s ceiling" held for
the MoE models; a dense 9B genuinely sits below that line on this GPU.
## Verdict for your Hermes backend
- **Default stays gpt-oss-20b.** Gen speed is already the UX pain point (~6 vs ~20 tok/s is the
difference between usable and painful for interactive agent loops), and Ornith doesn't win on
quality to justify being 3× slower.
- **Keep Ornith as a menu specialist.** It's MIT, purpose-built for agentic coding, refreshingly
concise (less token waste on overthinking), and tool-calls cleanly. Good for one-shot coding
asks where you'll wait for quality.
- **The real coding upgrade you already have on disk:** `Qwen3-Coder-30B-A3B-Instruct-UD-Q2_K_XL.gguf`
is a **coder-tuned MoE (3B active)** — it would be both *faster* than Ornith (MoE) *and*
code-specialized. If you want a faster coding model than gpt-oss, that's the one to benchmark
next, not Ornith.
## Caveats
- Benchmarks are a small hand-written suite (4 tasks), not SWE-bench. Ornith's headline 69.4
SWE-Bench Verified is **self-reported/unverified** by DeepReinforce.
- The `deep_prefill` row measures prefill speed only — both models hit the 256-token cap mid-think
(`finish=length`, empty `content`), which is expected for that probe.
- Raw responses saved on valhalla at `/tmp/ornith-bench/` (one `.json` per model+test).
## Open decision for you
The repo config change (`stacks/llm/llama-swap-config.yaml`) is **deployed to the host but NOT
committed/pushed** — I don't push without your say-so, and pushing would trigger a Portainer
redeploy of stack 34. Tell me to commit+push if you want the repo (canonical source) to match the
live host, or to revert the menu entry if you'd rather not keep Ornith around.