# Ornith-1.0-9B vs gpt-oss-20b — P100 benchmark (2026-06-27)

**TL;DR:** Ornith-1.0-9B works cleanly as a Hermes backend (tool-calls, `<think>`,
content all correct) and matches gpt-oss-20b on **code quality** — but it is a **dense
9B**, so on the compute-bound P100 it generates **~2–3× slower** than the MoE gpt-oss-20b
(~6.3 vs ~13–23 tok/s). **Keep gpt-oss-20b as the default daily driver.** Ornith earns a
spot in the menu as a coding *specialist* to A/B on real tasks, but doesn't displace it.

## What was done
- Downloaded `ornith-1.0-9b-Q5_K_M.gguf` (6.47 GB, exact byte match) → `/storage1/labdata/llm/models/`.
- Added `ornith-1.0-9b` to `stacks/llm/llama-swap-config.yaml` (same Pascal macro: q8/q8 KV,
  `--parallel 1`, `--flash-attn on`, `--jinja`, 64k ctx; native 256k so no YaRN). Deployed to
  `/config/llm/` + restarted llama-swap. **Live in the menu now.**
- Quant choice: **Q5_K_M** (safe overnight, no OOM). Loads using only **7.7 GB VRAM** at 64k —
  huge headroom, so Q6_K or Q8_0 (9.5 GB, near-lossless) would also fit easily if you want more quality.

## Results (identical prompts, temp 0.6 / top_p 0.95 / top_k 20)

| Test | Metric | **ornith-1.0-9b** | **gpt-oss-20b** | Winner |
|---|---|---|---|---|
| Tool call | emits valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie |
| Codegen (merge_intervals) | correct? | ✅ correct, non-mutating, tests pass | ✅ correct (mutates input list) | ornith (slightly cleaner) |
| Debug (second_largest) | correct? | ✅ correct + edge-case guard | ✅ correct + O(n) alt | tie (gpt-oss more thorough) |
| **Gen speed** | tok/s (codegen) | **6.4** | **21.7** | **gpt-oss 3.4×** |
| **Gen speed** | tok/s (debug) | **6.4** | **18.9** | **gpt-oss 2.9×** |
| Prefill (small) | tok/s | 49–104 | 133–188 | gpt-oss |
| Prefill (deep ~25–30k ctx) | tok/s | 101.9 (n=30295) | 154.9 (n=23422) | gpt-oss 1.5× |
| Reasoning verbosity | think chars (codegen/debug) | 433 / 452 (concise) | 1671 / 2118 (verbose) | — |
| VRAM @ 64k | MiB | **7684** | ~12600 | ornith (more headroom) |

## Why the speed gap (the important takeaway)
gpt-oss-20b is **MoE with ~3.6B active params/token**; Ornith-9B is **dense (9B active/token)**.
The P100 (GP100, cc 6.0) is compute-bound, so per-token cost scales with *active* params —
2.5× more active params ≈ the ~2.5–3× slower generation we measured. This is architectural,
not a config problem; no amount of tuning closes it. Your memory's "~5 tok/s ceiling" held for
the MoE models; a dense 9B genuinely sits below that line on this GPU.

## Verdict for your Hermes backend
- **Default stays gpt-oss-20b.** Gen speed is already the UX pain point (~6 vs ~20 tok/s is the
  difference between usable and painful for interactive agent loops), and Ornith doesn't win on
  quality to justify being 3× slower.
- **Keep Ornith as a menu specialist.** It's MIT, purpose-built for agentic coding, refreshingly
  concise (less token waste on overthinking), and tool-calls cleanly. Good for one-shot coding
  asks where you'll wait for quality.
- **The real coding upgrade you already have on disk:** `Qwen3-Coder-30B-A3B-Instruct-UD-Q2_K_XL.gguf`
  is a **coder-tuned MoE (3B active)** — it would be both *faster* than Ornith (MoE) *and*
  code-specialized. If you want a faster coding model than gpt-oss, that's the one to benchmark
  next, not Ornith.

## Caveats
- Benchmarks are a small hand-written suite (4 tasks), not SWE-bench. Ornith's headline 69.4
  SWE-Bench Verified is **self-reported/unverified** by DeepReinforce.
- The `deep_prefill` row measures prefill speed only — both models hit the 256-token cap mid-think
  (`finish=length`, empty `content`), which is expected for that probe.
- Raw responses saved on valhalla at `/tmp/ornith-bench/` (one `.json` per model+test).

## Open decision for you
The repo config change (`stacks/llm/llama-swap-config.yaml`) is **deployed to the host but NOT
committed/pushed** — I don't push without your say-so, and pushing would trigger a Portainer
redeploy of stack 34. Tell me to commit+push if you want the repo (canonical source) to match the
live host, or to revert the menu entry if you'd rather not keep Ornith around.