ginnoirandClaude Opus 4.8
009a474e90
DeepReinforce Ornith-1.0 (dense 9B on Qwen 3.5, Q5_K_M, MIT), an agentic-coding model. Tool-calls + <think> work under --jinja; native 256k so no YaRN. Loads at ~7.7GB VRAM @ 64k. Benchmark (docs/2026-06-27-ornith-9b-benchmark.md): quality ties gpt-oss-20b but gen is ~2.5-3x slower (dense 9B active vs gpt-oss MoE 3.6B active on the compute-bound P100). Default stays gpt-oss-20b; ornith kept as a coding specialist in the menu. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
62 lines
4.1 KiB
Markdown
62 lines
4.1 KiB
Markdown
# Ornith-1.0-9B vs gpt-oss-20b — P100 benchmark (2026-06-27)
|
||
|
||
**TL;DR:** Ornith-1.0-9B works cleanly as a Hermes backend (tool-calls, `<think>`,
|
||
content all correct) and matches gpt-oss-20b on **code quality** — but it is a **dense
|
||
9B**, so on the compute-bound P100 it generates **~2–3× slower** than the MoE gpt-oss-20b
|
||
(~6.3 vs ~13–23 tok/s). **Keep gpt-oss-20b as the default daily driver.** Ornith earns a
|
||
spot in the menu as a coding *specialist* to A/B on real tasks, but doesn't displace it.
|
||
|
||
## What was done
|
||
- Downloaded `ornith-1.0-9b-Q5_K_M.gguf` (6.47 GB, exact byte match) → `/storage1/labdata/llm/models/`.
|
||
- Added `ornith-1.0-9b` to `stacks/llm/llama-swap-config.yaml` (same Pascal macro: q8/q8 KV,
|
||
`--parallel 1`, `--flash-attn on`, `--jinja`, 64k ctx; native 256k so no YaRN). Deployed to
|
||
`/config/llm/` + restarted llama-swap. **Live in the menu now.**
|
||
- Quant choice: **Q5_K_M** (safe overnight, no OOM). Loads using only **7.7 GB VRAM** at 64k —
|
||
huge headroom, so Q6_K or Q8_0 (9.5 GB, near-lossless) would also fit easily if you want more quality.
|
||
|
||
## Results (identical prompts, temp 0.6 / top_p 0.95 / top_k 20)
|
||
|
||
| Test | Metric | **ornith-1.0-9b** | **gpt-oss-20b** | Winner |
|
||
|---|---|---|---|---|
|
||
| Tool call | emits valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie |
|
||
| Codegen (merge_intervals) | correct? | ✅ correct, non-mutating, tests pass | ✅ correct (mutates input list) | ornith (slightly cleaner) |
|
||
| Debug (second_largest) | correct? | ✅ correct + edge-case guard | ✅ correct + O(n) alt | tie (gpt-oss more thorough) |
|
||
| **Gen speed** | tok/s (codegen) | **6.4** | **21.7** | **gpt-oss 3.4×** |
|
||
| **Gen speed** | tok/s (debug) | **6.4** | **18.9** | **gpt-oss 2.9×** |
|
||
| Prefill (small) | tok/s | 49–104 | 133–188 | gpt-oss |
|
||
| Prefill (deep ~25–30k ctx) | tok/s | 101.9 (n=30295) | 154.9 (n=23422) | gpt-oss 1.5× |
|
||
| Reasoning verbosity | think chars (codegen/debug) | 433 / 452 (concise) | 1671 / 2118 (verbose) | — |
|
||
| VRAM @ 64k | MiB | **7684** | ~12600 | ornith (more headroom) |
|
||
|
||
## Why the speed gap (the important takeaway)
|
||
gpt-oss-20b is **MoE with ~3.6B active params/token**; Ornith-9B is **dense (9B active/token)**.
|
||
The P100 (GP100, cc 6.0) is compute-bound, so per-token cost scales with *active* params —
|
||
2.5× more active params ≈ the ~2.5–3× slower generation we measured. This is architectural,
|
||
not a config problem; no amount of tuning closes it. Your memory's "~5 tok/s ceiling" held for
|
||
the MoE models; a dense 9B genuinely sits below that line on this GPU.
|
||
|
||
## Verdict for your Hermes backend
|
||
- **Default stays gpt-oss-20b.** Gen speed is already the UX pain point (~6 vs ~20 tok/s is the
|
||
difference between usable and painful for interactive agent loops), and Ornith doesn't win on
|
||
quality to justify being 3× slower.
|
||
- **Keep Ornith as a menu specialist.** It's MIT, purpose-built for agentic coding, refreshingly
|
||
concise (less token waste on overthinking), and tool-calls cleanly. Good for one-shot coding
|
||
asks where you'll wait for quality.
|
||
- **The real coding upgrade you already have on disk:** `Qwen3-Coder-30B-A3B-Instruct-UD-Q2_K_XL.gguf`
|
||
is a **coder-tuned MoE (3B active)** — it would be both *faster* than Ornith (MoE) *and*
|
||
code-specialized. If you want a faster coding model than gpt-oss, that's the one to benchmark
|
||
next, not Ornith.
|
||
|
||
## Caveats
|
||
- Benchmarks are a small hand-written suite (4 tasks), not SWE-bench. Ornith's headline 69.4
|
||
SWE-Bench Verified is **self-reported/unverified** by DeepReinforce.
|
||
- The `deep_prefill` row measures prefill speed only — both models hit the 256-token cap mid-think
|
||
(`finish=length`, empty `content`), which is expected for that probe.
|
||
- Raw responses saved on valhalla at `/tmp/ornith-bench/` (one `.json` per model+test).
|
||
|
||
## Open decision for you
|
||
The repo config change (`stacks/llm/llama-swap-config.yaml`) is **deployed to the host but NOT
|
||
committed/pushed** — I don't push without your say-so, and pushing would trigger a Portainer
|
||
redeploy of stack 34. Tell me to commit+push if you want the repo (canonical source) to match the
|
||
live host, or to revert the menu entry if you'd rather not keep Ornith around.
|