diff --git a/docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md b/docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md index e2dae43..a869896 100644 --- a/docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md +++ b/docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md @@ -68,15 +68,20 @@ Single service `llama-server`: built image with `-DCMAKE_CUDA_ARCHITECTURES=60`. - **GPU:** `devices: ["nvidia.com/gpu=0"]` (CDI). - **Volumes:** `/storage1/labdata/llm/models:/models` (bind). -- **Command / args (32k baseline):** +- **Command / args (AS DEPLOYED — 64k, required by Hermes' 64K minimum):** - `-m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf` - - `-ngl 99` (full offload — 14B Q4 fits in VRAM) - - `--ctx-size 32768` (native context, no rope scaling) - - `-fa` (flash attention — required for quantized KV cache; works on Pascal) - - `--cache-type-k q8_0 --cache-type-v q8_0` (KV cache q8_0 ≈ 6 GB at 32k) - - `--host 0.0.0.0 --port 8080` - - `--api-key ${LLM_API_KEY}` - `--alias qwen2.5-14b-instruct` (stable model name Hermes references) + - `--parallel 1` (one slot gets the FULL context; default 4 slots split it to 32k/seq → fails Hermes) + - `-ngl 99` (full offload — 14B Q4 fits in VRAM) + - `--ctx-size 65536` + - `--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768` (YaRN extends Qwen2.5's 32k native → 64k) + - `--override-kv qwen2.context_length=int:65536` (raises GGUF training-context metadata so + llama-server does NOT cap the slot back to 32768 — without this the slot is capped and Hermes still sees 32k) + - `--flash-attn on` (this build needs the explicit `on` value; a bare `-fa` swallows the next arg) + - `--cache-type-k q8_0 --cache-type-v q8_0` (**both q8_0** — q4_0 V-cache is pathological on Pascal: + 1.28 tok/s gen at 5–8% GPU util. q8_0/q8_0 → 9.2 tok/s and still fits 64k.) + - `--host 0.0.0.0 --port 8080` + - API key via `LLAMA_API_KEY` env (env_file) — NOT a CLI flag (no `${VAR}` interpolation; llama-server reads the env var natively) - **Ports:** `"172.20.0.1:8090:8080"` (reachable by Hermes on host + by containers). - **Networks:** private `llm` net only (no `edge` — no Caddy endpoint this round). - **restart:** `unless-stopped`. **Healthcheck:** GET `/health` on 8080. @@ -142,35 +147,56 @@ create via MCP/Portainer, poll `StackList` to confirm, use the stacks' working f PAT for git creds. Pure `env_file` (empty Portainer UI env). After registration, normal git-push → 5-min poll redeploys apply. -## Verification +## Final deployed state (verified live 2026-06-26) -1. `docker logs llama-server` shows model loaded, all layers offloaded to GPU, server - listening on 8080; healthcheck green. -2. `nvidia-smi` shows the llama-server process holding ~15–16 GB. -3. `curl http://172.20.0.1:8090/v1/chat/completions` (with API key) returns a completion. -4. Hermes, repointed, produces a completion served locally. -5. **64k stretch (post-baseline):** re-run with `--ctx-size 65536`, YaRN rope-scaling, and - `--cache-type-k q8_0 --cache-type-v q4_0` (or both q4_0), possibly a smaller weight quant - (Q4_K_S/IQ4_XS) for headroom. Drive a long prompt and watch VRAM; keep the largest - context that holds without OOM under load. Revert to 32k if 64k is unstable. +Portainer stack `llm` (id 34), container `llama-server` healthy. Config: 64k / q8_0 KV / +YaRN / `--parallel 1` / `--override-kv qwen2.context_length=int:65536`. VRAM 15.3 GB used, +~0.9 GB free. Hermes `model:` block points at provider `custom` → `http://172.20.0.1:8090/v1` +(matched to the `valhalla-p100` entry in the `providers:` list); active model +`qwen2.5-14b-instruct`. Original config backed up at `~/.hermes/config.yaml.bak.*`. + +### Measured performance (Qwen2.5-14B-Q4_K_M, 64k q8/q8, P100) + +- **Generation: ~9.2 tok/s** (memory-bound; fine for a personal assistant). +- **Prefill: ~54 tok/s** on a large prompt (the misleading ~10 tok/s figure is small-prompt + overhead, not throughput). +- **Hermes system prompt ≈ 16,400 tokens** → first (cold) turn ≈ **5 min** (all prefill). +- **Prompt cache makes it usable:** llama-server matches by longest-common-prefix + (`sim_best = 0.999`), so subsequent turns — even new conversations sharing the stable + system prompt — reuse the prefix and respond in **~20 s**. The 5 min is a one-time + post-restart warmup. + +### Hard-won config gotchas (all verified the slow/broken way first) + +1. **`-fa` needs an explicit value** in this build: use `--flash-attn on`. A bare `-fa` + swallows the next arg (`--cache-type-k`) and crash-loops. +2. **`--parallel 1`** — the default 4 slots split `--ctx-size` to 32k/sequence, which fails + Hermes' 64K minimum. One slot serves the full window. +3. **`--override-kv qwen2.context_length=int:65536`** — without it, llama-server *caps the + slot back to the GGUF training context (32768)* even with YaRN set, so per-seq stays 32k. +4. **q8_0 V-cache, NOT q4_0** — q4_0 V-cache is pathological on the GP100 (cc 6.0, no DP4A): + **1.28 tok/s** generation at 5–8% GPU util. q8_0/q8_0 → 9.2 tok/s and *still* fits 64k. +5. **Hermes requires ≥64K context** and rejects smaller models outright (or set + `model.context_length` to override — but then the server must actually serve it). +6. **Hermes provider wiring:** a `providers:` *list* entry is a "named custom provider", + activated only by setting the `model:` block to `provider: custom` + matching `base_url`. + It is NOT selectable via `--provider ` (that path wants a `providers:` *dict*). ## Non-goals / out of scope - vLLM (ruled out by hardware — see decision above). - SSO/Authentik on the endpoint (LAN/host-only, API-keyed). -- A public `llm.ginnoir.com` Caddy endpoint (declined this round; easy to add later by - joining `edge` + an `internal_only` block). +- A public `llm.ginnoir.com` Caddy endpoint (declined; easy to add later via `edge` + + `internal_only`). - Multi-GPU / 2nd P100 install. -## Risks +## Open risks / follow-ups -- ~~**Prebuilt image may lack sm_60**~~ → **RESOLVED (verified live 2026-06-26).** - `ghcr.io/ggml-org/llama.cpp:server-cuda` (digest - `sha256:ce294a4561e6…f0f2a9`) loaded Qwen2.5-0.5B-Instruct-Q4_K_M with `-ngl 99` on the - P100: `nvidia-smi` showed the server process holding ~1.1 GB of GPU VRAM, model loaded, - server listening, no arch/assert errors. Prebuilt image works on Pascal — no source build - needed. -- **32k @ q8_0 is tight (~15.7 GB)** → if compute buffers push it over, drop V cache to - q4_0 or context to 24k. -- **Pascal `-fa` performance** → flash-attn works on Pascal but is slower than on - Volta+; acceptable for single/few-user agent use, measured during verification. +- **VRAM is tight (~0.9 GB free).** A full 64k prefill held under real load (16k-token Hermes + prompt succeeded), but watch for OOM if other GPU users appear; fall back to `--ctx-size + 60000` or a smaller weight quant for margin. +- **Cold-start latency (~5 min).** Inherent to a 16k system prompt at Pascal prefill speed. + Mitigation if it annoys: trim Hermes' prompt (disable `environment_probe`, fewer toolsets) + to shrink the cached prefix. +- **Prompt-cache persistence across restarts** is in-memory; a container restart re-pays the + cold prefill once.