docs(llm): record final 64k config, perf, and Pascal/Hermes gotchas
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ginnoir
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
4aa2cd9468
commit
eba51a52a7
@@ -68,15 +68,20 @@ Single service `llama-server`:
|
||||
built image with `-DCMAKE_CUDA_ARCHITECTURES=60`.
|
||||
- **GPU:** `devices: ["nvidia.com/gpu=0"]` (CDI).
|
||||
- **Volumes:** `/storage1/labdata/llm/models:/models` (bind).
|
||||
- **Command / args (32k baseline):**
|
||||
- **Command / args (AS DEPLOYED — 64k, required by Hermes' 64K minimum):**
|
||||
- `-m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf`
|
||||
- `-ngl 99` (full offload — 14B Q4 fits in VRAM)
|
||||
- `--ctx-size 32768` (native context, no rope scaling)
|
||||
- `-fa` (flash attention — required for quantized KV cache; works on Pascal)
|
||||
- `--cache-type-k q8_0 --cache-type-v q8_0` (KV cache q8_0 ≈ 6 GB at 32k)
|
||||
- `--host 0.0.0.0 --port 8080`
|
||||
- `--api-key ${LLM_API_KEY}`
|
||||
- `--alias qwen2.5-14b-instruct` (stable model name Hermes references)
|
||||
- `--parallel 1` (one slot gets the FULL context; default 4 slots split it to 32k/seq → fails Hermes)
|
||||
- `-ngl 99` (full offload — 14B Q4 fits in VRAM)
|
||||
- `--ctx-size 65536`
|
||||
- `--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768` (YaRN extends Qwen2.5's 32k native → 64k)
|
||||
- `--override-kv qwen2.context_length=int:65536` (raises GGUF training-context metadata so
|
||||
llama-server does NOT cap the slot back to 32768 — without this the slot is capped and Hermes still sees 32k)
|
||||
- `--flash-attn on` (this build needs the explicit `on` value; a bare `-fa` swallows the next arg)
|
||||
- `--cache-type-k q8_0 --cache-type-v q8_0` (**both q8_0** — q4_0 V-cache is pathological on Pascal:
|
||||
1.28 tok/s gen at 5–8% GPU util. q8_0/q8_0 → 9.2 tok/s and still fits 64k.)
|
||||
- `--host 0.0.0.0 --port 8080`
|
||||
- API key via `LLAMA_API_KEY` env (env_file) — NOT a CLI flag (no `${VAR}` interpolation; llama-server reads the env var natively)
|
||||
- **Ports:** `"172.20.0.1:8090:8080"` (reachable by Hermes on host + by containers).
|
||||
- **Networks:** private `llm` net only (no `edge` — no Caddy endpoint this round).
|
||||
- **restart:** `unless-stopped`. **Healthcheck:** GET `/health` on 8080.
|
||||
@@ -142,35 +147,56 @@ create via MCP/Portainer, poll `StackList` to confirm, use the stacks' working f
|
||||
PAT for git creds. Pure `env_file` (empty Portainer UI env). After registration, normal
|
||||
git-push → 5-min poll redeploys apply.
|
||||
|
||||
## Verification
|
||||
## Final deployed state (verified live 2026-06-26)
|
||||
|
||||
1. `docker logs llama-server` shows model loaded, all layers offloaded to GPU, server
|
||||
listening on 8080; healthcheck green.
|
||||
2. `nvidia-smi` shows the llama-server process holding ~15–16 GB.
|
||||
3. `curl http://172.20.0.1:8090/v1/chat/completions` (with API key) returns a completion.
|
||||
4. Hermes, repointed, produces a completion served locally.
|
||||
5. **64k stretch (post-baseline):** re-run with `--ctx-size 65536`, YaRN rope-scaling, and
|
||||
`--cache-type-k q8_0 --cache-type-v q4_0` (or both q4_0), possibly a smaller weight quant
|
||||
(Q4_K_S/IQ4_XS) for headroom. Drive a long prompt and watch VRAM; keep the largest
|
||||
context that holds without OOM under load. Revert to 32k if 64k is unstable.
|
||||
Portainer stack `llm` (id 34), container `llama-server` healthy. Config: 64k / q8_0 KV /
|
||||
YaRN / `--parallel 1` / `--override-kv qwen2.context_length=int:65536`. VRAM 15.3 GB used,
|
||||
~0.9 GB free. Hermes `model:` block points at provider `custom` → `http://172.20.0.1:8090/v1`
|
||||
(matched to the `valhalla-p100` entry in the `providers:` list); active model
|
||||
`qwen2.5-14b-instruct`. Original config backed up at `~/.hermes/config.yaml.bak.*`.
|
||||
|
||||
### Measured performance (Qwen2.5-14B-Q4_K_M, 64k q8/q8, P100)
|
||||
|
||||
- **Generation: ~9.2 tok/s** (memory-bound; fine for a personal assistant).
|
||||
- **Prefill: ~54 tok/s** on a large prompt (the misleading ~10 tok/s figure is small-prompt
|
||||
overhead, not throughput).
|
||||
- **Hermes system prompt ≈ 16,400 tokens** → first (cold) turn ≈ **5 min** (all prefill).
|
||||
- **Prompt cache makes it usable:** llama-server matches by longest-common-prefix
|
||||
(`sim_best = 0.999`), so subsequent turns — even new conversations sharing the stable
|
||||
system prompt — reuse the prefix and respond in **~20 s**. The 5 min is a one-time
|
||||
post-restart warmup.
|
||||
|
||||
### Hard-won config gotchas (all verified the slow/broken way first)
|
||||
|
||||
1. **`-fa` needs an explicit value** in this build: use `--flash-attn on`. A bare `-fa`
|
||||
swallows the next arg (`--cache-type-k`) and crash-loops.
|
||||
2. **`--parallel 1`** — the default 4 slots split `--ctx-size` to 32k/sequence, which fails
|
||||
Hermes' 64K minimum. One slot serves the full window.
|
||||
3. **`--override-kv qwen2.context_length=int:65536`** — without it, llama-server *caps the
|
||||
slot back to the GGUF training context (32768)* even with YaRN set, so per-seq stays 32k.
|
||||
4. **q8_0 V-cache, NOT q4_0** — q4_0 V-cache is pathological on the GP100 (cc 6.0, no DP4A):
|
||||
**1.28 tok/s** generation at 5–8% GPU util. q8_0/q8_0 → 9.2 tok/s and *still* fits 64k.
|
||||
5. **Hermes requires ≥64K context** and rejects smaller models outright (or set
|
||||
`model.context_length` to override — but then the server must actually serve it).
|
||||
6. **Hermes provider wiring:** a `providers:` *list* entry is a "named custom provider",
|
||||
activated only by setting the `model:` block to `provider: custom` + matching `base_url`.
|
||||
It is NOT selectable via `--provider <name>` (that path wants a `providers:` *dict*).
|
||||
|
||||
## Non-goals / out of scope
|
||||
|
||||
- vLLM (ruled out by hardware — see decision above).
|
||||
- SSO/Authentik on the endpoint (LAN/host-only, API-keyed).
|
||||
- A public `llm.ginnoir.com` Caddy endpoint (declined this round; easy to add later by
|
||||
joining `edge` + an `internal_only` block).
|
||||
- A public `llm.ginnoir.com` Caddy endpoint (declined; easy to add later via `edge` +
|
||||
`internal_only`).
|
||||
- Multi-GPU / 2nd P100 install.
|
||||
|
||||
## Risks
|
||||
## Open risks / follow-ups
|
||||
|
||||
- ~~**Prebuilt image may lack sm_60**~~ → **RESOLVED (verified live 2026-06-26).**
|
||||
`ghcr.io/ggml-org/llama.cpp:server-cuda` (digest
|
||||
`sha256:ce294a4561e6…f0f2a9`) loaded Qwen2.5-0.5B-Instruct-Q4_K_M with `-ngl 99` on the
|
||||
P100: `nvidia-smi` showed the server process holding ~1.1 GB of GPU VRAM, model loaded,
|
||||
server listening, no arch/assert errors. Prebuilt image works on Pascal — no source build
|
||||
needed.
|
||||
- **32k @ q8_0 is tight (~15.7 GB)** → if compute buffers push it over, drop V cache to
|
||||
q4_0 or context to 24k.
|
||||
- **Pascal `-fa` performance** → flash-attn works on Pascal but is slower than on
|
||||
Volta+; acceptable for single/few-user agent use, measured during verification.
|
||||
- **VRAM is tight (~0.9 GB free).** A full 64k prefill held under real load (16k-token Hermes
|
||||
prompt succeeded), but watch for OOM if other GPU users appear; fall back to `--ctx-size
|
||||
60000` or a smaller weight quant for margin.
|
||||
- **Cold-start latency (~5 min).** Inherent to a 16k system prompt at Pascal prefill speed.
|
||||
Mitigation if it annoys: trim Hermes' prompt (disable `environment_probe`, fewer toolsets)
|
||||
to shrink the cached prefix.
|
||||
- **Prompt-cache persistence across restarts** is in-memory; a container restart re-pays the
|
||||
cold prefill once.
|
||||
|
||||
Reference in New Issue
Block a user