feat(llm): add llama.cpp inference stack for Hermes (Qwen2.5-14B on P100)
New stacks/llm/ serves Qwen2.5-14B-Instruct (Q4_K_M GGUF) via llama.cpp's OpenAI-compatible server on the Tesla P100 (CDI nvidia.com/gpu=0), published on 172.20.0.1:8090 for the host-side Hermes agent. vLLM was rejected: the P100 (cc 6.0) lacks the DP4A INT8 instructions its AWQ/GPTQ kernels need. Includes design spec and implementation plan under docs/superpowers/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ginnoir
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
847edff1f8
commit
8e7682985d
@@ -0,0 +1,176 @@
|
||||
# LLM inference backend for Hermes (valhalla) — design
|
||||
|
||||
**Date:** 2026-06-26
|
||||
**Status:** Approved (pending spec review)
|
||||
**Stack:** new `stacks/llm/`
|
||||
|
||||
## Goal
|
||||
|
||||
Stand up a local, OpenAI-compatible LLM inference endpoint on valhalla and point the
|
||||
existing **Hermes** agent backend at it, so Hermes runs against a self-hosted 14B model
|
||||
instead of an external provider.
|
||||
|
||||
## Key decision: llama.cpp, not vLLM
|
||||
|
||||
The request was "set up vLLM," but valhalla's GPU is a **Tesla P100-PCIE-16GB**, die
|
||||
**GP100, compute capability 6.0**. That rules out vLLM for the desired model class:
|
||||
|
||||
- The GP100 (cc 6.0) lacks the **DP4A INT8** instructions that vLLM's AWQ/GPTQ kernels
|
||||
require (those need Turing 7.5 / Ampere). So vLLM **cannot run quantized 13B+** here.
|
||||
- An unquantized 13B in fp16 is ~26 GB → does not fit 16 GB.
|
||||
- Current vLLM refuses cc < 7.0 outright; even a ~7B fp16 would need a pinned old vLLM +
|
||||
`--dtype float16` + `VLLM_ATTENTION_BACKEND=XFORMERS`, and still no 13B.
|
||||
|
||||
Since the user wants a **13B+ class model** and only needs an **OpenAI-compatible** API
|
||||
(confirmed), the right engine is **llama.cpp's `llama-server`**: rock-solid Pascal (sm_60)
|
||||
support, GGUF quantization, and a native OpenAI-compatible `/v1` API that Hermes consumes
|
||||
unchanged.
|
||||
|
||||
## Hardware / host facts (verified live 2026-06-26)
|
||||
|
||||
- GPU: 1× Tesla P100-PCIE-16GB, cc 6.0, driver 580.159.04, CUDA 13. Idle. (2nd staged P100
|
||||
not installed — single-GPU design.)
|
||||
- Docker 29.5.2, NVIDIA Container Toolkit 1.19.1.
|
||||
- **CDI already configured:** `nvidia.com/gpu=0` is a valid device. No host runtime changes
|
||||
needed — the compose just references the CDI device.
|
||||
- `/storage1` (ZFS, virtiofs): 17 TB free. Weights live here; VM root is only 200 GB.
|
||||
- **Hermes** runs on the host (not Docker) from `/home/ginnoir/.hermes/`:
|
||||
- `hermes dashboard --host 172.20.0.1 --port 9119` (gateway/agent API+WS, pid 1116)
|
||||
- `hermes_cli.main gateway run` (pid 1119)
|
||||
- `hermes-webui/server.py` on `172.20.0.1:8787` (pid 1128)
|
||||
- Bound to `172.20.0.1` = the Docker bridge gateway, so host↔container reachability is
|
||||
trivial: a container published on `172.20.0.1:<port>` is reachable by Hermes and by
|
||||
other containers. No Caddy hop in the inference path.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Hermes agent/gateway (host, 172.20.0.1)
|
||||
│ OpenAI base_url → http://172.20.0.1:8090/v1 (api_key = LLM_API_KEY)
|
||||
▼
|
||||
llama-server container (stacks/llm) ──CDI nvidia.com/gpu=0──▶ Tesla P100
|
||||
model: Qwen2.5-14B-Instruct-Q4_K_M.gguf
|
||||
weights bind-mounted from /storage1/labdata/llm/models
|
||||
```
|
||||
|
||||
No Caddy endpoint (Hermes-only, per decision). Endpoint is unauthenticated-but-API-keyed
|
||||
and only reachable on the host/bridge — matching the Pattern-B fallback for internal tools.
|
||||
|
||||
## The stack — `stacks/llm/docker-compose.yml`
|
||||
|
||||
Single service `llama-server`:
|
||||
|
||||
- **Image:** `ghcr.io/ggml-org/llama.cpp:server-cuda` at a pinned tag.
|
||||
- Infra-pinned per repo convention: label `com.centurylabs.watchtower.enable=false` so
|
||||
Watchtower won't drift it.
|
||||
- **Verification gate:** confirm the pinned prebuilt image includes Pascal `sm_60`
|
||||
kernels and is CUDA ≤ 13 compatible. If it errors on the P100, fall back to a locally
|
||||
built image with `-DCMAKE_CUDA_ARCHITECTURES=60`.
|
||||
- **GPU:** `devices: ["nvidia.com/gpu=0"]` (CDI).
|
||||
- **Volumes:** `/storage1/labdata/llm/models:/models` (bind).
|
||||
- **Command / args (32k baseline):**
|
||||
- `-m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf`
|
||||
- `-ngl 99` (full offload — 14B Q4 fits in VRAM)
|
||||
- `--ctx-size 32768` (native context, no rope scaling)
|
||||
- `-fa` (flash attention — required for quantized KV cache; works on Pascal)
|
||||
- `--cache-type-k q8_0 --cache-type-v q8_0` (KV cache q8_0 ≈ 6 GB at 32k)
|
||||
- `--host 0.0.0.0 --port 8080`
|
||||
- `--api-key ${LLM_API_KEY}`
|
||||
- `--alias qwen2.5-14b-instruct` (stable model name Hermes references)
|
||||
- **Ports:** `"172.20.0.1:8090:8080"` (reachable by Hermes on host + by containers).
|
||||
- **Networks:** private `llm` net only (no `edge` — no Caddy endpoint this round).
|
||||
- **restart:** `unless-stopped`. **Healthcheck:** GET `/health` on 8080.
|
||||
- **`env_file: stack.env`** per repo convention.
|
||||
|
||||
### `stacks/llm/stack.env`
|
||||
|
||||
- `LLM_API_KEY=<generated>` (committed per repo policy — secrets are versioned here).
|
||||
|
||||
### VRAM budget (Qwen2.5-14B, GQA: 48 layers, 8 KV heads, head_dim 128)
|
||||
|
||||
- KV cache ≈ 0.375 MiB/token fp16 → **q8_0 halves to ≈ 0.1875 MiB/token**.
|
||||
- Weights Q4_K_M ≈ 9.0 GB; reserve ~0.8 GB compute buffers.
|
||||
- 32k @ q8_0 KV ≈ 6.0 GB → **~15.7 GB total, fits** (tight but safe at 16 GB).
|
||||
|
||||
## Model acquisition (one-time)
|
||||
|
||||
Download `Qwen2.5-14B-Instruct-Q4_K_M.gguf` (~9 GB) from
|
||||
`bartowski/Qwen2.5-14B-Instruct-GGUF` into `/storage1/labdata/llm/models/` on the host
|
||||
(e.g. `huggingface-cli download` or `wget` the single GGUF). Documented host-side step,
|
||||
done before first stack deploy.
|
||||
|
||||
## Hermes integration (host-side, not in git)
|
||||
|
||||
Hermes is the **Nous Research Hermes agent** (`hermes-agent.nousresearch.com`). Its config
|
||||
is `~/.hermes/config.yaml`, which already has a `providers:` list whose entries are exactly
|
||||
OpenAI-compatible upstreams — there's a working `ollama` provider in it today
|
||||
(`type: openai`, `base_url: http://192.168.1.73:11434/v1`). Adding the P100 is one more
|
||||
entry of the same shape; no new integration surface.
|
||||
|
||||
1. **Add a provider** to `providers:` in `~/.hermes/config.yaml`:
|
||||
```yaml
|
||||
- name: valhalla-p100
|
||||
type: openai
|
||||
base_url: http://172.20.0.1:8090/v1
|
||||
api_key: <LLM_API_KEY>
|
||||
models:
|
||||
- qwen2.5-14b-instruct
|
||||
```
|
||||
Use `hermes config` / the `hermes` CLI where possible; a direct YAML edit + restart is
|
||||
the fallback (the CLI is the source of truth for `_config_version`).
|
||||
2. **Select the model** as the active one via `hermes model` (interactive) — or set
|
||||
`model.default: qwen2.5-14b-instruct` (+ matching provider) if it should be the gateway
|
||||
default rather than a switchable option. The current default is `gpt-5.5` /
|
||||
`openai-codex`; we add ours alongside and let the user choose, rather than silently
|
||||
replacing the default.
|
||||
3. **Restart** the three Hermes processes (gateway dashboard pid-class, `gateway run`,
|
||||
webui) so the new provider/model is live.
|
||||
4. **Verify** end-to-end: a Hermes prompt routed to `qwen2.5-14b-instruct` produces a
|
||||
completion served by the P100 (confirm via `nvidia-smi` showing the llama-server process
|
||||
holding VRAM during generation).
|
||||
|
||||
These host-side steps are documented in the plan (and worth a note in CLAUDE.md
|
||||
known-quirks), not committed as repo changes — Hermes isn't in compose, and the alias
|
||||
`qwen2.5-14b-instruct` set via `--alias` is the contract between llama-server and this
|
||||
provider entry.
|
||||
|
||||
## Deployment
|
||||
|
||||
`stacks/llm/` is a new Portainer git stack → must be **registered once** (new stacks aren't
|
||||
auto-created by the poller). Per repo precedent (memory: portainer-new-stack-registration):
|
||||
create via MCP/Portainer, poll `StackList` to confirm, use the stacks' working fine-grained
|
||||
PAT for git creds. Pure `env_file` (empty Portainer UI env). After registration, normal
|
||||
git-push → 5-min poll redeploys apply.
|
||||
|
||||
## Verification
|
||||
|
||||
1. `docker logs llama-server` shows model loaded, all layers offloaded to GPU, server
|
||||
listening on 8080; healthcheck green.
|
||||
2. `nvidia-smi` shows the llama-server process holding ~15–16 GB.
|
||||
3. `curl http://172.20.0.1:8090/v1/chat/completions` (with API key) returns a completion.
|
||||
4. Hermes, repointed, produces a completion served locally.
|
||||
5. **64k stretch (post-baseline):** re-run with `--ctx-size 65536`, YaRN rope-scaling, and
|
||||
`--cache-type-k q8_0 --cache-type-v q4_0` (or both q4_0), possibly a smaller weight quant
|
||||
(Q4_K_S/IQ4_XS) for headroom. Drive a long prompt and watch VRAM; keep the largest
|
||||
context that holds without OOM under load. Revert to 32k if 64k is unstable.
|
||||
|
||||
## Non-goals / out of scope
|
||||
|
||||
- vLLM (ruled out by hardware — see decision above).
|
||||
- SSO/Authentik on the endpoint (LAN/host-only, API-keyed).
|
||||
- A public `llm.ginnoir.com` Caddy endpoint (declined this round; easy to add later by
|
||||
joining `edge` + an `internal_only` block).
|
||||
- Multi-GPU / 2nd P100 install.
|
||||
|
||||
## Risks
|
||||
|
||||
- ~~**Prebuilt image may lack sm_60**~~ → **RESOLVED (verified live 2026-06-26).**
|
||||
`ghcr.io/ggml-org/llama.cpp:server-cuda` (digest
|
||||
`sha256:ce294a4561e6…f0f2a9`) loaded Qwen2.5-0.5B-Instruct-Q4_K_M with `-ngl 99` on the
|
||||
P100: `nvidia-smi` showed the server process holding ~1.1 GB of GPU VRAM, model loaded,
|
||||
server listening, no arch/assert errors. Prebuilt image works on Pascal — no source build
|
||||
needed.
|
||||
- **32k @ q8_0 is tight (~15.7 GB)** → if compute buffers push it over, drop V cache to
|
||||
q4_0 or context to 24k.
|
||||
- **Pascal `-fa` performance** → flash-attn works on Pascal but is slower than on
|
||||
Volta+; acceptable for single/few-user agent use, measured during verification.
|
||||
Reference in New Issue
Block a user