homelabstack/docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md

LLM inference backend for Hermes (valhalla) — design

Date: 2026-06-26 Status: Approved (pending spec review) Stack: new stacks/llm/

Goal

Stand up a local, OpenAI-compatible LLM inference endpoint on valhalla and point the existing Hermes agent backend at it, so Hermes runs against a self-hosted 14B model instead of an external provider.

Key decision: llama.cpp, not vLLM

The request was "set up vLLM," but valhalla's GPU is a Tesla P100-PCIE-16GB, die GP100, compute capability 6.0. That rules out vLLM for the desired model class:

• The GP100 (cc 6.0) lacks the DP4A INT8 instructions that vLLM's AWQ/GPTQ kernels require (those need Turing 7.5 / Ampere). So vLLM cannot run quantized 13B+ here.
• An unquantized 13B in fp16 is ~26 GB → does not fit 16 GB.
• Current vLLM refuses cc < 7.0 outright; even a ~7B fp16 would need a pinned old vLLM + --dtype float16 + VLLM_ATTENTION_BACKEND=XFORMERS, and still no 13B.

Since the user wants a 13B+ class model and only needs an OpenAI-compatible API (confirmed), the right engine is llama.cpp's llama-server: rock-solid Pascal (sm_60) support, GGUF quantization, and a native OpenAI-compatible /v1 API that Hermes consumes unchanged.

Hardware / host facts (verified live 2026-06-26)

• GPU: 1× Tesla P100-PCIE-16GB, cc 6.0, driver 580.159.04, CUDA 13. Idle. (2nd staged P100 not installed — single-GPU design.)
• Docker 29.5.2, NVIDIA Container Toolkit 1.19.1.
• CDI already configured: nvidia.com/gpu=0 is a valid device. No host runtime changes needed — the compose just references the CDI device.
• /storage1 (ZFS, virtiofs): 17 TB free. Weights live here; VM root is only 200 GB.
• Hermes runs on the host (not Docker) from /home/ginnoir/.hermes/:
  • hermes dashboard --host 172.20.0.1 --port 9119 (gateway/agent API+WS, pid 1116)
  • hermes_cli.main gateway run (pid 1119)
  • hermes-webui/server.py on 172.20.0.1:8787 (pid 1128)
  • Bound to 172.20.0.1 = the Docker bridge gateway, so host↔container reachability is trivial: a container published on 172.20.0.1:<port> is reachable by Hermes and by other containers. No Caddy hop in the inference path.

Architecture

Hermes agent/gateway (host, 172.20.0.1)
        │  OpenAI base_url → http://172.20.0.1:8090/v1   (api_key = LLM_API_KEY)
        ▼
llama-server container (stacks/llm)  ──CDI nvidia.com/gpu=0──▶  Tesla P100
   model: Qwen2.5-14B-Instruct-Q4_K_M.gguf
   weights bind-mounted from /storage1/labdata/llm/models

No Caddy endpoint (Hermes-only, per decision). Endpoint is unauthenticated-but-API-keyed and only reachable on the host/bridge — matching the Pattern-B fallback for internal tools.

The stack — stacks/llm/docker-compose.yml

Single service llama-server:

• Image: ghcr.io/ggml-org/llama.cpp:server-cuda at a pinned tag.
  • Infra-pinned per repo convention: label com.centurylabs.watchtower.enable=false so Watchtower won't drift it.
  • Verification gate: confirm the pinned prebuilt image includes Pascal sm_60 kernels and is CUDA ≤ 13 compatible. If it errors on the P100, fall back to a locally built image with -DCMAKE_CUDA_ARCHITECTURES=60.
• GPU: devices: ["nvidia.com/gpu=0"] (CDI).
• Volumes: /storage1/labdata/llm/models:/models (bind).
• Command / args (AS DEPLOYED — 64k, required by Hermes' 64K minimum):
  • -m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf
  • --alias qwen2.5-14b-instruct (stable model name Hermes references)
  • --parallel 1 (one slot gets the FULL context; default 4 slots split it to 32k/seq → fails Hermes)
  • -ngl 99 (full offload — 14B Q4 fits in VRAM)
  • --ctx-size 65536
  • --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768 (YaRN extends Qwen2.5's 32k native → 64k)
  • --override-kv qwen2.context_length=int:65536 (raises GGUF training-context metadata so llama-server does NOT cap the slot back to 32768 — without this the slot is capped and Hermes still sees 32k)
  • --flash-attn on (this build needs the explicit on value; a bare -fa swallows the next arg)
  • --cache-type-k q8_0 --cache-type-v q8_0 (both q8_0 — q4_0 V-cache is pathological on Pascal: 1.28 tok/s gen at 5–8% GPU util. q8_0/q8_0 → 9.2 tok/s and still fits 64k.)
  • --host 0.0.0.0 --port 8080
  • API key via LLAMA_API_KEY env (env_file) — NOT a CLI flag (no ${VAR} interpolation; llama-server reads the env var natively)
• Ports: "172.20.0.1:8090:8080" (reachable by Hermes on host + by containers).
• Networks: private llm net only (no edge — no Caddy endpoint this round).
• restart: unless-stopped. Healthcheck: GET /health on 8080.
• env_file: stack.env per repo convention.

stacks/llm/stack.env

• LLM_API_KEY=<generated> (committed per repo policy — secrets are versioned here).

VRAM budget (Qwen2.5-14B, GQA: 48 layers, 8 KV heads, head_dim 128)

• KV cache ≈ 0.375 MiB/token fp16 → q8_0 halves to ≈ 0.1875 MiB/token.
• Weights Q4_K_M ≈ 9.0 GB; reserve ~0.8 GB compute buffers.
• 32k @ q8_0 KV ≈ 6.0 GB → ~15.7 GB total, fits (tight but safe at 16 GB).

Model acquisition (one-time)

Download Qwen2.5-14B-Instruct-Q4_K_M.gguf (~9 GB) from bartowski/Qwen2.5-14B-Instruct-GGUF into /storage1/labdata/llm/models/ on the host (e.g. huggingface-cli download or wget the single GGUF). Documented host-side step, done before first stack deploy.

Hermes integration (host-side, not in git)

Hermes is the Nous Research Hermes agent (hermes-agent.nousresearch.com). Its config is ~/.hermes/config.yaml, which already has a providers: list whose entries are exactly OpenAI-compatible upstreams — there's a working ollama provider in it today (type: openai, base_url: http://192.168.1.73:11434/v1). Adding the P100 is one more entry of the same shape; no new integration surface.

1. Add a provider to providers: in ~/.hermes/config.yaml:

   - name: valhalla-p100
     type: openai
     base_url: http://172.20.0.1:8090/v1
     api_key: <LLM_API_KEY>
     models:
     - qwen2.5-14b-instruct
   

   Use hermes config / the hermes CLI where possible; a direct YAML edit + restart is the fallback (the CLI is the source of truth for _config_version).
2. Select the model as the active one via hermes model (interactive) — or set model.default: qwen2.5-14b-instruct (+ matching provider) if it should be the gateway default rather than a switchable option. The current default is gpt-5.5 / openai-codex; we add ours alongside and let the user choose, rather than silently replacing the default.
3. Restart the three Hermes processes (gateway dashboard pid-class, gateway run, webui) so the new provider/model is live.
4. Verify end-to-end: a Hermes prompt routed to qwen2.5-14b-instruct produces a completion served by the P100 (confirm via nvidia-smi showing the llama-server process holding VRAM during generation).

These host-side steps are documented in the plan (and worth a note in CLAUDE.md known-quirks), not committed as repo changes — Hermes isn't in compose, and the alias qwen2.5-14b-instruct set via --alias is the contract between llama-server and this provider entry.

Deployment

stacks/llm/ is a new Portainer git stack → must be registered once (new stacks aren't auto-created by the poller). Per repo precedent (memory: portainer-new-stack-registration): create via MCP/Portainer, poll StackList to confirm, use the stacks' working fine-grained PAT for git creds. Pure env_file (empty Portainer UI env). After registration, normal git-push → 5-min poll redeploys apply.

Final deployed state (verified live 2026-06-26)

Portainer stack llm (id 34), container llama-server healthy. Config: 64k / q8_0 KV / YaRN / --parallel 1 / --override-kv qwen2.context_length=int:65536. VRAM 15.3 GB used, ~0.9 GB free. Hermes model: block points at provider custom → http://172.20.0.1:8090/v1 (matched to the valhalla-p100 entry in the providers: list); active model qwen2.5-14b-instruct. Original config backed up at ~/.hermes/config.yaml.bak.*.

Measured performance (Qwen2.5-14B-Q4_K_M, 64k q8/q8, P100)

• Generation: ~9.2 tok/s (memory-bound; fine for a personal assistant).
• Prefill: ~54 tok/s on a large prompt (the misleading ~10 tok/s figure is small-prompt overhead, not throughput).
• Hermes system prompt ≈ 16,400 tokens → first (cold) turn ≈ 5 min (all prefill).
• Prompt cache makes it usable: llama-server matches by longest-common-prefix (sim_best = 0.999), so subsequent turns — even new conversations sharing the stable system prompt — reuse the prefix and respond in ~20 s. The 5 min is a one-time post-restart warmup.

Hard-won config gotchas (all verified the slow/broken way first)

1. -fa needs an explicit value in this build: use --flash-attn on. A bare -fa swallows the next arg (--cache-type-k) and crash-loops.
2. --parallel 1 — the default 4 slots split --ctx-size to 32k/sequence, which fails Hermes' 64K minimum. One slot serves the full window.
3. --override-kv qwen2.context_length=int:65536 — without it, llama-server caps the slot back to the GGUF training context (32768) even with YaRN set, so per-seq stays 32k.
4. q8_0 V-cache, NOT q4_0 — q4_0 V-cache is pathological on the GP100 (cc 6.0, no DP4A): 1.28 tok/s generation at 5–8% GPU util. q8_0/q8_0 → 9.2 tok/s and still fits 64k.
5. Hermes requires ≥64K context and rejects smaller models outright (or set model.context_length to override — but then the server must actually serve it).
6. Hermes provider wiring: a providers: list entry is a "named custom provider", activated only by setting the model: block to provider: custom + matching base_url. It is NOT selectable via --provider <name> (that path wants a providers: dict).

Non-goals / out of scope

• vLLM (ruled out by hardware — see decision above).
• SSO/Authentik on the endpoint (LAN/host-only, API-keyed).
• A public llm.ginnoir.com Caddy endpoint (declined; easy to add later via edge + internal_only).
• Multi-GPU / 2nd P100 install.

Open risks / follow-ups

• VRAM is tight (~0.9 GB free). A full 64k prefill held under real load (16k-token Hermes prompt succeeded), but watch for OOM if other GPU users appear; fall back to --ctx-size 60000 or a smaller weight quant for margin.
• Cold-start latency (~5 min). Inherent to a 16k system prompt at Pascal prefill speed. Mitigation if it annoys: trim Hermes' prompt (disable environment_probe, fewer toolsets) to shrink the cached prefix.
• Prompt-cache persistence across restarts is in-memory; a container restart re-pays the cold prefill once.