homelabstack/docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md

LLM inference backend for Hermes (valhalla) — design

Date: 2026-06-26 Status: Approved (pending spec review) Stack: new stacks/llm/

Goal

Stand up a local, OpenAI-compatible LLM inference endpoint on valhalla and point the existing Hermes agent backend at it, so Hermes runs against a self-hosted 14B model instead of an external provider.

Key decision: llama.cpp, not vLLM

The request was "set up vLLM," but valhalla's GPU is a Tesla P100-PCIE-16GB, die GP100, compute capability 6.0. That rules out vLLM for the desired model class:

• The GP100 (cc 6.0) lacks the DP4A INT8 instructions that vLLM's AWQ/GPTQ kernels require (those need Turing 7.5 / Ampere). So vLLM cannot run quantized 13B+ here.
• An unquantized 13B in fp16 is ~26 GB → does not fit 16 GB.
• Current vLLM refuses cc < 7.0 outright; even a ~7B fp16 would need a pinned old vLLM + --dtype float16 + VLLM_ATTENTION_BACKEND=XFORMERS, and still no 13B.

Since the user wants a 13B+ class model and only needs an OpenAI-compatible API (confirmed), the right engine is llama.cpp's llama-server: rock-solid Pascal (sm_60) support, GGUF quantization, and a native OpenAI-compatible /v1 API that Hermes consumes unchanged.

Hardware / host facts (verified live 2026-06-26)

• GPU: 1× Tesla P100-PCIE-16GB, cc 6.0, driver 580.159.04, CUDA 13. Idle. (2nd staged P100 not installed — single-GPU design.)
• Docker 29.5.2, NVIDIA Container Toolkit 1.19.1.
• CDI already configured: nvidia.com/gpu=0 is a valid device. No host runtime changes needed — the compose just references the CDI device.
• /storage1 (ZFS, virtiofs): 17 TB free. Weights live here; VM root is only 200 GB.
• Hermes runs on the host (not Docker) from /home/ginnoir/.hermes/:
  • hermes dashboard --host 172.20.0.1 --port 9119 (gateway/agent API+WS, pid 1116)
  • hermes_cli.main gateway run (pid 1119)
  • hermes-webui/server.py on 172.20.0.1:8787 (pid 1128)
  • Bound to 172.20.0.1 = the Docker bridge gateway, so host↔container reachability is trivial: a container published on 172.20.0.1:<port> is reachable by Hermes and by other containers. No Caddy hop in the inference path.

Architecture

Hermes agent/gateway (host, 172.20.0.1)
        │  OpenAI base_url → http://172.20.0.1:8090/v1   (api_key = LLM_API_KEY)
        ▼
llama-server container (stacks/llm)  ──CDI nvidia.com/gpu=0──▶  Tesla P100
   model: Qwen2.5-14B-Instruct-Q4_K_M.gguf
   weights bind-mounted from /storage1/labdata/llm/models

No Caddy endpoint (Hermes-only, per decision). Endpoint is unauthenticated-but-API-keyed and only reachable on the host/bridge — matching the Pattern-B fallback for internal tools.

The stack — stacks/llm/docker-compose.yml

Single service llama-server:

• Image: ghcr.io/ggml-org/llama.cpp:server-cuda at a pinned tag.
  • Infra-pinned per repo convention: label com.centurylabs.watchtower.enable=false so Watchtower won't drift it.
  • Verification gate: confirm the pinned prebuilt image includes Pascal sm_60 kernels and is CUDA ≤ 13 compatible. If it errors on the P100, fall back to a locally built image with -DCMAKE_CUDA_ARCHITECTURES=60.
• GPU: devices: ["nvidia.com/gpu=0"] (CDI).
• Volumes: /storage1/labdata/llm/models:/models (bind).
• Command / args (32k baseline):
  • -m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf
  • -ngl 99 (full offload — 14B Q4 fits in VRAM)
  • --ctx-size 32768 (native context, no rope scaling)
  • -fa (flash attention — required for quantized KV cache; works on Pascal)
  • --cache-type-k q8_0 --cache-type-v q8_0 (KV cache q8_0 ≈ 6 GB at 32k)
  • --host 0.0.0.0 --port 8080
  • --api-key ${LLM_API_KEY}
  • --alias qwen2.5-14b-instruct (stable model name Hermes references)
• Ports: "172.20.0.1:8090:8080" (reachable by Hermes on host + by containers).
• Networks: private llm net only (no edge — no Caddy endpoint this round).
• restart: unless-stopped. Healthcheck: GET /health on 8080.
• env_file: stack.env per repo convention.

stacks/llm/stack.env

• LLM_API_KEY=<generated> (committed per repo policy — secrets are versioned here).

VRAM budget (Qwen2.5-14B, GQA: 48 layers, 8 KV heads, head_dim 128)

• KV cache ≈ 0.375 MiB/token fp16 → q8_0 halves to ≈ 0.1875 MiB/token.
• Weights Q4_K_M ≈ 9.0 GB; reserve ~0.8 GB compute buffers.
• 32k @ q8_0 KV ≈ 6.0 GB → ~15.7 GB total, fits (tight but safe at 16 GB).

Model acquisition (one-time)

Download Qwen2.5-14B-Instruct-Q4_K_M.gguf (~9 GB) from bartowski/Qwen2.5-14B-Instruct-GGUF into /storage1/labdata/llm/models/ on the host (e.g. huggingface-cli download or wget the single GGUF). Documented host-side step, done before first stack deploy.

Hermes integration (host-side, not in git)

Hermes is the Nous Research Hermes agent (hermes-agent.nousresearch.com). Its config is ~/.hermes/config.yaml, which already has a providers: list whose entries are exactly OpenAI-compatible upstreams — there's a working ollama provider in it today (type: openai, base_url: http://192.168.1.73:11434/v1). Adding the P100 is one more entry of the same shape; no new integration surface.

1. Add a provider to providers: in ~/.hermes/config.yaml:

   - name: valhalla-p100
     type: openai
     base_url: http://172.20.0.1:8090/v1
     api_key: <LLM_API_KEY>
     models:
     - qwen2.5-14b-instruct
   

   Use hermes config / the hermes CLI where possible; a direct YAML edit + restart is the fallback (the CLI is the source of truth for _config_version).
2. Select the model as the active one via hermes model (interactive) — or set model.default: qwen2.5-14b-instruct (+ matching provider) if it should be the gateway default rather than a switchable option. The current default is gpt-5.5 / openai-codex; we add ours alongside and let the user choose, rather than silently replacing the default.
3. Restart the three Hermes processes (gateway dashboard pid-class, gateway run, webui) so the new provider/model is live.
4. Verify end-to-end: a Hermes prompt routed to qwen2.5-14b-instruct produces a completion served by the P100 (confirm via nvidia-smi showing the llama-server process holding VRAM during generation).

These host-side steps are documented in the plan (and worth a note in CLAUDE.md known-quirks), not committed as repo changes — Hermes isn't in compose, and the alias qwen2.5-14b-instruct set via --alias is the contract between llama-server and this provider entry.

Deployment

stacks/llm/ is a new Portainer git stack → must be registered once (new stacks aren't auto-created by the poller). Per repo precedent (memory: portainer-new-stack-registration): create via MCP/Portainer, poll StackList to confirm, use the stacks' working fine-grained PAT for git creds. Pure env_file (empty Portainer UI env). After registration, normal git-push → 5-min poll redeploys apply.

Verification

1. docker logs llama-server shows model loaded, all layers offloaded to GPU, server listening on 8080; healthcheck green.
2. nvidia-smi shows the llama-server process holding ~15–16 GB.
3. curl http://172.20.0.1:8090/v1/chat/completions (with API key) returns a completion.
4. Hermes, repointed, produces a completion served locally.
5. 64k stretch (post-baseline): re-run with --ctx-size 65536, YaRN rope-scaling, and --cache-type-k q8_0 --cache-type-v q4_0 (or both q4_0), possibly a smaller weight quant (Q4_K_S/IQ4_XS) for headroom. Drive a long prompt and watch VRAM; keep the largest context that holds without OOM under load. Revert to 32k if 64k is unstable.

Non-goals / out of scope

• vLLM (ruled out by hardware — see decision above).
• SSO/Authentik on the endpoint (LAN/host-only, API-keyed).
• A public llm.ginnoir.com Caddy endpoint (declined this round; easy to add later by joining edge + an internal_only block).
• Multi-GPU / 2nd P100 install.

Risks

• Prebuilt image may lack sm_60 → RESOLVED (verified live 2026-06-26). ghcr.io/ggml-org/llama.cpp:server-cuda (digest sha256:ce294a4561e6…f0f2a9) loaded Qwen2.5-0.5B-Instruct-Q4_K_M with -ngl 99 on the P100: nvidia-smi showed the server process holding ~1.1 GB of GPU VRAM, model loaded, server listening, no arch/assert errors. Prebuilt image works on Pascal — no source build needed.
• 32k @ q8_0 is tight (~15.7 GB) → if compute buffers push it over, drop V cache to q4_0 or context to 24k.
• Pascal -fa performance → flash-attn works on Pascal but is slower than on Volta+; acceptable for single/few-user agent use, measured during verification.