homelabstack/docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md

# LLM inference backend for Hermes (valhalla) — design

**Date:** 2026-06-26
**Status:** Approved (pending spec review)
**Stack:** new `stacks/llm/`

## Goal

Stand up a local, OpenAI-compatible LLM inference endpoint on valhalla and point the
existing **Hermes** agent backend at it, so Hermes runs against a self-hosted 14B model
instead of an external provider.

## Key decision: llama.cpp, not vLLM

The request was "set up vLLM," but valhalla's GPU is a **Tesla P100-PCIE-16GB**, die
**GP100, compute capability 6.0**. That rules out vLLM for the desired model class:

- The GP100 (cc 6.0) lacks the **DP4A INT8** instructions that vLLM's AWQ/GPTQ kernels
  require (those need Turing 7.5 / Ampere). So vLLM **cannot run quantized 13B+** here.
- An unquantized 13B in fp16 is ~26 GB → does not fit 16 GB.
- Current vLLM refuses cc < 7.0 outright; even a ~7B fp16 would need a pinned old vLLM +
  `--dtype float16` + `VLLM_ATTENTION_BACKEND=XFORMERS`, and still no 13B.

Since the user wants a **13B+ class model** and only needs an **OpenAI-compatible** API
(confirmed), the right engine is **llama.cpp's `llama-server`**: rock-solid Pascal (sm_60)
support, GGUF quantization, and a native OpenAI-compatible `/v1` API that Hermes consumes
unchanged.

## Hardware / host facts (verified live 2026-06-26)

- GPU: 1× Tesla P100-PCIE-16GB, cc 6.0, driver 580.159.04, CUDA 13. Idle. (2nd staged P100
  not installed — single-GPU design.)
- Docker 29.5.2, NVIDIA Container Toolkit 1.19.1.
- **CDI already configured:** `nvidia.com/gpu=0` is a valid device. No host runtime changes
  needed — the compose just references the CDI device.
- `/storage1` (ZFS, virtiofs): 17 TB free. Weights live here; VM root is only 200 GB.
- **Hermes** runs on the host (not Docker) from `/home/ginnoir/.hermes/`:
  - `hermes dashboard --host 172.20.0.1 --port 9119` (gateway/agent API+WS, pid 1116)
  - `hermes_cli.main gateway run` (pid 1119)
  - `hermes-webui/server.py` on `172.20.0.1:8787` (pid 1128)
  - Bound to `172.20.0.1` = the Docker bridge gateway, so host↔container reachability is
    trivial: a container published on `172.20.0.1:<port>` is reachable by Hermes and by
    other containers. No Caddy hop in the inference path.

## Architecture

```
Hermes agent/gateway (host, 172.20.0.1)
        │  OpenAI base_url → http://172.20.0.1:8090/v1   (api_key = LLM_API_KEY)
        ▼
llama-server container (stacks/llm)  ──CDI nvidia.com/gpu=0──▶  Tesla P100
   model: Qwen2.5-14B-Instruct-Q4_K_M.gguf
   weights bind-mounted from /storage1/labdata/llm/models
```

No Caddy endpoint (Hermes-only, per decision). Endpoint is unauthenticated-but-API-keyed
and only reachable on the host/bridge — matching the Pattern-B fallback for internal tools.

## The stack — `stacks/llm/docker-compose.yml`

Single service `llama-server`:

- **Image:** `ghcr.io/ggml-org/llama.cpp:server-cuda` at a pinned tag.
  - Infra-pinned per repo convention: label `com.centurylabs.watchtower.enable=false` so
    Watchtower won't drift it.
  - **Verification gate:** confirm the pinned prebuilt image includes Pascal `sm_60`
    kernels and is CUDA ≤ 13 compatible. If it errors on the P100, fall back to a locally
    built image with `-DCMAKE_CUDA_ARCHITECTURES=60`.
- **GPU:** `devices: ["nvidia.com/gpu=0"]` (CDI).
- **Volumes:** `/storage1/labdata/llm/models:/models` (bind).
- **Command / args (32k baseline):**
  - `-m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf`
  - `-ngl 99` (full offload — 14B Q4 fits in VRAM)
  - `--ctx-size 32768` (native context, no rope scaling)
  - `-fa` (flash attention — required for quantized KV cache; works on Pascal)
  - `--cache-type-k q8_0 --cache-type-v q8_0` (KV cache q8_0 ≈ 6 GB at 32k)
  - `--host 0.0.0.0 --port 8080`
  - `--api-key ${LLM_API_KEY}`
  - `--alias qwen2.5-14b-instruct` (stable model name Hermes references)
- **Ports:** `"172.20.0.1:8090:8080"` (reachable by Hermes on host + by containers).
- **Networks:** private `llm` net only (no `edge` — no Caddy endpoint this round).
- **restart:** `unless-stopped`. **Healthcheck:** GET `/health` on 8080.
- **`env_file: stack.env`** per repo convention.

### `stacks/llm/stack.env`

- `LLM_API_KEY=<generated>` (committed per repo policy — secrets are versioned here).

### VRAM budget (Qwen2.5-14B, GQA: 48 layers, 8 KV heads, head_dim 128)

- KV cache ≈ 0.375 MiB/token fp16 → **q8_0 halves to ≈ 0.1875 MiB/token**.
- Weights Q4_K_M ≈ 9.0 GB; reserve ~0.8 GB compute buffers.
- 32k @ q8_0 KV ≈ 6.0 GB → **~15.7 GB total, fits** (tight but safe at 16 GB).

## Model acquisition (one-time)

Download `Qwen2.5-14B-Instruct-Q4_K_M.gguf` (~9 GB) from
`bartowski/Qwen2.5-14B-Instruct-GGUF` into `/storage1/labdata/llm/models/` on the host
(e.g. `huggingface-cli download` or `wget` the single GGUF). Documented host-side step,
done before first stack deploy.

## Hermes integration (host-side, not in git)

Hermes is the **Nous Research Hermes agent** (`hermes-agent.nousresearch.com`). Its config
is `~/.hermes/config.yaml`, which already has a `providers:` list whose entries are exactly
OpenAI-compatible upstreams — there's a working `ollama` provider in it today
(`type: openai`, `base_url: http://192.168.1.73:11434/v1`). Adding the P100 is one more
entry of the same shape; no new integration surface.

1. **Add a provider** to `providers:` in `~/.hermes/config.yaml`:
   ```yaml
   - name: valhalla-p100
     type: openai
     base_url: http://172.20.0.1:8090/v1
     api_key: <LLM_API_KEY>
     models:
     - qwen2.5-14b-instruct
   ```
   Use `hermes config` / the `hermes` CLI where possible; a direct YAML edit + restart is
   the fallback (the CLI is the source of truth for `_config_version`).
2. **Select the model** as the active one via `hermes model` (interactive) — or set
   `model.default: qwen2.5-14b-instruct` (+ matching provider) if it should be the gateway
   default rather than a switchable option. The current default is `gpt-5.5` /
   `openai-codex`; we add ours alongside and let the user choose, rather than silently
   replacing the default.
3. **Restart** the three Hermes processes (gateway dashboard pid-class, `gateway run`,
   webui) so the new provider/model is live.
4. **Verify** end-to-end: a Hermes prompt routed to `qwen2.5-14b-instruct` produces a
   completion served by the P100 (confirm via `nvidia-smi` showing the llama-server process
   holding VRAM during generation).

These host-side steps are documented in the plan (and worth a note in CLAUDE.md
known-quirks), not committed as repo changes — Hermes isn't in compose, and the alias
`qwen2.5-14b-instruct` set via `--alias` is the contract between llama-server and this
provider entry.

## Deployment

`stacks/llm/` is a new Portainer git stack → must be **registered once** (new stacks aren't
auto-created by the poller). Per repo precedent (memory: portainer-new-stack-registration):
create via MCP/Portainer, poll `StackList` to confirm, use the stacks' working fine-grained
PAT for git creds. Pure `env_file` (empty Portainer UI env). After registration, normal
git-push → 5-min poll redeploys apply.

## Verification

1. `docker logs llama-server` shows model loaded, all layers offloaded to GPU, server
   listening on 8080; healthcheck green.
2. `nvidia-smi` shows the llama-server process holding ~15–16 GB.
3. `curl http://172.20.0.1:8090/v1/chat/completions` (with API key) returns a completion.
4. Hermes, repointed, produces a completion served locally.
5. **64k stretch (post-baseline):** re-run with `--ctx-size 65536`, YaRN rope-scaling, and
   `--cache-type-k q8_0 --cache-type-v q4_0` (or both q4_0), possibly a smaller weight quant
   (Q4_K_S/IQ4_XS) for headroom. Drive a long prompt and watch VRAM; keep the largest
   context that holds without OOM under load. Revert to 32k if 64k is unstable.

## Non-goals / out of scope

- vLLM (ruled out by hardware — see decision above).
- SSO/Authentik on the endpoint (LAN/host-only, API-keyed).
- A public `llm.ginnoir.com` Caddy endpoint (declined this round; easy to add later by
  joining `edge` + an `internal_only` block).
- Multi-GPU / 2nd P100 install.

## Risks

- ~~**Prebuilt image may lack sm_60**~~ → **RESOLVED (verified live 2026-06-26).**
  `ghcr.io/ggml-org/llama.cpp:server-cuda` (digest
  `sha256:ce294a4561e6…f0f2a9`) loaded Qwen2.5-0.5B-Instruct-Q4_K_M with `-ngl 99` on the
  P100: `nvidia-smi` showed the server process holding ~1.1 GB of GPU VRAM, model loaded,
  server listening, no arch/assert errors. Prebuilt image works on Pascal — no source build
  needed.
- **32k @ q8_0 is tight (~15.7 GB)** → if compute buffers push it over, drop V cache to
  q4_0 or context to 24k.
- **Pascal `-fa` performance** → flash-attn works on Pascal but is slower than on
  Volta+; acceptable for single/few-user agent use, measured during verification.