feat(llm): add llama.cpp inference stack for Hermes (Qwen2.5-14B on P100)

New stacks/llm/ serves Qwen2.5-14B-Instruct (Q4_K_M GGUF) via llama.cpp's OpenAI-compatible server on the Tesla P100 (CDI nvidia.com/gpu=0), published on 172.20.0.1:8090 for the host-side Hermes agent. vLLM was rejected: the P100 (cc 6.0) lacks the DP4A INT8 instructions its AWQ/GPTQ kernels need. Includes design spec and implementation plan under docs/superpowers/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 16:22:57 -05:00
parent 847edff1f8
commit 8e7682985d
4 changed files with 625 additions and 0 deletions
@@ -0,0 +1,385 @@
+# LLM inference backend for Hermes — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Stand up a llama.cpp OpenAI-compatible inference server in a new `stacks/llm/` Portainer stack, serving Qwen2.5-14B-Instruct on valhalla's Tesla P100, and add it as a provider in the Hermes agent.
+
+**Architecture:** Single `llama-server` container (image `ghcr.io/ggml-org/llama.cpp:server-cuda`, verified to run on the P100's sm_60) gets the GPU via CDI (`nvidia.com/gpu=0`), loads a GGUF from `/storage1/labdata/llm/models`, and publishes its OpenAI `/v1` API on the host at `172.20.0.1:8090`. Host-side Hermes (systemd-managed) reaches it directly — no Caddy. Config follows repo conventions: pure `env_file`, no `${VAR}` interpolation, infra image pinned out of Watchtower.
+
+**Tech Stack:** Docker Compose (Portainer git stack), llama.cpp server, CUDA/CDI, Gitea-polled deploy, Hermes (Nous Research agent) YAML config.
+
+**Reference spec:** `docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md`
+
+---
+
+## Pre-verified facts (do not re-derive)
+
+- GPU: Tesla P100-PCIE-16GB, cc 6.0; CDI device `nvidia.com/gpu=0` valid. Prebuilt image runs with full GPU offload (tested live 2026-06-26).
+- `/storage1/labdata` is root-owned; `ginnoir` has **passwordless sudo**.
+- `edge` network gateway = `172.20.0.1` (host IP on `br-b5aa55c3fedf`). Hermes binds here.
+- Image contains `curl` and `bash`.
+- Hermes services: `hermes-dashboard.service`, `hermes-gateway.service`, `hermes-webui.service` (system systemd). Config: `~/.hermes/config.yaml` with a `providers:` list (existing `ollama` entry as a template).
+- Repo deploy: app stacks deploy via git push → Portainer polls Gitea every 5 min. **New** stacks must be registered once in Portainer (see memory `portainer-new-stack-registration`).
+- `.gitattributes` forces LF — ensure `stack.env` / compose are LF on commit.
+
+## File structure
+
+- **Create** `stacks/llm/docker-compose.yml` — the llama-server service (one responsibility: serve the model on the GPU).
+- **Create** `stacks/llm/stack.env` — `LLAMA_API_KEY` only (committed per repo policy).
+- **Host-side (not in repo):** `/storage1/labdata/llm/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf`; one new `providers:` entry in `~/.hermes/config.yaml`.
+
+---
+
+### Task 1: Pre-stage the model on the host (must precede deploy)
+
+The container crash-loops if the GGUF is absent, so download it before Portainer deploys the stack.
+
+**Files:** none in repo (host filesystem only).
+
+- [ ] **Step 1: Create the model directory**
+
+Run:
+```bash
+ssh ginnoir@valhalla "sudo mkdir -p /storage1/labdata/llm/models && sudo ls -ld /storage1/labdata/llm/models"
+```
+Expected: directory exists.
+
+- [ ] **Step 2: Download Qwen2.5-14B-Instruct Q4_K_M (~9 GB)**
+
+Run:
+```bash
+ssh ginnoir@valhalla "cd /storage1/labdata/llm/models && sudo curl -fL -o Qwen2.5-14B-Instruct-Q4_K_M.gguf https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-GGUF/resolve/main/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
+```
+(Run in background if it's slow; it's a single ~9 GB file.)
+
+- [ ] **Step 3: Verify the download**
+
+Run:
+```bash
+ssh ginnoir@valhalla "sudo ls -lh /storage1/labdata/llm/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
+```
+Expected: file ~8.9–9.0 GB. If the size is wildly off (e.g. a few KB), it's an HTML error page — re-download.
+
+---
+
+### Task 2: Create `stacks/llm/stack.env`
+
+**Files:**
+- Create: `stacks/llm/stack.env`
+
+- [ ] **Step 1: Generate an API key**
+
+Run:
+```bash
+openssl rand -hex 32
+```
+Copy the output for the next step.
+
+- [ ] **Step 2: Write the file** (replace `<HEX>` with the generated key)
+
+`stacks/llm/stack.env`:
+```dotenv
+# llm stack secrets — read directly by the container via env_file.
+# llama.cpp's server reads LLAMA_API_KEY from the environment (no --api-key flag,
+# no ${VAR} interpolation), matching the repo's pure-env_file convention.
+LLAMA_API_KEY=<HEX>
+```
+
+- [ ] **Step 3: Confirm LF line endings**
+
+Run:
+```bash
+git check-attr text eol -- stacks/llm/stack.env
+```
+Expected: `eol: lf` (enforced by `.gitattributes`).
+
+---
+
+### Task 3: Create `stacks/llm/docker-compose.yml`
+
+**Files:**
+- Create: `stacks/llm/docker-compose.yml`
+
+- [ ] **Step 1: Write the compose file**
+
+`stacks/llm/docker-compose.yml`:
+```yaml
+# llm stack — local LLM inference backend for the Hermes agent.
+#
+# Single service: llama.cpp's OpenAI-compatible server (llama-server) serving
+# Qwen2.5-14B-Instruct (Q4_K_M GGUF) on the host's Tesla P100-16GB via CDI.
+# Chosen over vLLM because the P100 (GP100, compute capability 6.0) lacks the
+# DP4A INT8 instructions vLLM's AWQ/GPTQ kernels require — see
+# docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md.
+#
+# Pure env_file (LLAMA_API_KEY) — no Portainer UI env, no ${VAR} interpolation.
+# Image is infra-pinned out of Watchtower (manual tag bumps only).
+#
+# The server's OpenAI API is published on the host at 172.20.0.1:8090 (the edge
+# bridge gateway, a local host IP). Host-side Hermes reaches it there directly;
+# no Caddy block this round. Model weights live on the ZFS tier; the
+# /storage1/labdata/llm/models dir is pre-created with the GGUF before deploy.
+services:
+  llama-server:
+    image: ghcr.io/ggml-org/llama.cpp:server-cuda
+    container_name: llama-server
+    restart: unless-stopped
+    labels:
+      - "com.centurylabs.watchtower.enable=false"
+    networks: [llm]
+    env_file:
+      - stack.env
+    devices:
+      - "nvidia.com/gpu=0"
+    volumes:
+      - /storage1/labdata/llm/models:/models
+    command:
+      - "-m"
+      - "/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
+      - "--alias"
+      - "qwen2.5-14b-instruct"
+      - "-ngl"
+      - "99"
+      - "--ctx-size"
+      - "32768"
+      - "-fa"
+      - "--cache-type-k"
+      - "q8_0"
+      - "--cache-type-v"
+      - "q8_0"
+      - "--host"
+      - "0.0.0.0"
+      - "--port"
+      - "8080"
+    ports:
+      - "172.20.0.1:8090:8080"
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
+      start_period: 180s
+
+networks:
+  llm:
+    name: llm
+    driver: bridge
+```
+
+- [ ] **Step 2: Validate compose syntax locally**
+
+Run:
+```bash
+docker compose -f stacks/llm/docker-compose.yml config -q
+```
+Expected: no output (valid). If it errors on the CDI `devices` entry, that's a local-CLI version quirk, not a deploy blocker — the daemon on valhalla (Docker 29.5.2) supports it; proceed.
+
+---
+
+### Task 4: Commit and push the stack to Gitea
+
+**Files:** none new (commits Tasks 2–3).
+
+- [ ] **Step 1: Stage and commit**
+
+```bash
+git add stacks/llm/docker-compose.yml stacks/llm/stack.env docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md docs/superpowers/plans/2026-06-26-llm-backend-hermes.md
+git commit -m "feat(llm): add llama.cpp inference stack for Hermes (Qwen2.5-14B on P100)"
+```
+
+- [ ] **Step 2: Push (deploys nothing yet — stack isn't registered)**
+
+Use the **homelab-apply** skill's push path (push to Gitea; GitHub is the mirror). A `stacks/llm/*` change only redeploys once the stack is registered (Task 5).
+
+Run (per repo convention — token via one-off http.extraheader, never in git config):
+```bash
+git push   # to the configured remote(s); Gitea is primary
+```
+Expected: push succeeds; Portainer cannot yet act on `stacks/llm` because no stack references it.
+
+---
+
+### Task 5: Register the new Portainer git stack (one-time)
+
+New stacks aren't auto-created by polling — register once, then future pushes redeploy. See memory `portainer-new-stack-registration`.
+
+**Files:** none (Portainer state).
+
+- [ ] **Step 1: Read an existing git stack's config to copy repo URL + credential reference**
+
+Use the portainer MCP (invoke `get_guidance` first per the portainer-mcp-hygiene skill). Inspect a working app stack (e.g. `roms`) to copy the exact Gitea repo URL, ref (`refs/heads/main`), and the working fine-grained PAT/credential the other stacks use:
+```
+mcp__portainer__StackList  (select: name, GitConfig)
+mcp__portainer__StackInspect  on the roms stack id
+```
+
+- [ ] **Step 2: Create the stack from the git repository**
+
+Create a Docker standalone stack from the Gitea repo with:
+- compose path: `stacks/llm/docker-compose.yml`
+- ref: `refs/heads/main`
+- auto-update / git polling: **on** (match other app stacks)
+- env: **empty** (pure env_file)
+- credentials: the same working fine-grained PAT the other stacks use (the runner PAT cannot clone)
+
+Use `mcp__portainer__StackCreateDockerStandaloneRepository`. Per the memory note, the MCP call may time out but still succeed.
+
+- [ ] **Step 3: Verify the stack registered and deployed**
+
+Poll:
+```
+mcp__portainer__StackList  (select: [].{name:Name,status:Status})
+```
+Expected: a `llm` stack appears. Then confirm the container is running:
+```bash
+ssh ginnoir@valhalla "docker ps --filter name=llama-server --format '{{.Names}} {{.Status}}'"
+```
+Expected: `llama-server  Up … (health: starting|healthy)`.
+
+---
+
+### Task 6: Verify deploy — health, GPU offload, OpenAI API
+
+**Files:** none.
+
+- [ ] **Step 1: Confirm the model loaded on the GPU**
+
+Run:
+```bash
+ssh ginnoir@valhalla "docker logs --tail 60 llama-server 2>&1 | grep -iE 'P100|model loaded|listening|error|assert|cache_type|n_ctx'"
+```
+Expected: `Tesla P100`, `model loaded`, `server is listening`, no asserts. If logs show a `-fa` parse error, edit the compose to replace `-fa` with `--flash-attn` + `on` (two list items), re-commit/push, and let it redeploy.
+
+- [ ] **Step 2: Confirm VRAM is held (real offload, not CPU)**
+
+Run:
+```bash
+ssh ginnoir@valhalla "nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader; nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader"
+```
+Expected: a `llama-server`-owned process holding ~14–16 GB; free memory small. If `memory.used` is near 0 and the model is in RAM, GPU offload failed — recheck the CDI `devices` entry deployed correctly (`docker inspect llama-server --format '{{json .HostConfig.Devices}}{{json .HostConfig.DeviceRequests}}'`).
+
+- [ ] **Step 3: Confirm healthcheck is green**
+
+Run:
+```bash
+ssh ginnoir@valhalla "docker inspect llama-server --format '{{.State.Health.Status}}'"
+```
+Expected: `healthy` (allow up to `start_period` = 3 min).
+
+- [ ] **Step 4: Exercise the OpenAI endpoint from the host (as Hermes will)**
+
+Run (substitute the real key from `stacks/llm/stack.env`):
+```bash
+ssh ginnoir@valhalla "curl -fsS http://172.20.0.1:8090/v1/chat/completions -H 'Authorization: Bearer <LLAMA_API_KEY>' -H 'Content-Type: application/json' -d '{\"model\":\"qwen2.5-14b-instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: OK\"}],\"max_tokens\":8}'"
+```
+Expected: a JSON chat completion containing `OK`. A 401 means the key is wrong; a connection refused means the port publish/bind is wrong.
+
+---
+
+### Task 7: Stretch to 64k context (live tuning)
+
+Attempt the larger context now that the baseline works; keep it only if VRAM holds under load.
+
+**Files:**
+- Modify: `stacks/llm/docker-compose.yml` (only if 64k holds)
+
+- [ ] **Step 1: Try 64k with YaRN + lighter V cache, ephemerally**
+
+Run a throwaway container (doesn't touch the deployed stack), driving a long context:
+```bash
+ssh ginnoir@valhalla "docker run --rm --device nvidia.com/gpu=0 -v /storage1/labdata/llm/models:/models -p 172.20.0.1:8091:8080 ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf --alias q -ngl 99 --ctx-size 65536 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768 -fa --cache-type-k q8_0 --cache-type-v q4_0 --host 0.0.0.0 --port 8080 > /tmp/llm_64k.log 2>&1 & sleep 60; nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader; grep -iE 'model loaded|error|assert|out of memory|failed to allocate' /tmp/llm_64k.log | head; docker ps --filter publish=8091 -q | xargs -r docker rm -f"
+```
+Expected to decide: if `model loaded` with `memory.free` > ~300 MiB and no allocation failures, 64k is viable. If it OOMs / fails to allocate, 64k at this quant doesn't fit — keep 32k (stop here, leave the committed config as-is).
+
+- [ ] **Step 2 (only if 64k held): promote the 64k args into the stack**
+
+Edit `stacks/llm/docker-compose.yml` `command:` to: `--ctx-size 65536`, add `--rope-scaling yarn`, `--rope-scale 2`, `--yarn-orig-ctx 32768`, and set `--cache-type-v q4_0` (keep `--cache-type-k q8_0`). Then:
+```bash
+git add stacks/llm/docker-compose.yml
+git commit -m "feat(llm): raise llama-server context to 64k (YaRN + q4 V-cache)"
+git push
+```
+Let Portainer redeploy, then re-run Task 6 Steps 2–4. If the live 14B OOMs under a real long prompt, revert this commit.
+
+---
+
+### Task 8: Wire the provider into Hermes and verify end-to-end
+
+**Files:** host-side `~/.hermes/config.yaml` (not in repo).
+
+- [ ] **Step 1: Back up the Hermes config**
+
+Run:
+```bash
+ssh ginnoir@valhalla "cp ~/.hermes/config.yaml ~/.hermes/config.yaml.bak.$(date +%s) && ls -l ~/.hermes/config.yaml.bak.*"
+```
+
+- [ ] **Step 2: Add the provider entry under `providers:`**
+
+Append this entry to the `providers:` list in `~/.hermes/config.yaml` (same shape as the existing `ollama` entry; substitute the real key):
+```yaml
+- name: valhalla-p100
+  type: openai
+  base_url: http://172.20.0.1:8090/v1
+  api_key: <LLAMA_API_KEY>
+  models:
+  - qwen2.5-14b-instruct
+```
+Edit by reading the file, inserting the entry, and writing it back (preserve indentation exactly). Do **not** change the `model:` default block — we add the provider alongside the current default rather than silently replacing it (per spec).
+
+- [ ] **Step 3: Restart Hermes**
+
+Run:
+```bash
+ssh ginnoir@valhalla "sudo systemctl restart hermes-dashboard hermes-gateway hermes-webui && sleep 5 && systemctl is-active hermes-dashboard hermes-gateway hermes-webui"
+```
+Expected: three `active` lines. If any failed, check `journalctl -u hermes-gateway -n 50` — a YAML error means the edit broke indentation; restore the backup and retry.
+
+- [ ] **Step 4: Confirm Hermes sees the model and routes to the P100**
+
+Run a one-shot prompt forcing the new provider/model:
+```bash
+ssh ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes -z 'Reply with exactly: HELLO FROM P100' -m qwen2.5-14b-instruct --provider valhalla-p100 2>&1 | tail -20"
+```
+Expected: a completion containing the phrase. Simultaneously, `nvidia-smi` (separate shell) should show llama-server utilization spike during generation.
+
+- [ ] **Step 5: (Optional) make it the default**
+
+If ginnoir wants the P100 model as Hermes' default rather than a per-call choice, run interactively:
+```bash
+ssh -t ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes model"
+```
+and select `valhalla-p100` / `qwen2.5-14b-instruct`. Leave the default unchanged otherwise.
+
+---
+
+### Task 9: Cleanup and documentation
+
+**Files:** possibly `CLAUDE.md` (known-quirks note).
+
+- [ ] **Step 1: Remove the tiny test model**
+
+Run:
+```bash
+ssh ginnoir@valhalla "sudo rm -f /storage1/labdata/llm/models/qwen2.5-0.5b-instruct-q4_k_m.gguf && sudo ls /storage1/labdata/llm/models"
+```
+Expected: only the 14B GGUF remains.
+
+- [ ] **Step 2: Remove the config backup once verified (optional)**
+
+```bash
+ssh ginnoir@valhalla "ls ~/.hermes/config.yaml.bak.*"
+```
+Keep the most recent backup until the setup is confirmed stable, then remove.
+
+- [ ] **Step 3: Add a known-quirks note (optional, if desired)**
+
+Add a short bullet to `CLAUDE.md` under "External services" / "Known quirks": the `llm` stack serves Qwen2.5-14B on the P100 via llama.cpp; Hermes points at it via the `valhalla-p100` provider in `~/.hermes/config.yaml`; vLLM was rejected due to the P100's cc 6.0. Commit if added.
+
+---
+
+## Self-review notes
+
+- **Spec coverage:** engine (Task 3), model + storage (Tasks 1, 3), 32k baseline + q8 KV (Task 3), 64k stretch (Task 7), CDI GPU (Task 3, verified Task 6), `172.20.0.1:8090` publish (Task 3, verified Task 6), Hermes provider entry (Task 8), new-stack registration (Task 5), pure env_file / no `${VAR}` (Tasks 2–3), Watchtower pin (Task 3), no Caddy/no SSO (by omission), tiny-model cleanup (Task 9). All covered.
+- **No placeholders:** the only `<...>` tokens are the generated API key and (in Task 5) the repo URL/credential copied from an existing stack — both are runtime secrets/values, not undefined behavior.
+- **Consistency:** the alias `qwen2.5-14b-instruct` is the single contract used by the compose `--alias`, the curl test, and the Hermes provider `models:` / `-m` flag throughout.
@@ -0,0 +1,176 @@
+# LLM inference backend for Hermes (valhalla) — design
+
+**Date:** 2026-06-26
+**Status:** Approved (pending spec review)
+**Stack:** new `stacks/llm/`
+
+## Goal
+
+Stand up a local, OpenAI-compatible LLM inference endpoint on valhalla and point the
+existing **Hermes** agent backend at it, so Hermes runs against a self-hosted 14B model
+instead of an external provider.
+
+## Key decision: llama.cpp, not vLLM
+
+The request was "set up vLLM," but valhalla's GPU is a **Tesla P100-PCIE-16GB**, die
+**GP100, compute capability 6.0**. That rules out vLLM for the desired model class:
+
+- The GP100 (cc 6.0) lacks the **DP4A INT8** instructions that vLLM's AWQ/GPTQ kernels
+  require (those need Turing 7.5 / Ampere). So vLLM **cannot run quantized 13B+** here.
+- An unquantized 13B in fp16 is ~26 GB → does not fit 16 GB.
+- Current vLLM refuses cc < 7.0 outright; even a ~7B fp16 would need a pinned old vLLM +
+  `--dtype float16` + `VLLM_ATTENTION_BACKEND=XFORMERS`, and still no 13B.
+
+Since the user wants a **13B+ class model** and only needs an **OpenAI-compatible** API
+(confirmed), the right engine is **llama.cpp's `llama-server`**: rock-solid Pascal (sm_60)
+support, GGUF quantization, and a native OpenAI-compatible `/v1` API that Hermes consumes
+unchanged.
+
+## Hardware / host facts (verified live 2026-06-26)
+
+- GPU: 1× Tesla P100-PCIE-16GB, cc 6.0, driver 580.159.04, CUDA 13. Idle. (2nd staged P100
+  not installed — single-GPU design.)
+- Docker 29.5.2, NVIDIA Container Toolkit 1.19.1.
+- **CDI already configured:** `nvidia.com/gpu=0` is a valid device. No host runtime changes
+  needed — the compose just references the CDI device.
+- `/storage1` (ZFS, virtiofs): 17 TB free. Weights live here; VM root is only 200 GB.
+- **Hermes** runs on the host (not Docker) from `/home/ginnoir/.hermes/`:
+  - `hermes dashboard --host 172.20.0.1 --port 9119` (gateway/agent API+WS, pid 1116)
+  - `hermes_cli.main gateway run` (pid 1119)
+  - `hermes-webui/server.py` on `172.20.0.1:8787` (pid 1128)
+  - Bound to `172.20.0.1` = the Docker bridge gateway, so host↔container reachability is
+    trivial: a container published on `172.20.0.1:<port>` is reachable by Hermes and by
+    other containers. No Caddy hop in the inference path.
+
+## Architecture
+
+```
+Hermes agent/gateway (host, 172.20.0.1)
+        │  OpenAI base_url → http://172.20.0.1:8090/v1   (api_key = LLM_API_KEY)
+        ▼
+llama-server container (stacks/llm)  ──CDI nvidia.com/gpu=0──▶  Tesla P100
+   model: Qwen2.5-14B-Instruct-Q4_K_M.gguf
+   weights bind-mounted from /storage1/labdata/llm/models
+```
+
+No Caddy endpoint (Hermes-only, per decision). Endpoint is unauthenticated-but-API-keyed
+and only reachable on the host/bridge — matching the Pattern-B fallback for internal tools.
+
+## The stack — `stacks/llm/docker-compose.yml`
+
+Single service `llama-server`:
+
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-cuda` at a pinned tag.
+  - Infra-pinned per repo convention: label `com.centurylabs.watchtower.enable=false` so
+    Watchtower won't drift it.
+  - **Verification gate:** confirm the pinned prebuilt image includes Pascal `sm_60`
+    kernels and is CUDA ≤ 13 compatible. If it errors on the P100, fall back to a locally
+    built image with `-DCMAKE_CUDA_ARCHITECTURES=60`.
+- **GPU:** `devices: ["nvidia.com/gpu=0"]` (CDI).
+- **Volumes:** `/storage1/labdata/llm/models:/models` (bind).
+- **Command / args (32k baseline):**
+  - `-m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf`
+  - `-ngl 99` (full offload — 14B Q4 fits in VRAM)
+  - `--ctx-size 32768` (native context, no rope scaling)
+  - `-fa` (flash attention — required for quantized KV cache; works on Pascal)
+  - `--cache-type-k q8_0 --cache-type-v q8_0` (KV cache q8_0 ≈ 6 GB at 32k)
+  - `--host 0.0.0.0 --port 8080`
+  - `--api-key ${LLM_API_KEY}`
+  - `--alias qwen2.5-14b-instruct` (stable model name Hermes references)
+- **Ports:** `"172.20.0.1:8090:8080"` (reachable by Hermes on host + by containers).
+- **Networks:** private `llm` net only (no `edge` — no Caddy endpoint this round).
+- **restart:** `unless-stopped`. **Healthcheck:** GET `/health` on 8080.
+- **`env_file: stack.env`** per repo convention.
+
+### `stacks/llm/stack.env`
+
+- `LLM_API_KEY=<generated>` (committed per repo policy — secrets are versioned here).
+
+### VRAM budget (Qwen2.5-14B, GQA: 48 layers, 8 KV heads, head_dim 128)
+
+- KV cache ≈ 0.375 MiB/token fp16 → **q8_0 halves to ≈ 0.1875 MiB/token**.
+- Weights Q4_K_M ≈ 9.0 GB; reserve ~0.8 GB compute buffers.
+- 32k @ q8_0 KV ≈ 6.0 GB → **~15.7 GB total, fits** (tight but safe at 16 GB).
+
+## Model acquisition (one-time)
+
+Download `Qwen2.5-14B-Instruct-Q4_K_M.gguf` (~9 GB) from
+`bartowski/Qwen2.5-14B-Instruct-GGUF` into `/storage1/labdata/llm/models/` on the host
+(e.g. `huggingface-cli download` or `wget` the single GGUF). Documented host-side step,
+done before first stack deploy.
+
+## Hermes integration (host-side, not in git)
+
+Hermes is the **Nous Research Hermes agent** (`hermes-agent.nousresearch.com`). Its config
+is `~/.hermes/config.yaml`, which already has a `providers:` list whose entries are exactly
+OpenAI-compatible upstreams — there's a working `ollama` provider in it today
+(`type: openai`, `base_url: http://192.168.1.73:11434/v1`). Adding the P100 is one more
+entry of the same shape; no new integration surface.
+
+1. **Add a provider** to `providers:` in `~/.hermes/config.yaml`:
+   ```yaml
+   - name: valhalla-p100
+     type: openai
+     base_url: http://172.20.0.1:8090/v1
+     api_key: <LLM_API_KEY>
+     models:
+     - qwen2.5-14b-instruct
+   ```
+   Use `hermes config` / the `hermes` CLI where possible; a direct YAML edit + restart is
+   the fallback (the CLI is the source of truth for `_config_version`).
+2. **Select the model** as the active one via `hermes model` (interactive) — or set
+   `model.default: qwen2.5-14b-instruct` (+ matching provider) if it should be the gateway
+   default rather than a switchable option. The current default is `gpt-5.5` /
+   `openai-codex`; we add ours alongside and let the user choose, rather than silently
+   replacing the default.
+3. **Restart** the three Hermes processes (gateway dashboard pid-class, `gateway run`,
+   webui) so the new provider/model is live.
+4. **Verify** end-to-end: a Hermes prompt routed to `qwen2.5-14b-instruct` produces a
+   completion served by the P100 (confirm via `nvidia-smi` showing the llama-server process
+   holding VRAM during generation).
+
+These host-side steps are documented in the plan (and worth a note in CLAUDE.md
+known-quirks), not committed as repo changes — Hermes isn't in compose, and the alias
+`qwen2.5-14b-instruct` set via `--alias` is the contract between llama-server and this
+provider entry.
+
+## Deployment
+
+`stacks/llm/` is a new Portainer git stack → must be **registered once** (new stacks aren't
+auto-created by the poller). Per repo precedent (memory: portainer-new-stack-registration):
+create via MCP/Portainer, poll `StackList` to confirm, use the stacks' working fine-grained
+PAT for git creds. Pure `env_file` (empty Portainer UI env). After registration, normal
+git-push → 5-min poll redeploys apply.
+
+## Verification
+
+1. `docker logs llama-server` shows model loaded, all layers offloaded to GPU, server
+   listening on 8080; healthcheck green.
+2. `nvidia-smi` shows the llama-server process holding ~15–16 GB.
+3. `curl http://172.20.0.1:8090/v1/chat/completions` (with API key) returns a completion.
+4. Hermes, repointed, produces a completion served locally.
+5. **64k stretch (post-baseline):** re-run with `--ctx-size 65536`, YaRN rope-scaling, and
+   `--cache-type-k q8_0 --cache-type-v q4_0` (or both q4_0), possibly a smaller weight quant
+   (Q4_K_S/IQ4_XS) for headroom. Drive a long prompt and watch VRAM; keep the largest
+   context that holds without OOM under load. Revert to 32k if 64k is unstable.
+
+## Non-goals / out of scope
+
+- vLLM (ruled out by hardware — see decision above).
+- SSO/Authentik on the endpoint (LAN/host-only, API-keyed).
+- A public `llm.ginnoir.com` Caddy endpoint (declined this round; easy to add later by
+  joining `edge` + an `internal_only` block).
+- Multi-GPU / 2nd P100 install.
+
+## Risks
+
+- ~~**Prebuilt image may lack sm_60**~~ → **RESOLVED (verified live 2026-06-26).**
+  `ghcr.io/ggml-org/llama.cpp:server-cuda` (digest
+  `sha256:ce294a4561e6…f0f2a9`) loaded Qwen2.5-0.5B-Instruct-Q4_K_M with `-ngl 99` on the
+  P100: `nvidia-smi` showed the server process holding ~1.1 GB of GPU VRAM, model loaded,
+  server listening, no arch/assert errors. Prebuilt image works on Pascal — no source build
+  needed.
+- **32k @ q8_0 is tight (~15.7 GB)** → if compute buffers push it over, drop V cache to
+  q4_0 or context to 24k.
+- **Pascal `-fa` performance** → flash-attn works on Pascal but is slower than on
+  Volta+; acceptable for single/few-user agent use, measured during verification.