feat(llm): add llama.cpp inference stack for Hermes (Qwen2.5-14B on P100)

New stacks/llm/ serves Qwen2.5-14B-Instruct (Q4_K_M GGUF) via llama.cpp's OpenAI-compatible server on the Tesla P100 (CDI nvidia.com/gpu=0), published on 172.20.0.1:8090 for the host-side Hermes agent. vLLM was rejected: the P100 (cc 6.0) lacks the DP4A INT8 instructions its AWQ/GPTQ kernels need. Includes design spec and implementation plan under docs/superpowers/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 16:22:57 -05:00
parent 847edff1f8
commit 8e7682985d
4 changed files with 625 additions and 0 deletions
@@ -0,0 +1,385 @@
+# LLM inference backend for Hermes — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Stand up a llama.cpp OpenAI-compatible inference server in a new `stacks/llm/` Portainer stack, serving Qwen2.5-14B-Instruct on valhalla's Tesla P100, and add it as a provider in the Hermes agent.
+
+**Architecture:** Single `llama-server` container (image `ghcr.io/ggml-org/llama.cpp:server-cuda`, verified to run on the P100's sm_60) gets the GPU via CDI (`nvidia.com/gpu=0`), loads a GGUF from `/storage1/labdata/llm/models`, and publishes its OpenAI `/v1` API on the host at `172.20.0.1:8090`. Host-side Hermes (systemd-managed) reaches it directly — no Caddy. Config follows repo conventions: pure `env_file`, no `${VAR}` interpolation, infra image pinned out of Watchtower.
+
+**Tech Stack:** Docker Compose (Portainer git stack), llama.cpp server, CUDA/CDI, Gitea-polled deploy, Hermes (Nous Research agent) YAML config.
+
+**Reference spec:** `docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md`
+
+---
+
+## Pre-verified facts (do not re-derive)
+
+- GPU: Tesla P100-PCIE-16GB, cc 6.0; CDI device `nvidia.com/gpu=0` valid. Prebuilt image runs with full GPU offload (tested live 2026-06-26).
+- `/storage1/labdata` is root-owned; `ginnoir` has **passwordless sudo**.
+- `edge` network gateway = `172.20.0.1` (host IP on `br-b5aa55c3fedf`). Hermes binds here.
+- Image contains `curl` and `bash`.
+- Hermes services: `hermes-dashboard.service`, `hermes-gateway.service`, `hermes-webui.service` (system systemd). Config: `~/.hermes/config.yaml` with a `providers:` list (existing `ollama` entry as a template).
+- Repo deploy: app stacks deploy via git push → Portainer polls Gitea every 5 min. **New** stacks must be registered once in Portainer (see memory `portainer-new-stack-registration`).
+- `.gitattributes` forces LF — ensure `stack.env` / compose are LF on commit.
+
+## File structure
+
+- **Create** `stacks/llm/docker-compose.yml` — the llama-server service (one responsibility: serve the model on the GPU).
+- **Create** `stacks/llm/stack.env` — `LLAMA_API_KEY` only (committed per repo policy).
+- **Host-side (not in repo):** `/storage1/labdata/llm/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf`; one new `providers:` entry in `~/.hermes/config.yaml`.
+
+---
+
+### Task 1: Pre-stage the model on the host (must precede deploy)
+
+The container crash-loops if the GGUF is absent, so download it before Portainer deploys the stack.
+
+**Files:** none in repo (host filesystem only).
+
+- [ ] **Step 1: Create the model directory**
+
+Run:
+```bash
+ssh ginnoir@valhalla "sudo mkdir -p /storage1/labdata/llm/models && sudo ls -ld /storage1/labdata/llm/models"
+```
+Expected: directory exists.
+
+- [ ] **Step 2: Download Qwen2.5-14B-Instruct Q4_K_M (~9 GB)**
+
+Run:
+```bash
+ssh ginnoir@valhalla "cd /storage1/labdata/llm/models && sudo curl -fL -o Qwen2.5-14B-Instruct-Q4_K_M.gguf https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-GGUF/resolve/main/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
+```
+(Run in background if it's slow; it's a single ~9 GB file.)
+
+- [ ] **Step 3: Verify the download**
+
+Run:
+```bash
+ssh ginnoir@valhalla "sudo ls -lh /storage1/labdata/llm/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
+```
+Expected: file ~8.9–9.0 GB. If the size is wildly off (e.g. a few KB), it's an HTML error page — re-download.
+
+---
+
+### Task 2: Create `stacks/llm/stack.env`
+
+**Files:**
+- Create: `stacks/llm/stack.env`
+
+- [ ] **Step 1: Generate an API key**
+
+Run:
+```bash
+openssl rand -hex 32
+```
+Copy the output for the next step.
+
+- [ ] **Step 2: Write the file** (replace `<HEX>` with the generated key)
+
+`stacks/llm/stack.env`:
+```dotenv
+# llm stack secrets — read directly by the container via env_file.
+# llama.cpp's server reads LLAMA_API_KEY from the environment (no --api-key flag,
+# no ${VAR} interpolation), matching the repo's pure-env_file convention.
+LLAMA_API_KEY=<HEX>
+```
+
+- [ ] **Step 3: Confirm LF line endings**
+
+Run:
+```bash
+git check-attr text eol -- stacks/llm/stack.env
+```
+Expected: `eol: lf` (enforced by `.gitattributes`).
+
+---
+
+### Task 3: Create `stacks/llm/docker-compose.yml`
+
+**Files:**
+- Create: `stacks/llm/docker-compose.yml`
+
+- [ ] **Step 1: Write the compose file**
+
+`stacks/llm/docker-compose.yml`:
+```yaml
+# llm stack — local LLM inference backend for the Hermes agent.
+#
+# Single service: llama.cpp's OpenAI-compatible server (llama-server) serving
+# Qwen2.5-14B-Instruct (Q4_K_M GGUF) on the host's Tesla P100-16GB via CDI.
+# Chosen over vLLM because the P100 (GP100, compute capability 6.0) lacks the
+# DP4A INT8 instructions vLLM's AWQ/GPTQ kernels require — see
+# docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md.
+#
+# Pure env_file (LLAMA_API_KEY) — no Portainer UI env, no ${VAR} interpolation.
+# Image is infra-pinned out of Watchtower (manual tag bumps only).
+#
+# The server's OpenAI API is published on the host at 172.20.0.1:8090 (the edge
+# bridge gateway, a local host IP). Host-side Hermes reaches it there directly;
+# no Caddy block this round. Model weights live on the ZFS tier; the
+# /storage1/labdata/llm/models dir is pre-created with the GGUF before deploy.
+services:
+  llama-server:
+    image: ghcr.io/ggml-org/llama.cpp:server-cuda
+    container_name: llama-server
+    restart: unless-stopped
+    labels:
+      - "com.centurylabs.watchtower.enable=false"
+    networks: [llm]
+    env_file:
+      - stack.env
+    devices:
+      - "nvidia.com/gpu=0"
+    volumes:
+      - /storage1/labdata/llm/models:/models
+    command:
+      - "-m"
+      - "/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
+      - "--alias"
+      - "qwen2.5-14b-instruct"
+      - "-ngl"
+      - "99"
+      - "--ctx-size"
+      - "32768"
+      - "-fa"
+      - "--cache-type-k"
+      - "q8_0"
+      - "--cache-type-v"
+      - "q8_0"
+      - "--host"
+      - "0.0.0.0"
+      - "--port"
+      - "8080"
+    ports:
+      - "172.20.0.1:8090:8080"
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
+      start_period: 180s
+
+networks:
+  llm:
+    name: llm
+    driver: bridge
+```
+
+- [ ] **Step 2: Validate compose syntax locally**
+
+Run:
+```bash
+docker compose -f stacks/llm/docker-compose.yml config -q
+```
+Expected: no output (valid). If it errors on the CDI `devices` entry, that's a local-CLI version quirk, not a deploy blocker — the daemon on valhalla (Docker 29.5.2) supports it; proceed.
+
+---
+
+### Task 4: Commit and push the stack to Gitea
+
+**Files:** none new (commits Tasks 2–3).
+
+- [ ] **Step 1: Stage and commit**
+
+```bash
+git add stacks/llm/docker-compose.yml stacks/llm/stack.env docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md docs/superpowers/plans/2026-06-26-llm-backend-hermes.md
+git commit -m "feat(llm): add llama.cpp inference stack for Hermes (Qwen2.5-14B on P100)"
+```
+
+- [ ] **Step 2: Push (deploys nothing yet — stack isn't registered)**
+
+Use the **homelab-apply** skill's push path (push to Gitea; GitHub is the mirror). A `stacks/llm/*` change only redeploys once the stack is registered (Task 5).
+
+Run (per repo convention — token via one-off http.extraheader, never in git config):
+```bash
+git push   # to the configured remote(s); Gitea is primary
+```
+Expected: push succeeds; Portainer cannot yet act on `stacks/llm` because no stack references it.
+
+---
+
+### Task 5: Register the new Portainer git stack (one-time)
+
+New stacks aren't auto-created by polling — register once, then future pushes redeploy. See memory `portainer-new-stack-registration`.
+
+**Files:** none (Portainer state).
+
+- [ ] **Step 1: Read an existing git stack's config to copy repo URL + credential reference**
+
+Use the portainer MCP (invoke `get_guidance` first per the portainer-mcp-hygiene skill). Inspect a working app stack (e.g. `roms`) to copy the exact Gitea repo URL, ref (`refs/heads/main`), and the working fine-grained PAT/credential the other stacks use:
+```
+mcp__portainer__StackList  (select: name, GitConfig)
+mcp__portainer__StackInspect  on the roms stack id
+```
+
+- [ ] **Step 2: Create the stack from the git repository**
+
+Create a Docker standalone stack from the Gitea repo with:
+- compose path: `stacks/llm/docker-compose.yml`
+- ref: `refs/heads/main`
+- auto-update / git polling: **on** (match other app stacks)
+- env: **empty** (pure env_file)
+- credentials: the same working fine-grained PAT the other stacks use (the runner PAT cannot clone)
+
+Use `mcp__portainer__StackCreateDockerStandaloneRepository`. Per the memory note, the MCP call may time out but still succeed.
+
+- [ ] **Step 3: Verify the stack registered and deployed**
+
+Poll:
+```
+mcp__portainer__StackList  (select: [].{name:Name,status:Status})
+```
+Expected: a `llm` stack appears. Then confirm the container is running:
+```bash
+ssh ginnoir@valhalla "docker ps --filter name=llama-server --format '{{.Names}} {{.Status}}'"
+```
+Expected: `llama-server  Up … (health: starting|healthy)`.
+
+---
+
+### Task 6: Verify deploy — health, GPU offload, OpenAI API
+
+**Files:** none.
+
+- [ ] **Step 1: Confirm the model loaded on the GPU**
+
+Run:
+```bash
+ssh ginnoir@valhalla "docker logs --tail 60 llama-server 2>&1 | grep -iE 'P100|model loaded|listening|error|assert|cache_type|n_ctx'"
+```
+Expected: `Tesla P100`, `model loaded`, `server is listening`, no asserts. If logs show a `-fa` parse error, edit the compose to replace `-fa` with `--flash-attn` + `on` (two list items), re-commit/push, and let it redeploy.
+
+- [ ] **Step 2: Confirm VRAM is held (real offload, not CPU)**
+
+Run:
+```bash
+ssh ginnoir@valhalla "nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader; nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader"
+```
+Expected: a `llama-server`-owned process holding ~14–16 GB; free memory small. If `memory.used` is near 0 and the model is in RAM, GPU offload failed — recheck the CDI `devices` entry deployed correctly (`docker inspect llama-server --format '{{json .HostConfig.Devices}}{{json .HostConfig.DeviceRequests}}'`).
+
+- [ ] **Step 3: Confirm healthcheck is green**
+
+Run:
+```bash
+ssh ginnoir@valhalla "docker inspect llama-server --format '{{.State.Health.Status}}'"
+```
+Expected: `healthy` (allow up to `start_period` = 3 min).
+
+- [ ] **Step 4: Exercise the OpenAI endpoint from the host (as Hermes will)**
+
+Run (substitute the real key from `stacks/llm/stack.env`):
+```bash
+ssh ginnoir@valhalla "curl -fsS http://172.20.0.1:8090/v1/chat/completions -H 'Authorization: Bearer <LLAMA_API_KEY>' -H 'Content-Type: application/json' -d '{\"model\":\"qwen2.5-14b-instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: OK\"}],\"max_tokens\":8}'"
+```
+Expected: a JSON chat completion containing `OK`. A 401 means the key is wrong; a connection refused means the port publish/bind is wrong.
+
+---
+
+### Task 7: Stretch to 64k context (live tuning)
+
+Attempt the larger context now that the baseline works; keep it only if VRAM holds under load.
+
+**Files:**
+- Modify: `stacks/llm/docker-compose.yml` (only if 64k holds)
+
+- [ ] **Step 1: Try 64k with YaRN + lighter V cache, ephemerally**
+
+Run a throwaway container (doesn't touch the deployed stack), driving a long context:
+```bash
+ssh ginnoir@valhalla "docker run --rm --device nvidia.com/gpu=0 -v /storage1/labdata/llm/models:/models -p 172.20.0.1:8091:8080 ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf --alias q -ngl 99 --ctx-size 65536 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768 -fa --cache-type-k q8_0 --cache-type-v q4_0 --host 0.0.0.0 --port 8080 > /tmp/llm_64k.log 2>&1 & sleep 60; nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader; grep -iE 'model loaded|error|assert|out of memory|failed to allocate' /tmp/llm_64k.log | head; docker ps --filter publish=8091 -q | xargs -r docker rm -f"
+```
+Expected to decide: if `model loaded` with `memory.free` > ~300 MiB and no allocation failures, 64k is viable. If it OOMs / fails to allocate, 64k at this quant doesn't fit — keep 32k (stop here, leave the committed config as-is).
+
+- [ ] **Step 2 (only if 64k held): promote the 64k args into the stack**
+
+Edit `stacks/llm/docker-compose.yml` `command:` to: `--ctx-size 65536`, add `--rope-scaling yarn`, `--rope-scale 2`, `--yarn-orig-ctx 32768`, and set `--cache-type-v q4_0` (keep `--cache-type-k q8_0`). Then:
+```bash
+git add stacks/llm/docker-compose.yml
+git commit -m "feat(llm): raise llama-server context to 64k (YaRN + q4 V-cache)"
+git push
+```
+Let Portainer redeploy, then re-run Task 6 Steps 2–4. If the live 14B OOMs under a real long prompt, revert this commit.
+
+---
+
+### Task 8: Wire the provider into Hermes and verify end-to-end
+
+**Files:** host-side `~/.hermes/config.yaml` (not in repo).
+
+- [ ] **Step 1: Back up the Hermes config**
+
+Run:
+```bash
+ssh ginnoir@valhalla "cp ~/.hermes/config.yaml ~/.hermes/config.yaml.bak.$(date +%s) && ls -l ~/.hermes/config.yaml.bak.*"
+```
+
+- [ ] **Step 2: Add the provider entry under `providers:`**
+
+Append this entry to the `providers:` list in `~/.hermes/config.yaml` (same shape as the existing `ollama` entry; substitute the real key):
+```yaml
+- name: valhalla-p100
+  type: openai
+  base_url: http://172.20.0.1:8090/v1
+  api_key: <LLAMA_API_KEY>
+  models:
+  - qwen2.5-14b-instruct
+```
+Edit by reading the file, inserting the entry, and writing it back (preserve indentation exactly). Do **not** change the `model:` default block — we add the provider alongside the current default rather than silently replacing it (per spec).
+
+- [ ] **Step 3: Restart Hermes**
+
+Run:
+```bash
+ssh ginnoir@valhalla "sudo systemctl restart hermes-dashboard hermes-gateway hermes-webui && sleep 5 && systemctl is-active hermes-dashboard hermes-gateway hermes-webui"
+```
+Expected: three `active` lines. If any failed, check `journalctl -u hermes-gateway -n 50` — a YAML error means the edit broke indentation; restore the backup and retry.
+
+- [ ] **Step 4: Confirm Hermes sees the model and routes to the P100**
+
+Run a one-shot prompt forcing the new provider/model:
+```bash
+ssh ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes -z 'Reply with exactly: HELLO FROM P100' -m qwen2.5-14b-instruct --provider valhalla-p100 2>&1 | tail -20"
+```
+Expected: a completion containing the phrase. Simultaneously, `nvidia-smi` (separate shell) should show llama-server utilization spike during generation.
+
+- [ ] **Step 5: (Optional) make it the default**
+
+If ginnoir wants the P100 model as Hermes' default rather than a per-call choice, run interactively:
+```bash
+ssh -t ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes model"
+```
+and select `valhalla-p100` / `qwen2.5-14b-instruct`. Leave the default unchanged otherwise.
+
+---
+
+### Task 9: Cleanup and documentation
+
+**Files:** possibly `CLAUDE.md` (known-quirks note).
+
+- [ ] **Step 1: Remove the tiny test model**
+
+Run:
+```bash
+ssh ginnoir@valhalla "sudo rm -f /storage1/labdata/llm/models/qwen2.5-0.5b-instruct-q4_k_m.gguf && sudo ls /storage1/labdata/llm/models"
+```
+Expected: only the 14B GGUF remains.
+
+- [ ] **Step 2: Remove the config backup once verified (optional)**
+
+```bash
+ssh ginnoir@valhalla "ls ~/.hermes/config.yaml.bak.*"
+```
+Keep the most recent backup until the setup is confirmed stable, then remove.
+
+- [ ] **Step 3: Add a known-quirks note (optional, if desired)**
+
+Add a short bullet to `CLAUDE.md` under "External services" / "Known quirks": the `llm` stack serves Qwen2.5-14B on the P100 via llama.cpp; Hermes points at it via the `valhalla-p100` provider in `~/.hermes/config.yaml`; vLLM was rejected due to the P100's cc 6.0. Commit if added.
+
+---
+
+## Self-review notes
+
+- **Spec coverage:** engine (Task 3), model + storage (Tasks 1, 3), 32k baseline + q8 KV (Task 3), 64k stretch (Task 7), CDI GPU (Task 3, verified Task 6), `172.20.0.1:8090` publish (Task 3, verified Task 6), Hermes provider entry (Task 8), new-stack registration (Task 5), pure env_file / no `${VAR}` (Tasks 2–3), Watchtower pin (Task 3), no Caddy/no SSO (by omission), tiny-model cleanup (Task 9). All covered.
+- **No placeholders:** the only `<...>` tokens are the generated API key and (in Task 5) the repo URL/credential copied from an existing stack — both are runtime secrets/values, not undefined behavior.
+- **Consistency:** the alias `qwen2.5-14b-instruct` is the single contract used by the compose `--alias`, the curl test, and the Hermes provider `models:` / `-m` flag throughout.