homelabstack/docs/superpowers/plans/2026-06-26-llm-backend-hermes.md

# LLM inference backend for Hermes — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Stand up a llama.cpp OpenAI-compatible inference server in a new `stacks/llm/` Portainer stack, serving Qwen2.5-14B-Instruct on valhalla's Tesla P100, and add it as a provider in the Hermes agent.

**Architecture:** Single `llama-server` container (image `ghcr.io/ggml-org/llama.cpp:server-cuda`, verified to run on the P100's sm_60) gets the GPU via CDI (`nvidia.com/gpu=0`), loads a GGUF from `/storage1/labdata/llm/models`, and publishes its OpenAI `/v1` API on the host at `172.20.0.1:8090`. Host-side Hermes (systemd-managed) reaches it directly — no Caddy. Config follows repo conventions: pure `env_file`, no `${VAR}` interpolation, infra image pinned out of Watchtower.

**Tech Stack:** Docker Compose (Portainer git stack), llama.cpp server, CUDA/CDI, Gitea-polled deploy, Hermes (Nous Research agent) YAML config.

**Reference spec:** `docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md`

---

## Pre-verified facts (do not re-derive)

- GPU: Tesla P100-PCIE-16GB, cc 6.0; CDI device `nvidia.com/gpu=0` valid. Prebuilt image runs with full GPU offload (tested live 2026-06-26).
- `/storage1/labdata` is root-owned; `ginnoir` has **passwordless sudo**.
- `edge` network gateway = `172.20.0.1` (host IP on `br-b5aa55c3fedf`). Hermes binds here.
- Image contains `curl` and `bash`.
- Hermes services: `hermes-dashboard.service`, `hermes-gateway.service`, `hermes-webui.service` (system systemd). Config: `~/.hermes/config.yaml` with a `providers:` list (existing `ollama` entry as a template).
- Repo deploy: app stacks deploy via git push → Portainer polls Gitea every 5 min. **New** stacks must be registered once in Portainer (see memory `portainer-new-stack-registration`).
- `.gitattributes` forces LF — ensure `stack.env` / compose are LF on commit.

## File structure

- **Create** `stacks/llm/docker-compose.yml` — the llama-server service (one responsibility: serve the model on the GPU).
- **Create** `stacks/llm/stack.env` — `LLAMA_API_KEY` only (committed per repo policy).
- **Host-side (not in repo):** `/storage1/labdata/llm/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf`; one new `providers:` entry in `~/.hermes/config.yaml`.

---

### Task 1: Pre-stage the model on the host (must precede deploy)

The container crash-loops if the GGUF is absent, so download it before Portainer deploys the stack.

**Files:** none in repo (host filesystem only).

- [ ] **Step 1: Create the model directory**

Run:
```bash
ssh ginnoir@valhalla "sudo mkdir -p /storage1/labdata/llm/models && sudo ls -ld /storage1/labdata/llm/models"
```
Expected: directory exists.

- [ ] **Step 2: Download Qwen2.5-14B-Instruct Q4_K_M (~9 GB)**

Run:
```bash
ssh ginnoir@valhalla "cd /storage1/labdata/llm/models && sudo curl -fL -o Qwen2.5-14B-Instruct-Q4_K_M.gguf https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-GGUF/resolve/main/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
```
(Run in background if it's slow; it's a single ~9 GB file.)

- [ ] **Step 3: Verify the download**

Run:
```bash
ssh ginnoir@valhalla "sudo ls -lh /storage1/labdata/llm/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
```
Expected: file ~8.9–9.0 GB. If the size is wildly off (e.g. a few KB), it's an HTML error page — re-download.

---

### Task 2: Create `stacks/llm/stack.env`

**Files:**
- Create: `stacks/llm/stack.env`

- [ ] **Step 1: Generate an API key**

Run:
```bash
openssl rand -hex 32
```
Copy the output for the next step.

- [ ] **Step 2: Write the file** (replace `<HEX>` with the generated key)

`stacks/llm/stack.env`:
```dotenv
# llm stack secrets — read directly by the container via env_file.
# llama.cpp's server reads LLAMA_API_KEY from the environment (no --api-key flag,
# no ${VAR} interpolation), matching the repo's pure-env_file convention.
LLAMA_API_KEY=<HEX>
```

- [ ] **Step 3: Confirm LF line endings**

Run:
```bash
git check-attr text eol -- stacks/llm/stack.env
```
Expected: `eol: lf` (enforced by `.gitattributes`).

---

### Task 3: Create `stacks/llm/docker-compose.yml`

**Files:**
- Create: `stacks/llm/docker-compose.yml`

- [ ] **Step 1: Write the compose file**

`stacks/llm/docker-compose.yml`:
```yaml
# llm stack — local LLM inference backend for the Hermes agent.
#
# Single service: llama.cpp's OpenAI-compatible server (llama-server) serving
# Qwen2.5-14B-Instruct (Q4_K_M GGUF) on the host's Tesla P100-16GB via CDI.
# Chosen over vLLM because the P100 (GP100, compute capability 6.0) lacks the
# DP4A INT8 instructions vLLM's AWQ/GPTQ kernels require — see
# docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md.
#
# Pure env_file (LLAMA_API_KEY) — no Portainer UI env, no ${VAR} interpolation.
# Image is infra-pinned out of Watchtower (manual tag bumps only).
#
# The server's OpenAI API is published on the host at 172.20.0.1:8090 (the edge
# bridge gateway, a local host IP). Host-side Hermes reaches it there directly;
# no Caddy block this round. Model weights live on the ZFS tier; the
# /storage1/labdata/llm/models dir is pre-created with the GGUF before deploy.
services:
  llama-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    container_name: llama-server
    restart: unless-stopped
    labels:
      - "com.centurylabs.watchtower.enable=false"
    networks: [llm]
    env_file:
      - stack.env
    devices:
      - "nvidia.com/gpu=0"
    volumes:
      - /storage1/labdata/llm/models:/models
    command:
      - "-m"
      - "/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
      - "--alias"
      - "qwen2.5-14b-instruct"
      - "-ngl"
      - "99"
      - "--ctx-size"
      - "32768"
      - "-fa"
      - "--cache-type-k"
      - "q8_0"
      - "--cache-type-v"
      - "q8_0"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "8080"
    ports:
      - "172.20.0.1:8090:8080"
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 180s

networks:
  llm:
    name: llm
    driver: bridge
```

- [ ] **Step 2: Validate compose syntax locally**

Run:
```bash
docker compose -f stacks/llm/docker-compose.yml config -q
```
Expected: no output (valid). If it errors on the CDI `devices` entry, that's a local-CLI version quirk, not a deploy blocker — the daemon on valhalla (Docker 29.5.2) supports it; proceed.

---

### Task 4: Commit and push the stack to Gitea

**Files:** none new (commits Tasks 2–3).

- [ ] **Step 1: Stage and commit**

```bash
git add stacks/llm/docker-compose.yml stacks/llm/stack.env docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md docs/superpowers/plans/2026-06-26-llm-backend-hermes.md
git commit -m "feat(llm): add llama.cpp inference stack for Hermes (Qwen2.5-14B on P100)"
```

- [ ] **Step 2: Push (deploys nothing yet — stack isn't registered)**

Use the **homelab-apply** skill's push path (push to Gitea; GitHub is the mirror). A `stacks/llm/*` change only redeploys once the stack is registered (Task 5).

Run (per repo convention — token via one-off http.extraheader, never in git config):
```bash
git push   # to the configured remote(s); Gitea is primary
```
Expected: push succeeds; Portainer cannot yet act on `stacks/llm` because no stack references it.

---

### Task 5: Register the new Portainer git stack (one-time)

New stacks aren't auto-created by polling — register once, then future pushes redeploy. See memory `portainer-new-stack-registration`.

**Files:** none (Portainer state).

- [ ] **Step 1: Read an existing git stack's config to copy repo URL + credential reference**

Use the portainer MCP (invoke `get_guidance` first per the portainer-mcp-hygiene skill). Inspect a working app stack (e.g. `roms`) to copy the exact Gitea repo URL, ref (`refs/heads/main`), and the working fine-grained PAT/credential the other stacks use:
```
mcp__portainer__StackList  (select: name, GitConfig)
mcp__portainer__StackInspect  on the roms stack id
```

- [ ] **Step 2: Create the stack from the git repository**

Create a Docker standalone stack from the Gitea repo with:
- compose path: `stacks/llm/docker-compose.yml`
- ref: `refs/heads/main`
- auto-update / git polling: **on** (match other app stacks)
- env: **empty** (pure env_file)
- credentials: the same working fine-grained PAT the other stacks use (the runner PAT cannot clone)

Use `mcp__portainer__StackCreateDockerStandaloneRepository`. Per the memory note, the MCP call may time out but still succeed.

- [ ] **Step 3: Verify the stack registered and deployed**

Poll:
```
mcp__portainer__StackList  (select: [].{name:Name,status:Status})
```
Expected: a `llm` stack appears. Then confirm the container is running:
```bash
ssh ginnoir@valhalla "docker ps --filter name=llama-server --format '{{.Names}} {{.Status}}'"
```
Expected: `llama-server  Up … (health: starting|healthy)`.

---

### Task 6: Verify deploy — health, GPU offload, OpenAI API

**Files:** none.

- [ ] **Step 1: Confirm the model loaded on the GPU**

Run:
```bash
ssh ginnoir@valhalla "docker logs --tail 60 llama-server 2>&1 | grep -iE 'P100|model loaded|listening|error|assert|cache_type|n_ctx'"
```
Expected: `Tesla P100`, `model loaded`, `server is listening`, no asserts. If logs show a `-fa` parse error, edit the compose to replace `-fa` with `--flash-attn` + `on` (two list items), re-commit/push, and let it redeploy.

- [ ] **Step 2: Confirm VRAM is held (real offload, not CPU)**

Run:
```bash
ssh ginnoir@valhalla "nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader; nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader"
```
Expected: a `llama-server`-owned process holding ~14–16 GB; free memory small. If `memory.used` is near 0 and the model is in RAM, GPU offload failed — recheck the CDI `devices` entry deployed correctly (`docker inspect llama-server --format '{{json .HostConfig.Devices}}{{json .HostConfig.DeviceRequests}}'`).

- [ ] **Step 3: Confirm healthcheck is green**

Run:
```bash
ssh ginnoir@valhalla "docker inspect llama-server --format '{{.State.Health.Status}}'"
```
Expected: `healthy` (allow up to `start_period` = 3 min).

- [ ] **Step 4: Exercise the OpenAI endpoint from the host (as Hermes will)**

Run (substitute the real key from `stacks/llm/stack.env`):
```bash
ssh ginnoir@valhalla "curl -fsS http://172.20.0.1:8090/v1/chat/completions -H 'Authorization: Bearer <LLAMA_API_KEY>' -H 'Content-Type: application/json' -d '{\"model\":\"qwen2.5-14b-instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: OK\"}],\"max_tokens\":8}'"
```
Expected: a JSON chat completion containing `OK`. A 401 means the key is wrong; a connection refused means the port publish/bind is wrong.

---

### Task 7: Stretch to 64k context (live tuning)

Attempt the larger context now that the baseline works; keep it only if VRAM holds under load.

**Files:**
- Modify: `stacks/llm/docker-compose.yml` (only if 64k holds)

- [ ] **Step 1: Try 64k with YaRN + lighter V cache, ephemerally**

Run a throwaway container (doesn't touch the deployed stack), driving a long context:
```bash
ssh ginnoir@valhalla "docker run --rm --device nvidia.com/gpu=0 -v /storage1/labdata/llm/models:/models -p 172.20.0.1:8091:8080 ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf --alias q -ngl 99 --ctx-size 65536 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768 -fa --cache-type-k q8_0 --cache-type-v q4_0 --host 0.0.0.0 --port 8080 > /tmp/llm_64k.log 2>&1 & sleep 60; nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader; grep -iE 'model loaded|error|assert|out of memory|failed to allocate' /tmp/llm_64k.log | head; docker ps --filter publish=8091 -q | xargs -r docker rm -f"
```
Expected to decide: if `model loaded` with `memory.free` > ~300 MiB and no allocation failures, 64k is viable. If it OOMs / fails to allocate, 64k at this quant doesn't fit — keep 32k (stop here, leave the committed config as-is).

- [ ] **Step 2 (only if 64k held): promote the 64k args into the stack**

Edit `stacks/llm/docker-compose.yml` `command:` to: `--ctx-size 65536`, add `--rope-scaling yarn`, `--rope-scale 2`, `--yarn-orig-ctx 32768`, and set `--cache-type-v q4_0` (keep `--cache-type-k q8_0`). Then:
```bash
git add stacks/llm/docker-compose.yml
git commit -m "feat(llm): raise llama-server context to 64k (YaRN + q4 V-cache)"
git push
```
Let Portainer redeploy, then re-run Task 6 Steps 2–4. If the live 14B OOMs under a real long prompt, revert this commit.

---

### Task 8: Wire the provider into Hermes and verify end-to-end

**Files:** host-side `~/.hermes/config.yaml` (not in repo).

- [ ] **Step 1: Back up the Hermes config**

Run:
```bash
ssh ginnoir@valhalla "cp ~/.hermes/config.yaml ~/.hermes/config.yaml.bak.$(date +%s) && ls -l ~/.hermes/config.yaml.bak.*"
```

- [ ] **Step 2: Add the provider entry under `providers:`**

Append this entry to the `providers:` list in `~/.hermes/config.yaml` (same shape as the existing `ollama` entry; substitute the real key):
```yaml
- name: valhalla-p100
  type: openai
  base_url: http://172.20.0.1:8090/v1
  api_key: <LLAMA_API_KEY>
  models:
  - qwen2.5-14b-instruct
```
Edit by reading the file, inserting the entry, and writing it back (preserve indentation exactly). Do **not** change the `model:` default block — we add the provider alongside the current default rather than silently replacing it (per spec).

- [ ] **Step 3: Restart Hermes**

Run:
```bash
ssh ginnoir@valhalla "sudo systemctl restart hermes-dashboard hermes-gateway hermes-webui && sleep 5 && systemctl is-active hermes-dashboard hermes-gateway hermes-webui"
```
Expected: three `active` lines. If any failed, check `journalctl -u hermes-gateway -n 50` — a YAML error means the edit broke indentation; restore the backup and retry.

- [ ] **Step 4: Confirm Hermes sees the model and routes to the P100**

Run a one-shot prompt forcing the new provider/model:
```bash
ssh ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes -z 'Reply with exactly: HELLO FROM P100' -m qwen2.5-14b-instruct --provider valhalla-p100 2>&1 | tail -20"
```
Expected: a completion containing the phrase. Simultaneously, `nvidia-smi` (separate shell) should show llama-server utilization spike during generation.

- [ ] **Step 5: (Optional) make it the default**

If ginnoir wants the P100 model as Hermes' default rather than a per-call choice, run interactively:
```bash
ssh -t ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes model"
```
and select `valhalla-p100` / `qwen2.5-14b-instruct`. Leave the default unchanged otherwise.

---

### Task 9: Cleanup and documentation

**Files:** possibly `CLAUDE.md` (known-quirks note).

- [ ] **Step 1: Remove the tiny test model**

Run:
```bash
ssh ginnoir@valhalla "sudo rm -f /storage1/labdata/llm/models/qwen2.5-0.5b-instruct-q4_k_m.gguf && sudo ls /storage1/labdata/llm/models"
```
Expected: only the 14B GGUF remains.

- [ ] **Step 2: Remove the config backup once verified (optional)**

```bash
ssh ginnoir@valhalla "ls ~/.hermes/config.yaml.bak.*"
```
Keep the most recent backup until the setup is confirmed stable, then remove.

- [ ] **Step 3: Add a known-quirks note (optional, if desired)**

Add a short bullet to `CLAUDE.md` under "External services" / "Known quirks": the `llm` stack serves Qwen2.5-14B on the P100 via llama.cpp; Hermes points at it via the `valhalla-p100` provider in `~/.hermes/config.yaml`; vLLM was rejected due to the P100's cc 6.0. Commit if added.

---

## Self-review notes

- **Spec coverage:** engine (Task 3), model + storage (Tasks 1, 3), 32k baseline + q8 KV (Task 3), 64k stretch (Task 7), CDI GPU (Task 3, verified Task 6), `172.20.0.1:8090` publish (Task 3, verified Task 6), Hermes provider entry (Task 8), new-stack registration (Task 5), pure env_file / no `${VAR}` (Tasks 2–3), Watchtower pin (Task 3), no Caddy/no SSO (by omission), tiny-model cleanup (Task 9). All covered.
- **No placeholders:** the only `<...>` tokens are the generated API key and (in Task 5) the repo URL/credential copied from an existing stack — both are runtime secrets/values, not undefined behavior.
- **Consistency:** the alias `qwen2.5-14b-instruct` is the single contract used by the compose `--alias`, the curl test, and the Hermes provider `models:` / `-m` flag throughout.