homelabstack/docs/superpowers/plans/2026-06-26-llm-backend-hermes.md

LLM inference backend for Hermes — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Stand up a llama.cpp OpenAI-compatible inference server in a new stacks/llm/ Portainer stack, serving Qwen2.5-14B-Instruct on valhalla's Tesla P100, and add it as a provider in the Hermes agent.

Architecture: Single llama-server container (image ghcr.io/ggml-org/llama.cpp:server-cuda, verified to run on the P100's sm_60) gets the GPU via CDI (nvidia.com/gpu=0), loads a GGUF from /storage1/labdata/llm/models, and publishes its OpenAI /v1 API on the host at 172.20.0.1:8090. Host-side Hermes (systemd-managed) reaches it directly — no Caddy. Config follows repo conventions: pure env_file, no ${VAR} interpolation, infra image pinned out of Watchtower.

Tech Stack: Docker Compose (Portainer git stack), llama.cpp server, CUDA/CDI, Gitea-polled deploy, Hermes (Nous Research agent) YAML config.

Reference spec: docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md

---

Pre-verified facts (do not re-derive)

• GPU: Tesla P100-PCIE-16GB, cc 6.0; CDI device nvidia.com/gpu=0 valid. Prebuilt image runs with full GPU offload (tested live 2026-06-26).
• /storage1/labdata is root-owned; ginnoir has passwordless sudo.
• edge network gateway = 172.20.0.1 (host IP on br-b5aa55c3fedf). Hermes binds here.
• Image contains curl and bash.
• Hermes services: hermes-dashboard.service, hermes-gateway.service, hermes-webui.service (system systemd). Config: ~/.hermes/config.yaml with a providers: list (existing ollama entry as a template).
• Repo deploy: app stacks deploy via git push → Portainer polls Gitea every 5 min. New stacks must be registered once in Portainer (see memory portainer-new-stack-registration).
• .gitattributes forces LF — ensure stack.env / compose are LF on commit.

File structure

• Create stacks/llm/docker-compose.yml — the llama-server service (one responsibility: serve the model on the GPU).
• Create stacks/llm/stack.env — LLAMA_API_KEY only (committed per repo policy).
• Host-side (not in repo): /storage1/labdata/llm/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf; one new providers: entry in ~/.hermes/config.yaml.

---

Task 1: Pre-stage the model on the host (must precede deploy)

The container crash-loops if the GGUF is absent, so download it before Portainer deploys the stack.

Files: none in repo (host filesystem only).

• Step 1: Create the model directory

Run:

ssh ginnoir@valhalla "sudo mkdir -p /storage1/labdata/llm/models && sudo ls -ld /storage1/labdata/llm/models"

Expected: directory exists.

• Step 2: Download Qwen2.5-14B-Instruct Q4_K_M (~9 GB)

Run:

ssh ginnoir@valhalla "cd /storage1/labdata/llm/models && sudo curl -fL -o Qwen2.5-14B-Instruct-Q4_K_M.gguf https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-GGUF/resolve/main/Qwen2.5-14B-Instruct-Q4_K_M.gguf"

(Run in background if it's slow; it's a single ~9 GB file.)

• Step 3: Verify the download

Run:

ssh ginnoir@valhalla "sudo ls -lh /storage1/labdata/llm/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"

Expected: file ~8.9–9.0 GB. If the size is wildly off (e.g. a few KB), it's an HTML error page — re-download.

---

Task 2: Create stacks/llm/stack.env

Files:

• Create: stacks/llm/stack.env

• Step 1: Generate an API key

Run:

openssl rand -hex 32

Copy the output for the next step.

• Step 2: Write the file (replace <HEX> with the generated key)

stacks/llm/stack.env:

# llm stack secrets — read directly by the container via env_file.
# llama.cpp's server reads LLAMA_API_KEY from the environment (no --api-key flag,
# no ${VAR} interpolation), matching the repo's pure-env_file convention.
LLAMA_API_KEY=<HEX>

• Step 3: Confirm LF line endings

Run:

git check-attr text eol -- stacks/llm/stack.env

Expected: eol: lf (enforced by .gitattributes).

---

Task 3: Create stacks/llm/docker-compose.yml

Files:

• Create: stacks/llm/docker-compose.yml

• Step 1: Write the compose file

stacks/llm/docker-compose.yml:

# llm stack — local LLM inference backend for the Hermes agent.
#
# Single service: llama.cpp's OpenAI-compatible server (llama-server) serving
# Qwen2.5-14B-Instruct (Q4_K_M GGUF) on the host's Tesla P100-16GB via CDI.
# Chosen over vLLM because the P100 (GP100, compute capability 6.0) lacks the
# DP4A INT8 instructions vLLM's AWQ/GPTQ kernels require — see
# docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md.
#
# Pure env_file (LLAMA_API_KEY) — no Portainer UI env, no ${VAR} interpolation.
# Image is infra-pinned out of Watchtower (manual tag bumps only).
#
# The server's OpenAI API is published on the host at 172.20.0.1:8090 (the edge
# bridge gateway, a local host IP). Host-side Hermes reaches it there directly;
# no Caddy block this round. Model weights live on the ZFS tier; the
# /storage1/labdata/llm/models dir is pre-created with the GGUF before deploy.
services:
  llama-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    container_name: llama-server
    restart: unless-stopped
    labels:
      - "com.centurylabs.watchtower.enable=false"
    networks: [llm]
    env_file:
      - stack.env
    devices:
      - "nvidia.com/gpu=0"
    volumes:
      - /storage1/labdata/llm/models:/models
    command:
      - "-m"
      - "/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
      - "--alias"
      - "qwen2.5-14b-instruct"
      - "-ngl"
      - "99"
      - "--ctx-size"
      - "32768"
      - "-fa"
      - "--cache-type-k"
      - "q8_0"
      - "--cache-type-v"
      - "q8_0"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "8080"
    ports:
      - "172.20.0.1:8090:8080"
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 180s

networks:
  llm:
    name: llm
    driver: bridge

• Step 2: Validate compose syntax locally

Run:

docker compose -f stacks/llm/docker-compose.yml config -q

Expected: no output (valid). If it errors on the CDI devices entry, that's a local-CLI version quirk, not a deploy blocker — the daemon on valhalla (Docker 29.5.2) supports it; proceed.

---

Task 4: Commit and push the stack to Gitea

Files: none new (commits Tasks 2–3).

• Step 1: Stage and commit

git add stacks/llm/docker-compose.yml stacks/llm/stack.env docs/superpowers/specs/2026-06-26-llm-backend-hermes-design.md docs/superpowers/plans/2026-06-26-llm-backend-hermes.md
git commit -m "feat(llm): add llama.cpp inference stack for Hermes (Qwen2.5-14B on P100)"

• Step 2: Push (deploys nothing yet — stack isn't registered)

Use the homelab-apply skill's push path (push to Gitea; GitHub is the mirror). A stacks/llm/* change only redeploys once the stack is registered (Task 5).

Run (per repo convention — token via one-off http.extraheader, never in git config):

git push   # to the configured remote(s); Gitea is primary

Expected: push succeeds; Portainer cannot yet act on stacks/llm because no stack references it.

---

Task 5: Register the new Portainer git stack (one-time)

New stacks aren't auto-created by polling — register once, then future pushes redeploy. See memory portainer-new-stack-registration.

Files: none (Portainer state).

• Step 1: Read an existing git stack's config to copy repo URL + credential reference

Use the portainer MCP (invoke get_guidance first per the portainer-mcp-hygiene skill). Inspect a working app stack (e.g. roms) to copy the exact Gitea repo URL, ref (refs/heads/main), and the working fine-grained PAT/credential the other stacks use:

mcp__portainer__StackList  (select: name, GitConfig)
mcp__portainer__StackInspect  on the roms stack id

• Step 2: Create the stack from the git repository

Create a Docker standalone stack from the Gitea repo with:

• compose path: stacks/llm/docker-compose.yml
• ref: refs/heads/main
• auto-update / git polling: on (match other app stacks)
• env: empty (pure env_file)
• credentials: the same working fine-grained PAT the other stacks use (the runner PAT cannot clone)

Use mcp__portainer__StackCreateDockerStandaloneRepository. Per the memory note, the MCP call may time out but still succeed.

• Step 3: Verify the stack registered and deployed

Poll:

mcp__portainer__StackList  (select: [].{name:Name,status:Status})

Expected: a llm stack appears. Then confirm the container is running:

ssh ginnoir@valhalla "docker ps --filter name=llama-server --format '{{.Names}} {{.Status}}'"

Expected: llama-server Up … (health: starting|healthy).

---

Task 6: Verify deploy — health, GPU offload, OpenAI API

Files: none.

• Step 1: Confirm the model loaded on the GPU

Run:

ssh ginnoir@valhalla "docker logs --tail 60 llama-server 2>&1 | grep -iE 'P100|model loaded|listening|error|assert|cache_type|n_ctx'"

Expected: Tesla P100, model loaded, server is listening, no asserts. If logs show a -fa parse error, edit the compose to replace -fa with --flash-attn + on (two list items), re-commit/push, and let it redeploy.

• Step 2: Confirm VRAM is held (real offload, not CPU)

Run:

ssh ginnoir@valhalla "nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader; nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader"

Expected: a llama-server-owned process holding ~14–16 GB; free memory small. If memory.used is near 0 and the model is in RAM, GPU offload failed — recheck the CDI devices entry deployed correctly (docker inspect llama-server --format '{{json .HostConfig.Devices}}{{json .HostConfig.DeviceRequests}}').

• Step 3: Confirm healthcheck is green

Run:

ssh ginnoir@valhalla "docker inspect llama-server --format '{{.State.Health.Status}}'"

Expected: healthy (allow up to start_period = 3 min).

• Step 4: Exercise the OpenAI endpoint from the host (as Hermes will)

Run (substitute the real key from stacks/llm/stack.env):

ssh ginnoir@valhalla "curl -fsS http://172.20.0.1:8090/v1/chat/completions -H 'Authorization: Bearer <LLAMA_API_KEY>' -H 'Content-Type: application/json' -d '{\"model\":\"qwen2.5-14b-instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: OK\"}],\"max_tokens\":8}'"

Expected: a JSON chat completion containing OK. A 401 means the key is wrong; a connection refused means the port publish/bind is wrong.

---

Task 7: Stretch to 64k context (live tuning)

Attempt the larger context now that the baseline works; keep it only if VRAM holds under load.

Files:

• Modify: stacks/llm/docker-compose.yml (only if 64k holds)

• Step 1: Try 64k with YaRN + lighter V cache, ephemerally

Run a throwaway container (doesn't touch the deployed stack), driving a long context:

ssh ginnoir@valhalla "docker run --rm --device nvidia.com/gpu=0 -v /storage1/labdata/llm/models:/models -p 172.20.0.1:8091:8080 ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Qwen2.5-14B-Instruct-Q4_K_M.gguf --alias q -ngl 99 --ctx-size 65536 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 32768 -fa --cache-type-k q8_0 --cache-type-v q4_0 --host 0.0.0.0 --port 8080 > /tmp/llm_64k.log 2>&1 & sleep 60; nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader; grep -iE 'model loaded|error|assert|out of memory|failed to allocate' /tmp/llm_64k.log | head; docker ps --filter publish=8091 -q | xargs -r docker rm -f"

Expected to decide: if model loaded with memory.free > ~300 MiB and no allocation failures, 64k is viable. If it OOMs / fails to allocate, 64k at this quant doesn't fit — keep 32k (stop here, leave the committed config as-is).

• Step 2 (only if 64k held): promote the 64k args into the stack

Edit stacks/llm/docker-compose.yml command: to: --ctx-size 65536, add --rope-scaling yarn, --rope-scale 2, --yarn-orig-ctx 32768, and set --cache-type-v q4_0 (keep --cache-type-k q8_0). Then:

git add stacks/llm/docker-compose.yml
git commit -m "feat(llm): raise llama-server context to 64k (YaRN + q4 V-cache)"
git push

Let Portainer redeploy, then re-run Task 6 Steps 2–4. If the live 14B OOMs under a real long prompt, revert this commit.

---

Task 8: Wire the provider into Hermes and verify end-to-end

Files: host-side ~/.hermes/config.yaml (not in repo).

• Step 1: Back up the Hermes config

Run:

ssh ginnoir@valhalla "cp ~/.hermes/config.yaml ~/.hermes/config.yaml.bak.$(date +%s) && ls -l ~/.hermes/config.yaml.bak.*"

• Step 2: Add the provider entry under providers:

Append this entry to the providers: list in ~/.hermes/config.yaml (same shape as the existing ollama entry; substitute the real key):

- name: valhalla-p100
  type: openai
  base_url: http://172.20.0.1:8090/v1
  api_key: <LLAMA_API_KEY>
  models:
  - qwen2.5-14b-instruct

Edit by reading the file, inserting the entry, and writing it back (preserve indentation exactly). Do not change the model: default block — we add the provider alongside the current default rather than silently replacing it (per spec).

• Step 3: Restart Hermes

Run:

ssh ginnoir@valhalla "sudo systemctl restart hermes-dashboard hermes-gateway hermes-webui && sleep 5 && systemctl is-active hermes-dashboard hermes-gateway hermes-webui"

Expected: three active lines. If any failed, check journalctl -u hermes-gateway -n 50 — a YAML error means the edit broke indentation; restore the backup and retry.

• Step 4: Confirm Hermes sees the model and routes to the P100

Run a one-shot prompt forcing the new provider/model:

ssh ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes -z 'Reply with exactly: HELLO FROM P100' -m qwen2.5-14b-instruct --provider valhalla-p100 2>&1 | tail -20"

Expected: a completion containing the phrase. Simultaneously, nvidia-smi (separate shell) should show llama-server utilization spike during generation.

• Step 5: (Optional) make it the default

If ginnoir wants the P100 model as Hermes' default rather than a per-call choice, run interactively:

ssh -t ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes model"

and select valhalla-p100 / qwen2.5-14b-instruct. Leave the default unchanged otherwise.

---

Task 9: Cleanup and documentation

Files: possibly CLAUDE.md (known-quirks note).

• Step 1: Remove the tiny test model

Run:

ssh ginnoir@valhalla "sudo rm -f /storage1/labdata/llm/models/qwen2.5-0.5b-instruct-q4_k_m.gguf && sudo ls /storage1/labdata/llm/models"

Expected: only the 14B GGUF remains.

• Step 2: Remove the config backup once verified (optional)

ssh ginnoir@valhalla "ls ~/.hermes/config.yaml.bak.*"

Keep the most recent backup until the setup is confirmed stable, then remove.

• Step 3: Add a known-quirks note (optional, if desired)

Add a short bullet to CLAUDE.md under "External services" / "Known quirks": the llm stack serves Qwen2.5-14B on the P100 via llama.cpp; Hermes points at it via the valhalla-p100 provider in ~/.hermes/config.yaml; vLLM was rejected due to the P100's cc 6.0. Commit if added.

---

Self-review notes

• Spec coverage: engine (Task 3), model + storage (Tasks 1, 3), 32k baseline + q8 KV (Task 3), 64k stretch (Task 7), CDI GPU (Task 3, verified Task 6), 172.20.0.1:8090 publish (Task 3, verified Task 6), Hermes provider entry (Task 8), new-stack registration (Task 5), pure env_file / no ${VAR} (Tasks 2–3), Watchtower pin (Task 3), no Caddy/no SSO (by omission), tiny-model cleanup (Task 9). All covered.
• No placeholders: the only <...> tokens are the generated API key and (in Task 5) the repo URL/credential copied from an existing stack — both are runtime secrets/values, not undefined behavior.
• Consistency: the alias qwen2.5-14b-instruct is the single contract used by the compose --alias, the curl test, and the Hermes provider models: / -m flag throughout.