Compare commits

..
40 Commits
Author SHA1 Message Date
ginnoir 53d99c7cd1 fix(streaming): reach Plex HTTP via host :32402 proxy 2026-08-05 16:20:03 -05:00
ginnoir f7c30c0962 fix(streaming): restore SLEEP_DURATION and LOG_LEVEL dropped in dry-run flip 2026-08-05 16:02:25 -05:00
ginnoir 7f3c275516 feat(streaming): enable jellyplex-watched writes after dry-run
Dry-run connected to both servers with matching ginnoir usernames.
Library counts already at/above Plex baseline while the rescan finishes.
2026-08-05 15:59:47 -05:00
ginnoir 45ab885834 fix(streaming): bind JF 10.11 metadata path; fix Caddy deploy
Deploy Caddyfile to valhalla / deploy (push) Failing after 44s
Jellyfin 10.11 writes metadata to /config/data/metadata, not
/config/metadata — the old bind left bulk artwork on the root disk.
Moved existing metadata to /storage1 and corrected the compose mount.
Also sets TranscodingTempPath=/transcode.

Caddy deploy: job containers never saw /config/caddy; write via a
host-bind docker run instead. apply-compose.ps1 -Caddy stages through
/tmp + sudo. GITHUB_STACKS_PAT updated to the live Gitea PAT.
2026-08-05 15:50:25 -05:00
ginnoir 966063d34d chore: regenerate bookmarks for streaming stack 2026-08-05 13:29:05 -05:00
ginnoir 18f26634b6 feat(streaming): jellyfin GPU CDI access and jellyplex token
Decode-only P100 access for trickplay. Gives jellyplex-watched its
Jellyfin API key (still DRYRUN=True until mapping verified).
2026-08-05 13:29:04 -05:00
ginnoir ccbce57ab9 feat(caddy): add jellyfin, wizarr and jellystat sites
Deploy Caddyfile to valhalla / deploy (push) Failing after 1m37s
jellyfin and wizarr are public by design — they replace Plex's own
remote access. jellystat is internal_gate + forward_auth, matching the
tautulli pattern.
2026-08-05 13:17:48 -05:00
ginnoir de759345ec feat(streaming): add jellyfin stack
New Portainer stack: jellyfin, jellystat + postgres, wizarr and
jellyplex-watched. Bulk data (metadata, trickplay, cache, transcodes)
binds to /storage1; only SQLite config lands on the root disk.

Trickplay gets an explicit sub-bind because 10.11 moved it to
/config/data/trickplay, which is not relocatable from the UI. Transcodes
are disk-backed rather than tmpfs because jellyfin#16608 orphans segments
on the Remux/DirectStream jobs that are routine on this server.
2026-08-05 13:17:48 -05:00
ginnoir fea3a6b29f docs: correct Plex data and library paths in CLAUDE.md
/var/lib/plexmediaserver does not exist; the real path is /var/lib/plex,
itself a symlink to /storage1/labdata/plex. Media libraries are
/storage1/{Movies,TV,Anime} with /storage1/Media/* as symlinks to them.
2026-08-05 13:15:49 -05:00
ginnoir 942fefacb8 docs(plan): record root disk audit findings 2026-08-05 13:13:41 -05:00
ginnoirandClaude Opus 5 d028fb4e1b docs: add Jellyfin migration implementation plan
25 tasks across 7 phases, written to be executed in Cursor rather than
Claude Code — every action is a literal shell command, file edit or named
UI step, with no dependency on this repo's Claude Code skills or MCP
servers.

Phase 0 is a hard gate: the root disk is at 97% with /config on it, so
nothing deploys until space is reclaimed and free-space alerting exists.
Plex runs untouched through Task 22, and rollback stays a single command
until the 30-day archive expires.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 13:05:31 -05:00
ginnoirandClaude Opus 5 d24498e440 docs: add Jellyfin-replaces-Plex migration design
Plex now gates remote playback behind Plex Pass / Remote Watch Pass and
valhalla has no Plex Pass, so remote viewers are being pushed to pay for
access to a server they don't own.

Design covers a new stacks/streaming Portainer stack (jellyfin, jellystat
+ postgres, wizarr, jellyplex-watched), a storage layout that keeps bulk
data off the 97%-full root disk, and eight phases with Plex left running
and untouched through Phase 6.

Findings that shaped it, measured rather than assumed:
- root disk is at 97% (5.7G free) and /config lives on it — Phase 0
- Tesla P100 is GP100: no NVENC, decode only, so transcoding is CPU-bound
- 48% of streams transcode, peak 7 concurrent, but 1080p SDR with no
  subtitle burn-in and no HDR tone-mapping, which 52 threads can carry
- /var/lib/plex is a symlink to /storage1/labdata/plex holding 383G; the
  15G Metadata dir is reusable via Plexyfin, the 365G of BIF preview
  thumbnails is not (format-incompatible with Jellyfin trickplay)
- jellyfin-plugin-sso is browser-only, so Roku/Fire TV/tvOS cannot use it
  — native Jellyfin accounts, a deliberate exception to the auth standard

Also notes that CLAUDE.md documents a Plex data path that does not exist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 12:53:29 -05:00
ginnoirandClaude Opus 5 28455a6787 docs: correct valhalla OS in the remaining agent-facing docs
Same fix as 317a424, applied to the files an agent actually reads before
touching the host: AGENTS.md and both copies of the homelab-ssh skill.
They all claimed a headless Ubuntu server; valhalla is EndeavourOS (Arch).

Also record two things that cost time this session: there is no apt/apt-get
on the host, and ginnoir has passwordless sudo (needed for anything writing
to root-owned paths like /config/caddy/Caddyfile).

Historical records under plans/ and docs/superpowers/plans/ are left alone
on purpose — they document what was true when the work was done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 12:10:32 -05:00
ginnoirandClaude Opus 5 317a424a77 docs: correct valhalla OS and Plex packaging in CLAUDE.md
valhalla is EndeavourOS (Arch), not Ubuntu — there is no apt, only pacman.
Verified via /etc/os-release. This bit during a live session: `apt-get`
was not found when trying to install a package on the host.

Plex is likewise not a deb — it is the AUR/pacman package
plex-media-server 1.43.2.10687-1, confirmed with `pacman -Q`. Version,
systemd unit and binary path in that section were already correct.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 12:09:06 -05:00
ginnoirandClaude Opus 5 61cea7ee04 fix(caddy): restore the IP gate on nine sites where it was dead code
Deploy Caddyfile to valhalla / deploy (push) Failing after 43s
Caddy sorts the `route` directive ahead of `respond`, so on any site that
wrapped its handlers in a catch-all `route {}`, the `import internal_only`
above it never executed — the route matched everything and handled the
request before the 403 was reached. Confirmed in the adapted JSON: the
route was sub[0] with match="<always>" and the 403 was sub[1], unreachable.

sonarr, radarr, bazarr, prowlarr, tautulli, qbittorrent, nzbget, whisparr
and stash were all affected. They were still behind Authentik, so this was
not an open door, but the LAN/tailnet restriction was doing nothing.

Add an (internal_gate) snippet with the same IP check, intended to be
imported as the first directive INSIDE a route, where directives run in
written order. Verified in the adapted config: the 403 is now [0] inside
the route, ahead of the outpost proxy and the handle blocks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 11:38:58 -05:00
ginnoirandClaude Opus 5 cc960eb3eb fix(caddy): let nextcloud public share links work off-LAN
Deploy Caddyfile to valhalla / deploy (push) Successful in 44s
files.ginnoir.com imported internal_only, which 403s every non-LAN/tailnet
client — including anyone opening a public share link. Replace the blanket
import with an inline @blocked matcher that keeps the same IP gate but exempts
Nextcloud's public-share surface: /s/*, /public.php*, the files_sharing app,
and the static assets the share page loads.

Everything else on the host (file browser, /settings, /remote.php/dav) still
403s externally. The nextcloud.ginnoir.com alias is unchanged and stays fully
internal-only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 11:13:34 -05:00
ginnoirandClaude Opus 5 f70963e166 fix(proxy): pin public resolvers on caddy so ACME DNS-01 works
Deploy Caddyfile to valhalla / deploy (push) Successful in 1m32s
The LAN resolver (OPNsense Unbound) is authoritative for ginnoir.com via the
split-horizon override, but its local-zone holds only A records, so
`SOA ginnoir.com` returns NODATA. certmagic's zone lookup walks up the label
chain hunting for an SOA, finds none at ginnoir.com, climbs to `com.`, and asks
Cloudflare for a `com` zone:

  adding temporary record for zone "com.": expected 1 zone, got 0 for com.

Every DNS-01 renewal has failed for ~17 days (attempt 91 on the oldest), and
Caddy fell back to the LE staging endpoint. Certs began expiring as they rolled
off: 5etools (-4d), files (-1.9d), auth (-15.6h), fam (-15.6h), with ~20 more
queued behind them.

The expired auth.ginnoir.com cert is what broke Nextcloud SSO: its server-side
discovery fetch fails TLS verification ("certificate has expired"), and
user_oidc's LoginController catches that and returns 404 "provider unreachable".
Browsers were unaffected because they let you click through an expired cert;
PHP/curl does not.

Pinning 1.1.1.1/1.0.0.1 on the caddy container sends the SOA lookup to public
DNS. Verified Docker's embedded resolver still serves container names and the
gitea.ginnoir.com alias, which take priority over the external servers.

Also corrects the imgstudio comment, which credited its working TLS to avoiding
a "false .com" in the hostname. The actual reason was its explicit `resolvers`
line -- the same fix, applied to one site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 09:51:54 -05:00
ginnoir 914cd47a8f Bump RomM from floating :4 (4.9.2) to :5 (5.0.0).
Pre-upgrade dump at /config/backup/dumps/romm-pre-5.0.0-20260721.sql on valhalla.
2026-07-21 03:01:13 -05:00
ginnoir 8a3330a700 chore(famapp): set HOUSEHOLD_TIMEZONE for assistant dates 2026-07-08 21:40:57 -05:00
ginnoir ae71a2172b fix(caddy): imgstudio proxy with working TLS for Mac ComfyUI
Deploy Caddyfile to valhalla / deploy (push) Failing after 54s
Use imgstudio.ginnoir.com instead of comfyui.* — the ACME challenge FQDN
embeds a false .com that breaks Cloudflare DNS-01. Add per-site resolvers
1.1.1.1 to bypass LAN split-DNS during cert issuance.
2026-07-05 04:08:11 -05:00
ginnoir ec7174c255 feat(caddy): expose Mac ComfyUI at comfyui.ginnoir.com
Deploy Caddyfile to valhalla / deploy (push) Failing after 45s
Proxy the full ComfyUI editor through valhalla for LAN/tailnet access while
inference stays on the Mac at 192.168.1.121:8188.
2026-07-05 03:53:35 -05:00
ginnoir 754f6482d6 Update famapp LLM API key for homelab inference endpoint. 2026-07-05 01:10:24 -05:00
ginnoir 72c565c397 fix: persist second runner config path
Set CONFIG_FILE after initial registration so act_runner daemon survives restarts.
2026-07-04 23:42:33 -05:00
ginnoir e0619b6272 fix: let second Gitea runner auto-register on first boot
Remove CONFIG_FILE override so act_runner writes its initial config to /data.
2026-07-04 23:40:53 -05:00
ginnoir 58b2fd30c7 infra: add second Gitea Actions runner for parallel CI
Registers valhalla-ci-runner-2 with the same ubuntu-latest labels to drain the job queue faster.
2026-07-04 23:40:16 -05:00
ginnoir b3d4e59b3f fix(famapp): pull from self-hosted registry again 2026-07-04 23:02:56 -05:00
ginnoir 53d8cb9f2a fix(famapp): pull image from ghcr where ci publishes releases 2026-07-04 22:59:31 -05:00
ginnoir 0df08794bc feat(famapp): add llm assistant env matching dev 2026-07-04 22:52:34 -05:00
ginnoirandClaude Opus 4.8 15c654445c chore(authentik): bump 2026.5.2 -> 2026.5.3 (bugfix/security patch)
Pure bugfix/security patch per upstream release notes — no breaking
changes or migration steps. Bumps both authentik-server and
authentik-worker (embedded outpost updates with the server).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 20:56:00 -05:00
ginnoirandClaude Opus 4.8 70bc8fd26b fix(dev): stop BookStack inheriting Plane's DATABASE_URL
The dev stack shares one stack.env across all services via env_file, so
BookStack inherited Plane's DATABASE_URL (postgresql://...@postgres_plane).
Laravel prioritizes DATABASE_URL over the individual DB_* vars, so BookStack
connected to Plane's Postgres and 500'd with "could not find driver" (the
LSIO image ships pdo_mysql, not pdo_pgsql). Pin DATABASE_URL to BookStack's
own MariaDB DSN in its environment block.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 20:35:26 -05:00
ginnoirandClaude Sonnet 5 10997e4b16 feat(scripts): add uptime-check retry + Obsidian variant, add LLM benchmark docs
check_uptime.js gets a fetchWithRetry wrapper (3 attempts, 2s backoff)
for transient failures against the status page/heartbeat API.
check_uptime_to_obsidian.js is a variant that logs results into the
Obsidian vault instead of stdout. Also adds two benchmark writeups
(gpt-oss-20b on Ollama, 73-node Ollama fleet).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 02:17:53 -05:00
ginnoirandClaude Sonnet 5 1792dd964b fix(caddy): remove dead pf2e/ffttsystems site blocks
Deploy Caddyfile to valhalla / deploy (push) Successful in 56s
Both root paths (/srv/aon, /srv/ffttsystems) no longer exist on disk
-- confirmed during the recent disk cleanup audit -- so these blocks
were serving errors to any visitor. Content was already gone; only
the stale Caddyfile entries remained.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 02:13:06 -05:00
ginnoirandClaude Sonnet 5 13e29d9040 fix(logs): cap json-file log size on authentik/immich/mariadb
These three containers were writing multi-GB/day of unbounded logs
(authentik_server 3.5GB, immich-server 2.3GB, owncloud_mariadb 2.1GB),
a major contributor to a recent 100% disk-full incident. Daemon-wide
default (max-size 10m, max-file 3) was also added to
/etc/docker/daemon.json on valhalla, but that only covers newly
created containers going forward -- these three need the explicit
per-service override since they're long-running and won't be
recreated otherwise.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 02:11:38 -05:00
ginnoirandClaude Opus 4.8 d199b10ba4 feat(camofox): stack files (PARKED — not deployed, upstream browser bug)
Image camofox-browser:135.0.1-x86_64 builds + serves, but camoufox fails to
launch: "cannot open display: [object Promise]" (upstream un-awaited Promise in
the virtual_display path). HTTP /health ok, browserRunning:false. NOT deployed:
no Caddy route, no Portainer registration, no Hermes wiring. Resume once the
upstream display bug is fixed (re-add Caddy block + register stack).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 15:00:43 -05:00
ginnoirandClaude Opus 4.8 73ff2e92c8 docs: revise Hermes spec from live-CLI findings (v0.17 native features)
Executing Phase 1 revealed Hermes v0.17 ships native curator (managing 74
skills), native browser plugins (browserbase stealth), native acp editor mode,
and insights. Drop curator-evolver (redundant), re-evaluate camofox, keep acp
skill (installed+enabled) and eagle-eye (elevated).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 13:55:51 -05:00
ginnoirandClaude Opus 4.8 009a474e90 feat(llm): add ornith-1.0-9b coding model to llama-swap menu
DeepReinforce Ornith-1.0 (dense 9B on Qwen 3.5, Q5_K_M, MIT), an
agentic-coding model. Tool-calls + <think> work under --jinja; native
256k so no YaRN. Loads at ~7.7GB VRAM @ 64k.

Benchmark (docs/2026-06-27-ornith-9b-benchmark.md): quality ties
gpt-oss-20b but gen is ~2.5-3x slower (dense 9B active vs gpt-oss MoE
3.6B active on the compute-bound P100). Default stays gpt-oss-20b;
ornith kept as a coding specialist in the menu.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 13:51:19 -05:00
ginnoirandClaude Opus 4.8 dc2225d384 docs: fold ecosystem expansion + delegation fabric into Hermes plan
Add hermes-motif (skill discovery; complementary to curator, not a rival),
hermes-web-search-plus (mature multi-provider search, pairs with camofox),
optional llmtrim/rtk context efficiency, and the claude/codex/cursor/antigravity
delegation fabric. Record that Claude Code + Codex are now installed on valhalla.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 05:45:34 -05:00
ginnoirandClaude Opus 4.8 3cd843072d docs: resolve Hermes integration decisions (both UIs, report-only, eagle-eye)
UI: trial hermes-ui + hermes-workspace in parallel, keep winner (new Task 9).
Curator stays report-only. eagle-eye trial confirmed (only direct skill-router
in the ecosystem; llmtrim noted as complementary). camofox minimal 2-tool skill.
All phases now ungated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 05:30:37 -05:00
ginnoirandClaude Opus 4.8 98c4a2a02b docs: implementation plan for Hermes ecosystem integrations
Phased, reversible plan. Phase 1 (acp-skill + curator-evolver, report-only)
fully actionable; Phases 2-3 (camofox stack, eagle-eye trial, UI) gated on
the spec's open decisions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 05:18:21 -05:00
ginnoirandClaude Opus 4.8 d794c98505 docs: triage spec for Hermes ecosystem integrations (7 repos)
Research + fit analysis for hermes-agent-acp-skill, hermes-curator-evolver,
eagle-eye, camofox-browser, hermes-ui, hermes-workspace, mission-control.
Phased rollout; open decisions flagged for review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 05:14:15 -05:00
32 changed files with 4063 additions and 56 deletions
+1 -1
View File
@@ -5,7 +5,7 @@ description: Connect to and run commands on the valhalla homelab server (ginnoir
# homelab-ssh
The production homelab runs on a headless Ubuntu host reachable at `ssh ginnoir@valhalla` (key-based auth; resolves and works from this Windows host directly).
The production homelab runs on a headless **EndeavourOS (Arch)** host reachable at `ssh ginnoir@valhalla` (key-based auth; resolves and works from this Windows host directly). Package manager is `pacman` — there is no `apt`/`apt-get`. `ginnoir` has passwordless `sudo`.
## Connect (non-interactive / safe for automation)
+1 -1
View File
@@ -5,7 +5,7 @@ description: Connect to and run commands on the valhalla homelab server (ginnoir
# homelab-ssh
The production homelab runs on a headless Ubuntu host reachable at `ssh ginnoir@valhalla` (key-based auth; resolves and works from this Windows host directly).
The production homelab runs on a headless **EndeavourOS (Arch)** host reachable at `ssh ginnoir@valhalla` (key-based auth; resolves and works from this Windows host directly). Package manager is `pacman` — there is no `apt`/`apt-get`. `ginnoir` has passwordless `sudo`.
## Connect (non-interactive / safe for automation)
+2 -1
View File
@@ -111,7 +111,8 @@ GITHUB_RUNNER_ACCESS_TOKEN=github_pat_11ACRHQAI0q3n1svrl9jmT_ai2bQlXWSxY8HJjEI6q
# Used by StackCreateDockerStandaloneRepository and StackGitRedeploy.
# Pass as RepositoryAuthorizationType:0 (Basic), RepositoryUsername=ginnoir.
# Runner PAT above does NOT work for this — different scopes.
GITHUB_STACKS_PAT=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
# Gitea PAT for Portainer git stacks (env var name kept from GitHub era)
GITHUB_STACKS_PAT=54ea93904439c2919f1d8dab1ba545e54a2a3e86
# =============================================================
# DEV STACK
+10 -1
View File
@@ -17,6 +17,15 @@ jobs:
- name: Push Caddyfile and reload Caddy
run: |
# Job containers do NOT inherit the runner's /config/caddy bind.
# Docker-from-Docker with a host bind is the reliable path: the
# volume source is resolved on valhalla, not inside the job container.
# (Direct `cp /config/caddy/...` only works on the runner container
# itself, which is not where this step runs.)
apt-get update -qq && apt-get install -y -qq docker.io
cp Caddyfile /config/caddy/Caddyfile
docker run --rm \
-v /config/caddy:/dest \
-v "$PWD/Caddyfile:/src/Caddyfile:ro" \
alpine:3.20 \
cp /src/Caddyfile /dest/Caddyfile
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
+3 -3
View File
@@ -9,9 +9,9 @@ from your tool's user-level instructions (`~/.codex/AGENTS.md` / `~/.gemini/GEMI
## Essentials (full detail in `CLAUDE.md`)
- **What this is:** deployment config for a live, single-host Docker homelab on a headless Ubuntu
server reachable at `ssh ginnoir@valhalla`. ~50 containers in per-domain Portainer-managed stacks.
This is **not** application code.
- **What this is:** deployment config for a live, single-host Docker homelab on a headless
**EndeavourOS (Arch)** server reachable at `ssh ginnoir@valhalla``pacman`, not `apt`.
~50 containers in per-domain Portainer-managed stacks. This is **not** application code.
- **The repo is canonical.** Portainer polls `main` every 5 min and redeploys any app stack whose
`stacks/<domain>/*` files changed. Editing here changes nothing until you `git push`.
- **Deployment channels:**
+8 -5
View File
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## What this repo is
Deployment configuration for a **live, single-host Docker homelab** running on a headless Ubuntu server reachable at `ssh ginnoir@valhalla`. This repo is **not application code** — it is the canonical source for ~50 containers organized into per-domain Portainer-managed stacks.
Deployment configuration for a **live, single-host Docker homelab** running on a headless **EndeavourOS (Arch)** server reachable at `ssh ginnoir@valhalla` — package manager is `pacman`, there is no `apt`. This repo is **not application code** — it is the canonical source for ~50 containers organized into per-domain Portainer-managed stacks.
**The repo is canonical.** Portainer polls the **Gitea** repo `https://gitea.ginnoir.com/ginnoir/homelabstack` on `main` every 5 min and redeploys any application stack whose source files have changed. Editing here changes nothing until you push (or, for the management plane, run `apply-compose.ps1 -Portainer`). GitHub is a temporary push mirror only.
@@ -173,12 +173,15 @@ These run on valhalla but are managed outside of Portainer/Docker.
### Plex Media Server
- **Install:** deb package (`plexmediaserver`), managed by systemd. Not containerised.
- **Version:** 1.43.2 (Watchtower doesn't touch it — update via `apt upgrade plexmediaserver` or the Plex UI).
- **Install:** AUR/pacman package (`plex-media-server`), managed by systemd. Not containerised.
- **Version:** 1.43.2 (Watchtower doesn't touch it — update via `pacman -Syu plex-media-server` or the Plex UI).
- **Service:** `systemctl {start,stop,restart,status} plexmediaserver`
- **Binary:** `/usr/lib/plexmediaserver/`
- **App data / metadata / DB:** `/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/`
- **Media libraries:** `/storage1/Media/{Movies,TV,Anime}` — same ZFS pool bind-mounted into the *arr containers.
- **App data / metadata / DB:** `/var/lib/plex/Plex Media Server/` — note `/var/lib/plex` is a
**symlink to `/storage1/labdata/plex`**, so Plex's 383G of app data lives on the ZFS array, not
the root disk. (`/var/lib/plexmediaserver/` does not exist.)
- **Media libraries:** `/storage1/{Movies,TV,Anime}` — `/storage1/Media/*` are symlinks to these.
Same ZFS pool bind-mounted into the *arr containers.
- **Tautulli** (`tautulli` container in the media stack) reads PMS logs from `/storage1/Media` and connects to Plex over the LAN. It is **not** behind Authentik — Plex login is its own auth.
- **Caddy:** Plex is not reverse-proxied through Caddy (Plex handles its own relay/direct connections). No `plex.ginnoir.com` block exists.
+75 -20
View File
@@ -17,6 +17,18 @@
respond @blocked "Access denied" 403
}
# Same IP gate as (internal_only), but for use INSIDE a route{} block.
# Caddy sorts the `route` directive ahead of `respond`, so an `import
# internal_only` sitting above a catch-all `route {}` is dead code — the route
# matches everything and handles the request before the 403 is ever reached.
# Inside a route, directives run in written order, so importing this as the
# first line of the route makes the gate fire. Use this, not internal_only,
# on any site that wraps its handlers in route{} (i.e. the Authentik sites).
(internal_gate) {
@blocked_ip not remote_ip 192.168.1.0/24 172.16.0.0/12 100.64.0.0/10 127.0.0.1
respond @blocked_ip "Access denied" 403
}
# TB-006 — Authentik forward auth (embedded outpost on authentik-server:9000).
# Wrap protected sites in `route { ... }` so bypass handles run before forward_auth.
(authentik_outpost) {
@@ -80,16 +92,24 @@ foundry2.ginnoir.com {
reverse_proxy 5etools:80
}
pf2e.ginnoir.com {
root * /srv/aon
file_server
}
# =============================================================
# FILE STORAGE — internal only (Nextcloud)
# =============================================================
files.ginnoir.com {
import internal_only
# Public share links must resolve for external users; the rest of Nextcloud
# stays LAN/tailnet-only. Can't use `import internal_only` here — this needs
# the same IP gate with a path-based exemption, so the matcher is inlined.
# Blocked = external client AND not a public-share path. The /core, /dist,
# /css, /js and theming paths are the assets the share page itself loads;
# without them an external visitor gets an unstyled, non-functional page.
@blocked {
not remote_ip 192.168.1.0/24 172.16.0.0/12 100.64.0.0/10 127.0.0.1
not path /s/* /index.php/s/* /public.php /public.php/*
not path /apps/files_sharing/* /index.php/apps/files_sharing/* /ocs/v2.php/apps/files_sharing/*
not path /core/* /dist/* /css/* /js/* /themes/* /apps/theming/* /index.php/apps/theming/*
}
respond @blocked "Access denied" 403
redir /.well-known/carddav /remote.php/dav 301
redir /.well-known/caldav /remote.php/dav 301
reverse_proxy nextcloud:80 {
@@ -101,11 +121,6 @@ files.ginnoir.com {
# =============================================================
# STATIC SITES — public
# =============================================================
ffttsystems.ginnoir.com {
root * /srv/ffttsystems
file_server
}
wa4.ginnoir.com {
root * /srv/wa4
file_server
@@ -118,13 +133,24 @@ requests.ginnoir.com {
reverse_proxy seerr:5055
}
# =============================================================
# STREAMING — public (replaces Plex's own remote access)
# =============================================================
jellyfin.ginnoir.com {
reverse_proxy jellyfin:8096
}
wizarr.ginnoir.com {
reverse_proxy wizarr:5690
}
# =============================================================
# MEDIA MANAGEMENT — internal only
# =============================================================
sonarr.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle /api/* {
reverse_proxy sonarr:8989
@@ -137,8 +163,8 @@ sonarr.ginnoir.com {
}
radarr.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle /api/* {
reverse_proxy radarr:7878
@@ -151,8 +177,8 @@ radarr.ginnoir.com {
}
bazarr.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle /api/* {
reverse_proxy bazarr:6767
@@ -165,8 +191,8 @@ bazarr.ginnoir.com {
}
prowlarr.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle /api/* {
reverse_proxy prowlarr:9696
@@ -179,8 +205,8 @@ prowlarr.ginnoir.com {
}
tautulli.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle {
import authentik_forward_auth
@@ -189,12 +215,23 @@ tautulli.ginnoir.com {
}
}
jellystat.ginnoir.com {
route {
import internal_gate
import authentik_outpost
handle {
import authentik_forward_auth
reverse_proxy jellystat:3000
}
}
}
# =============================================================
# DOWNLOAD CLIENTS — internal only
# =============================================================
qbittorrent.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle {
import authentik_forward_auth
@@ -204,8 +241,8 @@ qbittorrent.ginnoir.com {
}
nzbget.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle {
import authentik_forward_auth
@@ -215,8 +252,8 @@ nzbget.ginnoir.com {
}
whisparr.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle /api/* {
reverse_proxy whisparr:6969
@@ -229,8 +266,8 @@ whisparr.ginnoir.com {
}
stash.ginnoir.com {
import internal_only
route {
import internal_gate
import authentik_outpost
handle {
import authentik_forward_auth
@@ -409,6 +446,24 @@ webui.ginnoir.com {
reverse_proxy 172.20.0.1:8787
}
# ComfyUI node editor — inference runs on the Mac at 192.168.1.121 (Metal/MPS).
# LAN/tailnet only; no Authentik (WebSocket queue/progress breaks under forward_auth).
#
# The explicit `resolvers` below is what made DNS-01 work here, not the imgstudio
# hostname (an earlier comment blamed a "false .com" in comfyui.ginnoir.com —
# that was wrong). The LAN resolver returns NODATA for `SOA ginnoir.com`, so
# certmagic's zone lookup climbs to `com.` and Cloudflare rejects it. The caddy
# container now pins public resolvers stack-wide (stacks/proxy/docker-compose.yml),
# making this block redundant; kept as belt-and-braces.
imgstudio.ginnoir.com {
import internal_only
tls {
dns cloudflare {env.CF_API_TOKEN}
resolvers 1.1.1.1
}
reverse_proxy 192.168.1.121:8188
}
backup.ginnoir.com {
route {
import authentik_outpost
+4 -1
View File
@@ -41,8 +41,11 @@ if ($EnvFile) {
}
if ($Caddy) {
# /config/caddy is root-owned on valhalla — scp directly fails with
# "Permission denied". Stage via /tmp and sudo-install.
Write-Host "Pushing Caddyfile ..."
scp $caddyLocal "${server}:/config/caddy/Caddyfile"
scp $caddyLocal "${server}:/tmp/Caddyfile.new"
ssh $server "sudo cp /tmp/Caddyfile.new /config/caddy/Caddyfile && sudo chown root:root /config/caddy/Caddyfile && rm -f /tmp/Caddyfile.new"
Write-Host "Reloading Caddy ..."
ssh $server "docker exec caddy caddy reload --config /etc/caddy/Caddyfile"
}
+24 -8
View File
@@ -15,17 +15,16 @@
<DT><H3>TABLETOP TOOLS (public)</H3>
<DL><p>
<DT><A HREF="https://5etools.ginnoir.com">5etools</A>
<DT><A HREF="https://pf2e.ginnoir.com">pf2e</A>
</DL><p>
<DT><H3>FILE STORAGE — public (Authentik on admin UI; shares + WebDAV bypass) (public)</H3>
<DT><H3>FILE STORAGE — internal only (Nextcloud) (internal)</H3>
<DL><p>
<DT><A HREF="https://files.ginnoir.com">files</A>
</DL><p>
<DT><H3>STATIC SITES (public)</H3>
<DL><p>
<DT><A HREF="https://ffttsystems.ginnoir.com">ffttsystems</A>
<DT><A HREF="https://wa4.ginnoir.com">wa4</A>
</DL><p>
<DT><H3>MEDIA REQUESTS (public)</H3>
@@ -33,26 +32,33 @@
<DT><A HREF="https://requests.ginnoir.com">requests</A>
</DL><p>
<DT><H3>MEDIA MANAGEMENT (internal)</H3>
<DT><H3>STREAMING — public (replaces Plex's own remote access) (public)</H3>
<DL><p>
<DT><A HREF="https://jellyfin.ginnoir.com">jellyfin</A>
<DT><A HREF="https://wizarr.ginnoir.com">wizarr</A>
</DL><p>
<DT><H3>MEDIA MANAGEMENT (public)</H3>
<DL><p>
<DT><A HREF="https://sonarr.ginnoir.com">sonarr</A>
<DT><A HREF="https://radarr.ginnoir.com">radarr</A>
<DT><A HREF="https://bazarr.ginnoir.com">bazarr</A>
<DT><A HREF="https://prowlarr.ginnoir.com">prowlarr</A>
<DT><A HREF="https://tautulli.ginnoir.com">tautulli</A>
<DT><A HREF="https://jellystat.ginnoir.com">jellystat</A>
</DL><p>
<DT><H3>DOWNLOAD CLIENTS (internal)</H3>
<DT><H3>DOWNLOAD CLIENTS</H3>
<DL><p>
<DT><A HREF="https://qbittorrent.ginnoir.com">qbittorrent</A>
<DT><A HREF="https://deluge.ginnoir.com">deluge</A>
<DT><A HREF="https://nzbget.ginnoir.com">nzbget</A>
<DT><A HREF="https://whisparr.ginnoir.com">whisparr</A>
<DT><A HREF="https://stash.ginnoir.com">stash</A>
<DT><A HREF="https://roms.ginnoir.com">roms</A>
<DT><A HREF="https://romhacks.ginnoir.com">romhacks</A>
<DT><A HREF="https://jd.ginnoir.com">jd</A>
<DT><A HREF="https://romhacks-files.ginnoir.com">romhacks-files</A>
<DT><A HREF="https://jd.ginnoir.com">jd [internal]</A>
<DT><A HREF="https://idlegame.ginnoir.com">idlegame [internal]</A>
<DT><A HREF="https://romhacks-files.ginnoir.com">romhacks-files [internal]</A>
</DL><p>
<DT><H3>NOTIFICATIONS &amp; RSS — public (ntfy native auth; FreshRSS HTTP auth via Authentik) (public)</H3>
@@ -85,6 +91,9 @@
<DT><A HREF="https://portainer.ginnoir.com">portainer</A>
<DT><A HREF="https://uptime.ginnoir.com">uptime</A>
<DT><A HREF="https://homarr.ginnoir.com">homarr [internal]</A>
<DT><A HREF="https://hermes.ginnoir.com">hermes</A>
<DT><A HREF="https://webui.ginnoir.com">webui [internal]</A>
<DT><A HREF="https://imgstudio.ginnoir.com">imgstudio [internal]</A>
<DT><A HREF="https://backup.ginnoir.com">backup</A>
<DT><A HREF="https://router.ginnoir.com">router [internal]</A>
</DL><p>
@@ -99,6 +108,13 @@
<DT><A HREF="https://vault.ginnoir.com">vault</A>
<DT><A HREF="https://docs.ginnoir.com">docs [internal]</A>
<DT><A HREF="https://plane.ginnoir.com">plane</A>
<DT><A HREF="https://nextcloud.ginnoir.com">nextcloud [internal]</A>
<DT><A HREF="https://office.ginnoir.com">office [internal]</A>
</DL><p>
<DT><H3>PHOTOS (internal)</H3>
<DL><p>
<DT><A HREF="https://photos.ginnoir.com">photos</A>
</DL><p>
<DT><H3>NOTES / PKM — internal only (LiveSync over LAN or tailnet) (internal)</H3>
+13 -5
View File
@@ -18,10 +18,7 @@
<DT><H3>Dev</H3>
<DL><p>
<DT><A HREF="ssh://git@valhalla:2222">gitea :2222</A>
<DT><A HREF="http://valhalla:3030">gitea :3030</A>
<DT><A HREF="http://valhalla:8443">code_server</A>
<DT><A HREF="http://valhalla:5000">registry</A>
<DT><A HREF="http://valhalla:4224">dbx</A>
<DT><A HREF="http://valhalla:6875">bookstack :6875</A>
</DL><p>
@@ -42,13 +39,12 @@
<DT><H3>Media</H3>
<DL><p>
<DT><A HREF="http://valhalla:8112">deluge</A>
<DT><A HREF="http://valhalla:6789">nzbget</A>
<DT><A HREF="http://valhalla:8989">sonarr</A>
<DT><A HREF="http://valhalla:7878">radarr</A>
<DT><A HREF="http://valhalla:6767">bazarr</A>
<DT><A HREF="http://valhalla:9696">prowlarr</A>
<DT><A HREF="http://valhalla:5055">overseerr</A>
<DT><A HREF="http://valhalla:5055">seerr</A>
<DT><A HREF="http://valhalla:8181">tautulli</A>
<DT><A HREF="http://valhalla:3232">qbittorrent :3232</A>
<DT><A HREF="http://valhalla:6881">qbittorrent :6881 (TCP/UDP)</A>
@@ -78,6 +74,11 @@
<DT><A HREF="http://valhalla:8080">owncloud_server</A>
</DL><p>
<DT><H3>Photos</H3>
<DL><p>
<DT><A HREF="http://valhalla:2283">immich-server</A>
</DL><p>
<DT><H3>Proxy</H3>
<DL><p>
<DT><A HREF="http://valhalla:80">caddy :80</A>
@@ -108,5 +109,12 @@
<DT><A HREF="http://valhalla:8997">romm :8997</A>
</DL><p>
<DT><H3>Streaming</H3>
<DL><p>
<DT><A HREF="http://valhalla:8096">jellyfin</A>
<DT><A HREF="http://valhalla:3009">jellystat :3009</A>
<DT><A HREF="http://valhalla:5690">wizarr</A>
</DL><p>
</DL><p>
</DL><p>
+61
View File
@@ -0,0 +1,61 @@
# Ornith-1.0-9B vs gpt-oss-20b — P100 benchmark (2026-06-27)
**TL;DR:** Ornith-1.0-9B works cleanly as a Hermes backend (tool-calls, `<think>`,
content all correct) and matches gpt-oss-20b on **code quality** — but it is a **dense
9B**, so on the compute-bound P100 it generates **~23× slower** than the MoE gpt-oss-20b
(~6.3 vs ~1323 tok/s). **Keep gpt-oss-20b as the default daily driver.** Ornith earns a
spot in the menu as a coding *specialist* to A/B on real tasks, but doesn't displace it.
## What was done
- Downloaded `ornith-1.0-9b-Q5_K_M.gguf` (6.47 GB, exact byte match) → `/storage1/labdata/llm/models/`.
- Added `ornith-1.0-9b` to `stacks/llm/llama-swap-config.yaml` (same Pascal macro: q8/q8 KV,
`--parallel 1`, `--flash-attn on`, `--jinja`, 64k ctx; native 256k so no YaRN). Deployed to
`/config/llm/` + restarted llama-swap. **Live in the menu now.**
- Quant choice: **Q5_K_M** (safe overnight, no OOM). Loads using only **7.7 GB VRAM** at 64k —
huge headroom, so Q6_K or Q8_0 (9.5 GB, near-lossless) would also fit easily if you want more quality.
## Results (identical prompts, temp 0.6 / top_p 0.95 / top_k 20)
| Test | Metric | **ornith-1.0-9b** | **gpt-oss-20b** | Winner |
|---|---|---|---|---|
| Tool call | emits valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie |
| Codegen (merge_intervals) | correct? | ✅ correct, non-mutating, tests pass | ✅ correct (mutates input list) | ornith (slightly cleaner) |
| Debug (second_largest) | correct? | ✅ correct + edge-case guard | ✅ correct + O(n) alt | tie (gpt-oss more thorough) |
| **Gen speed** | tok/s (codegen) | **6.4** | **21.7** | **gpt-oss 3.4×** |
| **Gen speed** | tok/s (debug) | **6.4** | **18.9** | **gpt-oss 2.9×** |
| Prefill (small) | tok/s | 49104 | 133188 | gpt-oss |
| Prefill (deep ~2530k ctx) | tok/s | 101.9 (n=30295) | 154.9 (n=23422) | gpt-oss 1.5× |
| Reasoning verbosity | think chars (codegen/debug) | 433 / 452 (concise) | 1671 / 2118 (verbose) | — |
| VRAM @ 64k | MiB | **7684** | ~12600 | ornith (more headroom) |
## Why the speed gap (the important takeaway)
gpt-oss-20b is **MoE with ~3.6B active params/token**; Ornith-9B is **dense (9B active/token)**.
The P100 (GP100, cc 6.0) is compute-bound, so per-token cost scales with *active* params —
2.5× more active params ≈ the ~2.53× slower generation we measured. This is architectural,
not a config problem; no amount of tuning closes it. Your memory's "~5 tok/s ceiling" held for
the MoE models; a dense 9B genuinely sits below that line on this GPU.
## Verdict for your Hermes backend
- **Default stays gpt-oss-20b.** Gen speed is already the UX pain point (~6 vs ~20 tok/s is the
difference between usable and painful for interactive agent loops), and Ornith doesn't win on
quality to justify being 3× slower.
- **Keep Ornith as a menu specialist.** It's MIT, purpose-built for agentic coding, refreshingly
concise (less token waste on overthinking), and tool-calls cleanly. Good for one-shot coding
asks where you'll wait for quality.
- **The real coding upgrade you already have on disk:** `Qwen3-Coder-30B-A3B-Instruct-UD-Q2_K_XL.gguf`
is a **coder-tuned MoE (3B active)** — it would be both *faster* than Ornith (MoE) *and*
code-specialized. If you want a faster coding model than gpt-oss, that's the one to benchmark
next, not Ornith.
## Caveats
- Benchmarks are a small hand-written suite (4 tasks), not SWE-bench. Ornith's headline 69.4
SWE-Bench Verified is **self-reported/unverified** by DeepReinforce.
- The `deep_prefill` row measures prefill speed only — both models hit the 256-token cap mid-think
(`finish=length`, empty `content`), which is expected for that probe.
- Raw responses saved on valhalla at `/tmp/ornith-bench/` (one `.json` per model+test).
## Open decision for you
The repo config change (`stacks/llm/llama-swap-config.yaml`) is **deployed to the host but NOT
committed/pushed** — I don't push without your say-so, and pushing would trigger a Portainer
redeploy of stack 34. Tell me to commit+push if you want the repo (canonical source) to match the
live host, or to revert the menu entry if you'd rather not keep Ornith around.
@@ -0,0 +1,84 @@
# gpt-oss-20b — `.73` Ollama box vs valhalla P100 (2026-06-28)
**TL;DR:** The same model (`gpt-oss-20b`) runs **~1.31.5× faster on generation** and
**~1.62× faster on prefill** on ginnoir's `192.168.1.73` Ollama box than on valhalla's
Tesla P100 llama-swap backend — *despite* the `.73` box partially offloading to CPU at 64K
context. Tool-calls and code outputs are correct on both. For interactive Hermes use the
`.73` box is the better backend whenever it's powered on; the P100 remains the always-on
fallback. **Caveat:** at very deep context (~43K tokens) `.73` generation drops to ~8 tok/s
(KV no longer fully GPU-resident).
## Setup
- **Endpoint:** `http://192.168.1.73:11434` — Ollama **0.30.11**, model `gpt-oss:20b`,
quant **MXFP4** (OpenAI's native gpt-oss 4-bit format), reported 20.9B params.
- **Serving config:** `num_ctx 65536` (matches the P100's 64K and Hermes' hard ≥64K
requirement), sampling `temperature 0.6 / top_p 0.95 / top_k 20` (identical to the
2026-06-27 Ornith/P100 run).
- **Measured via** Ollama's native `/api/chat` (exact `prompt_eval`/`eval` token counts +
durations). Cold model load at 64K ctx took **~65 s** (one-time).
- **P100 baseline** = the gpt-oss-20b column from `docs/2026-06-27-ornith-9b-benchmark.md`
(llama-swap, q8/q8 KV, `--parallel 1`, `--jinja`, 64K).
- GPU on `.73` **could not be identified** — no SSH (port 22 filtered). Inferred from VRAM
behavior (below): a fast but VRAM-limited card (or unified-memory host).
## Results
| Test | Metric | **.73 Ollama (MXFP4)** | **valhalla P100 (GGUF)** | `.73` advantage |
|---|---|---|---|---|
| Tool call | valid `tool_calls`? | ✅ `get_weather({"city":"Tokyo"})` | ✅ identical | tie (both correct) |
| Codegen (`merge_intervals`) | correct? | ✅ correct, non-mutating, type-hinted | ✅ correct (mutates input) | `.73` slightly cleaner |
| Debug (`second_largest`) | correct? | ✅ correct single-pass, handles dup max | ✅ correct | tie |
| Deep-prefill | answered? | ✅ "quick brown fox" | ✅ (capped probe) | tie |
| **Gen speed** | tok/s — tool_call | **33.1** | 23.0 | **1.44×** |
| **Gen speed** | tok/s — codegen | **27.4** | 21.7 | **1.26×** |
| **Gen speed** | tok/s — debug | **27.6** | 18.9 | **1.46×** |
| **Prefill** | tok/s — shallow (~100135 tok) | **240287** | 133188 | ~1.51.8× |
| **Prefill** | tok/s — deep | **2542** (n=43.8k) | 155 (n=23.4k) | far faster (diff depths) |
| Gen @ deep ctx | tok/s | 8.4 (n=43.8k ctx) | 12.7 (n=23.4k ctx) | **P100 wins at depth** |
| Reasoning verbosity | think chars (codegen/debug) | 1760 / 1317 | 1671 / 2118 | comparable |
| VRAM @ 64K | on-GPU / total | **8.82 / 14.16 GB** (partial CPU offload) | ~12.6 GB (100% GPU) | — |
## Reading the numbers
- **Shallow/typical depth is where `.73` wins decisively.** All three real tasks (tool-call,
codegen, debug) run at shallow context, and `.73` generates at **2733 tok/s vs the P100's
~1923** — roughly the difference between "comfortable" and "sluggish" for an interactive
agent loop. Prefill is also ~1.51.8× faster, so first-token latency improves too.
- **The `.73` box is VRAM-limited, not compute-limited.** At 64K ctx only **8.82 GB of the
14.16 GB** working set sits in VRAM — the rest (weights tail + deep KV) spills to system
RAM. It still beats the full-GPU P100, which means the card itself is much faster than the
P100; with more VRAM (or a smaller `num_ctx`) it would pull further ahead.
- **The one place the P100 wins: very deep context.** At ~43K resident tokens, `.73`
generation falls to **8.4 tok/s** because the KV cache is partly in CPU RAM (memory-
bandwidth-bound attention). The P100 holds its whole 64K KV in VRAM and degrades more
gracefully (12.7 tok/s at 23K). In practice Hermes' steady-state prompt is ~16K, so this
rarely bites — but long sessions on `.73` will slow down more than on the P100.
- **Quant differs**, so this isn't a pure hardware A/B: `.73` runs MXFP4 (gpt-oss's native,
near-lossless 4-bit) while the P100 GGUF quant is whatever llama-swap pulled. Both are
genuine gpt-oss-20b and both produced correct outputs; no quality regression observed.
## Verdict for the Hermes backend
- **Prefer `.73` when it's up.** It's the faster daily driver for gpt-oss-20b at the depths
Hermes actually runs at. Switch in-session with `/model --provider ollama --model gpt-oss:20b`.
- **Keep the P100 (`valhalla-p100`) as the always-on default.** It's a container on the
24/7 server; the `.73` box may be a desktop/workstation that isn't always powered. The
P100 also degrades more gracefully at very deep context.
- **If you want `.73` to be strictly better,** drop its `num_ctx` toward what Hermes needs
(it hard-requires ≥64K, so you can't go below that for Hermes) **or** put gpt-oss on a
bigger-VRAM card there — eliminating the CPU spill would lift both prefill and deep-context
generation.
## Caveats
- Small hand-written suite (4 tasks), not SWE-bench — measures latency/throughput and basic
correctness, not deep code quality.
- The codegen prompt lost its back-ticked tokens to shell quoting during the run (prompt_n 95
vs the P100's 113); the model still produced a correct `merge_intervals`, and gen tok/s is
prompt-content-independent, so the speed comparison stands.
- Deep-prefill rows use different context depths (43.8K on `.73` vs 23.4K on P100), so the
prefill-tok/s cells aren't directly comparable — read them as "each box's deep-prefill rate
at that depth," not a head-to-head ratio.
- Raw responses saved on valhalla at `/tmp/ollama-bench/` (one `.json` per task); P100
baselines at `/tmp/ornith-bench/`.
@@ -0,0 +1,87 @@
# `.73` Ollama fleet benchmark — all 9 models (2026-06-28)
**TL;DR:** Throughput across every model on `192.168.1.73`. Generation speed spans an
**~5× range** — from `gemma4:e4b` at **~93 tok/s** down to the big `qwen3:30b-a3b` at
**~18 tok/s**. **All 9 models emit valid tool-calls.** For an interactive agent backend the
sweet spot is **`gpt-oss:20b` (~29 tok/s)** or **`gemma4:12b` (~48 tok/s)** if 12B quality
suffices; the 30B-class Qwen MoEs are the slowest here (heavy CPU offload at 64K on this
VRAM-limited box). **Caveat:** code-correctness for the heavy *thinking* models is
indeterminate — they used the whole 768-token gen cap reasoning and never emitted code (see
Caveats); re-run with a bigger budget to judge quality.
## Setup
- Endpoint `http://192.168.1.73:11434`, Ollama 0.30.11. Each model served at **`num_ctx
65536`** (Hermes' ≥64K requirement), sampling `temp 0.6 / top_p 0.95 / top_k 20`.
- Native `/api/chat` timings. 4 tasks: tool-call, codegen (`merge_intervals`), debug
(`second_largest`), deep-prefill (~16K-token filler). Gen capped: 256 / 768 / 768 / 128.
- Same `.73` box as the gpt-oss head-to-head in
`docs/2026-06-28-gpt-oss-20b-ollama-benchmark.md` (GPU still unidentified — no SSH).
## Generation speed (tok/s) — the headline
Average of the three real tasks (tool-call / codegen / debug), sorted fastest first:
| Model | avg gen t/s | tool | codegen | debug | deep-ctx gen | cold load s | tool-call? |
|---|--:|--:|--:|--:|--:|--:|:--:|
| **gemma4:e4b** | **92.7** | 91.1 | 93.3 | 93.6 | 82.6 | 24.8 | ✅ |
| **gemma4:12b** | **48.2** | 46.8 | 49.0 | 48.8 | 45.7 | 10.6 | ✅ |
| **gpt-oss:20b** | **29.4** | 32.7 | 27.9 | 27.7 | 29.9 | 0.4¹ | ✅ |
| **qwen3.6:35b-a3b** | **27.3** | 29.1 | 26.4 | 26.4 | 28.2 | 35.9 | ✅ |
| **gemma4:26b** | **25.7** | 27.6 | 25.0 | 24.5 | 26.5 | 53.8 | ✅ |
| **glm-4.7-flash** | **21.2** | 24.2 | 19.7 | 19.7 | 21.4 | 34.8 | ✅ |
| **qwen3-vl:30b-a3b** | **19.1** | 22.7 | 16.5 | 18.2 | 20.6 | 29.8 | ✅ |
| **qwen3-coder:30b** | **19.0** | 22.8 | 17.1 | 17.2 | 20.4 | 25.3 | ✅ |
| **qwen3:30b-a3b** | **17.6** | 19.5 | 16.6 | 16.6 | 18.5 | 25.0 | ✅ |
¹ gpt-oss was already resident from the prior run; real cold load is ~65 s.
## Prefill speed (tok/s)
| Model | shallow (~100 tok) | deep (~16K tok) |
|---|--:|--:|
| gemma4:e4b | 11931799 | 7283 |
| gemma4:12b | 6671078 | 3318 |
| gpt-oss:20b | 281379 | 2655 |
| qwen3:30b-a3b | 63149 | 1030 |
| qwen3-coder:30b | 76270 | 979 |
| gemma4:26b | 110127² | 977 |
| qwen3.6:35b-a3b | 78209 | 620 |
| glm-4.7-flash | 70163 | 649 |
| qwen3-vl:30b-a3b | 68127 | 588 |
² gemma4:26b's first request after load measured 11.7 t/s (cold-cache artifact); ignore.
## What stands out
- **The two small gemmas are in a different league.** `gemma4:e4b` (~93 t/s) and
`gemma4:12b` (~48 t/s) are dense but small, so they sit fully on GPU and fly. If a 4B/12B
is smart enough for the job, they're the most responsive options by far.
- **gpt-oss:20b is the best "big-brain, still-fast" pick** (~29 t/s) — MoE ~3.6B active keeps
it quick despite 20B total. `qwen3.6:35b-a3b` nearly matches it (~27 t/s) and may be
stronger; worth A/B-ing on real tasks.
- **The 30B-a3b Qwen trio is the slowest** (~1719 t/s). Same "3B-active" MoE label, but
larger total weights → more spills to CPU RAM at 64K on this VRAM-limited box, dragging
generation below gpt-oss. `qwen3-coder` being this slow undercuts it as a *fast* coding
model here.
- **Every model tool-calls.** All 9 emitted a valid `get_weather({"city":"Tokyo"})`, so any
of them can drive Hermes' tool loop.
## Correctness (partial — see caveat)
- **Confirmed correct** code on the non-/light-thinking models that finished within the cap:
`gpt-oss:20b`, `gemma4:12b`, `qwen3-vl:30b-a3b`, `qwen3-coder:30b` (clean `def`,
`reason=stop` or code present), plus `gemma4:26b` & `qwen3:30b-a3b` on the task each
finished.
- **Indeterminate** (truncated mid-reasoning, `content=0`, `reason=length`): `glm-4.7-flash`
(both), `gemma4:e4b` (both), `qwen3.6:35b-a3b` (both), `gemma4:26b` (codegen),
`qwen3:30b-a3b` (debug). These spent all 768 gen tokens in the `thinking` channel — **not
wrong, just unfinished.** A re-run at `num_predict ~3072` is needed to grade their output.
## Caveats
- The 768-token gen cap was too low for heavy chain-of-thought models — it bounds runtime but
truncates their answers. Speed (tok/s) is unaffected and valid; code *quality* for the
truncated set is not measured here.
- Per-model VRAM split not captured (models unload after 2 min `keep_alive`); only
gpt-oss-20b is known (8.82 GB on-GPU / 14.16 GB total at 64K → partial CPU offload). The
slow 30B-class numbers are consistent with heavier offload.
- Small hand-written suite, not SWE-bench. Quants are each model's Ollama default.
- Raw per-task responses on valhalla at `/tmp/ollama-bench-all/` (`summary.json` + one JSON
per model/task); progress log `/tmp/ollama-bench-all/progress.txt`.
@@ -0,0 +1,780 @@
# Hermes Ecosystem Integration Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Integrate selected Hermes-ecosystem tools into ginnoir's live valhalla deployment — a host-side delegation skill and self-improvement plugin (Phase 1), a stealth-browser homelab stack and a skill pre-filter trial (Phase 2), and an optional UI upgrade (Phase 3) — each reversible and sized for the single-P100 / weak-local-model constraints.
**Architecture:** Two integration classes. **Class A** (host-side Hermes plugins/skills) install into `~/.hermes/` on the valhalla host and are applied by SSH + `hermes` CLI + `sudo systemctl restart hermes-gateway.service`; they are **host-managed, NOT committed to this repo** (tracked in project memory + the Obsidian vault, like the rustdesk/obsidian/xvfb units). **Class B** (Docker services) become `stacks/<name>/` entries deployed via the normal Gitea-poll path, fronted by Caddy `internal_only` + Authentik. The companion spec is `docs/superpowers/specs/2026-06-27-hermes-ecosystem-integration-design.md`.
**Tech Stack:** Hermes Agent v0.17.0 (host systemd), llama-swap/Tesla P100 backend (`gpt-oss-20b`, `--parallel 1`), Python 3.11 (`~/.hermes/hermes-agent/venv`), `uv`, SQLite, Docker Compose + Portainer (Gitea-polled), Caddy, Authentik, Codex + Claude Code CLIs.
---
## How to read this plan (operational, not codebase-TDD)
These are operational integrations against a live host and third-party services, so the TDD rhythm
is adapted: each task is **back up → change → verify with a smoke test → document/commit**. The
"test" is a real verification command with expected output. **Class A (host) changes are not git
commits** — their checkpoint is a backup + smoke test + a memory/vault note. **Class B (repo)
changes do commit** (and push triggers Portainer). Run every step from the Windows workstation;
host steps use `ssh -o BatchMode=yes ginnoir@valhalla "..."`.
**Global guardrails (apply to every task):**
- `hermes` is only on the **login-shell** PATH → over SSH call it by full path: `~/.local/bin/hermes`.
- Gateway restart needs root: `sudo systemctl restart hermes-gateway.service`.
- Always back up `~/.hermes/config.yaml` before editing (`cp ...bak.$(date +%s)`).
- **Never** load a second model onto the P100. Keep curator/eagle-eye semantic layers on CPU or off.
- Read third-party code before running it (curator writes skills; acp-skill spawns external agents; camofox automates a browser).
**Decisions (RESOLVED 2026-06-27 — all phases actionable):** trial **both** UIs and keep the winner
(Phase 3); curator stays **report-only** (Task 3); **trial eagle-eye** — it's the only direct
skill-router in the ecosystem (Task 6); camofox wired as a **minimal 2-tool skill** (Task 5 Step 8).
---
# PHASE 1 — Host-side, reversible, high-leverage (actionable now)
## Task 1: Pre-flight — capture current Hermes state
**Files:**
- Host only (no repo files).
- [ ] **Step 1: Verify host reachability and Hermes services are up**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "systemctl is-active hermes-gateway.service hermes-dashboard.service hermes-webui.service"
```
Expected: three lines, each `active`.
- [ ] **Step 2: Snapshot config + inventory skills/plugins/sessions dirs**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "cp ~/.hermes/config.yaml ~/.hermes/config.yaml.bak.preflight.$(date +%s) && ls -la ~/.hermes/skills ~/.hermes/plugins ~/.hermes/sessions 2>&1 | head -60 && ~/.local/bin/hermes --version"
```
Expected: a backup is created; directory listings print (note whether `~/.hermes/plugins` exists yet); `hermes` prints a version (≈ v0.17.0). Record the skills-dir path — confirms `~/.hermes/skills` is correct for later tasks.
- [x] **Step 3: Confirm Codex and Claude Code are invocable — DONE 2026-06-27 (installed this session)**
Both delegation CLIs were installed on valhalla this session:
- `claude``~/.local/bin/claude` **v2.1.195** (login pending)
- `codex``/usr/bin/codex` **v0.142.3** (login pending; harmless PATH-alias warning at install)
**Gotcha recorded:** `/home/ginnoir/.claude` existed as an empty **root-owned** dir (created Jun 17,
likely a prior sudo op) and blocked the installer until `sudo chown ginnoir:ginnoir ~/.claude`. Codex
global install needs `sudo` (npm global prefix is `/usr`). Re-verify any time with:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc 'claude --version; codex --version'"
```
**ginnoir must log in** to each (`claude`, then `codex login`) before Task 2's external delegation
smoke-tests will succeed. Cursor + Antigravity targets are added later in **Task 10**.
- [ ] **Step 4: Checkpoint**
No commit (host inventory only). Record findings (skills-dir path, whether `plugins/` exists, Codex/Claude availability) in the session notes for use in Tasks 23.
---
## Task 2: Install `hermes-agent-acp-skill` (multi-agent delegation)
**Files:**
- Host: `~/.hermes/skills/hermes-acp-orchestrator/` (skill files), `~/.hermes/config.yaml` (delegation block).
- Scratch: clone under `/storage1/hermes/workspace/clones/` (never root; see the disk gotcha).
- [ ] **Step 1: Clone and read the skill before installing**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "mkdir -p /storage1/hermes/workspace/clones && git -C /storage1/hermes/workspace/clones clone https://github.com/Rainhoole/hermes-agent-acp-skill && sed -n '1,200p' /storage1/hermes/workspace/clones/hermes-agent-acp-skill/SKILL.md"
```
Expected: repo clones; `SKILL.md` prints. **Read it** to confirm: the skill folder/name, how `delegate_task()` is wired, and whether it expects a specific install path or a config key. The README omits install steps, so the SKILL.md is authoritative — follow whatever placement it documents. If SKILL.md specifies a different mechanism than the manual copy below, use SKILL.md's.
- [ ] **Step 2: Place the skill into the Hermes skills directory**
Run (adjust the destination name to match SKILL.md's declared skill name):
```bash
ssh -o BatchMode=yes ginnoir@valhalla "mkdir -p ~/.hermes/skills/hermes-acp-orchestrator && cp -r /storage1/hermes/workspace/clones/hermes-agent-acp-skill/SKILL.md /storage1/hermes/workspace/clones/hermes-agent-acp-skill/README.md ~/.hermes/skills/hermes-acp-orchestrator/ && ls -la ~/.hermes/skills/hermes-acp-orchestrator/"
```
Expected: `SKILL.md` and `README.md` present in the new skill dir.
- [ ] **Step 3: Add the delegation config block (with a safe backup)**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "cp ~/.hermes/config.yaml ~/.hermes/config.yaml.bak.acp.$(date +%s) && printf '\ndelegation:\n external_timeout_seconds: 900\n external_max_output_chars: 24000\n' >> ~/.hermes/config.yaml && tail -8 ~/.hermes/config.yaml"
```
Expected: a `.bak.acp.*` backup exists; the `delegation:` block is appended and printed. (If SKILL.md says the block belongs under a different key or nesting, edit accordingly instead of this append.)
- [ ] **Step 4: Restart the gateway and confirm the skill registers**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo systemctl restart hermes-gateway.service && sleep 5 && systemctl is-active hermes-gateway.service && ~/.local/bin/hermes skills list 2>&1 | grep -i acp"
```
Expected: gateway `active`; the ACP/orchestrator skill appears in `hermes skills list`. (If the subcommand differs, use `~/.local/bin/hermes skills --help` to find the list command — verify on host.)
- [ ] **Step 5: Smoke-test a trivial delegation to the local hermes subagent first**
Run (a no-external-dependency delegation — routes to `hermes`, not Codex/Claude, to isolate the skill from CLI availability):
```bash
ssh -o BatchMode=yes ginnoir@valhalla "~/.local/bin/hermes run 'Use delegate_task to ask a hermes subagent to reply with exactly the word PONG, then report its output.' 2>&1 | tail -30"
```
Expected: the delegated subagent returns `PONG` and the parent reports it. **This proves the skill mechanics.** (Exact `hermes` one-shot invocation may differ — confirm the non-interactive run command via `~/.local/bin/hermes --help` in Step 1's read-through.)
- [ ] **Step 6: Smoke-test an external delegation (only if Codex/Claude were found in Task 1)**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "~/.local/bin/hermes run 'Use delegate_task with agent=claude-code to print the current working directory and nothing else, then report it.' 2>&1 | tail -40"
```
Expected: Claude Code is spawned within the 900 s timeout, returns the cwd, output is captured under the 24,000-char cap. **If it hangs or auths interactively**, the external CLI needs non-interactive credentials on the service env — note for ginnoir; the `hermes`-target path (Step 5) still works.
- [ ] **Step 7: Checkpoint (host note + reversibility recorded)**
No git commit. Record in session notes: skill installed at `~/.hermes/skills/hermes-acp-orchestrator/`, config backup at `~/.hermes/config.yaml.bak.acp.*`. **Rollback** = `rm -rf ~/.hermes/skills/hermes-acp-orchestrator`, restore the `.bak.acp.*`, restart gateway.
---
## Task 3: Install `hermes-curator-evolver` (self-improvement, report-only)
**Files:**
- Host: `~/.hermes/plugins/curator-evolver/` (plugin + `data/evidence.sqlite`), systemd **user** timer.
- [ ] **Step 1: Read the plugin source before installing (it can write to skills)**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "git -C /storage1/hermes/workspace/clones clone https://github.com/pingchesu/hermes-curator-evolver && sed -n '1,160p' /storage1/hermes/workspace/clones/hermes-curator-evolver/README.md"
```
Expected: repo clones; README prints. Confirm the apply path requires `--approve` (it does per the README) and that `auto-run` without `--apply-low-risk --approve-auto-apply` is **dry-run only**.
- [ ] **Step 2: Install the plugin (no semantic/embedding extras — keep it off the P100)**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc '~/.local/bin/hermes plugins install pingchesu/hermes-curator-evolver --enable && uv pip install --python ~/.hermes/hermes-agent/venv/bin/python -e ~/.hermes/plugins/curator-evolver && ~/.hermes/hermes-agent/venv/bin/hermes-curator-evolver bootstrap'"
```
Expected: plugin installs to `~/.hermes/plugins/curator-evolver`; editable pip install succeeds; `bootstrap` configures and installs a **systemd user timer**. **Do NOT pass `--semantic`** (that pulls Qwen/BGE models — CPU/VRAM cost we're avoiding for now; BM25/FTS ranking is the v1 default).
- [ ] **Step 3: Backfill recent sessions and generate the first dry-run report**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes-curator-evolver backfill-sessions --sessions-dir ~/.hermes/sessions --days 30 --format json 2>&1 | tail -20 && ~/.hermes/hermes-agent/venv/bin/hermes-curator-evolver report --days 7 --format json 2>&1 | tail -40"
```
Expected: evidence is mined into `~/.hermes/plugins/curator-evolver/data/evidence.sqlite`; `report` prints a JSON summary of candidate skill improvements. **No skill files are modified** (report is read-only).
- [ ] **Step 4: Generate a dry-run proposal for one skill and inspect it**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "~/.hermes/hermes-agent/venv/bin/hermes-curator-evolver auto-run --skills-dir ~/.hermes/skills --format json 2>&1 | tail -60"
```
Expected: a JSON set of **proposed** (not applied) edits. Confirm no files under `~/.hermes/skills` changed:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "find ~/.hermes/skills -newermt '-10 minutes' -type f 2>/dev/null"
```
Expected: empty output (nothing modified) — proves dry-run safety.
- [ ] **Step 5: Confirm the scheduled timer is report-only**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc 'systemctl --user list-timers \"hermes-curator-evolver*\" --all --no-pager' && ssh -o BatchMode=yes ginnoir@valhalla \"systemctl --user cat 'hermes-curator-evolver*' 2>&1 | grep -iE 'ExecStart|approve|apply'\""
```
Expected: a timer is listed; its `ExecStart` runs `auto-run` **without** `--apply-low-risk`/`--approve-auto-apply`. **If the bootstrap-installed unit includes those flags, override it** to remove them (the morning decision in spec §6.2 defaults to report-only). If user-lingering isn't enabled the timer won't fire across logout — enable with `sudo loginctl enable-linger ginnoir` (note for ginnoir).
- [ ] **Step 6: Checkpoint (host note + reversibility recorded)**
No git commit. Record: plugin at `~/.hermes/plugins/curator-evolver`, DB at `.../data/evidence.sqlite`, timer name, report-only confirmed. **Rollback** = `systemctl --user disable --now <timer>`, `~/.local/bin/hermes plugins uninstall curator-evolver` (verify exact uninstall verb), `rm -rf ~/.hermes/plugins/curator-evolver`.
---
## Task 3b: Install `hermes-motif` (skill DISCOVERY, proposal-only)
> Complements curator-evolver — does **not** compete with it (spec §7.1). motif *discovers new*
> skills by mining repeated tool sequences; curator *refines existing* ones. **Zero P100 cost**
> (makes no LLM calls). Together with eagle-eye (routing) they form a skill factory:
> motif creates → curator refines → eagle-eye routes.
**Files:**
- Host: `~/.hermes/plugins/` (motif plugin), `~/.hermes/plugins/<motif>/plugin/plugin.yaml`.
- [ ] **Step 1: Clone and read; confirm proposal-only config**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "git -C /storage1/hermes/workspace/clones clone https://github.com/Saurav0989/hermes-motif && sed -n '1,160p' /storage1/hermes/workspace/clones/hermes-motif/README.md && cat /storage1/hermes/workspace/clones/hermes-motif/plugin/plugin.yaml 2>&1"
```
Expected: README + `plugin.yaml` print. **Confirm `auto_install: false`** (proposal-only) and note
`min_occurrences` / sequence-length thresholds. Note the referenced Hermes trace bug (#12922) that
can affect trace completeness — acceptable for a proposal-only trial.
- [ ] **Step 2: Install the plugin**
Run (per its README — clone + pip + `scripts/install_plugin.sh`):
```bash
ssh -o BatchMode=yes ginnoir@valhalla "cd /storage1/hermes/workspace/clones/hermes-motif && bash scripts/install_plugin.sh 2>&1 | tail -20"
```
Expected: the plugin lands under `~/.hermes/plugins/` and registers. (If the script expects a
different layout, follow the README's exact steps.)
- [ ] **Step 3: Verify it mines and PROPOSES without modifying skills**
Restart the gateway, run the agent through a couple of repeated multi-tool workflows, then check for
proposals (drafts), confirming nothing under `~/.hermes/skills` was auto-written:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo systemctl restart hermes-gateway.service && sleep 5 && find ~/.hermes/plugins -iname '*propos*' -o -iname '*draft*' 2>/dev/null | head && find ~/.hermes/skills -newermt '-10 minutes' -type f 2>/dev/null"
```
Expected: proposal/draft artifacts may appear under the plugin dir; the second `find` is **empty**
(no skill files auto-modified) — proves `auto_install: false` safety.
- [ ] **Step 4: Checkpoint**
No repo commit (host-side). **Rollback** = remove the motif plugin dir + restart gateway. Record in
`memory/hermes-extensions.md` alongside curator (skill factory: motif=create, curator=refine).
---
## Task 4: Document Phase 1 in memory + vault (durable knowledge)
**Files:**
- Memory: `C:\Users\MattC\.claude\projects\C--Users-MattC-Documents-homelabstack\memory\hermes-extensions.md` + `MEMORY.md` pointer.
- Vault: append to the Hermes project note via Obsidian MCP (`mcp__obsidian__*`).
- [ ] **Step 1: Write the memory file**
Create `memory/hermes-extensions.md` (frontmatter `type: project`) recording: acp-skill installed (delegation to hermes/codex/claude-code/cursor/antigravity, 900s/24k caps); curator-evolver installed **report-only** (CPU ranking, no `--semantic`, no auto-apply flags); **motif** installed proposal-only (skill factory: motif creates → curator refines → eagle-eye routes); **claude v2.1.195 + codex v0.142.3 installed on valhalla 2026-06-27** (login pending; `~/.claude` was root-owned → chowned); exact paths and rollback commands; the host-vs-repo boundary. Link `[[llm-stack-hermes]]`, `[[multi-agent-tool-configs]]`, `[[obsidian-app-on-valhalla]]`.
- [ ] **Step 2: Add the MEMORY.md index pointer**
Append one line to `MEMORY.md`:
`- [Hermes host extensions](hermes-extensions.md) — acp delegation skill + curator-evolver (report-only) on valhalla; host-managed in ~/.hermes, not in the repo`
- [ ] **Step 3: Write back to the Obsidian vault**
Per the global rule, use the Obsidian MCP (never write CouchDB directly) to append a session note to the Hermes project folder summarizing Phase 1 (what, why report-only, rollback). If the MCP is unreachable, tell ginnoir and skip — do not hand-edit.
- [ ] **Step 4: Checkpoint**
No code commit required (memory files live outside the repo). Phase 1 complete and documented.
---
# PHASE 2 — New capability + experiment
> Decisions resolved (spec §6.3 eagle-eye trial; §6.4 camofox minimal). Actionable.
## Task 5: `camofox-browser` as a homelab stack (Class B)
**Files:**
- Create: `stacks/camofox/docker-compose.yml`, `stacks/camofox/stack.env`.
- Modify: `Caddyfile` (new site block), `bookmarks-domains.html` + `bookmarks-ports.html` (regenerated).
- Host (image): build under `/storage1/hermes/workspace/clones/camofox-browser`.
- [ ] **Step 1: Decide image provenance and build it**
camofox publishes **no registry image** (`make up` builds locally). Recommended default: build on
valhalla and tag `camofox-browser:local`, reference that tag from compose (Watchtower already
disabled for pinned infra). Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "git -C /storage1/hermes/workspace/clones clone https://github.com/jo-inc/camofox-browser && cd /storage1/hermes/workspace/clones/camofox-browser && docker build -t camofox-browser:local . 2>&1 | tail -20 && docker image ls camofox-browser:local"
```
Expected: image builds; `camofox-browser:local` is listed. **Alternative (if a reproducible/Gitea-Actions build is preferred, like famapp):** build + push to `registry.ginnoir.com/ginnoir/camofox-browser` and reference that instead — flag this choice for ginnoir.
- [ ] **Step 2: Write the stack compose**
Create `stacks/camofox/docker-compose.yml`:
```yaml
# camofox stack — stealth headless browser REST API for the Hermes agent.
# No published image: built on-host as camofox-browser:local (see plan Task 5).
# Internal-only; reachable by Caddy over edge and by host-side Hermes.
services:
camofox:
image: camofox-browser:local
container_name: camofox
restart: unless-stopped
labels:
- "com.centurylabs.watchtower.enable=false"
env_file:
- stack.env
networks: [edge, camofox]
volumes:
- /config/camofox/cookies:/home/node/.camofox/cookies
- /config/camofox/profiles:/home/node/.camofox/profiles
ports:
- "172.20.0.1:9377:9377"
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost:9377/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 40s
networks:
edge:
external: true
camofox:
name: camofox
driver: bridge
```
(The `172.20.0.1:9377` host-port mirrors the llm stack's pattern so host-side Hermes can reach it directly; Caddy reaches it over `edge` by container name.)
- [ ] **Step 3: Write `stack.env` (secrets; LF endings)**
Create `stacks/camofox/stack.env` with a generated bearer key (replace the value with a real secret before push):
```
CAMOFOX_ACCESS_KEY=GENERATE_A_LONG_RANDOM_KEY
CAMOFOX_ADMIN_KEY=GENERATE_A_SECOND_RANDOM_KEY
CAMOFOX_PORT=9377
```
Generate the keys: `ssh ... "openssl rand -hex 32"` (run twice). **Ensure LF line endings** (`.gitattributes` enforces this — verify the file isn't CRLF before committing). Leave `CAMOFOX_API_KEY` unset (cookie-import endpoint stays disabled).
- [ ] **Step 4: Add the Caddy site block (internal-only)**
Add to `Caddyfile` (place near other internal admin services). Since camofox enforces its own bearer auth and Hermes calls it machine-to-machine, gate by LAN only (no Authentik forward-auth, which would block the agent's API calls):
```caddy
camofox.ginnoir.com {
import internal_only
reverse_proxy camofox:9377
}
```
- [ ] **Step 5: Create host config dirs, regenerate bookmarks, then deploy**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo mkdir -p /config/camofox/cookies /config/camofox/profiles && sudo chown -R ginnoir:ginnoir /config/camofox"
```
Then regenerate bookmarks and push (Portainer must have the `stacks/camofox` git stack registered — see Step 6):
```powershell
./scripts/gen-bookmarks.ps1
git add stacks/camofox/ Caddyfile bookmarks-domains.html bookmarks-ports.html
git commit -m "feat(camofox): stealth browser stack for the Hermes agent"
git push
```
Expected: commit + push; Gitea Actions reloads Caddy (Caddyfile changed); Portainer redeploys the camofox stack within 5 min.
- [ ] **Step 6: Register the stack in Portainer if new, and verify it runs**
New stacks need one-time Portainer registration (see `portainer-new-stack-registration` memory). After deploy, verify:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "docker ps --filter name=camofox --format '{{.Names}} {{.Status}}' && curl -fsS http://172.20.0.1:9377/health"
```
Expected: container `Up (healthy)`; `/health` returns OK.
- [ ] **Step 7: Smoke-test the browser API end-to-end**
Run (creates a tab, snapshots a page):
```bash
ssh -o BatchMode=yes ginnoir@valhalla "K=\$(grep CAMOFOX_ACCESS_KEY /config/portainer/compose/*/stacks/camofox/stack.env | cut -d= -f2); ID=\$(curl -fsS -H \"Authorization: Bearer \$K\" -H 'Content-Type: application/json' -d '{\"userId\":\"smoke\",\"sessionKey\":\"t1\",\"url\":\"https://example.com\"}' http://172.20.0.1:9377/tabs | python3 -c 'import sys,json;print(json.load(sys.stdin)[\"id\"])'); curl -fsS -H \"Authorization: Bearer \$K\" \"http://172.20.0.1:9377/tabs/\$ID/snapshot?userId=smoke\" | head -20"
```
Expected: a tab id comes back; the snapshot returns accessibility text containing "Example Domain". (Adjust the JSON id field name to match the real response from Step 1's README read.)
- [ ] **Step 8: Wire camofox into Hermes as a minimal tool surface (per §6.4 decision)**
Default recommendation: a **small Hermes skill** (2 high-level tools — `browse(url)` and `search(query)`) that curls camofox, rather than exposing the full REST surface (respects the tool-budget that keeps gpt-oss-20b functional). Create `~/.hermes/skills/camofox-browse/SKILL.md` documenting the two operations against `http://172.20.0.1:9377` with the bearer key, restart the gateway, and smoke-test:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo systemctl restart hermes-gateway.service && sleep 5 && ~/.local/bin/hermes run 'browse https://example.com and tell me the page heading' 2>&1 | tail -20"
```
Expected: Hermes uses the camofox tool and reports "Example Domain". **If §6.4 chose an MCP shim instead**, build/register the MCP server and add it to `mcp_servers:` with a 2-tool `tools.include` allowlist (per the MCP-curation pattern in `llm-stack-hermes`).
- [ ] **Step 9: Checkpoint**
Repo changes are committed (Step 5). Update `memory/hermes-extensions.md` + the vault note with the camofox stack + tool wiring and the bearer-key location.
---
## Task 5b: Install `hermes-web-search-plus` (multi-provider search; pairs with camofox)
> Mature (v2.6.1, MIT, stdlib-only). Complements camofox (spec §7.2): search-plus *finds* via cheap
> provider APIs, camofox *browses/renders*. Lighter and higher-frequency — good default reach-for.
**Files:**
- Host: `~/.hermes/plugins/` (plugin), provider key(s) in `~/.hermes/config.yaml` (or the plugin's config).
- [ ] **Step 1: Install the plugin**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc '~/.local/bin/hermes plugins install robbyczgw-cla/hermes-web-search-plus --enable'"
```
Expected: plugin installs and enables.
- [ ] **Step 2: Configure at least one provider key (free tier)**
All provider keys are optional but ≥1 is needed to function. Pick a free-tier provider (e.g. Tavily,
Exa, or self-hosted SearXNG; Keenable has a keyless public tier). Add the key per the plugin's README
(read `/storage1/hermes/workspace/clones/...` or the plugin docs for the exact env/config key), then:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo systemctl restart hermes-gateway.service && sleep 5 && systemctl is-active hermes-gateway.service"
```
Expected: gateway `active`. **Mind the tool-budget rule** — if it exposes both `web_search_plus` +
extract, that's fine (2 tools); don't also enable redundant search MCPs.
- [ ] **Step 3: Smoke-test a search**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "~/.local/bin/hermes run 'search the web for the latest Hermes Agent release version and cite the source' 2>&1 | tail -20"
```
Expected: the agent calls the search tool, returns a current result with a source URL.
- [ ] **Step 4: Checkpoint**
No repo commit (host-side). Document the chosen provider + key location in `memory/hermes-extensions.md`.
---
## Task 6: Trial `eagle-eye` skill pre-filter (Class A, behind a switch)
> Confirmed the chosen tool: eagle-eye is the **only** direct skill-router in the Hermes ecosystem
> (per `awesome-hermes-agent` / Hermes Atlas). It directly serves the goal of "many skills installed,
> few injected per turn." Complementary (not a substitute) and worth a later look on the *tool* side:
> `llmtrim` (compresses tool schemas + MCP output before each request). `hermes-motif` overlaps
> curator-evolver (trace→micro-skill), not this router.
**Files:**
- Host: `~/.hermes/plugins/eagle-eye/` (or skills dir per its README), config toggle in `~/.hermes/config.yaml`.
- [ ] **Step 1: Clone and read; confirm graceful-degradation and the jieba dependency**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "git -C /storage1/hermes/workspace/clones clone https://github.com/willingning-coder/eagle-eye && sed -n '1,200p' /storage1/hermes/workspace/clones/eagle-eye/README.md"
```
Expected: README prints. Confirm the install hook, the on/off switch, and that L2L5 (incl. dense embeddings) are optional. **Plan to run with the dense layer disabled** (CPU/keep off the P100) — lean on L1 (hard triggers) + L2 (BM25) only for the trial.
- [ ] **Step 2: Install with an easy off-switch and minimal deps**
Install per the README (likely `~/.local/bin/hermes plugins install willingning-coder/eagle-eye --enable`), then restart the gateway:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc '~/.local/bin/hermes plugins install willingning-coder/eagle-eye --enable' && ssh -o BatchMode=yes ginnoir@valhalla 'sudo systemctl restart hermes-gateway.service && systemctl is-active hermes-gateway.service'"
```
Expected: plugin enabled; gateway `active`. (If install fails on `jieba`, `uv pip install --python ~/.hermes/hermes-agent/venv/bin/python jieba` then retry — note the foreign-language dep for maintenance.)
- [ ] **Step 3: A/B test skill selection on representative prompts**
Pick 5 prompts that should each map to a known skill and 2 that should map to none. Run each with eagle-eye enabled, then disable it (`~/.local/bin/hermes plugins disable eagle-eye` + gateway restart) and run the same 7. Record which skills each surfaced and whether the local model then picked the right one.
```bash
ssh -o BatchMode=yes ginnoir@valhalla "~/.local/bin/hermes run '<representative prompt>' 2>&1 | tail -25"
```
Expected: with eagle-eye on, the prompt's prompt-injected skill candidates are ≤5 and include the right one; the "no skill needed" prompts proceed without forced skill loading.
- [ ] **Step 4: Keep-or-cut decision**
**Keep only if** skill selection measurably improved (right skill surfaced more often AND/OR fewer wrong skills loaded) without regressions. Otherwise disable and uninstall:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "~/.local/bin/hermes plugins uninstall eagle-eye && sudo systemctl restart hermes-gateway.service"
```
Record the verdict + evidence in `memory/hermes-extensions.md`.
- [ ] **Step 5: Checkpoint**
No repo commit (host-side). Document the A/B result and final state (kept/cut) in memory + vault.
---
# PHASE 2b — Delegation fabric & context efficiency
> Extends acp-skill (Task 2) from 3 targets to 4 external agents, and adds optional token-trimming.
## Task 10: Wire Cursor + Antigravity into the delegation fabric
**Files:**
- Host: Cursor + `agy` binaries; acp-skill config or a generic shell-agent skill in `~/.hermes/skills/`.
- [ ] **Step 1: Install the Cursor CLI (official cursor.com)**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc 'curl https://cursor.com/install -fsS | bash' && ssh -o BatchMode=yes ginnoir@valhalla 'bash -lc \"command -v cursor-agent && cursor-agent --version\"'"
```
Expected: `cursor-agent` installs and prints a version. ginnoir logs in later.
- [ ] **Step 2: Install the Antigravity CLI (`agy`) from the OFFICIAL Google source**
Do **not** use blog-derived URLs. Get the exact installer from the official pages first:
`https://antigravity.google/download` and `https://antigravity.google/docs/gcli-migration`. Then run
the official one-line installer they document, e.g.:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc '<official agy installer from antigravity.google/docs>' && ssh -o BatchMode=yes ginnoir@valhalla 'bash -lc \"command -v agy && agy --version\"'"
```
Expected: `agy` (Go binary, ideal for headless SSH) installs and prints a version. Auth later via
keyring/Google sign-in or `ANTIGRAVITY_TOKEN`.
- [ ] **Step 3: Confirm each agent answers in headless mode (after ginnoir logs in)**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc 'claude -p \"say PONG\"; codex exec \"say PONG\"; cursor-agent -p --output-format json --trust \"say PONG\"; agy -p \"say PONG\"'"
```
Expected: each prints PONG-ish output. **Cursor caveat:** `-p` has a known hang bug — always pass
`--output-format json` and wrap with a timeout (`timeout 120 cursor-agent ...`).
- [ ] **Step 4: Extend acp-skill routing (or add a generic shell-agent skill)**
Read `~/.hermes/skills/hermes-acp-orchestrator/SKILL.md` to see if `agent=` routing is extensible.
- **If extensible:** add `cursor` and `antigravity` targets mapping to the Step 3 invocations
(with the cursor timeout + json flags), honoring the `delegation:` `external_timeout_seconds: 900`
/ `external_max_output_chars: 24000` caps.
- **If not:** add `~/.hermes/skills/shell-agent/SKILL.md` exposing **one** tool
`delegate(agent, goal)` that shells out to claude/codex/cursor/agy with the caps + cursor guard.
One tool keeps the surface within the tool-budget.
Then restart the gateway:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo systemctl restart hermes-gateway.service && sleep 5 && systemctl is-active hermes-gateway.service"
```
- [ ] **Step 5: End-to-end smoke-test each delegation target**
Run one delegated task per agent (e.g. `agent=cursor`, `agent=antigravity`) and confirm output is
captured under the cap. Record any that hang/auth-fail for follow-up.
- [ ] **Step 6: Checkpoint**
No repo commit (host-side). Document the four-target fabric + cursor caveat in `memory/hermes-extensions.md`.
---
## Task 11: (OPTIONAL) Context efficiency — llmtrim on the cloud-delegation path
> Opt-in. Start where the win is unambiguous and local-risk-free: trimming the **cloud** delegation
> agents' traffic (Claude Code/Codex/Cursor → Anthropic/OpenAI). Defer the llama-swap-fronting idea
> until validated. `rtk-hermes` (shell-output trimming) is a separate lighter opt-in.
**Files:**
- Host: llmtrim service + `HTTPS_PROXY` env for the delegation agents.
- [ ] **Step 1: Install llmtrim and run setup**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "bash -lc 'npm install -g @llmtrim/cli@latest && llmtrim setup' 2>&1 | tail -20"
```
Expected: installs; `setup` installs the name-constrained CA + background proxy. **Review the MITM-CA
trust implication first** — it's name-constrained to LLM API domains, but it's still a CA on the host.
- [ ] **Step 2: Point the cloud delegation agents through it; measure**
Ensure the delegation agents inherit `HTTPS_PROXY` (llmtrim sets this). Run a representative delegated
task via Claude Code/Codex and compare token counts / cost before vs after (llmtrim reports savings).
Keep only if the reduction is real with no quality loss.
- [ ] **Step 3: (Later, separate) Evaluate llama-swap fronting + rtk-hermes**
Document — do not implement here — the two deferred ideas: (a) llmtrim in front of `172.20.0.1:8090`
via `LLMTRIM_EXTRA_HOSTS` to shrink prompts and speed Pascal prefill (needs validation; unproven for
local OpenAI-compatible backends); (b) `rtk-hermes` (`pre_tool_call` shell rewrite) if the `rtk`
binary is available on Ubuntu.
- [ ] **Step 4: Checkpoint**
No repo commit. Record the decision + measured savings (or rejection) in `memory/hermes-extensions.md`.
`llmtrim uninstall` fully reverses the proxy + CA if rejected.
---
# PHASE 3 — UI trial: stand up BOTH, keep the winner
> Decision resolved (spec §6.1): trial `hermes-ui` (Task 7) **and** `hermes-workspace` (Task 8) in
> parallel, compare head-to-head against the bundled webui (Task 9), keep one and tear down the
> rest. `mission-control` is skipped.
## Task 7: Trial `hermes-ui` (lightweight, no build)
**Files:**
- Host: clone at `/storage1/hermes/workspace/clones/hermes-ui`; optional `hermes-ui.service` (host unit) or a tiny container; Caddy block if exposed.
- [ ] **Step 1: Clone and run the stdlib proxy against the live gateway**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "git -C /storage1/hermes/workspace/clones clone https://github.com/pyrate-llama/hermes-ui && cd /storage1/hermes/workspace/clones/hermes-ui && (~/.hermes/hermes-agent/venv/bin/python3 serve_lite.py >/tmp/hermes-ui.log 2>&1 &) && sleep 3 && curl -fsS http://127.0.0.1:3333/hermes-ui.html | head -5"
```
Expected: the proxy starts on :3333 (defaults to gateway `127.0.0.1:8642`, which matches your deployment), and the HTML serves. If your gateway port differs, edit the `HERMES` variable at the top of `serve_lite.py` (no env var exists).
- [ ] **Step 2: Expose it on the LAN for evaluation (don't finalize yet)**
Bind the proxy to the host IP so Caddy can reach it, add a temporary internal-only Caddy block, and
keep it running for the Task 9 comparison:
```caddy
hermes-ui.ginnoir.com {
import internal_only
reverse_proxy 172.20.0.1:3333
}
```
Run `serve_lite.py` bound appropriately (edit its bind host if it defaults to `127.0.0.1`), regenerate
bookmarks, push the Caddyfile change. **Do not** create the persistent `hermes-ui.service` yet — that
happens in Task 9 only for the winner.
- [ ] **Step 3: Checkpoint**
hermes-ui is reachable at `https://hermes-ui.ginnoir.com` (LAN) for the head-to-head. Leave the
final keep/revert + boot-persistence to Task 9.
---
## Task 8: Deploy `hermes-workspace` as a stack (Class B) — for evaluation
**Files:**
- Create: `stacks/hermes-workspace/docker-compose.yml`, `stacks/hermes-workspace/stack.env`.
- Modify: `Caddyfile` (Authentik-gated site block), `bookmarks-domains.html` + `bookmarks-ports.html`.
- Host (image): build under `/storage1/hermes/workspace/clones/hermes-workspace`.
- [ ] **Step 1: Clone and read its deployment docs (get exact build, ports, env)**
Run:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "git -C /storage1/hermes/workspace/clones clone https://github.com/outsourc-e/hermes-workspace && sed -n '1,200p' /storage1/hermes/workspace/clones/hermes-workspace/README.md && ls /storage1/hermes/workspace/clones/hermes-workspace/{Dockerfile,docker-compose*.yml,.env*} 2>&1"
```
Expected: README + a `Dockerfile`/compose appear. Record the exact image build command, the served
port, and the env var(s) that point the frontend at the gateway (`:8642`) and dashboard (`:9119`).
**Note the swarm caveat for Task 9:** Swarm Mode (tmux worker pools) can't parallelize inference on
one P100 — evaluate the workspace/observability features, not swarm.
- [ ] **Step 2: Resolve container→host-service reachability**
hermes-workspace (a container) must reach the host's gateway `:8642` and dashboard `:9119`. Check
what interface those bind to:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "ss -ltnp | grep -E ':8642|:9119'"
```
Expected: shows the bind address. **If bound to `127.0.0.1`**, the container can't reach them — pick
one: (a) add `extra_hosts: ["host.docker.internal:host-gateway"]` and target `host.docker.internal`,
or (b) rebind the Hermes services to the docker-bridge host IP `172.20.0.1` (config change + gateway
restart, with backup). Default recommendation: **(a)** (no Hermes config change; reversible).
- [ ] **Step 3: Build the image on-host**
Run (use the build command discovered in Step 1; tag locally since there's no published image):
```bash
ssh -o BatchMode=yes ginnoir@valhalla "cd /storage1/hermes/workspace/clones/hermes-workspace && docker build -t hermes-workspace:local . 2>&1 | tail -20 && docker image ls hermes-workspace:local"
```
Expected: `hermes-workspace:local` is built and listed.
- [ ] **Step 4: Write the stack compose**
Create `stacks/hermes-workspace/docker-compose.yml` (adjust the served port and gateway/dashboard env
keys to Step 1's findings; this uses host.docker.internal per Step 2 option (a)):
```yaml
# hermes-workspace stack — full web command center for the Hermes agent (trial).
# No published image: built on-host as hermes-workspace:local (see plan Task 8).
# Human-facing UI → Authentik-gated. Reaches host gateway :8642 + dashboard :9119
# via host.docker.internal.
services:
hermes-workspace:
image: hermes-workspace:local
container_name: hermes-workspace
restart: unless-stopped
labels:
- "com.centurylabs.watchtower.enable=false"
env_file:
- stack.env
networks: [edge]
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "172.20.0.1:8088:8088"
networks:
edge:
external: true
```
- [ ] **Step 5: Write `stack.env` (gateway/dashboard targets; LF endings)**
Create `stacks/hermes-workspace/stack.env` using the real env keys from Step 1, e.g.:
```
HERMES_GATEWAY_URL=http://host.docker.internal:8642
HERMES_DASHBOARD_URL=http://host.docker.internal:9119
PORT=8088
```
Verify LF endings before committing.
- [ ] **Step 6: Add an Authentik-gated Caddy block**
Unlike camofox (machine-to-machine), this is a human UI → gate with Authentik forward_auth (Pattern B):
```caddy
workspace.ginnoir.com {
import internal_only
route {
import authentik_outpost
import authentik_forward_auth
reverse_proxy hermes-workspace:8088
}
}
```
- [ ] **Step 7: Deploy and verify**
Run:
```powershell
./scripts/gen-bookmarks.ps1
git add stacks/hermes-workspace/ Caddyfile bookmarks-domains.html bookmarks-ports.html
git commit -m "feat(hermes-workspace): trial command-center stack (eval vs hermes-ui)"
git push
```
Register the stack in Portainer if new, then:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "docker ps --filter name=hermes-workspace --format '{{.Names}} {{.Status}}' && curl -fsS http://172.20.0.1:8088/ | head -5"
```
Expected: container `Up`; the workspace HTML serves; logging into `https://workspace.ginnoir.com`
via Authentik shows live chat/memory/skills wired to your gateway.
- [ ] **Step 8: Checkpoint**
Repo changes committed (Step 7). Leave the keep/tear-down decision to Task 9.
---
## Task 9: Head-to-head UI decision — keep one, tear down the rest
**Files:**
- Modify (on tear-down): `Caddyfile`, `stacks/...` (remove the loser), bookmarks; host unit for the winner.
- [ ] **Step 1: Compare bundled webui vs hermes-ui vs hermes-workspace**
Use all three live for representative work (chat/streaming, tasks/kanban, files, terminal, skills,
MCP browser, cron, memory, health). Score against: does it surface your curated tools cleanly, does
it stay responsive against the P100's latency, and does it add real value over the bundled webui.
Record the verdict in the vault.
- [ ] **Step 2: Make the winner permanent**
- If **hermes-ui** wins: create host unit `hermes-ui.service` (host-managed, like `obsidian.service`
— NOT in this repo), `After=hermes-gateway.service`, `Restart=on-failure`; keep its Caddy block.
- If **hermes-workspace** wins: keep its stack + Authentik block as-is.
- If **bundled webui** wins: keep status quo.
- [ ] **Step 3: Tear down the losers (reversible, clean)**
- Remove the hermes-workspace stack if it lost: delete `stacks/hermes-workspace/`, its Caddy block,
regenerate bookmarks, commit + push, then delete the stack in Portainer and
`docker rm -f hermes-workspace`, `docker image rm hermes-workspace:local`.
- Stop/remove hermes-ui if it lost: `pkill -f 'serve_lite[.]py'` (bracket trick), remove its Caddy
block + clone, commit the Caddyfile change.
- [ ] **Step 4: Checkpoint**
One UI kept and documented in memory + vault; losers fully removed; repo reflects the final state.
---
## Self-Review (completed)
- **Spec coverage:** Original 7 repos — acp-skill (T2), curator-evolver (T3), camofox (T5),
eagle-eye (T6), hermes-ui (T7), hermes-workspace (T8 deploy) + keep-one decision (T9);
mission-control (skipped per spec §2.5/§5, intentional). **Ecosystem expansion (spec §7)**
hermes-motif (T3b), hermes-web-search-plus (T5b), delegation fabric for cursor+antigravity (T10),
optional llmtrim/rtk context efficiency (T11). Claude Code + Codex install is **done** (T1 Step 3).
Phase ordering, single-P100 discipline, host-vs-repo boundary, provenance (official installers
only — Antigravity URL verified to `antigravity.google`), reversibility, and the §6 decisions are
all reflected.
- **Placeholders:** None of the prohibited kinds. Where a third-party command form can't be verified
remotely (e.g. exact `hermes` subcommand spelling, acp-skill install mechanism, response field
names), the plan's **first step is a concrete "clone + read the README/SKILL.md" command** that
resolves it before use — a real action with expected output, not a TBD.
- **Consistency:** Paths and names are consistent throughout (`~/.hermes/skills`, `~/.hermes/plugins/curator-evolver`, `camofox-browser:local`, port 9377, gateway 8642, `172.20.0.1` host-IP publish pattern, `sudo systemctl restart hermes-gateway.service`).
- **Decision gates:** Phases 23 are clearly gated on spec §6 and must not start before ginnoir answers.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,376 @@
# Hermes Ecosystem Integration — Triage & Design
**Date:** 2026-06-27
**Author:** Claude (overnight autonomous research for ginnoir; for morning review)
**Status:** Draft — awaiting ginnoir's decisions on the flagged items below
---
## ⚠️ REVISION 2026-06-27 (during execution — live-CLI findings supersede README research)
This spec was written from the repos' READMEs. Executing Phase 1 against the **live** valhalla host
revealed **Hermes v0.17 already ships native equivalents** of several planned adds. Net changes:
- **`hermes-curator-evolver` → DROPPED.** Native `hermes curator` is already enabled, weekly, and
conservative (`consolidate: off`, prune-only, never deletes, auto-backups), managing **74
agent-created skills**. It satisfies the "report-only skill maintenance" intent. Installing the
plugin would put a second curator on the same `~/.hermes/skills/`. Use the native one (tune config
line 417 if desired).
- **`camofox-browser` → RE-EVALUATE.** Native bundled plugins `browser-browserbase`
(stealth/proxies/keep-alive, cloud) + `browser-browser-use`, plus `hermes acp --setup-browser`
(local Playwright), already cover browsing. camofox now only wins the *self-hosted-stealth-on-
valhalla, no-cloud-key* niche. Otherwise enable a native browser plugin (far less work).
- **`hermes-agent-acp-skill` → KEPT & INSTALLED (enabled).** Native `hermes acp` is an *editor-
integration server* (Zed/VS Code/JetBrains), distinct from this delegation skill; they coexist.
- **`eagle-eye` → KEPT / ELEVATED.** 74 agent skills + 59 bundled makes per-turn skill routing more
valuable (confirm native doesn't already pre-select).
- **`hermes-motif` → OPTIONAL / lower priority.** Still additive (native curator does lifecycle, not
trace→new-skill discovery), but the library is already large and well-managed.
- **`hermes-web-search-plus` → KEPT** (no native web search observed). **`llmtrim`** overlaps native
`hermes insights` for *measurement*; its trimming value stands.
**As-built so far:** acp-orchestrator skill installed + enabled. Cursor CLI installed; Claude Code +
Codex installed (login pending). `agy` (Antigravity) install blocked by the sandbox classifier — see
§7.4. The phase tables below are otherwise unchanged; treat the bullets above as the authoritative
deltas.
---
## 0. TL;DR for the morning
You asked me to research seven Hermes-ecosystem repos and plan how each fits. I verified all
seven are real and active, mapped each against your **actual** Hermes deployment (host systemd
install on valhalla, llama-swap/P100 backend, ~29-tool curated MCP surface, Codex + Claude Code
also installed), and triaged them into adopt / trial / decide / skip.
**Recommended path:**
| Phase | Repo | Verdict | Why |
|---|---|---|---|
| **1** | `hermes-agent-acp-skill` | **ADOPT** | Pure skill drop-in; lets the local 20B offload hard work to Codex/Claude Code, which you already run. Highest leverage, lowest risk. |
| **1** | `hermes-curator-evolver` | **ADOPT (dry-run first)** | Closes Hermes' self-improvement loop; read-only by default; aligns with your curation habit. |
| **2** | `camofox-browser` | **ADOPT (new homelab stack)** | Gives the local agent real web browsing; clean Docker/Caddy fit; independent of Hermes internals. |
| **2** | `eagle-eye` | **TRIAL / DEFER** | Solves the exact tool-overload pain you already hit — but very early (5★, 1 commit) and carries a required Chinese-tokenizer dep. Trial behind a flag; don't depend on it. |
| **3** | `hermes-ui` | **ADOPT (trial)** | Single-HTML + stdlib proxy; stood up next to the bundled webui to evaluate. |
| **3** | `hermes-workspace` | **ADOPT (trial)** | Full command center; stood up in parallel to compare head-to-head, then keep one. Swarm Mode won't parallelize on one P100 — value is the workspace/observability features. |
| **3** | `mission-control` | **SKIP** | Not Hermes-native (built for OpenClaw/CrewAI/LangGraph/AutoGen). Redundant with the two Hermes UIs. |
**Decisions — RESOLVED 2026-06-27 (see Section 6):** (1) trial **both** UIs, keep the winner;
(2) curator-evolver stays **report-only** for now; (3) **trial eagle-eye** — it's the only direct
skill-router in the ecosystem, no more-mature equivalent exists; (4) wire camofox as a **minimal
2-tool skill**.
---
## 1. The anchor: your current Hermes deployment
Everything below is sized against what you actually run (from project memory, verified against the
repo and host conventions):
- **Hermes Agent v0.17.0 (NousResearch)** runs on the **valhalla host**, not in Docker — systemd
units `hermes-gateway.service`, `hermes-dashboard.service`, `hermes-webui.service`. Editable
install at `~/.hermes/hermes-agent/`; config at `~/.hermes/config.yaml`; CLI `~/.local/bin/hermes`.
Workspace at `/storage1/hermes/workspace/`.
- **Backend:** llama-swap (`stacks/llm`, Portainer id 34) on a single **Tesla P100 16GB**, serving
one GGUF at a time, `--parallel 1`, 64k ctx, default `gpt-oss-20b`. OpenAI API at
`http://172.20.0.1:8090/v1`.
- **MCP surface deliberately curated to ~29 tools** because gpt-oss-20b drowns in tool overload.
This is the single most important constraint for everything below: **the local model is weak at
selection under load, and the GPU runs one model, one sequence at a time.**
- **You also run Codex (`~/.codex`) and Claude Code**, kept in sync with your Claude global config.
This is what makes the ACP delegation skill genuinely useful rather than theoretical.
### 1.1 Two integration classes (this drives the whole plan)
- **Class A — host-side Hermes extensions** (plugins & skills): live in `~/.hermes/` on valhalla,
applied by SSH + `hermes ...` + gateway restart. **These are NOT homelabstack repo content.**
They are host-managed exactly like the rustdesk/obsidian/xvfb units — tracked in project memory
and the Obsidian vault, not committed here. Fully reversible (config backups + plugin uninstall).
`hermes-agent-acp-skill`, `hermes-curator-evolver`, `eagle-eye`.
- **Class B — Docker services**: become `stacks/<name>/` entries, deployed via the normal
Gitea-poll path, fronted by Caddy (`internal_only` + Authentik), secrets in `stack.env`,
infra images pinned out of Watchtower.
`camofox-browser`, and (if adopted) `hermes-ui` / `hermes-workspace` / `mission-control`.
---
## 2. Per-repo analysis
### 2.1 `hermes-agent-acp-skill` (Rainhoole) — **ADOPT, Phase 1**
- **What:** A Hermes *skill* (`SKILL.md` + `README.md`, MIT) implementing `delegate_task()`
multi-agent delegation across Hermes internal subagents, **Codex**, and **Claude Code**. Handles
agent routing (`agent=...`), context isolation, and safety controls (recommended 900 s external
timeout, 24,000-char output cap).
- **Fit:** Strong and specific. Your local model is gpt-oss-20b — fine for orchestration and light
tasks, weak for real implementation/review. This skill lets Hermes *route the hard parts to the
frontier CLIs you already have installed and configured*. It turns the weak local brain into a
dispatcher.
- **Risk:** Low. It's a skill file, not running code you didn't write — but it *invokes external
agents*, so the real work is verifying Codex/Claude Code are launchable from the Hermes service
environment (PATH, auth, non-interactive flags) and that timeouts/output caps are sane.
- **Implementation sketch:** Drop into `~/.hermes` skills dir (or `hermes skills install` if the
repo packages it that way — verify on the host); set delegation targets to the Codex + Claude
Code invocations; restart gateway; smoke-test a trivial delegated task end-to-end.
### 2.2 `hermes-curator-evolver` (pingchesu) — **ADOPT, dry-run first, Phase 1**
- **What:** Local-first Hermes *plugin* (Python 3.11, SQLite, `hermes plugins install`). Mines
evidence from sessions/tool-calls, ranks skill-improvement candidates, and generates **dry-run,
reviewable** proposals. **Read-only by default; mutations require explicit `--approve`.** Only
touches *local agent-created* skills — skips bundled, hub-installed, and external skills. Size
guardrails, backup/rollback manifests, optional Qwen-embeddings + BGE-reranker for semantic
ranking, native scheduler (systemd) integration.
- **Fit:** This *is* Hermes' "self-improving" thesis made operational, and it matches how you
already work (curate, gate, keep things reversible). The default posture (report-only, dry-run,
human-review queue) is exactly the safe on-ramp.
- **Risk / caveats:**
- It writes to skills with `--approve`. **Recommendation: run report-only / dry-run indefinitely
at first**; promote to `--approve` only after you've seen a few proposal batches you'd have
accepted anyway.
- Optional embeddings/reranker want a model. **Do not load them onto the P100** alongside
gpt-oss — the GPU is single-model/`--parallel 1`. Run ranking on **CPU**, or skip the dense
layer initially (BM25/FTS ranking is enough to start).
- Schedule the mining pass **off-peak** via a systemd timer so it never contends with live
inference.
- **Implementation sketch:** `hermes plugins install` on the host; point it at the session store;
run one manual `collect``rank``report` cycle by hand; inspect the review queue; only then wire
a nightly systemd timer in report-only mode.
### 2.3 `eagle-eye` (willingning-coder) — **TRIAL / DEFER, Phase 2**
- **What:** Zero-invasive skill *pre-filter* plugin. Before the LLM call, narrows 50+ skills to the
top ~5 via a 5-layer cascade (hard triggers → FTS5/BM25 → synonyms → dense embeddings → RRF
fusion), degrading gracefully when optional deps are absent. "Not matching is a valid result."
- **Fit (conceptual):** Excellent — this is the **skill-side analog of the MCP tool-overload fix
you already did**. Fewer, better skill candidates in the prompt = better selection by a weak
model + less Pascal prefill. If it works, it directly improves gpt-oss-20b's behavior.
- **Risk / caveats (why it's a trial, not an adopt):**
- **Maturity:** 5 stars, single commit. Early. Treat as experimental.
- **Dependency smell:** `jieba` (Chinese tokenization) is a *required* dep; your skill library is
English. Works, but signals the author's primary use case differs from yours — watch for
English-tokenization edge cases and maintenance drift.
- Same P100 rule: keep its dense-embedding layer on **CPU** or disabled; lean on L1L2
(deterministic + BM25) first.
- **Implementation sketch:** Install behind an easy on/off switch on the host; A/B a handful of
representative prompts with it on vs off; keep it **only if** measured skill-selection quality
improves. Do not let any workflow depend on it.
### 2.4 `camofox-browser` (jo-inc) — **ADOPT as a homelab stack, Phase 2**
- **What:** Stealth headless browser (Camoufox/Firefox) exposing a **REST API for AI agents**
anti-fingerprinting, element-reference interaction (`e1`,`e2`), session isolation, proxy/GeoIP,
YouTube transcripts, Docker/Fly/Railway deploy. 7.2k★, actively maintained. **No Hermes coupling**
— it's a generic tool backend.
- **Fit:** Clean Class-B homelab fit and a real capability gap-filler: your agent currently has
freshrss + obsidian but no general web browser. As `stacks/camofox`, internal-only behind Caddy,
it gives Hermes (and Codex/Claude) reliable browsing.
- **Risk / caveats:**
- It's anti-detection tooling — appropriate for personal research; **keep it strictly LAN/SSO-gated
(`import internal_only` + Authentik)** and off the public internet. New attack surface; pin the
image and read the Dockerfile before running.
- Heavier service (~300 MB Camoufox binary, Firefox). Fine on valhalla, but it's not free.
- **Wiring to Hermes:** either a thin MCP shim around its REST API, or a Hermes skill that curls
it. Given your tool-overload sensitivity, expose it as **one or two** high-level tools (e.g.
`browse(url)`, `search(query)`), not its full surface.
- **Implementation sketch:** New `stacks/camofox/docker-compose.yml` (bind config to `/config/camofox`,
blobs/cache to `/storage1/labdata/camofox`, join `edge`), `stack.env`, a Caddy block
(`camofox.ginnoir.com`, `import internal_only`), regenerate bookmarks, push. Then add the minimal
Hermes tool/skill wrapper on the host and restart the gateway.
### 2.5 The three dashboards — pick **at most one** (Phase 3, DECISION)
You already run the bundled `hermes-webui.service`, so a UI is an *upgrade*, not a gap. These three
overlap heavily; adopting more than one is waste.
| | `hermes-ui` (pyrate-llama) | `hermes-workspace` (outsourc-e) | `mission-control` (builderz-labs) |
|---|---|---|---|
| Stars | 173 | 5.9k | 5.4k |
| Stack | 1 HTML file + Python **stdlib** proxy (`serve_lite.py`, :3333 → gateway :8642) | React/TS/**Vite build**, Node 22+, → gateway :8642 + dashboard :9119 | Next.js 16 / React 19, better-sqlite3 |
| Hermes-native? | **Yes** | **Yes** (best with vanilla hermes-agent) | **No** (OpenClaw/CrewAI/LangGraph/AutoGen/Claude SDK) |
| Effort to host | **Trivial** (no build) | Medium (build → container/stack) | Medium + **adapter work** |
| Headline features | chat, tasks/kanban, files, terminal, skills, MCP browser, cron, memory, health | all of the above + ops dashboards, "Conductor" missions, **Swarm Mode (tmux worker pools)** | 32 panels, RBAC, spend/trust scoring, multi-framework |
| Verdict | **Trial first** | **Consider as a proper stack later** | **Skip / defer** |
- **`hermes-ui` — recommended first move.** Lowest cost to evaluate a richer UI. Can run as a tiny
host service or a minimal container next to the bundled webui; no build pipeline. If it's better
than the bundled webui, keep it; if not, delete one file.
- **`hermes-workspace` — the ambitious option.** Genuinely the most capable, best run as a real
`stacks/hermes-workspace` Docker build behind Caddy + Authentik. **Caveat to flag:** its
marquee **Swarm Mode (parallel tmux worker pools) cannot actually parallelize inference on your
setup** — the single P100 runs one model with `--parallel 1`, so "swarm" workers serialize on the
GPU (or must use a different/remote backend per worker). Adopt for the workspace/observability
features, not for swarm, unless/until you add GPU capacity or point swarm workers at cloud
providers.
- **`mission-control` — skip unless requirements change.** It's framework-agnostic and **not
Hermes-native** — using it means writing a Hermes adapter for a dashboard whose job the two
Hermes-native UIs already do. Only worth it if you later want a single ops plane across *multiple*
agent frameworks (you don't run those today).
---
## 3. Recommended phased rollout
**Phase 1 — host-side, reversible, high-leverage (no repo changes):**
1. `hermes-agent-acp-skill` — install skill, wire Codex + Claude Code delegation targets, smoke-test.
2. `hermes-curator-evolver` — install plugin in **report-only/dry-run**, manual cycle, then nightly
systemd timer (report-only). CPU ranking; no P100 embeddings.
**Phase 2 — new capability + experiment:**
3. `camofox-browser` — new `stacks/camofox` (Class B), Caddy internal-only + Authentik, minimal
Hermes tool wrapper.
4. `eagle-eye` — host-side trial behind an on/off switch; A/B skill-selection; keep only if it helps.
**Phase 3 — UI decision (your call):**
5. Trial `hermes-ui` alongside the bundled webui. Evaluate `hermes-workspace` as a stack if the
workspace features justify the build. Skip `mission-control`.
Each phase is independently shippable and independently reversible. Stop after any phase.
---
## 4. Cross-cutting concerns (apply to every adoption)
- **Single-P100 discipline.** One GPU, one model, `--parallel 1`. Anything implying parallel or
extra model loads — curator embeddings, eagle-eye dense layer, workspace swarm — must run on
**CPU**, **off-peak**, or **not at all**. Never co-resident with gpt-oss on the P100.
- **Host vs repo boundary.** Class-A extensions are host-managed in `~/.hermes/` (document in memory
+ vault, like the rustdesk/obsidian units). Only Class-B services touch this repo. Don't
accidentally commit host config here.
- **Tool/skill surface minimalism.** The whole reason your MCP is curated to ~29 tools. Expose new
capabilities (camofox, delegation) as a *few high-level* tools, not raw surfaces, or you reintroduce
the overload you fixed.
- **Provenance & safety.** Several repos are small/young (eagle-eye 1 commit; acp-skill 1 commit).
**Read the code before running it**, especially anything that writes skills (curator) or executes
shell / external agents (acp-skill, camofox). **Pin to a reviewed commit/tag**; keep stack images
out of Watchtower auto-update.
- **Reversibility.** Back up `~/.hermes/config.yaml` before each Class-A change (you already do this);
snapshot before promoting curator to `--approve`.
- **Security posture.** Class-B services: `import internal_only` + Authentik, secrets in `stack.env`
(LF endings), never public.
---
## 5. What I am explicitly NOT proposing
- Not running curator-evolver in `--approve` (auto-skill-editing) mode out of the gate.
- Not loading any second model onto the P100 (no GPU embeddings, no GPU swarm).
- Not adopting more than one dashboard.
- Not committing host-side Hermes plugins/skills into the homelabstack repo.
- Not exposing camofox or any new UI to the public internet.
---
## 6. DECISIONS — RESOLVED 2026-06-27
1. **UI direction → trial BOTH.** Stand up `hermes-ui` *and* `hermes-workspace` in parallel,
evaluate head-to-head against the bundled webui, then keep the winner and remove the other.
(Phase 3 now builds both.)
2. **Curator autonomy → report-only for now.** `hermes-curator-evolver` runs dry-run/report only;
no `--approve` / `--apply-low-risk` / `--approve-auto-apply`. Revisit after a trust period.
3. **eagle-eye → trial it.** Rationale (ginnoir): wants a large skill library available without
weighting every turn — which is exactly eagle-eye's pre-LLM skill-selection job. **Research
result:** eagle-eye is the *only* direct skill-router in the Hermes ecosystem (per
`awesome-hermes-agent` / Hermes Atlas); the nominal "alternatives" do different jobs —
`llmtrim` (compresses tool schemas + MCP output; **complementary**, worth a later look for the
tool side), `rtk-hermes` (shell-output compression), `hermes-snow-search` (faster session
search), `hermes-motif` (mines traces into micro-skills; a curator-evolver *rival*, not a
router). So eagle-eye stands, with the trial caveat (1-commit maturity, `jieba` dep): keep only
if it measurably improves selection; run L1+L2 only (no P100 embeddings).
4. **camofox wiring → minimal.** A small 2-tool Hermes skill (`browse`, `search`) over the REST
API, not the full surface or an MCP shim — protects the curated tool budget.
All four resolved ⇒ **Phases 1, 2, and 3 are all actionable.** The implementation plan that
follows details every phase.
**Ecosystem expansion (added 2026-06-27, per ginnoir):** while researching eagle-eye alternatives I
surveyed the wider Hermes ecosystem and the external-agent landscape. The worthwhile additions and
the motif-vs-curator verdict are in **§7**. Claude Code + Codex are **already installed on valhalla**
(this session); Cursor + Antigravity are scoped for the delegation fabric.
---
## 7. Ecosystem expansion & the delegation fabric (added 2026-06-27)
### 7.1 `hermes-motif` vs `hermes-curator-evolver` — verdict: **complementary, run both**
I earlier mis-framed these as rivals. They do **different halves of one loop**:
- **`hermes-motif` (Saurav0989) — skill DISCOVERY.** Watches tool calls live, finds repeated
sequences (same tools, varying args → `{variable}`), and proposes them as **new** `SKILL.md`
drafts. **Makes no LLM calls** — pure pattern-mining, so **zero P100 cost**. `auto_install: false`
= proposal-only. Zero-to-one skill creation.
- **`hermes-curator-evolver` (pingchesu) — skill REFINEMENT.** Improves **existing** agent-created
skills from session evidence; bounded, reviewable edits; report-only default; more mature
(CI, systemd scheduler, backup/rollback).
**Fit for you:** your stated goal (eagle-eye rationale) is a *large* skill library without per-turn
weight. That's a **skill factory**: **motif manufactures** new skills from what you already do
repeatedly → **curator refines** them → **eagle-eye routes** among them at call time. They compose.
**Recommendation: run both in proposal/report-only mode.** If you insist on only one to start,
**motif** is the lower-risk, higher-immediate-value pick for *growing* a library (no model cost, no
edits to existing files) — but there's no reason to drop curator; it's already Phase 1 and free to
leave in report-only. Plan adds motif as **Task 3b**.
### 7.2 `hermes-web-search-plus` — **ADOPT** (pairs with camofox)
Mature (v2.6.1, MIT, stdlib-only) Hermes plugin: multi-provider web **search** + extraction with
intent-aware routing (news/shopping/docs/GitHub/academic/security) and a `research` mode. 14
providers, **all keys optional** — works with a single configured provider (free tiers exist: Tavily,
Exa, SearXNG self-host, Keenable keyless). **Complements camofox**, doesn't overlap it: search-plus
*finds* (cheap API calls), camofox *browses/interacts* (heavy headless Firefox). For a P100-limited
local agent, search-plus is the lighter, higher-frequency tool — arguably the one to reach for first;
camofox for when a page needs real rendering/interaction. Plan adds it as **Task 5b**, before/with
camofox. Install: `hermes plugins install robbyczgw-cla/hermes-web-search-plus --enable`.
### 7.3 Context-efficiency layer — **OPTIONAL**, helps the Pascal bottleneck
Your memory notes prefill is the P100 differentiator and context bloat is the enemy. Two opt-in tools:
- **`llmtrim` (fkiene, MPL-2.0)** — HTTPS-proxy / MCP / CLI that compresses **tool schemas + history
+ tool output** before requests (claims 31% input / 74% output tokens, BM25+ context ranking,
log-templating). Works with anything honoring `HTTPS_PROXY` (Claude Code, Cursor, Aider). **Two
distinct payoffs:** (a) in front of the **cloud delegation agents** (Claude Code/Codex/Cursor →
Anthropic/OpenAI) it cuts real $ cost; (b) potentially in front of **llama-swap** (via
`LLMTRIM_EXTRA_HOSTS`) to shrink prompts and speed Pascal prefill — needs validation, it's not
documented for local OpenAI-compatible backends. Caveat: it installs a name-constrained MITM CA;
acceptable but a real trust decision.
- **`rtk-hermes` (ogallotti, MIT)** — `pre_tool_call` hook that rewrites shell commands through the
`rtk` binary so terminal output arrives pre-filtered (cache-safe; preserves tool schema). Lighter,
shell-only. Needs the `rtk` binary on the host (`brew`/Linux build — verify availability).
**Recommendation:** defer both to an **opt-in Task 11**. Start llmtrim on the *cloud-delegation* path
(clear cost win, no local risk) before considering it in front of llama-swap.
### 7.4 The delegation fabric — claude-code · codex · cursor · antigravity
Your acp-skill (Task 2) natively routes to `codex`, `claude-code`, and `hermes` subagents. You also
want **Cursor** and **Antigravity** in the mix. All four have real **headless** CLIs usable as
delegation targets from the valhalla host:
| Agent | CLI | Headless invocation | Install (official) | Status on valhalla |
|---|---|---|---|---|
| Claude Code | `claude` | `claude -p` | `curl -fsSL https://claude.ai/install.sh \| bash``~/.local/bin/claude` | **Installed** v2.1.195 (login pending) |
| Codex | `codex` | `codex exec` / non-interactive | `npm i -g @openai/codex` (Node 22+) | **Installed** v0.142.3 (login pending) |
| Cursor | `cursor-agent` | `cursor-agent -p --output-format json --trust` | `curl https://cursor.com/install -fsS \| bash` | Scoped (Task 10) — note: known `-p` hang bug, set `--output-format json` + timeout |
| Antigravity | `agy` (ex-Gemini CLI, Go binary, **best for headless SSH**) | `agy -p` / `--headless --approve` | from **official** `antigravity.google/download` + `/docs/gcli-migration` (auth via keyring/Google sign-in or `ANTIGRAVITY_TOKEN`) | Scoped (Task 10) |
**Wiring approach (Task 10):** acp-skill supports `codex`/`claude-code`/`hermes` out of the box —
verify whether its `agent=` routing is **extensible** (read `SKILL.md`). If yes, add `cursor` and
`antigravity` targets pointing at the headless invocations above. If not, add a **thin generic
"shell-agent" skill** that shells out to any of the four with the timeout/output caps from the
`delegation:` config — keeping the exposed tool surface minimal (your standing tool-budget rule).
**Alternatives noted, not adopted:** `42-evey/hermes-plugins` ships `evey-bridge` (file inbox/outbox
+ MCP bridge to Claude Code) and `evey-delegate-model` (model routing w/ fallback) — heavier,
opinionated, unknown maturity; acp-skill is the simpler purpose-built choice. `evey-cost-guard`
(Langfuse budget enforcement) becomes worth a look **once cloud delegation is live** and spend
matters.
### 7.5 Updated rollout placement
- **Phase 1:** + **Task 3b `hermes-motif`** (proposal-only, zero P100 cost).
- **Phase 2:** + **Task 5b `hermes-web-search-plus`** (light search, pairs with camofox).
- **Phase 2b (new):** **Task 10 delegation fabric** (install cursor + agy, wire all four into
acp-skill) and **Task 11 (optional) context efficiency** (llmtrim on the cloud-delegation path;
rtk for shell).
- **Phase 3:** unchanged (both UIs → keep one).
@@ -0,0 +1,327 @@
# Jellyfin migration — design
**Date:** 2026-08-05
**Status:** approved, not yet implemented
**Supersedes:** `plans/homelab-improvements/tasks/TB-027-jellyfin.md` (stub — resolve it to "Do it" and link here)
---
## 1. Why
Plex now requires a Plex Pass (server owner) or a Remote Watch Pass (viewer) for remote
playback of personal media. Enforcement began on Roku in late 2025 and reaches Fire TV,
Android TV, Google TV, Apple TV and third-party clients through 2026. valhalla's Plex
server has **no Plex Pass**, so every remote viewer is being pushed toward paying for
access to a server they don't own.
Jellyfin has no such gate. This spec replaces host-native Plex with containerised Jellyfin,
managed in this repo like everything else.
## 2. Current state (measured 2026-08-05, not assumed)
### Plex
| Fact | Value |
|---|---|
| Install | AUR `plex-media-server` 1.43.2, systemd unit `plexmediaserver`, runs as `plex:plex` |
| Binary | `/usr/lib/plexmediaserver/` |
| Data dir | `/var/lib/plex`**symlink to `/storage1/labdata/plex`** (virtiofs, *not* the root disk) |
| Data size | **383G total**`Media` 249G, `localhost` 116G, `Metadata` 15G, `Plug-in Support` 2.1G |
| Reverse proxy | none — Plex handles its own relay/direct connect |
> `CLAUDE.md` documents the data dir as `/var/lib/plexmediaserver/...`. **That path does not
> exist.** Correcting it is a Phase 0 task.
### Libraries
Served from `/storage1`, exposed to the \*arrs through `/storage1/Media/*` symlinks:
| Library | Real path | Items |
|---|---|---|
| Movies | `/storage1/Movies` | 1357 |
| TV | `/storage1/TV` | 173 |
| Anime | `/storage1/Anime` | 361 |
`/storage1/Music`, `/storage1/Books` and `/storage1/LinuxISOs` (Stash's) are **out of scope**.
### Usage (Tautulli, trailing 90 days / 600-play sample)
| Metric | Value |
|---|---|
| Users | 37 registered, 29 flagged active — but **many are dormant in practice** |
| Stream decisions | 90 direct play · 3 direct stream · **84 transcode (~48%)** |
| Peak concurrent streams | **7** |
| Transcode drivers | audio 23/25 · video 15/25 · **subtitle 0/25** |
| Source content | h264 18 / hevc 7 · **100% SDR** |
| Platforms | Android 281 · tvOS 102 · browsers 53 · Roku 39 · iOS 18 · Tizen 2 · PlayStation 1 |
Two workloads that would have sunk a CPU-only plan are **absent**: no subtitle burn-in and
no HDR tone-mapping. The residual load is 1080p SDR, which 52 Xeon threads can carry.
### Host constraints
| Constraint | Detail |
|---|---|
| 🔴 Root disk | `/dev/sda2` 194G, **179G used, 5.7G free (97%)**. `/config` lives here. Docker holds 123.5G of images, 51G reclaimable. |
| `/storage1` | 94T, 78T used, **17T free**, virtiofs from the Proxmox host |
| GPU | Tesla P100-PCIE-16GB. **GP100 has no NVENC** — decode-only per NVIDIA's support matrix. Currently held by llama-swap. |
| CPU / RAM | 52 threads · 94G RAM (69G available) |
| Docker GPU | CDI configured (`/etc/cdi/nvidia.yaml`) |
## 3. Decisions
| # | Decision | Rationale |
|---|---|---|
| D1 | New Portainer stack `stacks/streaming/` | Jellyfin config changes must not redeploy sonarr/radarr/qbittorrent. Plex-era cleanup stays contained. |
| D2 | Public `jellyfin.ginnoir.com` via Caddy | Replaces Plex's relay. Tailscale-only or `share_gate` would break Roku/Fire TV/tvOS clients. |
| D3 | **No SSO for Jellyfin** | `jellyfin-plugin-sso` completes OIDC only inside a browser. Roku, Android TV/Fire TV and Swiftfin cannot use it at all — and that is the entire user base. Native Jellyfin accounts. **Deliberate exception to the project auth standard.** |
| D4 | Parallel run, then retire | Plex untouched and running through Phase 6. No failure before Phase 7 costs anything but time. |
| D5 | Plexyfin once, then removed | Copies the curated 15G of posters/backdrops and rebuilds collections from PMS. It deletes and replaces artwork wholesale on every run, so leaving it installed would destroy later Jellyfin customisations. |
| D6 | JellyPlex-Watched, two-way, scheduled | Users can move between servers freely during the overlap. |
| D7 | Jellystat replaces Tautulli | Closest 1:1 equivalent. Tautulli is Plex-only and dies with Plex. |
| D8 | CPU-only transcoding, measured | P100 cannot encode. Workload is 1080p SDR, no burn-in, no tone-mapping. Jellystat measures the truth. |
| D9 | Escape hatch: Intel Arc A310/A380 | ~$100-150, no session limit, AV1. Only if Phase 5 data shows CPU strain. |
| D10 | Seerr converted in place at cutover | `seerr-team/seerr` already supports Jellyfin natively. In-place conversion is unsupported upstream ([seerr#2522](https://github.com/seerr-team/seerr/issues/2522)) — mitigated by config backup and a fresh-config fallback. |
| D11 | Transcode dir on disk, **not tmpfs** | [jellyfin#16608](https://github.com/jellyfin/jellyfin/issues/16608): the segment cleaner first checks at 20s and the throttler at 5s, but Remux/DirectStream jobs finish in 2-7s, so cleanup never fires and segments orphan. DirectStream (copy video + transcode audio) is 8 of 25 sampled sessions — routine here, not an edge case. Plex already writes transcodes to virtiofs at a 48% transcode rate without trouble. |
| D12 | Trickplay at full quality | Plex's 365G of BIF thumbnails is **format-incompatible** with Jellyfin's tiled JPEG trickplay. None of it is reusable; Jellyfin regenerates from scratch. |
| D13 | Wizarr invites issued **manually and selectively** | Most Plex users are dormant. No bulk invite generation, no scripted account creation, no broadcast announcement. ginnoir sends individual invites as people ask or as he chooses. |
## 4. Architecture
New Portainer stack `stacks/streaming/` — private `streaming` network plus `edge`, all
config via `env_file: stack.env` per the repo convention, no Portainer UI env vars.
| Container | Image | Purpose |
|---|---|---|
| `jellyfin` | `lscr.io/linuxserver/jellyfin` | media server (PUID/PGID from `stack.env`) |
| `jellystat` | `cyfershepard/jellystat` | Tautulli replacement |
| `jellystat-db` | `postgres:16-alpine` | pinned + `com.centurylabs.watchtower.enable=false` per infra convention |
| `wizarr` | `ghcr.io/wizarrrr/wizarr` | invite + guided per-device onboarding, used ad-hoc |
| `jellyplex-watched` | `luigi311/jellyplex-watched` | scheduled two-way watch-state sync |
### Ordering constraint
Plexyfin and JellyPlex-Watched both read from the **live PMS API**. Everything that pulls
from Plex must complete before Plex is decommissioned. This is the hard sequencing rule of
the whole migration.
## 5. Storage layout
The root disk is the primary hazard, so Jellyfin's bulky data is split off deliberately.
| Container path | Host path | Filesystem | Rationale |
|---|---|---|---|
| `/config` | `/config/jellyfin` | root SSD | SQLite DBs — better fsync semantics, easy to back up. A few GB. |
| `/config/metadata` | `/storage1/labdata/jellyfin/metadata` | virtiofs | artwork + NFO, grows unbounded |
| `/config/data/trickplay` | `/storage1/labdata/jellyfin/trickplay` | virtiofs | **~300-400G.** 10.11 moved trickplay here and it is *not* relocatable from the UI, hence an explicit sub-bind. |
| `/config/cache` | `/storage1/labdata/jellyfin/cache` | virtiofs | image cache |
| `/transcode` | `/storage1/labdata/jellyfin/transcodes` | virtiofs | see D11 |
| `/data/movies` | `/storage1/Movies` | virtiofs, **`:ro`** | Jellyfin never writes to the library |
| `/data/tv` | `/storage1/TV` | virtiofs, **`:ro`** | |
| `/data/anime` | `/storage1/Anime` | virtiofs, **`:ro`** | |
Net effect: only a few GB ever lands on the 194G root disk.
### Note on SQLite and virtiofs
Plex has run its SQLite library DB on virtiofs here for months without incident, so this is
proven workable — the root-SSD placement is a preference, not a hard requirement.
It carries one accepted trade-off. If `/storage1` fails to mount again (as on 2026-07-06),
Jellyfin with its DB on root will **start up healthy against an empty library** and flag the
entire catalogue as missing — the RomM failure mode. With the DB on `/storage1` it would
simply fail to start, which is the louder and safer failure. `storage1-guard.service`
already blocks guest start when `/storage1` isn't real, so root SSD stands.
## 6. Caddy
```caddy
# =============================================================
# STREAMING — public
# =============================================================
jellyfin.ginnoir.com {
reverse_proxy jellyfin:8096
}
wizarr.ginnoir.com {
reverse_proxy wizarr:5690
}
# =============================================================
# STREAMING STATS — internal only
# =============================================================
jellystat.ginnoir.com {
route {
import internal_gate
import authentik_outpost
handle {
import authentik_forward_auth
reverse_proxy jellystat:3000
}
}
}
```
`jellyfin` and `wizarr` are public by design — remote users and invite recipients reach them
from anywhere. `jellystat` follows the tautulli/sonarr pattern: `internal_gate` first inside
the `route` block (an `import internal_only` above a catch-all route is dead code), then
Authentik forward auth.
In Jellyfin → Networking, register the Caddy container as a known proxy and enable forwarded
headers, so real client IPs reach Jellystat and the remote-vs-LAN bitrate rules.
## 7. Metadata strategy
1. Jellyfin scans and fetches fresh from TMDB/TVDB — the same sources Plex used, so baseline
quality is identical.
2. **Plexyfin runs once**, overwriting posters/backdrops with the curated Plex artwork and
rebuilding collections. This is where the 15G `Metadata` investment is preserved.
3. **Plexyfin is then removed.** See D5.
Plex's 365G of `Media` + `localhost` (BIF preview thumbnails) is not reusable — see D12.
## 8. Plugins
**Install**
| Plugin | Purpose | Note |
|---|---|---|
| Plexyfin | collections + curated artwork from PMS | 3rd-party repo. Temporary — remove after Phase 2. |
| Intro Skipper | skip intros/recaps/credits | **Verify 10.11 compatibility at install.** The original is unmaintained; use the active fork. |
| AniDB / AniList | anime metadata | Anime library only |
| Trakt | per-user scrobbling | optional |
**Deliberately skipped**
| Plugin | Why |
|---|---|
| `jellyfin-plugin-sso` | browser-only OIDC — useless for Roku/Fire TV/tvOS (D3) |
| Open Subtitles | Bazarr already writes subtitles to disk and does it better |
| Playback Reporting | Jellystat supersedes it |
## 9. Transcoding
CPU-only on 52 threads: `veryfast` preset, per-session thread cap, throttling enabled,
**segment deletion on with a ~360s keep window**. Measured worst case is 7 concurrent streams
at ~48% transcode, 1080p SDR, no burn-in, no tone-mapping.
Jellyfin's Android client (the heaviest platform here at 281 plays) direct-plays audio codecs
Plex's client refuses, so the transcode rate may **fall** after migration. Jellystat confirms
or refutes this during Phase 5.
If it does strain: Intel Arc A310/A380 (D9). Needs a PCIe slot, an R730XD shutdown, and an
IOMMU passthrough to the valhalla VM — check the new card's IOMMU group, as only the P100's
group 7 is known clean.
## 10. Phases
Plex stays running and untouched through Phase 6.
### Phase 0 — Disk remediation and baseline
The root disk is at 97%. Nothing else may proceed until it isn't.
- `docker image prune` / `builder prune` — ~51G reclaimable
- Audit the remaining 179G on `/dev/sda2`; identify anything else structurally growing
- Uptime Kuma monitor on root-disk free space + ntfy alert (this is the second root fill)
- Correct the Plex paths in `CLAUDE.md` (`/var/lib/plex``/storage1/labdata/plex`)
**Done when:** root disk has comfortable headroom and alerts before it doesn't.
### Phase 1 — Deploy the stack
- `stacks/streaming/docker-compose.yml` + `stack.env`, `jellyfin` only
- Storage layout per §5; register the stack in Portainer (one-time, per the new-stack procedure)
- Caddy site blocks; `scripts/gen-bookmarks.ps1`
- Libraries: Movies, TV, Anime (Anime with anime metadata providers)
- Verify `check-portainer-stack-env.ps1` before pushing
**Done when:** Jellyfin reachable at `jellyfin.ginnoir.com`, libraries scanned, no users.
**Rollback:** delete the stack.
### Phase 2 — Parity
- Plexyfin: sync collections + artwork from PMS, then **remove the plugin**
- **Trickplay generation** — full quality, thread-capped (~16-24 of 52), run before any users
arrive. Budget ~300-400G on `/storage1` and a multi-day first pass over ~1891 items.
Enable NVDEC for this: trickplay is decode + JPEG encode and needs no NVENC, so the P100
helps here even though it's useless for live transcode. Schedule when llama-swap is idle.
- Remaining plugins per §8
- Sonarr/Radarr/Bazarr → add Jellyfin library-refresh connections (leave the Plex ones for now)
**Done when:** Jellyfin's library visually matches Plex's, trickplay complete.
### Phase 3 — Watch-state sync
- `jellyplex-watched` container, two-way, every 6h
- Plex admin token + Jellyfin API key in `stack.env`; user-mapping file where names differ
- Verify against ginnoir's own account in both directions
**Done when:** a play on either server appears on the other within one cycle.
**Rollback:** stop the container.
### Phase 4 — Pilot
- Configure Wizarr (server connection, wizard steps per device)
- 3-5 willing users, **at least one each on Roku, Fire TV/Android TV and tvOS**
- Collect real feedback on playback, transcoding and the onboarding wizard
**Done when:** every target platform has a confirmed working client.
**Rollback:** pilot users return to Plex, which never stopped.
### Phase 5 — Selective migration
Per D13, this phase is **demand-driven, not a campaign.**
- Wizarr invites issued **individually**, as people ask or as ginnoir chooses
- Dormant accounts get nothing
- A short written explainer exists to accompany an invite — sent with invites, not broadcast
- Both servers live; run at least **4 weeks** to accumulate meaningful Jellystat data
- Watch transcode load and concurrency; decide on D9
**Done when:** the people who actually watch things are on Jellyfin and it's holding up.
### Phase 6 — Cutover
- Back up `/config/overseerr`, then reconfigure Seerr from Plex to Jellyfin (D10)
- Jellystat live at `jellystat.ginnoir.com`
- **Re-check Tautulli** for who is still on Plex and on what. Handle those users
individually — Tizen and PlayStation are the weak platforms (3 plays of 600 as of now).
- Remove Plex connections from Sonarr/Radarr/Bazarr
- Announce the Plex end date
**Done when:** Seerr requests flow against Jellyfin and nobody depends on Plex.
**Rollback:** restore `/config/overseerr`.
### Phase 7 — Decommission
- `systemctl stop plexmediaserver && systemctl disable plexmediaserver`
- **2-week soak** with the service stopped but installed
- `pacman -Rns plex-media-server`
- Retire the `tautulli` container and its Caddy block
- Tarball `/storage1/labdata/plex`, keep 30 days, then reclaim **383G**
- Update `CLAUDE.md` (drop the external-services Plex section), bookmarks, Uptime Kuma monitors
**Done when:** Plex is gone and nothing references it.
**Rollback (until the tarball expires):** reinstall the package, restore the data dir.
## 11. Risks
| Risk | Severity | Mitigation |
|---|---|---|
| Root disk at 97% | **high** | Phase 0 exists solely for this; monitor + alert added |
| Public Jellyfin is new attack surface Plex's relay hid | medium | Jellyfin's built-in brute-force lockout, strong admin credential, no anonymous access, monitoring |
| CPU transcode capacity at peak | medium | measured in Phase 5 via Jellystat; Arc A380 escape hatch (D9) |
| Seerr in-place conversion unsupported upstream | medium | config backup first, fresh-config fallback documented |
| Plexyfin is destructive | medium | run once in Phase 2, then remove (D5) |
| Trickplay generation starves live transcoding | low | thread-capped, runs before users arrive, NVDEC-assisted |
| Intro Skipper 10.11 compatibility | low | verify at install; it's optional polish |
| Holdouts on Tizen / PlayStation | low | 3 of 600 plays; handled individually at Phase 6 with fresh data |
## 12. Out of scope
- Music, Books and Stash content — Jellyfin serves Movies, TV and Anime only
- SSO for Jellyfin (D3)
- Migrating Seerr request history
- Replacing the P100 or restructuring GPU allocation
- The wider `/storage1` redundancy problem (tracked separately)
+23
View File
@@ -149,6 +149,29 @@ services:
labels:
- "com.centurylabs.watchtower.enable=false"
gitea-runner-ci:
container_name: gitea_runner_ci
image: gitea/act_runner:latest
restart: unless-stopped
networks:
- gitea_internal
environment:
- GITEA_INSTANCE_URL=https://gitea.ginnoir.com
- GITEA_RUNNER_REGISTRATION_TOKEN=${GITEA_RUNNER_REGISTRATION_TOKEN}
- GITEA_RUNNER_NAME=valhalla-ci-runner-2
- GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:24-bookworm,node-24:docker://node:24-bookworm
- CONFIG_FILE=/data/config.yaml
volumes:
- /config/gitea/act_runner_ci:/data
- /var/run/docker.sock:/var/run/docker.sock
- /home/ginnoir/valhalla-lab:/valhalla-lab
- /config/caddy:/config/caddy
depends_on:
gitea:
condition: service_started
labels:
- "com.centurylabs.watchtower.enable=false"
registry:
container_name: registry
image: registry:2
+19 -2
View File
@@ -1,10 +1,27 @@
const fs = require('fs');
const path = require('path');
async function fetchWithRetry(url, options = {}, retries = 3, backoff = 2000) {
for (let i = 0; i < retries; i++) {
try {
const response = await fetch(url, options);
if (response.ok) return response;
if (response.status >= 500) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
return response;
} catch (err) {
if (i === retries - 1) throw err;
console.warn(`Fetch to ${url} failed (attempt ${i + 1}/${retries}): ${err.message}. Retrying in ${backoff}ms...`);
await new Promise(resolve => setTimeout(resolve, backoff));
}
}
}
async function checkStatus() {
try {
// 1. Fetch status page HTML
const htmlResponse = await fetch('https://uptime.ginnoir.com/status/default');
const htmlResponse = await fetchWithRetry('https://uptime.ginnoir.com/status/default');
if (!htmlResponse.ok) {
throw new Error(`Failed to fetch status page: ${htmlResponse.statusText}`);
}
@@ -34,7 +51,7 @@ async function checkStatus() {
}
// 2. Fetch heartbeat JSON
const heartbeatResponse = await fetch('https://uptime.ginnoir.com/api/status-page/heartbeat/default');
const heartbeatResponse = await fetchWithRetry('https://uptime.ginnoir.com/api/status-page/heartbeat/default');
if (!heartbeatResponse.ok) {
throw new Error(`Failed to fetch heartbeat: ${heartbeatResponse.statusText}`);
}
+179
View File
@@ -0,0 +1,179 @@
const fs = require('fs');
const path = require('path');
async function fetchWithRetry(url, options = {}, retries = 3, backoff = 2000) {
for (let i = 0; i < retries; i++) {
try {
const response = await fetch(url, options);
if (response.ok) return response;
if (response.status >= 500) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
return response;
} catch (err) {
if (i === retries - 1) throw err;
console.warn(`Fetch to ${url} failed (attempt ${i + 1}/${retries}): ${err.message}. Retrying in ${backoff}ms...`);
await new Promise(resolve => setTimeout(resolve, backoff));
}
}
}
async function checkStatus() {
try {
// 1. Fetch status page HTML
const htmlResponse = await fetchWithRetry('https://uptime.ginnoir.com/status/default');
if (!htmlResponse.ok) {
throw new Error(`Failed to fetch status page: ${htmlResponse.statusText}`);
}
const html = await htmlResponse.text();
// Extract window.preloadData = ...;
const preloadRegex = /window\.preloadData\s*=\s*({.*?});/s;
const match = html.match(preloadRegex);
if (!match) {
throw new Error("Could not find window.preloadData in HTML");
}
let preloadData;
try {
preloadData = new Function(`return ${match[1]};`)();
} catch (e) {
throw new Error(`Failed to parse preloadData: ${e.message}`);
}
const monitors = [];
if (preloadData && preloadData.publicGroupList) {
for (const group of preloadData.publicGroupList) {
if (group.monitorList) {
monitors.push(...group.monitorList);
}
}
}
// 2. Fetch heartbeat JSON
const heartbeatResponse = await fetchWithRetry('https://uptime.ginnoir.com/api/status-page/heartbeat/default');
if (!heartbeatResponse.ok) {
throw new Error(`Failed to fetch heartbeat: ${heartbeatResponse.statusText}`);
}
const heartbeats = await heartbeatResponse.json();
// 3. Map status and print summary
const monitorMap = {};
for (const monitor of monitors) {
monitorMap[monitor.id] = {
name: monitor.name,
type: monitor.type,
status: 'UNKNOWN',
ping: null,
lastCheck: null,
msg: ''
};
}
const heartbeatList = heartbeats.heartbeatList || {};
for (const id in heartbeatList) {
const list = heartbeatList[id];
if (list && list.length > 0) {
const latest = list[list.length - 1];
if (monitorMap[id]) {
monitorMap[id].status = latest.status === 1 ? 'UP' : 'DOWN';
monitorMap[id].ping = latest.ping;
monitorMap[id].lastCheck = latest.time;
monitorMap[id].msg = latest.msg || '';
}
}
}
// Generate summary
const monitorValues = Object.values(monitorMap);
const total = monitorValues.length;
const up = monitorValues.filter(m => m.status === 'UP').length;
const down = monitorValues.filter(m => m.status === 'DOWN').length;
const unknown = monitorValues.filter(m => m.status === 'UNKNOWN').length;
let summaryMd = `### Uptime Status Summary (Checked at ${new Date().toLocaleString()})\n\n`;
if (down > 0) {
summaryMd += `⚠️ **Status: Degraded (${down}/${total} services DOWN)**\n\n`;
} else if (up === total) {
summaryMd += `✅ **Status: Healthy (All ${total} services UP)**\n\n`;
} else {
summaryMd += `️ **Status: Mixed (UP: ${up}, DOWN: ${down}, UNKNOWN: ${unknown})**\n\n`;
}
if (down > 0) {
summaryMd += `#### 🚨 DOWN Services:\n`;
monitorValues.filter(m => m.status === 'DOWN').forEach(m => {
summaryMd += `- **${m.name}** (${m.type}) - ${m.msg || 'No message'} (Last checked: ${m.lastCheck})\n`;
});
summaryMd += `\n`;
}
summaryMd += `#### 📋 Service Statuses:\n`;
summaryMd += `| Service | Status | Latency (ms) | Last Check |\n`;
summaryMd += `| :--- | :---: | :---: | :--- |\n`;
// Sort services by name
monitorValues.sort((a, b) => a.name.localeCompare(b.name));
for (const m of monitorValues) {
const statusIcon = m.status === 'UP' ? '🟢 UP' : m.status === 'DOWN' ? '🔴 DOWN' : '⚪ UNKNOWN';
const pingText = m.ping !== null ? `${m.ping} ms` : 'N/A';
summaryMd += `| ${m.name} | ${statusIcon} | ${pingText} | ${m.lastCheck || 'N/A'} |\n`;
}
// Output to stdout
console.log(summaryMd);
// 1. Write the latest report to Uptime Status Report.md
const reportPath = 'c:/Users/MattC/Documents/Obsidian Vault/Homelab/Uptime Status Report.md';
const reportContent = `---
project: Homelab
type: status
status: current
tags: [monitoring, uptime]
updated: ${new Date().toISOString().split('T')[0]}
---
# Live Uptime Status Report
${summaryMd}
`;
fs.writeFileSync(reportPath, reportContent, 'utf8');
console.log(`Wrote status report to ${reportPath}`);
// 2. Append to today's session log
const todayStr = new Date().toISOString().split('T')[0];
const sessionLogDir = 'c:/Users/MattC/Documents/Obsidian Vault/Homelab/Sessions';
if (!fs.existsSync(sessionLogDir)) {
fs.mkdirSync(sessionLogDir, { recursive: true });
}
const sessionLogPath = path.join(sessionLogDir, `${todayStr}-uptime-status-monitoring-schedule.md`);
let sessionContent = '';
if (!fs.existsSync(sessionLogPath)) {
sessionContent = `# Session Log — ${todayStr} — Uptime Status Monitoring Schedule
## What was done
- Automatically checked uptime status page and updated reports.
## Daily Log of Checks
`;
} else {
sessionContent = fs.readFileSync(sessionLogPath, 'utf8');
}
const timeStr = new Date().toLocaleTimeString();
const statusText = down > 0 ? `🚨 DEGRADED (${down}/${total} services DOWN)` : `✅ Healthy (${total}/${total} services UP)`;
const logEntry = `\n### Check at ${timeStr}\n- **Status**: ${statusText}\n`;
fs.writeFileSync(sessionLogPath, sessionContent + logEntry, 'utf8');
console.log(`Appended check entry to ${sessionLogPath}`);
} catch (error) {
console.error(`Error checking status: ${error.message}`);
process.exit(1);
}
}
checkStatus();
+7 -2
View File
@@ -8,7 +8,7 @@
services:
authentik-server:
container_name: authentik_server
image: ghcr.io/goauthentik/server:2026.5.2
image: ghcr.io/goauthentik/server:2026.5.3
restart: unless-stopped
command: server
networks: [authentik, edge]
@@ -23,6 +23,11 @@ services:
- ./blueprints:/blueprints/custom:ro
ports:
- "9200:9000"
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
depends_on:
authentik-db:
condition: service_healthy
@@ -31,7 +36,7 @@ services:
authentik-worker:
container_name: authentik_worker
image: ghcr.io/goauthentik/server:2026.5.2
image: ghcr.io/goauthentik/server:2026.5.3
restart: unless-stopped
command: worker
networks: [authentik]
+41
View File
@@ -0,0 +1,41 @@
# camofox stack — self-hosted stealth headless browser (Camoufox/Firefox) REST API for the
# Hermes agent. Chosen over the native Playwright/Browserbase plugins for self-hosted,
# no-cloud-key, anti-detection browsing on valhalla.
#
# No published image: built on-host as camofox-browser:135.0.1-x86_64 via `make build` in the
# upstream clone (/storage1/hermes/workspace/clones/camofox-browser). pull_policy: never so
# Portainer's git redeploy uses the local image instead of trying to pull a non-existent
# registry tag. To make it survive a host rebuild, push to registry.ginnoir.com later (famapp
# pattern) and swap the image ref.
#
# Internal-only: published on the edge host-IP (172.20.0.1:9377) so host-side Hermes can reach
# it directly, and joined to `edge` so Caddy reaches it by container name. Bearer-auth enforced
# by camofox itself (CAMOFOX_ACCESS_KEY). Image pinned out of Watchtower.
services:
camofox:
image: camofox-browser:135.0.1-x86_64
pull_policy: never
container_name: camofox
restart: unless-stopped
labels:
- "com.centurylabs.watchtower.enable=false"
env_file:
- stack.env
networks: [edge]
# Firefox-based browsers can crash with the default 64MB /dev/shm.
shm_size: "1gb"
volumes:
- /config/camofox/cookies:/data/cookies
- /config/camofox/profiles:/data/profiles
ports:
- "172.20.0.1:9377:9377"
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost:9377/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 40s
networks:
edge:
external: true
+13
View File
@@ -0,0 +1,13 @@
# camofox stack env — committed intentionally (homelab convention; secrets versioned in the
# private repo). LF line endings (enforced by .gitattributes).
#
# CAMOFOX_ACCESS_KEY — bearer required on all routes except /health, cookie-import, /stop.
# CAMOFOX_ADMIN_KEY — required for POST /stop.
# CAMOFOX_API_KEY — left UNSET on purpose: keeps the cookie-import endpoint disabled.
# CAMOFOX_COOKIES_DIR / CAMOFOX_PROFILE_DIR — explicit container paths (avoid home-dir ambiguity);
# mapped to /config/camofox/* binds on the host.
CAMOFOX_ACCESS_KEY=cc9dac55139c97431160468bcfa43bbe0ab862710990f762d04e5881da9574b3
CAMOFOX_ADMIN_KEY=117606acac488bf9661dab6ed276bc2cb465799eaf5676f2f7576b57e44b0ef4
CAMOFOX_PORT=9377
CAMOFOX_COOKIES_DIR=/data/cookies
CAMOFOX_PROFILE_DIR=/data/profiles
+3
View File
@@ -57,6 +57,9 @@ services:
- TZ
- APP_URL=https://docs.ginnoir.com
- APP_KEY
# stack.env leaks Plane's DATABASE_URL; Laravel prioritizes it over DB_* and
# would connect BookStack to postgres_plane. Pin it to BookStack's own MariaDB.
- DATABASE_URL=mysql://bookstack:bookstack_d3v_pass@mariadb_bookstack:3306/bookstack
- DB_HOST=mariadb_bookstack
- DB_PORT=3306
- DB_USER=bookstack
+8
View File
@@ -36,3 +36,11 @@ MINIO_BUCKET=garden
OPENPLANTBOOK_CLIENT_ID=5JdQLab69RhVaF4er7b7D29BCrkf3fo3OEiDsOQa
OPENPLANTBOOK_CLIENT_SECRET=kxPTrdQPAVta7sjVuIRf5EGIJ1fKvlMlqRSJpQtFxRftJhg7oeGnXzwH7ls6tluQLFUYVDYRMwISiAuKZGZLvkuJzUwFGCwhZw7UtW8OvcTGy4717WeNUinvuCDSsNXP
# LLM assistant (OpenAI-compatible — same as local dev for now)
LLM_PROVIDER=openai
LLM_BASE_URL=http://192.168.1.121:8080/v1
LLM_API_KEY=famapp-unc-f04f7ef0baf68d542d10a501
LLM_MODEL=auto
# IANA timezone for assistant relative dates ("Thursday at 2")
HOUSEHOLD_TIMEZONE=America/Chicago
+12 -3
View File
@@ -3,10 +3,10 @@
# only one fits in VRAM at a time, so selecting a different model triggers a
# brief reload. Default is chosen by Hermes (model.default = gpt-oss-20b).
#
# All four serve 64k context (Hermes' minimum). Args are the validated Pascal
# All serve 64k context (Hermes' minimum). Args are the validated Pascal
# config: q8_0 KV (q4_0 V-cache is pathological on GP100), flash-attn on,
# --parallel 1 so one sequence gets the full 64k. gpt-oss/gemma4 are natively
# >=128k so no YaRN/override-kv needed.
# --parallel 1 so one sequence gets the full 64k. gpt-oss/gemma4/ornith are
# natively >=128k so no YaRN/override-kv needed.
#
# Excluded: qwen3-30b-a3b-2507 (Q3) — OOMs at 64k in 16GB, so it can't meet
# Hermes' 64k minimum on this GPU.
@@ -27,6 +27,15 @@ models:
-m /models/gpt-oss-20b-mxfp4.gguf --alias gpt-oss-20b
${common}
"ornith-1.0-9b":
# DeepReinforce Ornith-1.0, dense ~9B on Qwen 3.5 (Q5_K_M). MIT. Agentic-coding
# tuned: <think> block (-> reasoning_content under --jinja) + Qwen3 XML tool calls.
# Native 256k so no YaRN. Recommended sampling: temp 0.6 / top_p 0.95 / top_k 20.
cmd: >
/app/llama-server --port ${PORT} --host 0.0.0.0
-m /models/ornith-1.0-9b-Q5_K_M.gguf --alias ornith-1.0-9b
${common}
"gemma-4-26b-a4b":
# gemma4 MoE, 4B active / 26B total (UD-Q3_K_M). Quality-leaning; ~147 tok/s prefill.
cmd: >
+5
View File
@@ -48,6 +48,11 @@ services:
environment:
- MYSQL_DATABASE=owncloud
command: ["--max-allowed-packet=128M", "--innodb-log-file-size=64M"]
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
healthcheck:
# $$ defers expansion to container runtime (env_file supplies the value)
test: ["CMD-SHELL", "mariadb-admin ping -u root --password=\"$$MYSQL_ROOT_PASSWORD\""]
+5
View File
@@ -16,6 +16,11 @@ services:
- /etc/localtime:/etc/localtime:ro
ports:
- "2283:2283"
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
depends_on:
- immich-redis
- immich-postgres
+15
View File
@@ -19,6 +19,21 @@ services:
restart: unless-stopped
labels:
- "com.centurylabs.watchtower.enable=false"
# Public resolvers for ACME DNS-01 zone detection.
#
# The LAN resolver (OPNsense Unbound, 192.168.1.1) is authoritative for
# ginnoir.com via the split-horizon override, but its local-zone only holds
# A records — so `SOA ginnoir.com` returns NODATA. certmagic's zone lookup
# walks up the label chain looking for an SOA, finds none at ginnoir.com,
# climbs to `com.`, and asks Cloudflare for a `com` zone. Result:
# "adding temporary record for zone \"com.\": expected 1 zone, got 0"
# and every DNS-01 renewal fails until certs expire.
#
# Docker's embedded DNS (127.0.0.11) still resolves container names and the
# gitea.ginnoir.com alias below; only external lookups go to Cloudflare.
dns:
- 1.1.1.1
- 1.0.0.1
networks:
edge:
aliases:
+3 -3
View File
@@ -22,9 +22,9 @@
services:
romm:
# Floating v4 tag (watchtower-tracked). The save-sync (/api/sync/*) + raw
# content_hash fixes that once forced a 4.9.0-beta.2 pin are now in stable 4.9.x.
image: rommapp/romm:4
# Floating v5 tag (watchtower-tracked). Bumped from :4 (4.9.2) → :5 (5.0.0)
# 2026-07-21. Pre-upgrade dump: /config/backup/dumps/romm-pre-5.0.0-20260721.sql
image: rommapp/romm:5
container_name: romm
restart: unless-stopped
networks: [roms, edge]
+96
View File
@@ -0,0 +1,96 @@
# streaming stack — Jellyfin and its satellites.
#
# Storage split is deliberate: SQLite config on the root SSD, all bulk data
# (metadata, trickplay, cache, transcodes) on /storage1. The root disk is only
# 194G and has filled twice; nothing here may grow unbounded on it.
#
# Jellyfin 10.11 (linuxserver) keeps metadata at /config/data/metadata — NOT
# /config/metadata. Trickplay is at /config/data/trickplay and is not
# relocatable from the UI. Transcodes are configured to /transcode.
#
# Transcodes are disk-backed, not tmpfs: jellyfin#16608 means Remux and
# DirectStream jobs finish before the segment cleaner's first check, orphaning
# segments. Plex already writes transcodes to virtiofs at a 48% transcode rate.
#
# Env convention: every service gets stack.env via env_file; var names there are
# exactly what the containers read. No ${VAR} interpolation, no Portainer UI
# env vars.
services:
jellyfin:
container_name: jellyfin
image: lscr.io/linuxserver/jellyfin:latest
restart: unless-stopped
networks: [streaming, edge]
env_file:
- stack.env
devices:
- nvidia.com/gpu=0
environment:
- JELLYFIN_PublishedServerUrl=https://jellyfin.ginnoir.com
volumes:
- /etc/localtime:/etc/localtime:ro
- /config/jellyfin:/config
- /storage1/labdata/jellyfin/metadata:/config/data/metadata
- /storage1/labdata/jellyfin/cache:/config/cache
- /storage1/labdata/jellyfin/trickplay:/config/data/trickplay
- /storage1/labdata/jellyfin/transcodes:/transcode
- /storage1/Movies:/data/movies:ro
- /storage1/TV:/data/tv:ro
- /storage1/Anime:/data/anime:ro
ports:
- "8096:8096"
jellystat-db:
container_name: jellystat-db
image: postgres:16-alpine
restart: unless-stopped
networks: [streaming]
env_file:
- stack.env
labels:
- "com.centurylabs.watchtower.enable=false"
volumes:
- /config/jellystat-db:/var/lib/postgresql/data
jellystat:
container_name: jellystat
image: cyfershepard/jellystat:latest
restart: unless-stopped
networks: [streaming, edge]
depends_on:
- jellystat-db
env_file:
- stack.env
volumes:
- /config/jellystat:/app/backend/backup-data
ports:
- "3009:3000"
wizarr:
container_name: wizarr
image: ghcr.io/wizarrrr/wizarr:latest
restart: unless-stopped
networks: [streaming, edge]
env_file:
- stack.env
volumes:
- /config/wizarr:/data/database
ports:
- "5690:5690"
jellyplex-watched:
container_name: jellyplex-watched
image: luigi311/jellyplex-watched:latest
restart: unless-stopped
networks: [streaming, edge]
env_file:
- stack.env
networks:
streaming:
name: streaming
driver: bridge
edge:
name: edge
external: true
+34
View File
@@ -0,0 +1,34 @@
# streaming stack config — injected into every service via env_file.
# Var names are exactly what the containers read; no ${VAR} interpolation,
# no Portainer UI env vars needed.
# --- shared ---
TZ=America/Chicago
PUID=1000
PGID=1000
# --- jellystat-db (postgres:16-alpine) ---
POSTGRES_DB=jfstat
POSTGRES_USER=jellystat
POSTGRES_PASSWORD=bf2d47d8e1a87baf2b4942704a5acfaf5c2aad336fdd480e
# --- jellystat ---
POSTGRES_IP=jellystat-db
POSTGRES_PORT=5432
JWT_SECRET=0dd9fcdb632277b76977dbada8226e547cc773e1deafc8d778d4038f8d59393b
# --- jellyplex-watched ---
# Plex is host-native. Plain HTTP on :32400 only answers on loopback; LAN HTTP
# gets an empty reply. Temporary host socat proxy plex-http-proxy.service
# exposes loopback Plex HTTP on :32402 for Docker clients (Plexyfin + jellyplex).
# Tear down that unit after Plex is decommissioned / Plexyfin is done.
PLEX_URL=http://192.168.1.69:32402
PLEX_TOKEN=hM9aycswHKSsUrxiHqwq
JELLYFIN_BASEURL=http://jellyfin:8096
JELLYFIN_TOKEN=a3d9ee827d5c42298cae087a2c31d3fa
SYNC_FROM_PLEX_TO_JELLYFIN=True
SYNC_FROM_JELLYFIN_TO_PLEX=True
# Start in dry-run. Task 16 flips this to False after the mapping is verified.
DRYRUN=False
SLEEP_DURATION=21600
LOG_LEVEL=INFO