Runs 137-139 all failed at "cp: can't stat '/dest/Caddyfile/Caddyfile'".
The step bound "$PWD/Caddyfile" into an alpine container, but the job
workspace is a Docker volume -- the daemon resolved that source on the
HOST, found nothing, and created an empty directory there instead (it
left a root-owned /workspace/ginnoir/homelabstack/Caddyfile/ on valhalla).
So alpine got a directory as its copy source and cp failed, leaving the
live Caddyfile stale since run 136.
The premise was wrong: act_runner's container.options already binds
/config/caddy and /var/run/docker.sock into every job container. Copy
straight into the bind and reload through the Docker Engine API over the
socket -- which also drops the 40s apt-get install of docker.io.
Uses cp rather than mv/install because /config/caddy/Caddyfile is bound
into the caddy container as a single file, so the bind follows the inode.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Point the service at registry.ginnoir.com/ginnoir/romhacks-orchestrator:latest
(built by the workflow added in 60121a5, verified pullable on valhalla) and
drop the `build:` context that Portainer never honoured.
This unfreezes stack 28, which has been stuck at 847edff since 2026-06-24:
the containers kept running, so nothing alerted, but no commit had deployed
in two months.
README: document that Portainer builds nothing here, and that re-adding a
`build:` would deadlock the stack again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Portainer's git auto-update only runs `docker compose pull`, never `build`,
so the orchestrator's `build:` context + local-only
`homelab/romhacks-orchestrator:latest` deadlocked stack 28: every 5-min poll
died on "pull access denied" and the stack sat frozen at 847edff since
2026-06-24 while every other stack moved on.
Build it in CI and push it to registry.ginnoir.com instead, same shape as
famapp's release.yml. Compose switches to the registry image in the next
commit, once the image actually exists.
Push creds come from the committed .env rather than Gitea Actions secrets
(famapp needs those only because its repo doesn't commit .env) — one source
of truth on rotation. Only the two REGISTRY_PUSH_* keys are extracted; the
vault material in that file stays out of the job env.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Watchtower pulled the Reactive Resume rewrite (sha-3c195dc, built
2026-08-24) and recreated the container. The new build ignores the
STORAGE_*/MinIO vars and writes to LOCAL_STORAGE_PATH=/app/data, which
/api/health confirms ("storage": {"type": "local"}).
The app service had no volumes at all, so every avatar and export lived
in the container's writable layer and would vanish on the next recreate.
Bind /storage1/labdata/resume/data (pre-created 1000:1000, matching the
container's node user) to /app/data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Jellyfin 10.11's data dir is /config/data/data, so trickplay resolves to
/config/data/data/trickplay. The bind targeted /config/data/trickplay, which
Docker happily created and Jellyfin never used -- 11G of trickplay tiles went
to the 194G root SSD instead of the ZFS array, with root at 84%.
Metadata is unaffected: MetadataPath is set independently and its bind at
/config/data/metadata is live.
Existing 11G rsynced to /storage1/labdata/jellyfin/trickplay; root 84% -> 78%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Jellyfin 10.11 writes metadata to /config/data/metadata, not
/config/metadata — the old bind left bulk artwork on the root disk.
Moved existing metadata to /storage1 and corrected the compose mount.
Also sets TranscodingTempPath=/transcode.
Caddy deploy: job containers never saw /config/caddy; write via a
host-bind docker run instead. apply-compose.ps1 -Caddy stages through
/tmp + sudo. GITHUB_STACKS_PAT updated to the live Gitea PAT.
jellyfin and wizarr are public by design — they replace Plex's own
remote access. jellystat is internal_gate + forward_auth, matching the
tautulli pattern.
New Portainer stack: jellyfin, jellystat + postgres, wizarr and
jellyplex-watched. Bulk data (metadata, trickplay, cache, transcodes)
binds to /storage1; only SQLite config lands on the root disk.
Trickplay gets an explicit sub-bind because 10.11 moved it to
/config/data/trickplay, which is not relocatable from the UI. Transcodes
are disk-backed rather than tmpfs because jellyfin#16608 orphans segments
on the Remux/DirectStream jobs that are routine on this server.
/var/lib/plexmediaserver does not exist; the real path is /var/lib/plex,
itself a symlink to /storage1/labdata/plex. Media libraries are
/storage1/{Movies,TV,Anime} with /storage1/Media/* as symlinks to them.
25 tasks across 7 phases, written to be executed in Cursor rather than
Claude Code — every action is a literal shell command, file edit or named
UI step, with no dependency on this repo's Claude Code skills or MCP
servers.
Phase 0 is a hard gate: the root disk is at 97% with /config on it, so
nothing deploys until space is reclaimed and free-space alerting exists.
Plex runs untouched through Task 22, and rollback stays a single command
until the 30-day archive expires.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Plex now gates remote playback behind Plex Pass / Remote Watch Pass and
valhalla has no Plex Pass, so remote viewers are being pushed to pay for
access to a server they don't own.
Design covers a new stacks/streaming Portainer stack (jellyfin, jellystat
+ postgres, wizarr, jellyplex-watched), a storage layout that keeps bulk
data off the 97%-full root disk, and eight phases with Plex left running
and untouched through Phase 6.
Findings that shaped it, measured rather than assumed:
- root disk is at 97% (5.7G free) and /config lives on it — Phase 0
- Tesla P100 is GP100: no NVENC, decode only, so transcoding is CPU-bound
- 48% of streams transcode, peak 7 concurrent, but 1080p SDR with no
subtitle burn-in and no HDR tone-mapping, which 52 threads can carry
- /var/lib/plex is a symlink to /storage1/labdata/plex holding 383G; the
15G Metadata dir is reusable via Plexyfin, the 365G of BIF preview
thumbnails is not (format-incompatible with Jellyfin trickplay)
- jellyfin-plugin-sso is browser-only, so Roku/Fire TV/tvOS cannot use it
— native Jellyfin accounts, a deliberate exception to the auth standard
Also notes that CLAUDE.md documents a Plex data path that does not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same fix as 317a424, applied to the files an agent actually reads before
touching the host: AGENTS.md and both copies of the homelab-ssh skill.
They all claimed a headless Ubuntu server; valhalla is EndeavourOS (Arch).
Also record two things that cost time this session: there is no apt/apt-get
on the host, and ginnoir has passwordless sudo (needed for anything writing
to root-owned paths like /config/caddy/Caddyfile).
Historical records under plans/ and docs/superpowers/plans/ are left alone
on purpose — they document what was true when the work was done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
valhalla is EndeavourOS (Arch), not Ubuntu — there is no apt, only pacman.
Verified via /etc/os-release. This bit during a live session: `apt-get`
was not found when trying to install a package on the host.
Plex is likewise not a deb — it is the AUR/pacman package
plex-media-server 1.43.2.10687-1, confirmed with `pacman -Q`. Version,
systemd unit and binary path in that section were already correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Caddy sorts the `route` directive ahead of `respond`, so on any site that
wrapped its handlers in a catch-all `route {}`, the `import internal_only`
above it never executed — the route matched everything and handled the
request before the 403 was reached. Confirmed in the adapted JSON: the
route was sub[0] with match="<always>" and the 403 was sub[1], unreachable.
sonarr, radarr, bazarr, prowlarr, tautulli, qbittorrent, nzbget, whisparr
and stash were all affected. They were still behind Authentik, so this was
not an open door, but the LAN/tailnet restriction was doing nothing.
Add an (internal_gate) snippet with the same IP check, intended to be
imported as the first directive INSIDE a route, where directives run in
written order. Verified in the adapted config: the 403 is now [0] inside
the route, ahead of the outpost proxy and the handle blocks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
files.ginnoir.com imported internal_only, which 403s every non-LAN/tailnet
client — including anyone opening a public share link. Replace the blanket
import with an inline @blocked matcher that keeps the same IP gate but exempts
Nextcloud's public-share surface: /s/*, /public.php*, the files_sharing app,
and the static assets the share page loads.
Everything else on the host (file browser, /settings, /remote.php/dav) still
403s externally. The nextcloud.ginnoir.com alias is unchanged and stays fully
internal-only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The LAN resolver (OPNsense Unbound) is authoritative for ginnoir.com via the
split-horizon override, but its local-zone holds only A records, so
`SOA ginnoir.com` returns NODATA. certmagic's zone lookup walks up the label
chain hunting for an SOA, finds none at ginnoir.com, climbs to `com.`, and asks
Cloudflare for a `com` zone:
adding temporary record for zone "com.": expected 1 zone, got 0 for com.
Every DNS-01 renewal has failed for ~17 days (attempt 91 on the oldest), and
Caddy fell back to the LE staging endpoint. Certs began expiring as they rolled
off: 5etools (-4d), files (-1.9d), auth (-15.6h), fam (-15.6h), with ~20 more
queued behind them.
The expired auth.ginnoir.com cert is what broke Nextcloud SSO: its server-side
discovery fetch fails TLS verification ("certificate has expired"), and
user_oidc's LoginController catches that and returns 404 "provider unreachable".
Browsers were unaffected because they let you click through an expired cert;
PHP/curl does not.
Pinning 1.1.1.1/1.0.0.1 on the caddy container sends the SOA lookup to public
DNS. Verified Docker's embedded resolver still serves container names and the
gitea.ginnoir.com alias, which take priority over the external servers.
Also corrects the imgstudio comment, which credited its working TLS to avoiding
a "false .com" in the hostname. The actual reason was its explicit `resolvers`
line -- the same fix, applied to one site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Use imgstudio.ginnoir.com instead of comfyui.* — the ACME challenge FQDN
embeds a false .com that breaks Cloudflare DNS-01. Add per-site resolvers
1.1.1.1 to bypass LAN split-DNS during cert issuance.
Pure bugfix/security patch per upstream release notes — no breaking
changes or migration steps. Bumps both authentik-server and
authentik-worker (embedded outpost updates with the server).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dev stack shares one stack.env across all services via env_file, so
BookStack inherited Plane's DATABASE_URL (postgresql://...@postgres_plane).
Laravel prioritizes DATABASE_URL over the individual DB_* vars, so BookStack
connected to Plane's Postgres and 500'd with "could not find driver" (the
LSIO image ships pdo_mysql, not pdo_pgsql). Pin DATABASE_URL to BookStack's
own MariaDB DSN in its environment block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
check_uptime.js gets a fetchWithRetry wrapper (3 attempts, 2s backoff)
for transient failures against the status page/heartbeat API.
check_uptime_to_obsidian.js is a variant that logs results into the
Obsidian vault instead of stdout. Also adds two benchmark writeups
(gpt-oss-20b on Ollama, 73-node Ollama fleet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both root paths (/srv/aon, /srv/ffttsystems) no longer exist on disk
-- confirmed during the recent disk cleanup audit -- so these blocks
were serving errors to any visitor. Content was already gone; only
the stale Caddyfile entries remained.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
These three containers were writing multi-GB/day of unbounded logs
(authentik_server 3.5GB, immich-server 2.3GB, owncloud_mariadb 2.1GB),
a major contributor to a recent 100% disk-full incident. Daemon-wide
default (max-size 10m, max-file 3) was also added to
/etc/docker/daemon.json on valhalla, but that only covers newly
created containers going forward -- these three need the explicit
per-service override since they're long-running and won't be
recreated otherwise.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Image camofox-browser:135.0.1-x86_64 builds + serves, but camoufox fails to
launch: "cannot open display: [object Promise]" (upstream un-awaited Promise in
the virtual_display path). HTTP /health ok, browserRunning:false. NOT deployed:
no Caddy route, no Portainer registration, no Hermes wiring. Resume once the
upstream display bug is fixed (re-add Caddy block + register stack).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DeepReinforce Ornith-1.0 (dense 9B on Qwen 3.5, Q5_K_M, MIT), an
agentic-coding model. Tool-calls + <think> work under --jinja; native
256k so no YaRN. Loads at ~7.7GB VRAM @ 64k.
Benchmark (docs/2026-06-27-ornith-9b-benchmark.md): quality ties
gpt-oss-20b but gen is ~2.5-3x slower (dense 9B active vs gpt-oss MoE
3.6B active on the compute-bound P100). Default stays gpt-oss-20b;
ornith kept as a coding specialist in the menu.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add hermes-motif (skill discovery; complementary to curator, not a rival),
hermes-web-search-plus (mature multi-provider search, pairs with camofox),
optional llmtrim/rtk context efficiency, and the claude/codex/cursor/antigravity
delegation fabric. Record that Claude Code + Codex are now installed on valhalla.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
UI: trial hermes-ui + hermes-workspace in parallel, keep winner (new Task 9).
Curator stays report-only. eagle-eye trial confirmed (only direct skill-router
in the ecosystem; llmtrim noted as complementary). camofox minimal 2-tool skill.
All phases now ungated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Research + fit analysis for hermes-agent-acp-skill, hermes-curator-evolver,
eagle-eye, camofox-browser, hermes-ui, hermes-workspace, mission-control.
Phased rollout; open decisions flagged for review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Portainer's git checkout auto-creates a relative repo-file bind as a directory,
breaking the /app/config.yaml mount. Use the absolute host path like the share
stack; repo copy stays canonical, mirrored to /config/llm on deploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace single llama-server with llama-swap so all benchmarked models are
selectable from Hermes' menu and hot-swapped on the one P100. Menu: gpt-oss-20b
(default, ~45s cold start), gemma-4-26b-a4b (MoE), gemma-4-12b, gemma-4-e4b.
qwen3-30b-a3b excluded (OOMs at 64k in 16GB). All 64k, q8/q8 KV, --parallel 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
