jellyfin and wizarr are public by design — they replace Plex's own
remote access. jellystat is internal_gate + forward_auth, matching the
tautulli pattern.
New Portainer stack: jellyfin, jellystat + postgres, wizarr and
jellyplex-watched. Bulk data (metadata, trickplay, cache, transcodes)
binds to /storage1; only SQLite config lands on the root disk.
Trickplay gets an explicit sub-bind because 10.11 moved it to
/config/data/trickplay, which is not relocatable from the UI. Transcodes
are disk-backed rather than tmpfs because jellyfin#16608 orphans segments
on the Remux/DirectStream jobs that are routine on this server.
/var/lib/plexmediaserver does not exist; the real path is /var/lib/plex,
itself a symlink to /storage1/labdata/plex. Media libraries are
/storage1/{Movies,TV,Anime} with /storage1/Media/* as symlinks to them.
25 tasks across 7 phases, written to be executed in Cursor rather than
Claude Code — every action is a literal shell command, file edit or named
UI step, with no dependency on this repo's Claude Code skills or MCP
servers.
Phase 0 is a hard gate: the root disk is at 97% with /config on it, so
nothing deploys until space is reclaimed and free-space alerting exists.
Plex runs untouched through Task 22, and rollback stays a single command
until the 30-day archive expires.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Plex now gates remote playback behind Plex Pass / Remote Watch Pass and
valhalla has no Plex Pass, so remote viewers are being pushed to pay for
access to a server they don't own.
Design covers a new stacks/streaming Portainer stack (jellyfin, jellystat
+ postgres, wizarr, jellyplex-watched), a storage layout that keeps bulk
data off the 97%-full root disk, and eight phases with Plex left running
and untouched through Phase 6.
Findings that shaped it, measured rather than assumed:
- root disk is at 97% (5.7G free) and /config lives on it — Phase 0
- Tesla P100 is GP100: no NVENC, decode only, so transcoding is CPU-bound
- 48% of streams transcode, peak 7 concurrent, but 1080p SDR with no
subtitle burn-in and no HDR tone-mapping, which 52 threads can carry
- /var/lib/plex is a symlink to /storage1/labdata/plex holding 383G; the
15G Metadata dir is reusable via Plexyfin, the 365G of BIF preview
thumbnails is not (format-incompatible with Jellyfin trickplay)
- jellyfin-plugin-sso is browser-only, so Roku/Fire TV/tvOS cannot use it
— native Jellyfin accounts, a deliberate exception to the auth standard
Also notes that CLAUDE.md documents a Plex data path that does not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same fix as 317a424, applied to the files an agent actually reads before
touching the host: AGENTS.md and both copies of the homelab-ssh skill.
They all claimed a headless Ubuntu server; valhalla is EndeavourOS (Arch).
Also record two things that cost time this session: there is no apt/apt-get
on the host, and ginnoir has passwordless sudo (needed for anything writing
to root-owned paths like /config/caddy/Caddyfile).
Historical records under plans/ and docs/superpowers/plans/ are left alone
on purpose — they document what was true when the work was done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
valhalla is EndeavourOS (Arch), not Ubuntu — there is no apt, only pacman.
Verified via /etc/os-release. This bit during a live session: `apt-get`
was not found when trying to install a package on the host.
Plex is likewise not a deb — it is the AUR/pacman package
plex-media-server 1.43.2.10687-1, confirmed with `pacman -Q`. Version,
systemd unit and binary path in that section were already correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Caddy sorts the `route` directive ahead of `respond`, so on any site that
wrapped its handlers in a catch-all `route {}`, the `import internal_only`
above it never executed — the route matched everything and handled the
request before the 403 was reached. Confirmed in the adapted JSON: the
route was sub[0] with match="<always>" and the 403 was sub[1], unreachable.
sonarr, radarr, bazarr, prowlarr, tautulli, qbittorrent, nzbget, whisparr
and stash were all affected. They were still behind Authentik, so this was
not an open door, but the LAN/tailnet restriction was doing nothing.
Add an (internal_gate) snippet with the same IP check, intended to be
imported as the first directive INSIDE a route, where directives run in
written order. Verified in the adapted config: the 403 is now [0] inside
the route, ahead of the outpost proxy and the handle blocks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
files.ginnoir.com imported internal_only, which 403s every non-LAN/tailnet
client — including anyone opening a public share link. Replace the blanket
import with an inline @blocked matcher that keeps the same IP gate but exempts
Nextcloud's public-share surface: /s/*, /public.php*, the files_sharing app,
and the static assets the share page loads.
Everything else on the host (file browser, /settings, /remote.php/dav) still
403s externally. The nextcloud.ginnoir.com alias is unchanged and stays fully
internal-only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The LAN resolver (OPNsense Unbound) is authoritative for ginnoir.com via the
split-horizon override, but its local-zone holds only A records, so
`SOA ginnoir.com` returns NODATA. certmagic's zone lookup walks up the label
chain hunting for an SOA, finds none at ginnoir.com, climbs to `com.`, and asks
Cloudflare for a `com` zone:
adding temporary record for zone "com.": expected 1 zone, got 0 for com.
Every DNS-01 renewal has failed for ~17 days (attempt 91 on the oldest), and
Caddy fell back to the LE staging endpoint. Certs began expiring as they rolled
off: 5etools (-4d), files (-1.9d), auth (-15.6h), fam (-15.6h), with ~20 more
queued behind them.
The expired auth.ginnoir.com cert is what broke Nextcloud SSO: its server-side
discovery fetch fails TLS verification ("certificate has expired"), and
user_oidc's LoginController catches that and returns 404 "provider unreachable".
Browsers were unaffected because they let you click through an expired cert;
PHP/curl does not.
Pinning 1.1.1.1/1.0.0.1 on the caddy container sends the SOA lookup to public
DNS. Verified Docker's embedded resolver still serves container names and the
gitea.ginnoir.com alias, which take priority over the external servers.
Also corrects the imgstudio comment, which credited its working TLS to avoiding
a "false .com" in the hostname. The actual reason was its explicit `resolvers`
line -- the same fix, applied to one site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Use imgstudio.ginnoir.com instead of comfyui.* — the ACME challenge FQDN
embeds a false .com that breaks Cloudflare DNS-01. Add per-site resolvers
1.1.1.1 to bypass LAN split-DNS during cert issuance.
Pure bugfix/security patch per upstream release notes — no breaking
changes or migration steps. Bumps both authentik-server and
authentik-worker (embedded outpost updates with the server).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dev stack shares one stack.env across all services via env_file, so
BookStack inherited Plane's DATABASE_URL (postgresql://...@postgres_plane).
Laravel prioritizes DATABASE_URL over the individual DB_* vars, so BookStack
connected to Plane's Postgres and 500'd with "could not find driver" (the
LSIO image ships pdo_mysql, not pdo_pgsql). Pin DATABASE_URL to BookStack's
own MariaDB DSN in its environment block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
check_uptime.js gets a fetchWithRetry wrapper (3 attempts, 2s backoff)
for transient failures against the status page/heartbeat API.
check_uptime_to_obsidian.js is a variant that logs results into the
Obsidian vault instead of stdout. Also adds two benchmark writeups
(gpt-oss-20b on Ollama, 73-node Ollama fleet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both root paths (/srv/aon, /srv/ffttsystems) no longer exist on disk
-- confirmed during the recent disk cleanup audit -- so these blocks
were serving errors to any visitor. Content was already gone; only
the stale Caddyfile entries remained.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
These three containers were writing multi-GB/day of unbounded logs
(authentik_server 3.5GB, immich-server 2.3GB, owncloud_mariadb 2.1GB),
a major contributor to a recent 100% disk-full incident. Daemon-wide
default (max-size 10m, max-file 3) was also added to
/etc/docker/daemon.json on valhalla, but that only covers newly
created containers going forward -- these three need the explicit
per-service override since they're long-running and won't be
recreated otherwise.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Image camofox-browser:135.0.1-x86_64 builds + serves, but camoufox fails to
launch: "cannot open display: [object Promise]" (upstream un-awaited Promise in
the virtual_display path). HTTP /health ok, browserRunning:false. NOT deployed:
no Caddy route, no Portainer registration, no Hermes wiring. Resume once the
upstream display bug is fixed (re-add Caddy block + register stack).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DeepReinforce Ornith-1.0 (dense 9B on Qwen 3.5, Q5_K_M, MIT), an
agentic-coding model. Tool-calls + <think> work under --jinja; native
256k so no YaRN. Loads at ~7.7GB VRAM @ 64k.
Benchmark (docs/2026-06-27-ornith-9b-benchmark.md): quality ties
gpt-oss-20b but gen is ~2.5-3x slower (dense 9B active vs gpt-oss MoE
3.6B active on the compute-bound P100). Default stays gpt-oss-20b;
ornith kept as a coding specialist in the menu.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add hermes-motif (skill discovery; complementary to curator, not a rival),
hermes-web-search-plus (mature multi-provider search, pairs with camofox),
optional llmtrim/rtk context efficiency, and the claude/codex/cursor/antigravity
delegation fabric. Record that Claude Code + Codex are now installed on valhalla.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
UI: trial hermes-ui + hermes-workspace in parallel, keep winner (new Task 9).
Curator stays report-only. eagle-eye trial confirmed (only direct skill-router
in the ecosystem; llmtrim noted as complementary). camofox minimal 2-tool skill.
All phases now ungated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Research + fit analysis for hermes-agent-acp-skill, hermes-curator-evolver,
eagle-eye, camofox-browser, hermes-ui, hermes-workspace, mission-control.
Phased rollout; open decisions flagged for review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Portainer's git checkout auto-creates a relative repo-file bind as a directory,
breaking the /app/config.yaml mount. Use the absolute host path like the share
stack; repo copy stays canonical, mirrored to /config/llm on deploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace single llama-server with llama-swap so all benchmarked models are
selectable from Hermes' menu and hot-swapped on the one P100. Menu: gpt-oss-20b
(default, ~45s cold start), gemma-4-26b-a4b (MoE), gemma-4-12b, gemma-4-e4b.
qwen3-30b-a3b excluded (OOMs at 64k in 16GB). All 64k, q8/q8 KV, --parallel 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The q4_0 V-cache + flash-attention path is pathological on the GP100: 1.28
tok/s generation at 5-8% GPU util. q8_0 V-cache gives 9.2 tok/s and still fits
64k context in 16GB (15.3GB used, ~950MB free).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
llama-server caps the slot to the GGUF training context (32768) and ignores the
YaRN-extended size, leaving per-seq context at 32k. Raise qwen2.context_length
metadata to 65536 so the full window is served per request.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the default 4 slots, llama-server splits ctx into 32k per sequence, which
fails Hermes' 64K minimum. One slot serves the full 65536 per request (serial
agent use; concurrent calls queue).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Hermes Agent rejects models with <64K context. Qwen2.5-14B is 32k native, so
enable YaRN rope-scaling (2x → 65536) and drop the V-cache to q4_0 for VRAM
headroom on the 16GB P100.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The server-cuda image parses -fa as --flash-attn [on|off|auto], so a bare -fa
swallowed the following --cache-type-k as its value and crash-looped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New stacks/llm/ serves Qwen2.5-14B-Instruct (Q4_K_M GGUF) via llama.cpp's
OpenAI-compatible server on the Tesla P100 (CDI nvidia.com/gpu=0), published on
172.20.0.1:8090 for the host-side Hermes agent. vLLM was rejected: the P100
(cc 6.0) lacks the DP4A INT8 instructions its AWQ/GPTQ kernels need.
Includes design spec and implementation plan under docs/superpowers/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SameSite=Strict caused Safari and Firefox to drop the freshly-set cookie on
the immediate same-site redirect when the top-level navigation originated
from an external app (Discord). SameSite=Lax explicitly permits the cookie
on same-site redirects regardless of the initiating context.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
{env.SHARE_KEY} is not resolved by Caddy's query matcher or CEL expression at
request time — the literal placeholder string was compared against the URL key,
causing every external request to fall through to 403. {$SHARE_KEY} is
substituted by the Caddyfile adapter at reload/start, baking the literal key
value into the compiled config.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sonarr/radarr/prowlarr/whisparr/bazarr were hitting the default Docker
nofile limit of 1024, causing EMFILE errors and excessive virtiofsd FD
churn on the /storage1 virtiofs mount.
ginnoir