valhalla is EndeavourOS (Arch), not Ubuntu — there is no apt, only pacman.
Verified via /etc/os-release. This bit during a live session: `apt-get`
was not found when trying to install a package on the host.
Plex is likewise not a deb — it is the AUR/pacman package
plex-media-server 1.43.2.10687-1, confirmed with `pacman -Q`. Version,
systemd unit and binary path in that section were already correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Caddy sorts the `route` directive ahead of `respond`, so on any site that
wrapped its handlers in a catch-all `route {}`, the `import internal_only`
above it never executed — the route matched everything and handled the
request before the 403 was reached. Confirmed in the adapted JSON: the
route was sub[0] with match="<always>" and the 403 was sub[1], unreachable.
sonarr, radarr, bazarr, prowlarr, tautulli, qbittorrent, nzbget, whisparr
and stash were all affected. They were still behind Authentik, so this was
not an open door, but the LAN/tailnet restriction was doing nothing.
Add an (internal_gate) snippet with the same IP check, intended to be
imported as the first directive INSIDE a route, where directives run in
written order. Verified in the adapted config: the 403 is now [0] inside
the route, ahead of the outpost proxy and the handle blocks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
files.ginnoir.com imported internal_only, which 403s every non-LAN/tailnet
client — including anyone opening a public share link. Replace the blanket
import with an inline @blocked matcher that keeps the same IP gate but exempts
Nextcloud's public-share surface: /s/*, /public.php*, the files_sharing app,
and the static assets the share page loads.
Everything else on the host (file browser, /settings, /remote.php/dav) still
403s externally. The nextcloud.ginnoir.com alias is unchanged and stays fully
internal-only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The LAN resolver (OPNsense Unbound) is authoritative for ginnoir.com via the
split-horizon override, but its local-zone holds only A records, so
`SOA ginnoir.com` returns NODATA. certmagic's zone lookup walks up the label
chain hunting for an SOA, finds none at ginnoir.com, climbs to `com.`, and asks
Cloudflare for a `com` zone:
adding temporary record for zone "com.": expected 1 zone, got 0 for com.
Every DNS-01 renewal has failed for ~17 days (attempt 91 on the oldest), and
Caddy fell back to the LE staging endpoint. Certs began expiring as they rolled
off: 5etools (-4d), files (-1.9d), auth (-15.6h), fam (-15.6h), with ~20 more
queued behind them.
The expired auth.ginnoir.com cert is what broke Nextcloud SSO: its server-side
discovery fetch fails TLS verification ("certificate has expired"), and
user_oidc's LoginController catches that and returns 404 "provider unreachable".
Browsers were unaffected because they let you click through an expired cert;
PHP/curl does not.
Pinning 1.1.1.1/1.0.0.1 on the caddy container sends the SOA lookup to public
DNS. Verified Docker's embedded resolver still serves container names and the
gitea.ginnoir.com alias, which take priority over the external servers.
Also corrects the imgstudio comment, which credited its working TLS to avoiding
a "false .com" in the hostname. The actual reason was its explicit `resolvers`
line -- the same fix, applied to one site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Use imgstudio.ginnoir.com instead of comfyui.* — the ACME challenge FQDN
embeds a false .com that breaks Cloudflare DNS-01. Add per-site resolvers
1.1.1.1 to bypass LAN split-DNS during cert issuance.
Pure bugfix/security patch per upstream release notes — no breaking
changes or migration steps. Bumps both authentik-server and
authentik-worker (embedded outpost updates with the server).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dev stack shares one stack.env across all services via env_file, so
BookStack inherited Plane's DATABASE_URL (postgresql://...@postgres_plane).
Laravel prioritizes DATABASE_URL over the individual DB_* vars, so BookStack
connected to Plane's Postgres and 500'd with "could not find driver" (the
LSIO image ships pdo_mysql, not pdo_pgsql). Pin DATABASE_URL to BookStack's
own MariaDB DSN in its environment block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
check_uptime.js gets a fetchWithRetry wrapper (3 attempts, 2s backoff)
for transient failures against the status page/heartbeat API.
check_uptime_to_obsidian.js is a variant that logs results into the
Obsidian vault instead of stdout. Also adds two benchmark writeups
(gpt-oss-20b on Ollama, 73-node Ollama fleet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both root paths (/srv/aon, /srv/ffttsystems) no longer exist on disk
-- confirmed during the recent disk cleanup audit -- so these blocks
were serving errors to any visitor. Content was already gone; only
the stale Caddyfile entries remained.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
These three containers were writing multi-GB/day of unbounded logs
(authentik_server 3.5GB, immich-server 2.3GB, owncloud_mariadb 2.1GB),
a major contributor to a recent 100% disk-full incident. Daemon-wide
default (max-size 10m, max-file 3) was also added to
/etc/docker/daemon.json on valhalla, but that only covers newly
created containers going forward -- these three need the explicit
per-service override since they're long-running and won't be
recreated otherwise.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Image camofox-browser:135.0.1-x86_64 builds + serves, but camoufox fails to
launch: "cannot open display: [object Promise]" (upstream un-awaited Promise in
the virtual_display path). HTTP /health ok, browserRunning:false. NOT deployed:
no Caddy route, no Portainer registration, no Hermes wiring. Resume once the
upstream display bug is fixed (re-add Caddy block + register stack).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DeepReinforce Ornith-1.0 (dense 9B on Qwen 3.5, Q5_K_M, MIT), an
agentic-coding model. Tool-calls + <think> work under --jinja; native
256k so no YaRN. Loads at ~7.7GB VRAM @ 64k.
Benchmark (docs/2026-06-27-ornith-9b-benchmark.md): quality ties
gpt-oss-20b but gen is ~2.5-3x slower (dense 9B active vs gpt-oss MoE
3.6B active on the compute-bound P100). Default stays gpt-oss-20b;
ornith kept as a coding specialist in the menu.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add hermes-motif (skill discovery; complementary to curator, not a rival),
hermes-web-search-plus (mature multi-provider search, pairs with camofox),
optional llmtrim/rtk context efficiency, and the claude/codex/cursor/antigravity
delegation fabric. Record that Claude Code + Codex are now installed on valhalla.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
UI: trial hermes-ui + hermes-workspace in parallel, keep winner (new Task 9).
Curator stays report-only. eagle-eye trial confirmed (only direct skill-router
in the ecosystem; llmtrim noted as complementary). camofox minimal 2-tool skill.
All phases now ungated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Research + fit analysis for hermes-agent-acp-skill, hermes-curator-evolver,
eagle-eye, camofox-browser, hermes-ui, hermes-workspace, mission-control.
Phased rollout; open decisions flagged for review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Portainer's git checkout auto-creates a relative repo-file bind as a directory,
breaking the /app/config.yaml mount. Use the absolute host path like the share
stack; repo copy stays canonical, mirrored to /config/llm on deploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace single llama-server with llama-swap so all benchmarked models are
selectable from Hermes' menu and hot-swapped on the one P100. Menu: gpt-oss-20b
(default, ~45s cold start), gemma-4-26b-a4b (MoE), gemma-4-12b, gemma-4-e4b.
qwen3-30b-a3b excluded (OOMs at 64k in 16GB). All 64k, q8/q8 KV, --parallel 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The q4_0 V-cache + flash-attention path is pathological on the GP100: 1.28
tok/s generation at 5-8% GPU util. q8_0 V-cache gives 9.2 tok/s and still fits
64k context in 16GB (15.3GB used, ~950MB free).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
llama-server caps the slot to the GGUF training context (32768) and ignores the
YaRN-extended size, leaving per-seq context at 32k. Raise qwen2.context_length
metadata to 65536 so the full window is served per request.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the default 4 slots, llama-server splits ctx into 32k per sequence, which
fails Hermes' 64K minimum. One slot serves the full 65536 per request (serial
agent use; concurrent calls queue).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Hermes Agent rejects models with <64K context. Qwen2.5-14B is 32k native, so
enable YaRN rope-scaling (2x → 65536) and drop the V-cache to q4_0 for VRAM
headroom on the 16GB P100.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The server-cuda image parses -fa as --flash-attn [on|off|auto], so a bare -fa
swallowed the following --cache-type-k as its value and crash-looped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New stacks/llm/ serves Qwen2.5-14B-Instruct (Q4_K_M GGUF) via llama.cpp's
OpenAI-compatible server on the Tesla P100 (CDI nvidia.com/gpu=0), published on
172.20.0.1:8090 for the host-side Hermes agent. vLLM was rejected: the P100
(cc 6.0) lacks the DP4A INT8 instructions its AWQ/GPTQ kernels need.
Includes design spec and implementation plan under docs/superpowers/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SameSite=Strict caused Safari and Firefox to drop the freshly-set cookie on
the immediate same-site redirect when the top-level navigation originated
from an external app (Discord). SameSite=Lax explicitly permits the cookie
on same-site redirects regardless of the initiating context.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
{env.SHARE_KEY} is not resolved by Caddy's query matcher or CEL expression at
request time — the literal placeholder string was compared against the URL key,
causing every external request to fall through to 403. {$SHARE_KEY} is
substituted by the Caddyfile adapter at reload/start, baking the literal key
value into the compiled config.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sonarr/radarr/prowlarr/whisparr/bazarr were hitting the default Docker
nofile limit of 1024, causing EMFILE errors and excessive virtiofsd FD
churn on the /storage1 virtiofs mount.
Capability-URL snippet: external users visit ?k=SHARE_KEY once, Caddy
sets a 30-day HttpOnly cookie and redirects to the clean path. Subsequent
visits check the cookie only. Internal IPs and tailnet pass through
unconditionally. Applied to roms.ginnoir.com and romhacks.ginnoir.com.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous commit used `hbbs -R 192.168.1.69` (uppercase -R sets
rendezvous/cluster servers), so hbbs advertised an empty relay list and
clients failed with "failed to connect to relay server". Lowercase -r is
the relay-server flag; hbbs now advertises 192.168.1.69 and LAN clients
relay correctly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The save-sync (/api/sync/*) and raw content_hash fixes that forced the
beta pin shipped in stable 4.9.x (now 4.9.2). Return to the watchtower-
tracked :4 tag per the original design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
R510 -> R730XD/Proxmox migration helpers (consistent DB dumps, B2/Kopia snapshot refresh, cutover final backup) plus the previously-untracked gitea-portainer-registry cutover plan under docs/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
