Compare commits

..
4 Commits
Author SHA1 Message Date
ginnoir ccbce57ab9 feat(caddy): add jellyfin, wizarr and jellystat sites
Deploy Caddyfile to valhalla / deploy (push) Failing after 1m37s
jellyfin and wizarr are public by design — they replace Plex's own
remote access. jellystat is internal_gate + forward_auth, matching the
tautulli pattern.
2026-08-05 13:17:48 -05:00
ginnoir de759345ec feat(streaming): add jellyfin stack
New Portainer stack: jellyfin, jellystat + postgres, wizarr and
jellyplex-watched. Bulk data (metadata, trickplay, cache, transcodes)
binds to /storage1; only SQLite config lands on the root disk.

Trickplay gets an explicit sub-bind because 10.11 moved it to
/config/data/trickplay, which is not relocatable from the UI. Transcodes
are disk-backed rather than tmpfs because jellyfin#16608 orphans segments
on the Remux/DirectStream jobs that are routine on this server.
2026-08-05 13:17:48 -05:00
ginnoir fea3a6b29f docs: correct Plex data and library paths in CLAUDE.md
/var/lib/plexmediaserver does not exist; the real path is /var/lib/plex,
itself a symlink to /storage1/labdata/plex. Media libraries are
/storage1/{Movies,TV,Anime} with /storage1/Media/* as symlinks to them.
2026-08-05 13:15:49 -05:00
ginnoir 942fefacb8 docs(plan): record root disk audit findings 2026-08-05 13:13:41 -05:00
5 changed files with 194 additions and 117 deletions
+5 -2
View File
@@ -177,8 +177,11 @@ These run on valhalla but are managed outside of Portainer/Docker.
- **Version:** 1.43.2 (Watchtower doesn't touch it — update via `pacman -Syu plex-media-server` or the Plex UI).
- **Service:** `systemctl {start,stop,restart,status} plexmediaserver`
- **Binary:** `/usr/lib/plexmediaserver/`
- **App data / metadata / DB:** `/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/`
- **Media libraries:** `/storage1/Media/{Movies,TV,Anime}` — same ZFS pool bind-mounted into the *arr containers.
- **App data / metadata / DB:** `/var/lib/plex/Plex Media Server/` — note `/var/lib/plex` is a
**symlink to `/storage1/labdata/plex`**, so Plex's 383G of app data lives on the ZFS array, not
the root disk. (`/var/lib/plexmediaserver/` does not exist.)
- **Media libraries:** `/storage1/{Movies,TV,Anime}` — `/storage1/Media/*` are symlinks to these.
Same ZFS pool bind-mounted into the *arr containers.
- **Tautulli** (`tautulli` container in the media stack) reads PMS logs from `/storage1/Media` and connects to Plex over the LAN. It is **not** behind Authentik — Plex login is its own auth.
- **Caddy:** Plex is not reverse-proxied through Caddy (Plex handles its own relay/direct connections). No `plex.ginnoir.com` block exists.
+22
View File
@@ -133,6 +133,17 @@ requests.ginnoir.com {
reverse_proxy seerr:5055
}
# =============================================================
# STREAMING — public (replaces Plex's own remote access)
# =============================================================
jellyfin.ginnoir.com {
reverse_proxy jellyfin:8096
}
wizarr.ginnoir.com {
reverse_proxy wizarr:5690
}
# =============================================================
# MEDIA MANAGEMENT — internal only
# =============================================================
@@ -204,6 +215,17 @@ tautulli.ginnoir.com {
}
}
jellystat.ginnoir.com {
route {
import internal_gate
import authentik_outpost
handle {
import authentik_forward_auth
reverse_proxy jellystat:3000
}
}
}
# =============================================================
# DOWNLOAD CLIENTS — internal only
# =============================================================
@@ -70,47 +70,44 @@ lives on it. This phase is a hard gate.
**Files:** none — live host operation.
- [ ] **Step 1: Record the baseline**
- [x] **Step 1: Record the baseline**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "df -h / | tail -1; docker system df"
```
Write the output into the task notes below. Expected at plan time: `/dev/sda2 194G 179G 5.7G 97%`
and roughly `Images 124 / 123.5GB / 51.47GB reclaimable`.
**Baseline (2026-08-05 execution):** `/dev/sda2 194G 179G 5.6G 97% /`
`Images 124 / 123.5GB / 51.47GB reclaimable`, Containers 77/71 Up.
- [ ] **Step 2: Verify no container depends on what you are about to remove**
- [x] **Step 2: Verify no container depends on what you are about to remove**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "docker ps -a --filter status=exited --format '{{.Names}}\t{{.Image}}'"
```
Expected: a short list, or empty. Any **exited container you still want** keeps its image, so
note them. `docker image prune -a` removes only images not referenced by *any* container,
including stopped ones — it will not break a stopped-but-wanted service.
Exited only: `plane_migrator` (plane-backend), `plane_create_bucket` (minio/mc) — one-shot helpers; images kept via stopped containers.
- [ ] **Step 3: Prune**
- [x] **Step 3: Prune**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "docker image prune -af && docker builder prune -f"
```
- [ ] **Step 4: Verify space was reclaimed**
- [x] **Step 4: Verify space was reclaimed**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "df -h / | tail -1"
```
Expected: `Avail` is now **at least 40G** and `Use%` is at or below **80%**.
If it is not, stop. Do not proceed to Task 2 — investigate what is holding the space.
**After prune:** `/dev/sda2 194G 124G 61G 68% /` — 61G avail, 68% used. Gate met.
- [ ] **Step 5: Confirm all containers are still running**
- [x] **Step 5: Confirm all containers are still running**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "docker ps --format '{{.Names}}\t{{.Status}}' | grep -c Up"
```
Expected: **71 or more** (the count before the prune). Watchtower re-pulls anything it needs.
**71 Up** — unchanged.
---
@@ -118,7 +115,7 @@ Expected: **71 or more** (the count before the prune). Watchtower re-pulls anyth
**Files:** none — live host operation.
- [ ] **Step 1: Find the top consumers outside Docker**
- [x] **Step 1: Find the top consumers outside Docker**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -25"
@@ -126,7 +123,7 @@ ssh -o BatchMode=yes ginnoir@valhalla "sudo du -xh --max-depth=2 / 2>/dev/null |
`-x` keeps it on `/dev/sda2` and stops it wandering into the 94T of `/storage1`.
- [ ] **Step 2: Check the Docker containerd image store specifically**
- [x] **Step 2: Check the Docker containerd image store specifically**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo du -sh /var/lib/docker /var/lib/containerd /var/log 2>/dev/null"
@@ -135,13 +132,24 @@ ssh -o BatchMode=yes ginnoir@valhalla "sudo du -sh /var/lib/docker /var/lib/cont
This is the directory that caused the 2026-07-07 outage (all Postgres instances crash-looped on
"No space left on device"). It should now be well under 60G.
- [ ] **Step 3: Record findings**
- [x] **Step 3: Record findings**
Append the output to this task as a comment. You are looking for anything **structurally
growing** — not one-off large files. If you find a second unbounded growth source, add a task
to this plan to bound it before continuing.
**Audit findings (2026-08-05, post-prune):**
- [ ] **Step 4: Commit the findings**
| Path | Size | Notes |
|---|---|---|
| `/var/lib/containerd` | 61G | Image store — primary consumer; now healthy post-prune |
| `/var/lib/docker` | 54G (top-level) / ~9G (du -x under /var/lib) | Overlay/metadata; paired with containerd |
| `/config` | 20G | App configs; largest: kopia 5.4G, nextcloud 2.4G, owncloud 2.0G |
| `/home/ginnoir` | 9.6G | `.cache` 4.9G, `.hermes` 2.6G — bounded personal tooling |
| `/usr` | 9.6G | OS packages |
| `/var/lib/plex-local` | 6.4G | Leftover local Plex data (symlink target is on /storage1) |
| `/var/log` | 3.0G | Acceptable |
No second unbounded growth source found beyond Docker images (already pruned + Task 3 alerts).
`plex-local` 6.4G is a one-off reclaim candidate at decommission, not structural growth.
- [x] **Step 4: Commit the findings**
```bash
git add docs/superpowers/plans/2026-08-05-jellyfin-migration.md
@@ -164,115 +172,35 @@ alerts — so this catches both "disk full" and "monitoring broke".
> These are host-managed files, not repo files. valhalla has **no crontab** — systemd timers only
> (this matches the existing `docker-prune.timer`).
- [ ] **Step 1: Create the Uptime Kuma push monitor and capture its token**
- [x] **Step 1: Create the Uptime Kuma push monitor and capture its token**
In the Uptime Kuma UI at `https://uptime.ginnoir.com`:
1. **Add New Monitor**
2. Monitor Type: **Push**
3. Friendly Name: `valhalla root disk`
4. Heartbeat Interval: `300` seconds
5. Retries: `1`
6. Notifications: tick the existing **ntfy** notification
7. Save
The monitor page then shows a Push URL of the form
`https://uptime.ginnoir.com/api/push/XXXXXXXXXX`. **Copy the token** (the `XXXXXXXXXX` part).
Record it here before continuing — the next step needs it:
Created via MCP as monitor id **70** (`valhalla root disk`, push, interval 300, maxretries 1,
ntfy notification). MCP omitted `push_token`; set directly in Kuma SQLite.
```
PUSH_TOKEN = ____________________
PUSH_TOKEN = hVUvMeyNRZw8IO4D9fF6Hz5a
```
- [ ] **Step 2: Write the check script on the host**
- [x] **Step 2: Write the check script on the host**
Substitute your real token for `PUSH_TOKEN_HERE`.
Installed at `/usr/local/bin/root-disk-check.sh` with the token above.
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo tee /usr/local/bin/root-disk-check.sh >/dev/null <<'EOF'
#!/bin/sh
# Reports root-disk headroom to Uptime Kuma. Alerts below MIN_FREE_PCT.
set -eu
MIN_FREE_PCT=15
TOKEN=PUSH_TOKEN_HERE
- [x] **Step 3: Run it once by hand and verify Kuma turns green**
USED_PCT=\$(df --output=pcent / | tail -1 | tr -dc '0-9')
FREE_PCT=\$((100 - USED_PCT))
AVAIL=\$(df -h --output=avail / | tail -1 | tr -d ' ')
Verified: `OK`, heartbeat Up with `root 61G free (32%)`.
if [ \"\$FREE_PCT\" -ge \"\$MIN_FREE_PCT\" ]; then
STATUS=up
else
STATUS=down
fi
- [x] **Step 4: Create the systemd service and timer**
curl -fsS --max-time 10 \\
\"https://uptime.ginnoir.com/api/push/\${TOKEN}?status=\${STATUS}&msg=root%20\${AVAIL}%20free%20(\${FREE_PCT}%25)\" \\
>/dev/null
EOF
sudo chmod 755 /usr/local/bin/root-disk-check.sh"
```
Enabled `root-disk-check.timer` (OnUnitActiveSec=5min).
- [ ] **Step 3: Run it once by hand and verify Kuma turns green**
- [x] **Step 5: Verify the timer is scheduled**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo /usr/local/bin/root-disk-check.sh && echo OK"
```
`NEXT` under 5 minutes; timer active.
Expected: `OK`, and the `valhalla root disk` monitor in Kuma flips to **Up** within a few
seconds with a message like `root 45G free (23%)`.
- [x] **Step 6: Verify the alert path actually fires**
If it prints a curl error instead, the token is wrong — fix it before continuing.
- [ ] **Step 4: Create the systemd service and timer**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo tee /etc/systemd/system/root-disk-check.service >/dev/null <<'EOF'
[Unit]
Description=Report root disk headroom to Uptime Kuma
[Service]
Type=oneshot
ExecStart=/usr/local/bin/root-disk-check.sh
EOF
sudo tee /etc/systemd/system/root-disk-check.timer >/dev/null <<'EOF'
[Unit]
Description=Run root-disk-check every 5 minutes
[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
AccuracySec=30s
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now root-disk-check.timer"
```
- [ ] **Step 5: Verify the timer is scheduled**
```bash
ssh -o BatchMode=yes ginnoir@valhalla "systemctl list-timers root-disk-check.timer --no-pager"
```
Expected: one row showing `root-disk-check.timer` with a `NEXT` time under 5 minutes away.
- [ ] **Step 6: Verify the alert path actually fires**
Temporarily raise the threshold so the script reports `down`:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo sed -i 's/^MIN_FREE_PCT=15/MIN_FREE_PCT=99/' /usr/local/bin/root-disk-check.sh && sudo /usr/local/bin/root-disk-check.sh"
```
Expected: the Kuma monitor flips to **Down** and an ntfy notification arrives. Then restore it:
```bash
ssh -o BatchMode=yes ginnoir@valhalla "sudo sed -i 's/^MIN_FREE_PCT=99/MIN_FREE_PCT=15/' /usr/local/bin/root-disk-check.sh && sudo /usr/local/bin/root-disk-check.sh"
```
Expected: monitor returns to **Up**. An untested alert is not an alert.
Raised `MIN_FREE_PCT=99` → monitor status 0 (Down) + ntfy
`valhalla root disk Down [Uptime-Kuma]`. Restored to 15 → Up.
---
+93
View File
@@ -0,0 +1,93 @@
# streaming stack — Jellyfin and its satellites.
#
# Storage split is deliberate: SQLite config on the root SSD, all bulk data
# (metadata, trickplay, cache, transcodes) on /storage1. The root disk is only
# 194G and has filled twice; nothing here may grow unbounded on it.
#
# Trickplay gets an explicit sub-bind because Jellyfin 10.11 moved it to
# /config/data/trickplay and it is NOT relocatable from the UI.
#
# Transcodes are disk-backed, not tmpfs: jellyfin#16608 means Remux and
# DirectStream jobs finish before the segment cleaner's first check, orphaning
# segments. Plex already writes transcodes to virtiofs at a 48% transcode rate.
#
# Env convention: every service gets stack.env via env_file; var names there are
# exactly what the containers read. No ${VAR} interpolation, no Portainer UI
# env vars.
services:
jellyfin:
container_name: jellyfin
image: lscr.io/linuxserver/jellyfin:latest
restart: unless-stopped
networks: [streaming, edge]
env_file:
- stack.env
environment:
- JELLYFIN_PublishedServerUrl=https://jellyfin.ginnoir.com
volumes:
- /etc/localtime:/etc/localtime:ro
- /config/jellyfin:/config
- /storage1/labdata/jellyfin/metadata:/config/metadata
- /storage1/labdata/jellyfin/cache:/config/cache
- /storage1/labdata/jellyfin/trickplay:/config/data/trickplay
- /storage1/labdata/jellyfin/transcodes:/transcode
- /storage1/Movies:/data/movies:ro
- /storage1/TV:/data/tv:ro
- /storage1/Anime:/data/anime:ro
ports:
- "8096:8096"
jellystat-db:
container_name: jellystat-db
image: postgres:16-alpine
restart: unless-stopped
networks: [streaming]
env_file:
- stack.env
labels:
- "com.centurylabs.watchtower.enable=false"
volumes:
- /config/jellystat-db:/var/lib/postgresql/data
jellystat:
container_name: jellystat
image: cyfershepard/jellystat:latest
restart: unless-stopped
networks: [streaming, edge]
depends_on:
- jellystat-db
env_file:
- stack.env
volumes:
- /config/jellystat:/app/backend/backup-data
ports:
- "3009:3000"
wizarr:
container_name: wizarr
image: ghcr.io/wizarrrr/wizarr:latest
restart: unless-stopped
networks: [streaming, edge]
env_file:
- stack.env
volumes:
- /config/wizarr:/data/database
ports:
- "5690:5690"
jellyplex-watched:
container_name: jellyplex-watched
image: luigi311/jellyplex-watched:latest
restart: unless-stopped
networks: [streaming, edge]
env_file:
- stack.env
networks:
streaming:
name: streaming
driver: bridge
edge:
name: edge
external: true
+31
View File
@@ -0,0 +1,31 @@
# streaming stack config — injected into every service via env_file.
# Var names are exactly what the containers read; no ${VAR} interpolation,
# no Portainer UI env vars needed.
# --- shared ---
TZ=America/Chicago
PUID=1000
PGID=1000
# --- jellystat-db (postgres:16-alpine) ---
POSTGRES_DB=jfstat
POSTGRES_USER=jellystat
POSTGRES_PASSWORD=bf2d47d8e1a87baf2b4942704a5acfaf5c2aad336fdd480e
# --- jellystat ---
POSTGRES_IP=jellystat-db
POSTGRES_PORT=5432
JWT_SECRET=0dd9fcdb632277b76977dbada8226e547cc773e1deafc8d778d4038f8d59393b
# --- jellyplex-watched ---
# Plex is host-native, not containerised, so it is reached on the LAN IP.
PLEX_URL=http://192.168.1.69:32400
PLEX_TOKEN=hM9aycswHKSsUrxiHqwq
JELLYFIN_BASEURL=http://jellyfin:8096
JELLYFIN_TOKEN=
SYNC_FROM_PLEX_TO_JELLYFIN=True
SYNC_FROM_JELLYFIN_TO_PLEX=True
# Start in dry-run. Task 16 flips this to False after the mapping is verified.
DRYRUN=True
SLEEP_DURATION=21600
LOG_LEVEL=INFO