docs+ops: rewrite CLAUDE.md, skills, runner for post-split topology
Post-monolith documentation and ops cleanup: - CLAUDE.md rewritten end-to-end. Documents the 11 Portainer-managed stacks + raw-compose management plane, the single shared `edge` network, the SSD-vs-ZFS bind-mount tiering, the four deployment channels (git push for app stacks, runner for Caddyfile, apply-compose.ps1 for mgmt plane / Caddyfile / vault), and the fact that the repo is now canonical. - homelab-apply skill rewritten for the new channels — no more `dc up -d`, no more monolith. - homelab-ssh skill rewritten — no `dc` alias, plain `docker` against container names; per-stack compose ops via /data/compose/<id>/... - homelab-sync skill + sync-prod.ps1 retired. The repo is canonical now; pulling from prod is the wrong direction. - .github/workflows/deploy.yml: drop the dc up -d steps, gate on paths:[Caddyfile, .github/workflows/deploy.yml], reload caddy via `docker exec` (no longer through compose). - apply-compose.ps1: drop -Compose and -DevStack flags; -Caddy now reloads via `docker exec caddy` (Caddy is in its own Portainer stack now). No live container is touched by this commit. The runner workflow is currently disabled at the repo level; re-enabling it makes Caddyfile pushes auto-deploy again.
This commit is contained in:
ginnoir
@@ -1,37 +1,67 @@
|
||||
---
|
||||
name: homelab-apply
|
||||
description: Push local config changes (docker-compose.yml, .env, Caddyfile) to the valhalla server and apply them to the live Docker stack. Use after editing homelab config to deploy it. This mutates the running production stack.
|
||||
description: Deploy changes to the valhalla homelab. For application stacks (foundry, media, owncloud, etc.), commit and push — Portainer git-polls every 5 min. For the management plane (portainer, vault, watchtower, runner), Caddyfile, or .env on the server, use the apply-compose.ps1 helper. Use after editing any homelab config to get it deployed.
|
||||
---
|
||||
|
||||
# homelab-apply
|
||||
|
||||
Deploys local repo changes to the live stack on valhalla. **This mutates production** — sync and review your diff first (see the `homelab-sync` skill).
|
||||
The repo is canonical. Different config layers deploy through different channels:
|
||||
|
||||
## Run it
|
||||
| Edit | Deployment channel | Lag |
|
||||
|---|---|---|
|
||||
| `stacks/<domain>/docker-compose.yml` or `stacks/<domain>/.env` | **git push** → Portainer polls every 5 min and redeploys that stack | up to 5 min |
|
||||
| `Caddyfile` | git push triggers `.github/workflows/deploy.yml` → runner pushes file + `docker exec caddy caddy reload` | seconds (if workflow is enabled) |
|
||||
| `portainer-compose.yml` (mgmt plane: portainer, vault, watchtower, runner) | `apply-compose.ps1 -Portainer` — Portainer can't manage itself | seconds |
|
||||
| `vault.hcl` | `apply-compose.ps1 -Portainer` (bundled with mgmt plane push) | seconds |
|
||||
| `.env` at `~/valhalla-lab/.env` (only used by mgmt plane + vault unseal) | `apply-compose.ps1 -EnvFile` | seconds |
|
||||
|
||||
## For application stack edits — just push
|
||||
|
||||
```powershell
|
||||
powershell -File apply-compose.ps1 # push .env + Caddyfile + compose, then apply
|
||||
powershell -File apply-compose.ps1 -EnvFile # .env only
|
||||
powershell -File apply-compose.ps1 -Caddy # Caddyfile only + hot-reload Caddy
|
||||
powershell -File apply-compose.ps1 -Compose # docker-compose.yml only + dc pull + dc up -d
|
||||
git add stacks/media/.env
|
||||
git commit -m "..."
|
||||
git push
|
||||
```
|
||||
|
||||
Flags combine (e.g. `-Caddy -Compose`); no flags pushes & applies all three.
|
||||
|
||||
## What each step does
|
||||
|
||||
- **`.env` / compose** → `scp` to `~/htpc-download-box/`, then `dc pull` (latest images) + `dc up -d` (recreates only containers whose resolved config changed).
|
||||
- **Caddyfile** → `scp` to `/config/caddy/Caddyfile`, then `dc exec caddy caddy reload …` — a hot reload with no downtime. If the new Caddyfile is invalid the reload fails and the old config keeps running; read the output.
|
||||
|
||||
## After applying, verify
|
||||
Portainer's per-stack auto-update will pick up the new commit within 5 minutes and redeploy. Watch progress:
|
||||
|
||||
```powershell
|
||||
ssh ginnoir@valhalla "bash -ic 'dc ps'"
|
||||
ssh ginnoir@valhalla "bash -ic 'dc logs --tail=50 <service>'"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker compose -f /data/compose/<id>/stacks/<domain>/docker-compose.yml ps"
|
||||
```
|
||||
|
||||
Or just `docker ps` and filter by name. To force an immediate redeploy without waiting, trigger it from the Portainer UI ("Update the stack").
|
||||
|
||||
## For Caddyfile edits — push and wait for the runner
|
||||
|
||||
Workflow is gated by `paths: [Caddyfile, .github/workflows/deploy.yml]` so only Caddyfile commits trigger it.
|
||||
|
||||
If you don't want to wait (or the workflow is disabled), the manual fallback is:
|
||||
|
||||
```powershell
|
||||
powershell -File apply-compose.ps1 -Caddy
|
||||
```
|
||||
|
||||
This `scp`s the local Caddyfile to `/config/caddy/Caddyfile` and runs `docker exec caddy caddy reload --config /etc/caddy/Caddyfile`. Invalid Caddyfile → reload fails, old config keeps running; read the output.
|
||||
|
||||
## For management-plane edits
|
||||
|
||||
```powershell
|
||||
powershell -File apply-compose.ps1 -Portainer # portainer-compose.yml + vault.hcl + up -d
|
||||
powershell -File apply-compose.ps1 -EnvFile # ~/valhalla-lab/.env only
|
||||
powershell -File apply-compose.ps1 -VaultUnseal # unseal vault after restart (keys from .env)
|
||||
```
|
||||
|
||||
No flags = `-EnvFile` + `-Caddy` (the two ad-hoc pushes you'd want for a quick manual deploy).
|
||||
|
||||
## Verify after applying
|
||||
|
||||
```powershell
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker ps --format 'table {{.Names}}\t{{.Status}}'"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker logs --tail 50 <container>"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Compose is **v1.27.4** (`docker-compose`, hyphenated); `dc` runs over `bash -ic` to load the alias.
|
||||
- Editing config without applying does nothing — the server only sees pushed files.
|
||||
- Cosmetic/whitespace-only compose edits are no-ops to running containers; `dc up -d` won't recreate them.
|
||||
- Compose is **v2** (`docker compose`, space).
|
||||
- There is no monolithic `dc` alias anymore — each stack has its own compose dir under `/data/compose/<id>/stacks/<domain>/`. Address services by container name with `docker logs`, `docker exec`, etc. — `docker compose ...` only makes sense when you explicitly point at a per-stack compose file.
|
||||
- A **GateGuard hook** blocks the first `Bash` and every `Write`/`Edit` until you state the user request + what the operation does/affects. State them, retry.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: homelab-ssh
|
||||
description: Connect to and run commands on the valhalla homelab server (ginnoir@valhalla) over SSH, including docker-compose operations through the `dc` alias. Use when inspecting the live stack, tailing logs, restarting a service, reloading Caddy, or checking container/file state on the server.
|
||||
description: Connect to and run commands on the valhalla homelab server (ginnoir@valhalla) over SSH. Use when inspecting the live per-domain stacks, tailing logs, restarting a container, reloading Caddy, or checking container/network/volume state. There is no `dc` alias anymore — services are addressed by container name with plain `docker` commands.
|
||||
---
|
||||
|
||||
# homelab-ssh
|
||||
@@ -15,37 +15,73 @@ Always pass `-o BatchMode=yes` so a missing key or password prompt fails fast in
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=8 ginnoir@valhalla "<command>"
|
||||
```
|
||||
|
||||
The first `Bash` command of a session is gated by a GateGuard hook — state the user request + what the command does, then retry.
|
||||
The first `Bash` of a session is gated by a GateGuard hook — state the user request + what the command does, then retry.
|
||||
|
||||
## docker-compose via the `dc` alias
|
||||
## The estate is now 11 Portainer-managed stacks + the management plane
|
||||
|
||||
The stack uses Docker Compose v2 (`docker compose` plugin, installed via apt). On the server, `dc` is aliased in `~/.bashrc` to:
|
||||
| Stack (Portainer project name) | Containers |
|
||||
|---|---|
|
||||
| `proxy` | `caddy` |
|
||||
| `media` | `deluge`, `nzbget`, `sonarr`, `radarr`, `bazarr`, `prowlarr`, `overseerr`, `tautulli`, `qbittorrent`, `whisparr`, `stash` |
|
||||
| `foundry` | `foundry`, `foundry2`, `5etools` |
|
||||
| `owncloud` | `owncloud_server`, `owncloud_mariadb`, `owncloud_redis` |
|
||||
| `resume` | `resume`, `postgres_resume`, `minio_resume`, `chrome` |
|
||||
| `famapp` | `famapp`, `postgres_famapp`, `minio_famapp` |
|
||||
| `authentik` | `authentik_server`, `authentik_worker`, `postgres_authentik`, `redis_authentik` |
|
||||
| `notify` | `ntfy`, `freshrss`, `vigilant` |
|
||||
| `monitoring` | `uptime_kuma`, `homarr` |
|
||||
| `remote` | `hbbr`, `hbbs` |
|
||||
| `dev` | `gitea`, `code_server`, `bookstack`, `mariadb_bookstack`, `dbx`, `registry`, `postgres_gitea`, `postgres_plane`, `redis_plane`, `plane_minio`, `plane_api`, `plane_web`, `plane_worker`, `plane_beat`, `plane_create_bucket` |
|
||||
| management plane (raw compose at `~/valhalla-lab/portainer-compose.yml`, project name `valhalla-lab`) | `portainer`, `vault`, `github_runner`, `watchtower` |
|
||||
|
||||
```
|
||||
docker compose -f ~/htpc-download-box/docker-compose.yml --env-file ~/htpc-download-box/.env
|
||||
```
|
||||
## Common ops by container name
|
||||
|
||||
It's a shell alias, so it only exists in an interactive shell. Invoke it with `bash -ic`:
|
||||
There's no umbrella `dc` alias. Use plain `docker`:
|
||||
|
||||
```powershell
|
||||
ssh ginnoir@valhalla "bash -ic 'dc ps'"
|
||||
ssh ginnoir@valhalla "bash -ic 'dc logs -f --tail=200 <service>'"
|
||||
ssh ginnoir@valhalla "bash -ic 'dc restart <service>'"
|
||||
ssh ginnoir@valhalla "bash -ic 'dc up -d'"
|
||||
ssh ginnoir@valhalla "bash -ic 'dc exec caddy caddy reload --config /etc/caddy/Caddyfile'"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Label \"com.docker.compose.project\"}}'"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker logs --tail 200 -f <container>"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker restart <container>"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker exec <container> <command>"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker exec caddy caddy reload --config /etc/caddy/Caddyfile"
|
||||
```
|
||||
|
||||
Use the compose **service name** (`sonarr`, `caddy`, `app`, …). Without `bash -ic`, `dc` is "command not found".
|
||||
## Per-stack `docker compose` operations
|
||||
|
||||
Each Portainer git stack lives at `/data/compose/<id>/stacks/<domain>/docker-compose.yml` (Portainer chooses the `<id>` when the stack is created). To inspect a stack's resolved config or run a one-off compose op:
|
||||
|
||||
```powershell
|
||||
# Pick the stack file from its label
|
||||
ssh ginnoir@valhalla "docker inspect <container> --format '{{ index .Config.Labels \"com.docker.compose.project.config_files\"}}'"
|
||||
# Then point compose at it
|
||||
ssh ginnoir@valhalla "docker compose -f /data/compose/14/stacks/owncloud/docker-compose.yml ps"
|
||||
```
|
||||
|
||||
For management plane only:
|
||||
|
||||
```powershell
|
||||
ssh ginnoir@valhalla "docker compose -f ~/valhalla-lab/portainer-compose.yml --env-file ~/valhalla-lab/.env ps"
|
||||
```
|
||||
|
||||
## Key paths on the server
|
||||
|
||||
- `~/htpc-download-box/docker-compose.yml`, `~/htpc-download-box/.env` — stack definition
|
||||
- `/config/caddy/Caddyfile` — Caddy config (mounted into the `caddy` container at `/etc/caddy/Caddyfile`)
|
||||
- `/config/<service>/` — per-service persisted config; `/storage1/` — media & data; plus named Docker volumes
|
||||
- `~/htpc-download-box/` contains only `docker-compose.yml` and `.env` — the legacy cruft has been cleaned out
|
||||
- `~/valhalla-lab/portainer-compose.yml` + `~/valhalla-lab/.env` — management plane (raw compose, applied via `apply-compose.ps1 -Portainer`)
|
||||
- `/config/caddy/Caddyfile` — Caddy config (mounted into `caddy` at `/etc/caddy/Caddyfile`)
|
||||
- `/config/<service>/` — per-service persisted config (SSD tier: configs + Postgres + Redis)
|
||||
- `/storage1/labdata/<service>/` — high-density data on the ZFS pool (MinIO buckets, registry layers, gitea repos)
|
||||
- `/storage1/` — media library (Books, Tabletop, etc.)
|
||||
- `/data/compose/<id>/stacks/<domain>/` — Portainer's working copy of each stack (Portainer auto-pulls these from git every 5 min)
|
||||
|
||||
## Networks
|
||||
|
||||
- `edge` — shared reverse-proxy network. `caddy` + every service Caddy proxies. Anything new that needs to be public must join this.
|
||||
- `<domain>` (e.g. `media`, `authentik`, `dev`) — per-stack private network for intra-stack traffic (app ↔ its db/cache/minio).
|
||||
- `portainer_proxy`, `valhalla-lab_default` — management-plane networks (legacy names, kept).
|
||||
- `bridge`, `host`, `none` — Docker defaults.
|
||||
|
||||
## Notes
|
||||
|
||||
- Read-only exploration of `/config`, `/storage1`, and container state is fine for answering questions.
|
||||
- Avoid destructive commands against the live stack without explicit confirmation.
|
||||
- To push config changes and apply them, use the `homelab-apply` skill; to pull prod config into the repo, use `homelab-sync`.
|
||||
- For destructive ops (`docker rm -f`, network/volume pruning), get explicit confirmation first.
|
||||
- To deploy config changes, use the `homelab-apply` skill (per-domain stacks → git push; mgmt plane / Caddyfile → `apply-compose.ps1`).
|
||||
- `htpc-download-box_*` and `valhalla-lab_*` named volumes still exist as a data safety net from the pre-split monolith — every per-domain stack uses bind mounts now, so those orphans can be reaped once you're confident in the new persistence.
|
||||
|
||||
@@ -1,38 +0,0 @@
|
||||
---
|
||||
name: homelab-sync
|
||||
description: Pull the live production configs (docker-compose.yml, .env, Caddyfile) from the valhalla server into this repo so local matches what's deployed. Use at the start of any session that will edit homelab deployment config, or to check for drift between repo and production.
|
||||
---
|
||||
|
||||
# homelab-sync
|
||||
|
||||
Production (valhalla) is the source of truth. Pull before editing so you never edit a stale copy.
|
||||
|
||||
## Run it
|
||||
|
||||
```powershell
|
||||
powershell -File sync-prod.ps1
|
||||
```
|
||||
|
||||
This `scp`s three files from valhalla, **overwriting** the local copies:
|
||||
|
||||
| From valhalla | Into repo |
|
||||
|---------------|-----------|
|
||||
| `~/htpc-download-box/docker-compose.yml` | `docker-compose.yml` |
|
||||
| `~/htpc-download-box/.env` | `.env` |
|
||||
| `/config/caddy/Caddyfile` | `Caddyfile` |
|
||||
|
||||
It overwrites local working copies — commit or stash anything you want to keep first.
|
||||
|
||||
## Check drift without overwriting
|
||||
|
||||
To see how local differs from prod before deciding direction (git is installed):
|
||||
|
||||
```powershell
|
||||
$tmp = Join-Path $env:TEMP 'valhalla-compose.yml'
|
||||
scp -o BatchMode=yes "ginnoir@valhalla:~/htpc-download-box/docker-compose.yml" $tmp
|
||||
git --no-pager diff --no-index -- $tmp .\docker-compose.yml
|
||||
```
|
||||
|
||||
(`.env` and `Caddyfile` can be compared the same way.)
|
||||
|
||||
After syncing, review `git status` / `git diff` to see what changed on the server since your last sync. To push local changes the other direction, use the `homelab-apply` skill.
|
||||
@@ -1,12 +1,21 @@
|
||||
name: Deploy to valhalla
|
||||
|
||||
# Post-split deploy. The application stacks (foundry, media, notify, monitoring,
|
||||
# remote, owncloud, resume, famapp, authentik, dev) live under stacks/<domain>/
|
||||
# and are deployed by Portainer git polling — the runner does NOT touch them.
|
||||
# Caddy lives in its own Portainer git stack (stacks/proxy/) and is likewise
|
||||
# deployed by Portainer; this workflow only pushes the Caddyfile and triggers
|
||||
# a hot reload on the running Caddy container.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- Caddyfile
|
||||
- .github/workflows/deploy.yml
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
|
||||
COMPOSE: docker compose -f /valhalla-lab/docker-compose.yml --env-file /valhalla-lab/.env
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
@@ -15,17 +24,7 @@ jobs:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Push .env + compose
|
||||
run: |
|
||||
cp .env /valhalla-lab/.env
|
||||
cp docker-compose.yml /valhalla-lab/docker-compose.yml
|
||||
|
||||
- name: Pull and apply compose
|
||||
run: |
|
||||
$COMPOSE pull
|
||||
$COMPOSE up -d
|
||||
|
||||
- name: Reload Caddyfile
|
||||
- name: Push Caddyfile and reload Caddy
|
||||
run: |
|
||||
cp Caddyfile /config/caddy/Caddyfile
|
||||
$COMPOSE exec -T caddy caddy reload --config /etc/caddy/Caddyfile
|
||||
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
|
||||
|
||||
@@ -4,92 +4,116 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
## What this repo is
|
||||
|
||||
Deployment configuration for a **live, single-host Docker homelab** running on a headless Ubuntu server reachable at `ssh ginnoir@valhalla`. This repo is **not application code** — it is a version-controlled mirror of three files that live on the server. Editing here changes nothing until the files are pushed to valhalla and the stack is re-applied.
|
||||
Deployment configuration for a **live, single-host Docker homelab** running on a headless Ubuntu server reachable at `ssh ginnoir@valhalla`. This repo is **not application code** — it is the canonical source for ~50 containers organized into per-domain Portainer-managed stacks.
|
||||
|
||||
Tracked files ↔ where they live on valhalla:
|
||||
**The repo is canonical.** Portainer polls this repo's `main` branch every 5 min and redeploys any application stack whose source files have changed. Editing here changes nothing until you push to GitHub (or, for the management plane, run `apply-compose.ps1 -Portainer`).
|
||||
|
||||
| Repo file | On valhalla | Consumed by |
|
||||
|-----------|-------------|-------------|
|
||||
| `docker-compose.yml` | `~/htpc-download-box/docker-compose.yml` | the whole stack |
|
||||
| `.env` | `~/htpc-download-box/.env` | compose variable substitution |
|
||||
| `Caddyfile` | `/config/caddy/Caddyfile` | the `caddy` container, mounted at `/etc/caddy/Caddyfile` |
|
||||
`.env` and `stacks/*/.env` are committed **intentionally** — the GitHub repo `ginnoir/homelabstack` is private and the stack's secrets are versioned with it. Do not scrub or gitignore them.
|
||||
|
||||
`.env` is committed **intentionally** — the GitHub repo `ginnoir/homelabstack` is private and the stack's secrets are versioned with it. Do not scrub or gitignore it.
|
||||
## Layout
|
||||
|
||||
The server's `~/htpc-download-box/` contains only `docker-compose.yml` and `.env` — the legacy cruft from the original *htpc-download-box* project has been cleaned out.
|
||||
```
|
||||
.
|
||||
├── stacks/ # one directory per Portainer-managed application stack
|
||||
│ ├── proxy/ # caddy
|
||||
│ ├── media/ # *arr stack, deluge, qbittorrent, stash, ...
|
||||
│ ├── foundry/ # foundry, foundry2, 5etools
|
||||
│ ├── owncloud/ # owncloud + its mariadb + redis
|
||||
│ ├── resume/ # reactive-resume + its postgres + minio + chrome
|
||||
│ ├── famapp/ # famapp + its postgres + minio
|
||||
│ ├── authentik/ # authentik server/worker + its postgres + redis
|
||||
│ ├── notify/ # ntfy, freshrss, vigilant
|
||||
│ ├── monitoring/ # uptime-kuma, homarr
|
||||
│ ├── remote/ # rustdesk hbbr + hbbs
|
||||
│ └── dev/ # gitea, code-server, registry, bookstack, dbx, plane
|
||||
├── portainer-compose.yml # management plane (raw compose, not a Portainer stack)
|
||||
├── vault.hcl # vault config (deployed alongside portainer-compose.yml)
|
||||
├── Caddyfile # caddy config (deployed via runner workflow or -Caddy)
|
||||
├── .env # mgmt-plane env + vault unseal keys + global vars
|
||||
├── apply-compose.ps1 # manual ops helper (mgmt plane + Caddyfile + vault unseal)
|
||||
└── .github/workflows/deploy.yml # runner: pushes Caddyfile + reloads caddy on push
|
||||
```
|
||||
|
||||
## The core workflow
|
||||
## Deployment channels — what triggers what
|
||||
|
||||
Production is the source of truth. **Sync before editing, apply after editing:**
|
||||
| Edit | Deploys via | Latency |
|
||||
|---|---|---|
|
||||
| `stacks/<domain>/*` | **git push** → Portainer polls every 5 min and redeploys that one stack | ≤ 5 min |
|
||||
| `Caddyfile` | git push → runner workflow `scp`s to `/config/caddy/Caddyfile` + `docker exec caddy caddy reload` | seconds (when workflow is enabled) |
|
||||
| `portainer-compose.yml` / `vault.hcl` | `apply-compose.ps1 -Portainer` (Portainer can't manage itself) | seconds |
|
||||
| `~/valhalla-lab/.env` (mgmt-plane env, vault unseal keys) | `apply-compose.ps1 -EnvFile` | seconds |
|
||||
|
||||
The runner workflow gate-filters on `paths: [Caddyfile, .github/workflows/deploy.yml]` so non-Caddyfile pushes don't trigger it.
|
||||
|
||||
## On-host topology
|
||||
|
||||
| Path on valhalla | Purpose |
|
||||
|---|---|
|
||||
| `~/valhalla-lab/portainer-compose.yml` + `.env` | management plane (raw compose; project name `valhalla-lab`) |
|
||||
| `/data/compose/<id>/stacks/<domain>/` | Portainer's working copy of each app stack (auto-pulled from git) |
|
||||
| `/config/<service>/` | SSD-tier persistence: app configs, Postgres data, Redis data |
|
||||
| `/storage1/labdata/<service>/` | ZFS-tier persistence: MinIO blob buckets, registry image layers, gitea repos/LFS |
|
||||
| `/storage1/` | media library (Books, Tabletop, plex-style media) |
|
||||
| `/config/caddy/Caddyfile` | live Caddy config (deployed from this repo) |
|
||||
| `/config/vault/` | Vault config + sealed data + audit logs |
|
||||
|
||||
The pre-split monolith's named volumes (`htpc-download-box_*` and `valhalla-lab_*`) are still on the host as a safety net — every app now uses bind mounts. Reap the orphans once you trust the new persistence.
|
||||
|
||||
## Networks
|
||||
|
||||
Single shared edge network, not Caddy-on-every-network:
|
||||
|
||||
- `edge` — the **only** reverse-proxy network. Caddy + everything Caddy proxies. A new public service must join this.
|
||||
- `<domain>` (e.g. `media`, `authentik`, `dev`) — per-stack private network for intra-stack traffic (app ↔ its db/cache/minio).
|
||||
- `portainer_proxy`, `valhalla-lab_default` — mgmt plane (`valhalla-lab_default` is auto-named after the project dir; it's mis-named but legitimate).
|
||||
- `bridge`, `host`, `none` — Docker defaults.
|
||||
|
||||
`reverse_proxy` upstreams use container name + container-internal port (e.g. `qbittorrent:3232`, `stash:6970`), resolving over `edge`.
|
||||
|
||||
## Common ops on valhalla
|
||||
|
||||
There is **no `dc` alias** anymore. Each stack has its own compose dir. Use plain `docker` against container names:
|
||||
|
||||
```powershell
|
||||
powershell -File sync-prod.ps1 # pull prod -> repo (OVERWRITES local copies)
|
||||
# ...edit docker-compose.yml / Caddyfile / .env...
|
||||
powershell -File apply-compose.ps1 # push repo -> prod and apply (mutates the LIVE stack)
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Label \"com.docker.compose.project\"}}'"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker logs --tail 200 -f <container>"
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker exec caddy caddy reload --config /etc/caddy/Caddyfile"
|
||||
```
|
||||
|
||||
`apply-compose.ps1` flags (combine freely; no flags = all three):
|
||||
|
||||
- `-EnvFile` — push `.env` only
|
||||
- `-Caddy` — push `Caddyfile`, then hot-reload Caddy (no downtime)
|
||||
- `-Compose` — push `docker-compose.yml`, then `dc pull` + `dc up -d`
|
||||
|
||||
Caddy changes hot-reload; on `dc up -d` only containers whose resolved config changed are recreated. Cosmetic/whitespace-only compose edits are therefore no-ops to the running containers.
|
||||
|
||||
## Running commands on valhalla
|
||||
|
||||
SSH is key-based and works from this Windows host directly. In automation, pass `-o BatchMode=yes` so it fails fast instead of hanging on a prompt.
|
||||
|
||||
All compose operations go through the **`dc` alias** in the server's `~/.bashrc`:
|
||||
|
||||
```
|
||||
dc = docker compose -f ~/htpc-download-box/docker-compose.yml --env-file ~/htpc-download-box/.env
|
||||
```
|
||||
|
||||
It's a shell alias, so it only exists in an **interactive** shell — invoke it over SSH with `bash -ic`:
|
||||
For per-stack compose ops, pick the file from the container's label:
|
||||
|
||||
```powershell
|
||||
ssh ginnoir@valhalla "bash -ic 'dc ps'"
|
||||
ssh ginnoir@valhalla "bash -ic 'dc logs -f --tail=100 sonarr'"
|
||||
ssh ginnoir@valhalla "bash -ic 'dc restart caddy'"
|
||||
ssh ginnoir@valhalla "bash -ic 'dc exec caddy caddy reload --config /etc/caddy/Caddyfile'"
|
||||
ssh ginnoir@valhalla "docker inspect <container> --format '{{ index .Config.Labels \"com.docker.compose.project.config_files\"}}'"
|
||||
ssh ginnoir@valhalla "docker compose -f /data/compose/14/stacks/owncloud/docker-compose.yml ps"
|
||||
```
|
||||
|
||||
Address services by their **compose service name** (`sonarr`, `caddy`, `app`, …) — this works regardless of how the container is named.
|
||||
## Adding a service
|
||||
|
||||
## docker-compose.yml — read before editing
|
||||
|
||||
- **Compose v2** (`docker compose`, space), installed as the `docker-compose-plugin` apt package and kept current via apt. Project name is `htpc-download-box` (from the directory — inconsequential since every service now has an explicit `container_name:`).
|
||||
- **Container names:** every service has an explicit `container_name:`. Disambiguated duplicates use `type_context` naming: `postgres_resume`, `postgres_famapp`, `postgres_authentik`, `minio_resume`, `minio_famapp`, `redis_authentik`. Always use the compose **service name** with `dc` commands — container names are just for `docker ps` readability.
|
||||
- **`${ROOT}`** (`.env`, default `/`) prefixes some media-stack volume mounts; other services hardcode `/config` and `/storage1`. Both forms resolve to the same real paths.
|
||||
- **Networks segment the stack:** `foundry`, `media`, `media_external`, `owncloud`, `resume`, `famapp`, `authentik`. `caddy` is the only container on all of them — a new service that Caddy must proxy has to share a network with `caddy`.
|
||||
- **Persistence lives on the server, not in this repo:** container state is in `/config/<service>` bind mounts, under `/storage1`, or in named volumes (`postgres_data`, `minio_data`, `authentik_db_data`, `garden_uploads`, …).
|
||||
1. Pick the stack it belongs to under `stacks/<domain>/` (or create a new stack and a new Portainer git stack for it).
|
||||
2. Add the service to `stacks/<domain>/docker-compose.yml`. Configs → `/config/<svc>` bind. Blobs → `/storage1/labdata/<svc>` bind. Add `edge` to its `networks:` if Caddy must reach it; otherwise just the per-stack private net.
|
||||
3. Add secrets to `stacks/<domain>/.env`.
|
||||
4. Add a site block to `Caddyfile` (`reverse_proxy <container_name>:<port>`); `import internal_only` for LAN-only.
|
||||
5. `git push`. Portainer redeploys the stack within 5 min; the runner reloads Caddy on the same push if `Caddyfile` changed.
|
||||
|
||||
## Caddyfile
|
||||
|
||||
- TLS uses the Cloudflare DNS-01 challenge (`acme_dns cloudflare {env.CF_API_TOKEN}`), so certs need no inbound ports. Most sites are `<name>.ginnoir.com → reverse_proxy <service>:<port>`.
|
||||
- The `(internal_only)` snippet 403s any client outside `192.168.1.0/24`. `import internal_only` is how admin UIs (sonarr, qbittorrent, …) stay LAN-only while public sites omit it.
|
||||
- Reverse-proxy upstreams use the compose **service name + container-internal port** (e.g. `qbittorrent:3232`, `stash:6970`), not the host-published port.
|
||||
|
||||
## Adding a service (the common change)
|
||||
|
||||
1. Add the service to `docker-compose.yml`; put it on a network `caddy` is also on if it needs proxying; bind config to `/config/<service>`.
|
||||
2. Add any secrets/vars to `.env`.
|
||||
3. Add a site block to `Caddyfile` (`reverse_proxy <service>:<port>`); add `import internal_only` for LAN-only access.
|
||||
4. Run `apply-compose.ps1` (or `-Compose -Caddy`), then verify with `dc ps` / `dc logs <service>`.
|
||||
- TLS uses the Cloudflare DNS-01 challenge (`acme_dns cloudflare {env.CF_API_TOKEN}`), so certs need no inbound ports.
|
||||
- The `(internal_only)` snippet 403s any client outside `192.168.1.0/24`. `import internal_only` is how admin UIs (sonarr, qbittorrent, ...) stay LAN-only while public sites omit it.
|
||||
- Upstreams use container name + container-internal port (e.g. `qbittorrent:3232`), not host-published ports.
|
||||
|
||||
## Known quirks / gotchas
|
||||
|
||||
- A **GateGuard hook blocks the first use of `Bash`, and every `Write`/`Edit`, until you state the required facts** (the user request + what the operation does/affects). State them, then retry the same call.
|
||||
- A **GateGuard hook** blocks the first use of `Bash`, and every `Write`/`Edit`, until you state the required facts (the user request + what the operation does/affects). State them, then retry the same call.
|
||||
- `Caddyfile` proxies `matrix.ginnoir.com → localhost:8008`, but there is **no Matrix/Synapse service in compose** — it's external/legacy. Likewise `dev.ginnoir.com → 192.168.1.74:3000` points at a different LAN host.
|
||||
- Reactive Resume's `app` reads `${RESUME_AUTH_SECRET}`, which is **not defined in `.env`** (resolves empty). `.env` also defines `AUTH_SECRET` three times — last definition wins (famapp's).
|
||||
- `watchtower` auto-updates `:latest` images, so a running image can drift ahead of what the last `apply` pulled.
|
||||
- Line endings: a `.gitattributes` forces **LF** so files stay Unix-clean. Pushing CRLF (especially in `.env`) to the Linux host would append stray `\r` to values and break things.
|
||||
- Line endings: `.gitattributes` forces **LF** so files stay Unix-clean. Pushing CRLF (especially `.env`) to the Linux host appends stray `\r` to values and breaks things.
|
||||
- The runner workflow (`.github/workflows/deploy.yml`) may be **disabled** at the GitHub repo level — check `gh workflow list` if Caddyfile pushes don't trigger a reload. Manual fallback: `apply-compose.ps1 -Caddy`.
|
||||
|
||||
## Skills
|
||||
|
||||
Project skills in `.claude/skills/` wrap the recurring operations:
|
||||
Project skills in `.claude/skills/` wrap the recurring ops:
|
||||
|
||||
- `homelab-sync` — pull prod configs into the repo
|
||||
- `homelab-apply` — push config + apply to the live stack
|
||||
- `homelab-ssh` — run commands / inspect state on valhalla
|
||||
- `homelab-apply` — deploy changes (git push for app stacks; `apply-compose.ps1` for mgmt plane / Caddyfile / vault)
|
||||
- `homelab-ssh` — inspect and operate the live containers from SSH
|
||||
|
||||
The pre-split `homelab-sync` skill (pull live config into repo) has been retired — the repo is now canonical, not a mirror.
|
||||
|
||||
+17
-43
@@ -1,25 +1,22 @@
|
||||
# Push .env, Caddyfile, and docker-compose.yml to valhalla, then apply them
|
||||
# to the live Docker stack.
|
||||
# Manual ops helper for the management plane + Caddyfile + Vault.
|
||||
#
|
||||
# On valhalla, `dc` is aliased (in ~/.bashrc) to:
|
||||
# docker compose -f ~/valhalla-lab/docker-compose.yml --env-file ~/valhalla-lab/.env
|
||||
# It's a shell alias, so we invoke it over SSH with `bash -ic` to load it.
|
||||
# Post-split topology: every application stack lives under stacks/<domain>/ and
|
||||
# is deployed by Portainer's git poll — there is no monolithic root compose any
|
||||
# more, so this script no longer pushes one. The runner workflow (deploy.yml)
|
||||
# handles Caddyfile reload on push automatically; -Caddy here is for ad-hoc
|
||||
# pushes when you don't want to wait for the runner.
|
||||
#
|
||||
# Usage (flags combine; no flags = push & apply all three):
|
||||
# powershell -File apply-compose.ps1 # .env + Caddyfile + compose
|
||||
# powershell -File apply-compose.ps1 -EnvFile # .env only
|
||||
# powershell -File apply-compose.ps1 -Caddy # Caddyfile only + hot-reload Caddy
|
||||
# powershell -File apply-compose.ps1 -Compose # docker-compose.yml only + pull + up -d
|
||||
# powershell -File apply-compose.ps1 -Portainer # portainer-compose.yml only + up -d
|
||||
# powershell -File apply-compose.ps1 -DevStack # dev-compose.yml + vault.hcl + up -d
|
||||
# powershell -File apply-compose.ps1 -VaultUnseal # unseal vault after restart (uses keys from .env)
|
||||
# Usage (flags combine; no flags = .env + Caddyfile):
|
||||
# powershell -File apply-compose.ps1 # .env + Caddyfile
|
||||
# powershell -File apply-compose.ps1 -EnvFile # .env only (mgmt plane / vault unseal)
|
||||
# powershell -File apply-compose.ps1 -Caddy # Caddyfile only + hot-reload Caddy
|
||||
# powershell -File apply-compose.ps1 -Portainer # portainer-compose.yml + vault.hcl + up -d
|
||||
# powershell -File apply-compose.ps1 -VaultUnseal # unseal vault after restart (keys from .env)
|
||||
|
||||
param(
|
||||
[switch]$Compose,
|
||||
[switch]$Caddy,
|
||||
[switch]$EnvFile,
|
||||
[switch]$Portainer,
|
||||
[switch]$DevStack,
|
||||
[switch]$VaultUnseal
|
||||
)
|
||||
|
||||
@@ -28,16 +25,14 @@ $ErrorActionPreference = "Stop"
|
||||
$PSNativeCommandUseErrorActionPreference = $true
|
||||
|
||||
$server = "ginnoir@valhalla"
|
||||
$composeLocal = Join-Path $PSScriptRoot "docker-compose.yml"
|
||||
$caddyLocal = Join-Path $PSScriptRoot "Caddyfile"
|
||||
$envLocal = Join-Path $PSScriptRoot ".env"
|
||||
$portainerLocal = Join-Path $PSScriptRoot "portainer-compose.yml"
|
||||
$devComposeLocal = Join-Path $PSScriptRoot "dev-compose.yml"
|
||||
$vaultConfigLocal = Join-Path $PSScriptRoot "vault.hcl"
|
||||
|
||||
# Default (no flags): push everything.
|
||||
if (-not $Compose -and -not $Caddy -and -not $EnvFile -and -not $Portainer -and -not $DevStack -and -not $VaultUnseal) {
|
||||
$EnvFile = $true; $Caddy = $true; $Compose = $true
|
||||
# Default (no flags): push .env + Caddyfile.
|
||||
if (-not $Caddy -and -not $EnvFile -and -not $Portainer -and -not $VaultUnseal) {
|
||||
$EnvFile = $true; $Caddy = $true
|
||||
}
|
||||
|
||||
if ($EnvFile) {
|
||||
@@ -49,40 +44,19 @@ if ($Caddy) {
|
||||
Write-Host "Pushing Caddyfile ..."
|
||||
scp $caddyLocal "${server}:/config/caddy/Caddyfile"
|
||||
Write-Host "Reloading Caddy ..."
|
||||
ssh $server "bash -ic 'dc exec caddy caddy reload --config /etc/caddy/Caddyfile'"
|
||||
}
|
||||
|
||||
if ($Compose) {
|
||||
Write-Host "Pushing docker-compose.yml ..."
|
||||
scp $composeLocal "${server}:~/valhalla-lab/docker-compose.yml"
|
||||
Write-Host "Pulling latest images ..."
|
||||
ssh $server "bash -ic 'dc pull'"
|
||||
Write-Host "Applying compose changes (recreates only changed containers) ..."
|
||||
ssh $server "bash -ic 'dc up -d'"
|
||||
ssh $server "docker exec caddy caddy reload --config /etc/caddy/Caddyfile"
|
||||
}
|
||||
|
||||
if ($Portainer) {
|
||||
Write-Host "Pushing portainer-compose.yml ..."
|
||||
scp $portainerLocal "${server}:~/valhalla-lab/portainer-compose.yml"
|
||||
Write-Host "Pushing vault.hcl (Vault now lives in the management plane) ..."
|
||||
Write-Host "Pushing vault.hcl (Vault lives in the management plane) ..."
|
||||
ssh $server "mkdir -p /config/vault/config /config/vault/data /config/vault/logs"
|
||||
scp $vaultConfigLocal "${server}:/config/vault/config/vault.hcl"
|
||||
Write-Host "Applying management plane (portainer, vault, github-runner, watchtower) ..."
|
||||
ssh $server "docker compose -f ~/valhalla-lab/portainer-compose.yml --env-file ~/valhalla-lab/.env up -d"
|
||||
}
|
||||
|
||||
if ($DevStack) {
|
||||
Write-Host "Pushing dev-compose.yml ..."
|
||||
scp $devComposeLocal "${server}:~/valhalla-lab/dev-compose.yml"
|
||||
Write-Host "Pushing vault.hcl ..."
|
||||
ssh $server "mkdir -p /config/vault/config /config/vault/data /config/vault/logs"
|
||||
scp $vaultConfigLocal "${server}:/config/vault/config/vault.hcl"
|
||||
Write-Host "Pulling latest dev images ..."
|
||||
ssh $server "docker compose -f ~/valhalla-lab/dev-compose.yml --env-file ~/valhalla-lab/.env pull"
|
||||
Write-Host "Applying dev stack (recreates only changed containers) ..."
|
||||
ssh $server "docker compose -f ~/valhalla-lab/dev-compose.yml --env-file ~/valhalla-lab/.env up -d"
|
||||
}
|
||||
|
||||
if ($VaultUnseal) {
|
||||
# Load the three unseal keys from .env
|
||||
$envContent = Get-Content $envLocal | Where-Object { $_ -match "^VAULT_UNSEAL_KEY_[123]=" }
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
# Pull the live production configs from valhalla into this repo, so local
|
||||
# matches what's actually deployed. Run at the start of any session where
|
||||
# you'll edit deployment config.
|
||||
#
|
||||
# Production is the source of truth. This OVERWRITES the local copies of
|
||||
# docker-compose.yml, .env, and Caddyfile -- commit or stash first.
|
||||
#
|
||||
# Usage: powershell -File sync-prod.ps1
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$PSNativeCommandUseErrorActionPreference = $true
|
||||
|
||||
$server = "ginnoir@valhalla"
|
||||
$repoDir = $PSScriptRoot
|
||||
|
||||
Write-Host "Syncing production configs from $server ..."
|
||||
|
||||
scp "${server}:~/valhalla-lab/docker-compose.yml" "$repoDir\docker-compose.yml"
|
||||
scp "${server}:~/valhalla-lab/.env" "$repoDir\.env"
|
||||
scp "${server}:/config/caddy/Caddyfile" "$repoDir\Caddyfile"
|
||||
|
||||
Write-Host "Done. Files synced:"
|
||||
Get-ChildItem $repoDir -File |
|
||||
Where-Object { $_.Name -in 'docker-compose.yml', '.env', 'Caddyfile' } |
|
||||
Select-Object Name, Length, LastWriteTime
|
||||
Reference in New Issue
Block a user