diff --git a/docs/superpowers/plans/2026-06-15-gitea-portainer-registry-cutover.md b/docs/superpowers/plans/2026-06-15-gitea-portainer-registry-cutover.md new file mode 100644 index 0000000..b180bb7 --- /dev/null +++ b/docs/superpowers/plans/2026-06-15-gitea-portainer-registry-cutover.md @@ -0,0 +1,1700 @@ +# Gitea Portainer Registry Cutover Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Move homelab deployment source control from GitHub to local Gitea, keep GitHub as a temporary rollback mirror, move Gitea and the Docker registry into the management plane, publish famapp images to `registry.ginnoir.com`, and recreate Portainer stacks with `stack.env` as the only app-stack environment source. + +**Architecture:** The management plane becomes the dependency root for Portainer, Vault, Gitea, Gitea runner, Docker registry, registry UI, and Watchtower. Portainer app stacks are recreated to poll `https://gitea.ginnoir.com/ginnoir/homelabstack.git`, with GitHub retained only as a temporary push mirror. Famapp release automation moves to Gitea Actions and publishes images to the self-hosted Docker registry. + +**Tech Stack:** Docker Compose, Portainer CE Git stacks, Gitea, Gitea Actions `act_runner`, Docker registry v2, Caddy, PowerShell, SSH to `ginnoir@valhalla`, famapp Next.js Docker image. + +--- + +## File Structure + +Homelab repo files: + +- Modify `portainer-compose.yml` + - Add Gitea, `postgres_gitea`, `gitea_runner`, `registry`, and `registry_ui` to the raw management compose file. + - Add registry auth bind mounts and environment. + - Keep `portainer`, `vault`, and `watchtower` in place. + - Remove `github-runner` only after Gitea Actions and Caddy reloads are proven. +- Modify `.env` + - Add management-plane registry credential variables and Gitea runner token variables. + - Retain GitHub token variables during the staged mirror period. +- Modify `stacks/dev/docker-compose.yml` + - Remove Gitea, `postgres_gitea`, `gitea_runner`, `registry`, and `registry_ui`. + - Convert remaining services away from `${VAR}` interpolation by storing container-native variables directly in `stacks/dev/stack.env`. +- Modify `stacks/dev/stack.env` + - Preserve existing secret values while adding the names that containers actually read: `APP_KEY`, `DB_PASS`, `MYSQL_ROOT_PASSWORD`, `MYSQL_PASSWORD`, `SECRET_KEY`, `DATABASE_URL`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `MINIO_USER`, `MINIO_PASSWORD`, `POSTGRES_PASSWORD`, `MINIO_ROOT_USER`, and `MINIO_ROOT_PASSWORD`. + - Remove Gitea-only variables after Gitea is moved to `.env`. +- Modify `stacks/resume/docker-compose.yml` + - Remove MinIO OIDC `${VAR}` interpolation by making `MINIO_IDENTITY_OPENID_CLIENT_ID` and `MINIO_IDENTITY_OPENID_CLIENT_SECRET` direct `stack.env` keys. +- Modify `stacks/resume/stack.env` + - Add `MINIO_IDENTITY_OPENID_CLIENT_ID` with the current `MINIO_OIDC_CLIENT_ID` value. + - Add `MINIO_IDENTITY_OPENID_CLIENT_SECRET` with the current `MINIO_OIDC_CLIENT_SECRET` value. +- Modify `stacks/famapp/docker-compose.yml` + - Change `famapp` image from `ghcr.io/ginnoir/famapp:latest` to `registry.ginnoir.com/ginnoir/famapp:latest`. + - Keep `pull_policy: always` during the staged cutover. +- Modify `stacks/proxy/docker-compose.yml` + - Update comments from GitHub runner reloads to Gitea Actions reloads after the workflow is proven. +- Modify `Caddyfile` + - Keep `registry.ginnoir.com` internal-only. + - Keep `gitea.ginnoir.com` Git/API bypasses. + - Adjust upstreams only if service names or networks change. +- Modify `CLAUDE.md` and `AGENTS.md` + - Replace GitHub-primary deploy language with Gitea-primary, GitHub-temporary-mirror language. + - Record that Portainer app stacks must have no Portainer UI env vars. +- Create `scripts/check-portainer-stack-env.ps1` + - Fail if any non-comment line in `stacks/*/docker-compose.yml` contains `${`. + - Fail if a service that requires `env_file: stack.env` lacks it. + - Allow explicit, reviewed exemptions in `scripts/portainer-stack-env-exemptions.json`. +- Create `scripts/portainer-stack-env-exemptions.json` + - List services that intentionally need no `stack.env`. +- Create `scripts/new-registry-auth.ps1` + - Generate registry push/pull credentials and a bcrypt `htpasswd` file without printing passwords to the terminal after creation. + +Famapp repo files: + +- Modify `.github/workflows/ci.yml` only if GitHub CI remains as temporary mirror validation. +- Create `.gitea/workflows/ci.yml` + - Mirror current CI checks in Gitea Actions. +- Create `.gitea/workflows/release.yml` + - Build and push `registry.ginnoir.com/ginnoir/famapp`. +- Modify `.release-it.json` + - Disable GitHub release creation or move release publication to Gitea-compatible tagging only. +- Modify `README.md`, `docs/RUNBOOK.md`, `docs/ENV.md`, `deploy/README.md`, and `deploy/compose.example.yaml` + - Replace GHCR image references with `registry.ginnoir.com/ginnoir/famapp`. + - Replace GitHub-primary clone/release text with Gitea-primary text. +- Verify `.git/config` + - Confirm there are no `manta-cyber` or `RTSOK` remotes. + - Leave GitHub `origin` only as a temporary mirror until final cleanup. + +Live system targets: + +- `ginnoir@valhalla` +- `~/valhalla-lab/portainer-compose.yml` +- `~/valhalla-lab/.env` +- `/config/registry/auth/htpasswd` +- `/config/portainer` +- `/storage1/labdata/registry` +- `/config/gitea` +- `/storage1/labdata/gitea/repositories` +- `/storage1/labdata/gitea/lfs` + +## Task 1: Baseline Live State and Protect Against Cross-Repo Drift + +**Files:** +- Read: `portainer-compose.yml` +- Read: `stacks/dev/docker-compose.yml` +- Read: `stacks/famapp/docker-compose.yml` +- Read: `C:\Users\MattC\Documents\famapp\.git\config` + +- [ ] **Step 1: Confirm homelab repo state** + +Run: + +```powershell +git -C C:\Users\MattC\Documents\homelabstack status --short --branch +git -C C:\Users\MattC\Documents\homelabstack remote -v +``` + +Expected: + +```text +## main...origin/main +gitea https://gitea.ginnoir.com/ginnoir/homelabstack.git (fetch) +gitea https://gitea.ginnoir.com/ginnoir/homelabstack.git (push) +origin https://github.com/ginnoir/homelabstack.git (fetch) +origin https://github.com/ginnoir/homelabstack.git (push) +origin https://gitea.ginnoir.com/ginnoir/homelabstack.git (push) +``` + +- [ ] **Step 2: Confirm famapp has no work-repo remote** + +Run: + +```powershell +git -C C:\Users\MattC\Documents\famapp remote -v +Select-String -Path C:\Users\MattC\Documents\famapp\.git\config -Pattern 'manta-cyber|RTSOK' +``` + +Expected: + +```text +gitea https://gitea.ginnoir.com/ginnoir/famapp.git (fetch) +gitea https://gitea.ginnoir.com/ginnoir/famapp.git (push) +origin https://github.com/ginnoir/famapp.git (fetch) +origin https://github.com/ginnoir/famapp.git (push) +origin https://gitea.ginnoir.com/ginnoir/famapp.git (push) +``` + +`Select-String` should return no matches. + +- [ ] **Step 3: Confirm live dependency containers** + +Run: + +```powershell +ssh -o BatchMode=yes -o ConnectTimeout=8 ginnoir@valhalla "docker ps --format '{{.Names}} {{.Image}} {{.Status}}' | egrep '^(portainer|gitea|postgres_gitea|gitea_runner|registry|registry_ui|github_runner|caddy|famapp) '" +``` + +Expected: + +```text +portainer ... Up ... +gitea ... Up ... +postgres_gitea ... Up ... +gitea_runner ... Up ... +registry ... Up ... +registry_ui ... Up ... +github_runner ... Up ... +caddy ... Up ... +famapp ... Up ... +``` + +- [ ] **Step 4: Confirm registry and Gitea endpoints** + +Run: + +```powershell +ssh -o BatchMode=yes -o ConnectTimeout=8 ginnoir@valhalla "curl -skI https://registry.ginnoir.com/v2/ | head -5; curl -skI https://gitea.ginnoir.com/ | head -5" +``` + +Expected before registry auth: + +```text +HTTP/2 200 +... +HTTP/2 200 +``` + +Expected after registry auth in later tasks: + +```text +HTTP/2 401 +www-authenticate: Basic realm="Valhalla Registry" +... +HTTP/2 200 +``` + +- [ ] **Step 5: Commit baseline-free state** + +No commit is needed in this task. If any command shows unexpected dirty files, stop and identify whether they are user changes before continuing. + +## Task 2: Add Stack Env Guardrails + +**Files:** +- Create: `scripts/check-portainer-stack-env.ps1` +- Create: `scripts/portainer-stack-env-exemptions.json` + +- [ ] **Step 1: Create the exemption file** + +Create `scripts/portainer-stack-env-exemptions.json`: + +```json +{ + "authentik": ["authentik-redis"], + "dev": ["dbx", "redis_plane"], + "foundry": ["5etools"], + "monitoring": ["uptime-kuma"], + "owncloud": ["redis"] +} +``` + +Do not include `registry` or `registry_ui` here because they will move to the management plane. + +- [ ] **Step 2: Create the guard script** + +Create `scripts/check-portainer-stack-env.ps1`: + +```powershell +[CmdletBinding()] +param( + [string]$StacksRoot = (Join-Path $PSScriptRoot "..\stacks"), + [string]$ExemptionsPath = (Join-Path $PSScriptRoot "portainer-stack-env-exemptions.json") +) + +$ErrorActionPreference = "Stop" +$failures = New-Object System.Collections.Generic.List[string] +$exemptions = @{} + +if (Test-Path $ExemptionsPath) { + $raw = Get-Content -Raw -LiteralPath $ExemptionsPath | ConvertFrom-Json + foreach ($property in $raw.PSObject.Properties) { + $exemptions[$property.Name] = @($property.Value) + } +} + +function Get-ServiceBlocks { + param([string[]]$Lines) + + $inServices = $false + $currentName = $null + $currentLines = New-Object System.Collections.Generic.List[string] + $blocks = New-Object System.Collections.Generic.List[object] + + foreach ($line in $Lines) { + if ($line -match '^services:\s*$') { + $inServices = $true + continue + } + + if (-not $inServices) { + continue + } + + if ($line -match '^[A-Za-z0-9_-]+:\s*$') { + break + } + + if ($line -match '^ ([A-Za-z0-9_-]+):\s*$') { + if ($null -ne $currentName) { + $blocks.Add([pscustomobject]@{ + Name = $currentName + Lines = @($currentLines) + }) + } + $currentName = $Matches[1] + $currentLines = New-Object System.Collections.Generic.List[string] + continue + } + + if ($null -ne $currentName) { + $currentLines.Add($line) + } + } + + if ($null -ne $currentName) { + $blocks.Add([pscustomobject]@{ + Name = $currentName + Lines = @($currentLines) + }) + } + + return $blocks +} + +Get-ChildItem -LiteralPath $StacksRoot -Directory | Sort-Object Name | ForEach-Object { + $stackName = $_.Name + $composePath = Join-Path $_.FullName "docker-compose.yml" + $stackEnvPath = Join-Path $_.FullName "stack.env" + + if (-not (Test-Path $composePath)) { + return + } + + if (-not (Test-Path $stackEnvPath)) { + $failures.Add("${stackName}: missing stack.env") + } + + $lines = Get-Content -LiteralPath $composePath + $interpolations = Select-String -LiteralPath $composePath -Pattern '\$\{' | + Where-Object { $_.Line -notmatch '^\s*#' } + + foreach ($match in $interpolations) { + $failures.Add("${stackName}: compose interpolation at $($match.Path):$($match.LineNumber): $($match.Line.Trim())") + } + + $allowedMissing = @() + if ($exemptions.ContainsKey($stackName)) { + $allowedMissing = @($exemptions[$stackName]) + } + + foreach ($service in Get-ServiceBlocks -Lines $lines) { + $hasEnvFile = $service.Lines | Where-Object { $_ -match '^\s+env_file:\s*$' -or $_ -match '^\s+- stack\.env\s*$' } + if (-not $hasEnvFile -and $allowedMissing -notcontains $service.Name) { + $failures.Add("${stackName}/${service.Name}: missing env_file: stack.env") + } + } +} + +if ($failures.Count -gt 0) { + $failures | ForEach-Object { Write-Error $_ } + exit 1 +} + +Write-Host "Portainer stack env checks passed." +``` + +- [ ] **Step 3: Run the guard and prove current failure** + +Run: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\check-portainer-stack-env.ps1 +``` + +Expected before fixes: + +```text +Write-Error: dev: compose interpolation at ... +Write-Error: resume: compose interpolation at ... +``` + +- [ ] **Step 4: Commit guardrails** + +Run: + +```powershell +git add scripts/check-portainer-stack-env.ps1 scripts/portainer-stack-env-exemptions.json +git commit -m "test: add portainer stack env guard" +``` + +Expected: commit exits `0` and prints a one-line summary for `test: add portainer stack env guard`. + +## Task 3: Prepare Registry Auth Material + +**Files:** +- Create: `scripts/new-registry-auth.ps1` +- Modify: `.env` + +- [ ] **Step 1: Create registry credential generator** + +Create `scripts/new-registry-auth.ps1`: + +```powershell +[CmdletBinding()] +param( + [string]$OutputDirectory = (Join-Path $env:TEMP "valhalla-registry-auth"), + [string]$PushUser = "registry_push", + [string]$PullUser = "registry_pull" +) + +$ErrorActionPreference = "Stop" + +function New-Secret { + $bytes = New-Object byte[] 32 + [System.Security.Cryptography.RandomNumberGenerator]::Fill($bytes) + return [Convert]::ToBase64String($bytes).TrimEnd("=") +} + +New-Item -ItemType Directory -Force -Path $OutputDirectory | Out-Null + +$pushPassword = New-Secret +$pullPassword = New-Secret +$htpasswdPath = Join-Path $OutputDirectory "htpasswd" +$envPath = Join-Path $OutputDirectory "registry.env" + +$pushLine = docker run --rm --entrypoint htpasswd httpd:2 -Bbn $PushUser $pushPassword +$pullLine = docker run --rm --entrypoint htpasswd httpd:2 -Bbn $PullUser $pullPassword + +Set-Content -LiteralPath $htpasswdPath -NoNewline -Value ($pushLine + "`n" + $pullLine + "`n") +Set-Content -LiteralPath $envPath -NoNewline -Value @" +REGISTRY_PUSH_USERNAME=$PushUser +REGISTRY_PUSH_PASSWORD=$pushPassword +REGISTRY_PULL_USERNAME=$PullUser +REGISTRY_PULL_PASSWORD=$pullPassword +"@ + +Write-Host "Created:" +Write-Host " $htpasswdPath" +Write-Host " $envPath" +Write-Host "Append registry.env values to .env, then copy htpasswd to valhalla:/config/registry/auth/htpasswd." +``` + +- [ ] **Step 2: Generate credentials** + +Run: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\new-registry-auth.ps1 +``` + +Expected: the script prints concrete paths under `$env:TEMP\valhalla-registry-auth` for `htpasswd` and `registry.env`. + +- [ ] **Step 3: Add generated registry credentials to `.env`** + +Open the generated `registry.env`, copy these four generated key/value lines into `.env` under a new section: + +Add this section to `.env`, using the exact generated values from `$env:TEMP\valhalla-registry-auth\registry.env`: + +```dotenv +# ============================================================= +# DOCKER REGISTRY +# ============================================================= +REGISTRY_PUSH_USERNAME=registry_push +REGISTRY_PUSH_PASSWORD=value generated by scripts/new-registry-auth.ps1 +REGISTRY_PULL_USERNAME=registry_pull +REGISTRY_PULL_PASSWORD=value generated by scripts/new-registry-auth.ps1 +``` + +Replace the two `value generated by...` strings with the actual generated password values before saving `.env`. + +- [ ] **Step 4: Commit generator and `.env` credential additions** + +Run: + +```powershell +git add scripts/new-registry-auth.ps1 .env +git commit -m "chore: add registry auth credentials" +``` + +Expected: commit exits `0` and prints a one-line summary for `chore: add registry auth credentials`. + +## Task 4: Move Gitea and Registry to the Management Plane + +**Files:** +- Modify: `portainer-compose.yml` +- Modify: `.env` +- Modify: `stacks/dev/docker-compose.yml` +- Modify: `stacks/dev/stack.env` + +- [ ] **Step 1: Add management services to `portainer-compose.yml`** + +In `portainer-compose.yml`, add these services after `vault` and before `github-runner`: + +```yaml + postgres_gitea: + container_name: postgres_gitea + image: postgres:16-alpine + restart: unless-stopped + labels: + - "com.centurylabs.watchtower.enable=false" + networks: + - gitea_internal + environment: + POSTGRES_USER: gitea + POSTGRES_PASSWORD: ${GITEA_DB_PASSWORD} + POSTGRES_DB: gitea + volumes: + - /config/gitea/postgres:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U gitea -d gitea"] + interval: 10s + timeout: 5s + retries: 5 + + gitea: + container_name: gitea + image: gitea/gitea:latest + restart: unless-stopped + networks: + - gitea_internal + - edge + environment: + - USER_UID=1000 + - USER_GID=1000 + - GITEA__database__DB_TYPE=postgres + - GITEA__database__HOST=postgres_gitea:5432 + - GITEA__database__NAME=gitea + - GITEA__database__USER=gitea + - GITEA__database__PASSWD=${GITEA_DB_PASSWORD} + - GITEA__server__DOMAIN=gitea.ginnoir.com + - GITEA__server__ROOT_URL=https://gitea.ginnoir.com + - GITEA__server__SSH_DOMAIN=gitea.ginnoir.com + - GITEA__server__SSH_PORT=2222 + - GITEA__repository__ROOT=/repos + - GITEA__lfs__PATH=/repos-lfs + volumes: + - /config/gitea:/data + - /storage1/labdata/gitea/repositories:/repos + - /storage1/labdata/gitea/lfs:/repos-lfs + - /etc/localtime:/etc/localtime:ro + ports: + - "3030:3000" + - "2222:22" + depends_on: + postgres_gitea: + condition: service_healthy + labels: + - "com.centurylabs.watchtower.enable=false" + + gitea-runner: + container_name: gitea_runner + image: gitea/act_runner:latest + restart: unless-stopped + networks: + - gitea_internal + environment: + - GITEA_INSTANCE_URL=https://gitea.ginnoir.com + - GITEA_RUNNER_REGISTRATION_TOKEN=${GITEA_RUNNER_REGISTRATION_TOKEN} + - GITEA_RUNNER_NAME=valhalla-management-runner + - GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:24-bookworm,node-24:docker://node:24-bookworm + volumes: + - /config/gitea/act_runner:/data + - /var/run/docker.sock:/var/run/docker.sock + - /home/ginnoir/valhalla-lab:/valhalla-lab + - /config/caddy:/config/caddy + depends_on: + gitea: + condition: service_started + labels: + - "com.centurylabs.watchtower.enable=false" + + registry: + container_name: registry + image: registry:2 + restart: unless-stopped + networks: + - registry_internal + - edge + environment: + - REGISTRY_AUTH=htpasswd + - REGISTRY_AUTH_HTPASSWD_REALM=Valhalla Registry + - REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd + - REGISTRY_STORAGE_DELETE_ENABLED=true + volumes: + - /storage1/labdata/registry:/var/lib/registry + - /config/registry/auth:/auth:ro + ports: + - "5000:5000" + labels: + - "com.centurylabs.watchtower.enable=false" + + registry-ui: + container_name: registry_ui + image: joxit/docker-registry-ui:latest + restart: unless-stopped + networks: + - registry_internal + - edge + environment: + - SINGLE_REGISTRY=true + - REGISTRY_TITLE=Valhalla Registry + - NGINX_PROXY_PASS_URL=http://registry:5000 + - SHOW_CONTENT_DIGEST=true + - DELETE_IMAGES=true + depends_on: + - registry + labels: + - "com.centurylabs.watchtower.enable=false" +``` + +- [ ] **Step 2: Add management networks to `portainer-compose.yml`** + +At the bottom of `portainer-compose.yml`, add: + +```yaml + gitea_internal: + name: gitea_internal + driver: bridge + registry_internal: + name: registry_internal + driver: bridge +``` + +- [ ] **Step 3: Remove moved services from `stacks/dev/docker-compose.yml`** + +Remove these service blocks from `stacks/dev/docker-compose.yml`: + +```text +gitea +gitea-runner +postgres_gitea +registry +registry-ui +``` + +Keep `code-server`, `bookstack`, `mariadb_bookstack`, `dbx`, and all Plane services in `stacks/dev/docker-compose.yml`. + +- [ ] **Step 4: Move Gitea variables out of `stacks/dev/stack.env`** + +Remove this key from `stacks/dev/stack.env` after confirming the same value exists in root `.env`: + +```dotenv +GITEA_DB_PASSWORD=... +``` + +Keep `GITEA_DB_PASSWORD` in root `.env` for management-plane compose interpolation. + +- [ ] **Step 5: Validate raw compose locally** + +Run: + +```powershell +docker compose -f .\portainer-compose.yml --env-file .\.env config | Out-Null +``` + +Expected: the command exits `0` with no error output. + +- [ ] **Step 6: Commit management-plane extraction** + +Run: + +```powershell +git add portainer-compose.yml stacks/dev/docker-compose.yml stacks/dev/stack.env .env +git commit -m "refactor: move gitea and registry to management plane" +``` + +Expected: commit exits `0` and prints a one-line summary for `refactor: move gitea and registry to management plane`. + +## Task 5: Convert Remaining App Stack Interpolation to `stack.env` + +**Files:** +- Modify: `stacks/dev/docker-compose.yml` +- Modify: `stacks/dev/stack.env` +- Modify: `stacks/resume/docker-compose.yml` +- Modify: `stacks/resume/stack.env` + +- [ ] **Step 1: Convert BookStack variable names** + +In `stacks/dev/stack.env`, add direct container keys preserving existing values: + +```dotenv +APP_KEY=copy the current BOOKSTACK_APP_KEY value +DB_PASS=copy the current BOOKSTACK_DB_PASSWORD value +MYSQL_ROOT_PASSWORD=copy the current BOOKSTACK_DB_ROOT_PASSWORD value +MYSQL_PASSWORD=copy the current BOOKSTACK_DB_PASSWORD value +``` + +Replace each `copy the current ... value` string with the exact current secret value from `stacks/dev/stack.env`. + +In `stacks/dev/docker-compose.yml`, replace BookStack environment entries that use `${...}` with direct names: + +```yaml + - PUID=1000 + - PGID=1000 + - TZ + - APP_URL=https://docs.ginnoir.com + - APP_KEY + - DB_HOST=mariadb_bookstack + - DB_PORT=3306 + - DB_USER=bookstack + - DB_PASS + - DB_DATABASE=bookstack + - AUTH_METHOD=oidc + - AUTH_AUTO_INITIATE=false + - OIDC_NAME=Authentik + - OIDC_ISSUER=https://auth.ginnoir.com/application/o/bookstack/ + - OIDC_ISSUER_DISCOVER=true + - OIDC_END_SESSION_ENDPOINT=true +``` + +For `mariadb_bookstack`, replace environment with: + +```yaml + - PUID=1000 + - PGID=1000 + - TZ + - MYSQL_ROOT_PASSWORD + - MYSQL_DATABASE=bookstack + - MYSQL_USER=bookstack + - MYSQL_PASSWORD +``` + +- [ ] **Step 2: Convert code-server `TZ` mapping** + +In `stacks/dev/docker-compose.yml`, replace: + +```yaml + - TZ=${TZ} +``` + +with: + +```yaml + - TZ +``` + +- [ ] **Step 3: Convert Plane variables** + +In `stacks/dev/stack.env`, add direct container keys preserving existing values: + +```dotenv +SECRET_KEY=copy the current PLANE_SECRET_KEY value +DATABASE_URL=postgresql://plane:copy-the-current-PLANE-DB-PASSWORD-value@postgres_plane:5432/plane +AWS_ACCESS_KEY_ID=copy the current PLANE_MINIO_USER value +AWS_SECRET_ACCESS_KEY=copy the current PLANE_MINIO_PASSWORD value +MINIO_USER=copy the current PLANE_MINIO_USER value +MINIO_PASSWORD=copy the current PLANE_MINIO_PASSWORD value +POSTGRES_PASSWORD=copy the current PLANE_DB_PASSWORD value +MINIO_ROOT_USER=copy the current PLANE_MINIO_USER value +MINIO_ROOT_PASSWORD=copy the current PLANE_MINIO_PASSWORD value +``` + +Replace each `copy the current ... value` string with the exact current secret value from `stacks/dev/stack.env`. In `DATABASE_URL`, replace only `copy-the-current-PLANE-DB-PASSWORD-value` with the actual `PLANE_DB_PASSWORD` value. + +In all Plane service environment blocks, replace `${...}` entries with direct variables. For example, `plane-api` should include: + +```yaml + - SECRET_KEY + - DEBUG=0 + - DATABASE_URL + - REDIS_URL=redis://redis_plane:6379/ + - CELERY_BROKER_URL=redis://redis_plane:6379/0 + - USE_MINIO=1 + - AWS_REGION=us-east-1 + - AWS_ACCESS_KEY_ID + - AWS_SECRET_ACCESS_KEY + - AWS_S3_ENDPOINT_URL=http://plane_minio:9000 + - AWS_S3_BUCKET_NAME=uploads + - WEB_URL=https://plane.ginnoir.com + - CORS_ALLOWED_ORIGINS=https://plane.ginnoir.com + - GUNICORN_WORKERS=2 +``` + +`postgres_plane` should use: + +```yaml + environment: + POSTGRES_USER: plane + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + POSTGRES_DB: plane +``` + +Then change it again to list form to avoid compose interpolation: + +```yaml + environment: + - POSTGRES_USER=plane + - POSTGRES_PASSWORD + - POSTGRES_DB=plane +``` + +`plane-create-bucket` should use: + +```yaml + environment: + - MINIO_USER + - MINIO_PASSWORD +``` + +`plane-minio` should use: + +```yaml + environment: + - MINIO_ROOT_USER + - MINIO_ROOT_PASSWORD +``` + +- [ ] **Step 4: Convert resume MinIO OIDC variables** + +In `stacks/resume/stack.env`, add direct container keys preserving existing values: + +```dotenv +MINIO_IDENTITY_OPENID_CLIENT_ID=copy the current MINIO_OIDC_CLIENT_ID value +MINIO_IDENTITY_OPENID_CLIENT_SECRET=copy the current MINIO_OIDC_CLIENT_SECRET value +``` + +Replace each `copy the current ... value` string with the exact current secret value from `stacks/resume/stack.env`. + +In `stacks/resume/docker-compose.yml`, replace: + +```yaml + - MINIO_IDENTITY_OPENID_CLIENT_ID=${MINIO_OIDC_CLIENT_ID} + - MINIO_IDENTITY_OPENID_CLIENT_SECRET=${MINIO_OIDC_CLIENT_SECRET} +``` + +with: + +```yaml + - MINIO_IDENTITY_OPENID_CLIENT_ID + - MINIO_IDENTITY_OPENID_CLIENT_SECRET +``` + +- [ ] **Step 5: Run the env guard** + +Run: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\check-portainer-stack-env.ps1 +``` + +Expected: + +```text +Portainer stack env checks passed. +``` + +- [ ] **Step 6: Validate all stack compose files** + +Run: + +```powershell +Get-ChildItem .\stacks -Directory | ForEach-Object { + $compose = Join-Path $_.FullName "docker-compose.yml" + $env = Join-Path $_.FullName "stack.env" + if ((Test-Path $compose) -and (Test-Path $env)) { + Push-Location $_.FullName + docker compose --env-file stack.env config | Out-Null + Pop-Location + } +} +``` + +Expected: the command exits `0` with no error output. + +- [ ] **Step 7: Commit env model cleanup** + +Run: + +```powershell +git add stacks/dev/docker-compose.yml stacks/dev/stack.env stacks/resume/docker-compose.yml stacks/resume/stack.env scripts/check-portainer-stack-env.ps1 scripts/portainer-stack-env-exemptions.json +git commit -m "fix: make portainer stacks use stack env only" +``` + +Expected: commit exits `0` and prints a one-line summary for `fix: make portainer stacks use stack env only`. + +## Task 6: Apply Management Plane and Verify Gitea/Registry Stability + +**Files:** +- Use: `apply-compose.ps1` +- Use: `portainer-compose.yml` +- Use: `.env` + +- [ ] **Step 1: Copy registry `htpasswd` to valhalla** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "mkdir -p /config/registry/auth" +scp "$env:TEMP\valhalla-registry-auth\htpasswd" "ginnoir@valhalla:/tmp/registry-htpasswd" +ssh -o BatchMode=yes ginnoir@valhalla "sudo mv /tmp/registry-htpasswd /config/registry/auth/htpasswd && sudo chmod 640 /config/registry/auth/htpasswd" +``` + +Expected: each command exits `0` with no error output. + +- [ ] **Step 2: Push management `.env` to valhalla** + +Run: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File .\apply-compose.ps1 -EnvFile +``` + +Expected: + +```text +Pushing .env ... +``` + +- [ ] **Step 3: Apply management plane** + +Run: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File .\apply-compose.ps1 -Portainer +``` + +Expected: + +```text +Applying management plane (portainer, vault, github-runner, watchtower) ... +``` + +The status text can still mention `github-runner` until docs/scripts are updated later. + +- [ ] **Step 4: Verify moved containers** + +Run: + +```powershell +ssh -o BatchMode=yes -o ConnectTimeout=8 ginnoir@valhalla "docker ps --format '{{.Names}} {{.Status}} {{.Label \"com.docker.compose.project\"}}' | egrep '^(gitea|postgres_gitea|gitea_runner|registry|registry_ui|portainer) '" +``` + +Expected: + +```text +gitea Up ... valhalla-lab +postgres_gitea Up ... valhalla-lab +gitea_runner Up ... valhalla-lab +registry Up ... valhalla-lab +registry_ui Up ... valhalla-lab +portainer Up ... valhalla-lab +``` + +- [ ] **Step 5: Verify registry auth challenge** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "curl -skI https://registry.ginnoir.com/v2/ | head -8" +``` + +Expected: + +```text +HTTP/2 401 +www-authenticate: Basic realm="Valhalla Registry" +docker-distribution-api-version: registry/2.0 +``` + +- [ ] **Step 6: Verify Gitea endpoint and SSH** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "curl -skI https://gitea.ginnoir.com/ | head -5; ssh -o BatchMode=yes -o StrictHostKeyChecking=no -p 2222 git@localhost 2>&1 | head -5" +``` + +Expected: + +```text +HTTP/2 200 +... +Hi there, ginnoir! You've successfully authenticated... +``` + +The SSH greeting username can differ if the key maps to a different Gitea account. + +## Task 7: Add Gitea Actions for Homelab Caddy Reload + +**Files:** +- Create: `.gitea/workflows/deploy-caddy.yml` +- Keep temporarily: `.github/workflows/deploy.yml` +- Modify later: `stacks/proxy/docker-compose.yml` +- Modify later: `CLAUDE.md` + +- [ ] **Step 1: Create Gitea deploy workflow** + +Create `.gitea/workflows/deploy-caddy.yml`: + +```yaml +name: Deploy Caddyfile to valhalla + +on: + push: + branches: [main] + paths: + - Caddyfile + - .gitea/workflows/deploy-caddy.yml + workflow_dispatch: + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Push Caddyfile and reload Caddy + run: | + cp Caddyfile /config/caddy/Caddyfile + docker exec caddy caddy reload --config /etc/caddy/Caddyfile +``` + +- [ ] **Step 2: Commit workflow** + +Run: + +```powershell +git add .gitea/workflows/deploy-caddy.yml +git commit -m "ci: add gitea caddy deploy workflow" +``` + +Expected: commit exits `0` and prints a one-line summary for `ci: add gitea caddy deploy workflow`. + +- [ ] **Step 3: Push to Gitea and GitHub mirror** + +Run: + +```powershell +git push origin main +``` + +Expected: + +```text +To https://github.com/ginnoir/homelabstack.git +To https://gitea.ginnoir.com/ginnoir/homelabstack.git +``` + +- [ ] **Step 4: Verify Gitea workflow run** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "docker logs --tail 200 gitea_runner" +``` + +Expected: + +```text +... Deploy Caddyfile to valhalla ... +... completed ... +``` + +- [ ] **Step 5: Verify Caddy is still serving** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "curl -skI https://gitea.ginnoir.com/ | head -5; curl -skI https://registry.ginnoir.com/v2/ | head -8" +``` + +Expected: + +```text +HTTP/2 200 +... +HTTP/2 401 +``` + +## Task 8: Recreate Portainer Stacks Against Gitea With No UI Env Vars + +**Files:** +- Use: `stacks/*/docker-compose.yml` +- Use: `stacks/*/stack.env` +- Use: `scripts/check-portainer-stack-env.ps1` + +- [ ] **Step 1: Run env guard before touching Portainer** + +Run: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\check-portainer-stack-env.ps1 +``` + +Expected: + +```text +Portainer stack env checks passed. +``` + +- [ ] **Step 2: Create a Gitea read-only PAT for Portainer** + +In Gitea, create a token for a deployment user or `ginnoir` with repository read access for `ginnoir/homelabstack`. + +Record these values for Portainer stack creation: + +```text +Repository URL: https://gitea.ginnoir.com/ginnoir/homelabstack.git +Repository reference: refs/heads/main +Authentication type: Basic +Username: ginnoir +Password/token: the Gitea read-only PAT created for Portainer +``` + +- [ ] **Step 3: Recreate one low-risk stack first** + +Use Portainer UI for the first stack to avoid API shape surprises: + +```text +Stack name: remote +Repository URL: https://gitea.ginnoir.com/ginnoir/homelabstack.git +Repository reference: refs/heads/main +Compose path: stacks/remote/docker-compose.yml +Authentication: Basic, ginnoir, Gitea read-only PAT +GitOps updates: enabled +Mechanism: polling +Fetch interval: 5 minutes +Environment variables in Portainer UI: empty +``` + +Expected: + +```text +remote stack deploys successfully +Portainer UI shows no manually declared environment variables +``` + +- [ ] **Step 4: Verify recreated low-risk stack** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "docker ps --format '{{.Names}} {{.Status}} {{.Label \"com.docker.compose.project\"}}' | egrep '^(hbbr|hbbs) '" +``` + +Expected: + +```text +hbbr Up ... remote +hbbs Up ... remote +``` + +- [ ] **Step 5: Recreate remaining app stacks** + +Repeat the same Portainer Git stack settings for each stack: + +```text +proxy stacks/proxy/docker-compose.yml +media stacks/media/docker-compose.yml +foundry stacks/foundry/docker-compose.yml +owncloud stacks/owncloud/docker-compose.yml +resume stacks/resume/docker-compose.yml +famapp stacks/famapp/docker-compose.yml +authentik stacks/authentik/docker-compose.yml +notify stacks/notify/docker-compose.yml +monitoring stacks/monitoring/docker-compose.yml +roms stacks/roms/docker-compose.yml +romhacks stacks/romhacks/docker-compose.yml +share stacks/share/docker-compose.yml +backup stacks/backup/docker-compose.yml +notes stacks/notes/docker-compose.yml +dev stacks/dev/docker-compose.yml +``` + +For each stack: + +```text +Environment variables in Portainer UI: empty +GitOps updates: enabled +Mechanism: polling +Fetch interval: 5 minutes +Force redeployment: disabled +Re-pull image: enabled for stacks using mutable image tags +``` + +- [ ] **Step 6: Verify stack env values landed from `stack.env`** + +For each recreated stack, inspect one representative container: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "docker inspect famapp --format '{{json .Config.Env}}' | jq -r '.[]' | sort | egrep '^(DATABASE_URL|AUTH_URL|MINIO_ROOT_USER)='" +ssh -o BatchMode=yes ginnoir@valhalla "docker inspect bookstack --format '{{json .Config.Env}}' | jq -r '.[]' | sort | egrep '^(APP_KEY|DB_PASS|OIDC_CLIENT_ID)='" +ssh -o BatchMode=yes ginnoir@valhalla "docker inspect plane_api --format '{{json .Config.Env}}' | jq -r '.[]' | sort | egrep '^(SECRET_KEY|DATABASE_URL|AWS_ACCESS_KEY_ID)='" +``` + +Expected: + +```text +The expected keys are present. +No value contains a trailing carriage return. +No value is empty unless intentionally empty in stack.env. +``` + +- [ ] **Step 7: Verify Portainer is polling Gitea** + +Make a harmless comment-only commit in `stacks/remote/docker-compose.yml`, push it, and watch Portainer redeploy `remote`. + +Run: + +```powershell +git commit --allow-empty -m "test: verify portainer gitea polling" +git push origin main +``` + +Expected within 5 minutes: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "docker ps --format '{{.Names}} {{.Status}} {{.Label \"com.docker.compose.project\"}}' | egrep '^(hbbr|hbbs) '" +``` + +The `remote` container creation time should refresh or Portainer activity should show a GitOps update from Gitea. + +## Task 9: Move Famapp Image Build and Release to Gitea Actions + +**Files in `C:\Users\MattC\Documents\famapp`:** +- Create: `.gitea/workflows/ci.yml` +- Create: `.gitea/workflows/release.yml` +- Modify: `.release-it.json` +- Modify: `README.md` +- Modify: `docs/RUNBOOK.md` +- Modify: `docs/ENV.md` +- Modify: `deploy/README.md` +- Modify: `deploy/compose.example.yaml` + +- [ ] **Step 1: Create Gitea CI workflow** + +Create `C:\Users\MattC\Documents\famapp\.gitea\workflows\ci.yml`: + +```yaml +name: CI + +on: + push: + branches: [main] + pull_request: + +jobs: + checks: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Enable pnpm + run: | + corepack enable + corepack prepare pnpm@10.33.3 --activate + + - name: Install dependencies + run: pnpm install --frozen-lockfile + + - name: Typecheck + run: pnpm typecheck + + - name: Lint + run: pnpm lint + + - name: Format check + run: pnpm format:check + + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Enable pnpm + run: | + corepack enable + corepack prepare pnpm@10.33.3 --activate + + - name: Install dependencies + run: pnpm install --frozen-lockfile + + - name: Build + run: pnpm build + env: + DATABASE_URL: postgres://ci_user:ci_password@localhost:5432/ci_database + AUTH_SECRET: ci-auth-secret-for-build + NEXT_PUBLIC_APP_URL: http://localhost:3000 +``` + +- [ ] **Step 2: Create Gitea release workflow** + +Create `C:\Users\MattC\Documents\famapp\.gitea\workflows\release.yml`: + +```yaml +name: Release Image + +on: + push: + tags: + - "v*" + workflow_dispatch: + +jobs: + build-and-push: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Derive image tags + id: meta + shell: bash + run: | + VERSION="${GITEA_REF_NAME#v}" + MAJOR_MINOR="$(printf '%s' "$VERSION" | awk -F. '{print $1"."$2}')" + { + echo "version=$VERSION" + echo "major_minor=$MAJOR_MINOR" + echo "image=registry.ginnoir.com/ginnoir/famapp" + } >> "$GITEA_OUTPUT" + + - name: Login to registry + run: | + echo "${{ secrets.REGISTRY_PUSH_PASSWORD }}" | docker login registry.ginnoir.com \ + --username "${{ secrets.REGISTRY_PUSH_USERNAME }}" \ + --password-stdin + + - name: Build image + run: | + docker build \ + -t "${{ steps.meta.outputs.image }}:${{ steps.meta.outputs.version }}" \ + -t "${{ steps.meta.outputs.image }}:${{ steps.meta.outputs.major_minor }}" \ + -t "${{ steps.meta.outputs.image }}:latest" \ + . + + - name: Push image + run: | + docker push "${{ steps.meta.outputs.image }}:${{ steps.meta.outputs.version }}" + docker push "${{ steps.meta.outputs.image }}:${{ steps.meta.outputs.major_minor }}" + docker push "${{ steps.meta.outputs.image }}:latest" +``` + +- [ ] **Step 3: Add Gitea repository secrets** + +In Gitea repository `ginnoir/famapp`, add repository-level Actions secrets: + +```text +REGISTRY_PUSH_USERNAME=copy the current homelabstack .env REGISTRY_PUSH_USERNAME value +REGISTRY_PUSH_PASSWORD=copy the current homelabstack .env REGISTRY_PUSH_PASSWORD value +``` + +Use the exact values from `C:\Users\MattC\Documents\homelabstack\.env` when saving the Gitea secrets. + +Expected: + +```text +Secrets are visible by name in Gitea repository settings. +Secret values are not visible after saving. +``` + +- [ ] **Step 4: Disable GitHub release creation in `.release-it.json`** + +In `C:\Users\MattC\Documents\famapp\.release-it.json`, replace: + +```json + "github": { + "release": true, + "releaseName": "v${version}" + }, +``` + +with: + +```json + "github": { + "release": false + }, +``` + +- [ ] **Step 5: Update famapp docs and compose example** + +Replace GHCR image references with: + +```text +registry.ginnoir.com/ginnoir/famapp +``` + +Required replacements: + +```text +ghcr.io/ginnoir/famapp:latest -> registry.ginnoir.com/ginnoir/famapp:latest +ghcr.io/ginnoir/famapp:v0.x.y -> registry.ginnoir.com/ginnoir/famapp:v0.x.y +FAMAPP_IMAGE=ghcr.io/ginnoir/famapp:v0.4.7 -> FAMAPP_IMAGE=registry.ginnoir.com/ginnoir/famapp:v0.4.7 +``` + +- [ ] **Step 6: Run local famapp checks** + +Run: + +```powershell +pnpm --dir C:\Users\MattC\Documents\famapp typecheck +pnpm --dir C:\Users\MattC\Documents\famapp lint +pnpm --dir C:\Users\MattC\Documents\famapp format:check +docker build -t registry.ginnoir.com/ginnoir/famapp:local-test C:\Users\MattC\Documents\famapp +``` + +Expected: + +```text +typecheck passes +lint exits 0 +format:check exits 0 +docker build completes +``` + +- [ ] **Step 7: Commit famapp workflow migration** + +Run: + +```powershell +git -C C:\Users\MattC\Documents\famapp add .gitea/workflows/ci.yml .gitea/workflows/release.yml .release-it.json README.md docs/RUNBOOK.md docs/ENV.md deploy/README.md deploy/compose.example.yaml +git -C C:\Users\MattC\Documents\famapp commit -m "ci: publish famapp images to valhalla registry" +``` + +Expected: commit exits `0` and prints a one-line summary for `ci: publish famapp images to valhalla registry`. + +- [ ] **Step 8: Push famapp to Gitea and GitHub mirror** + +Run: + +```powershell +git -C C:\Users\MattC\Documents\famapp push origin main +``` + +Expected: + +```text +To https://github.com/ginnoir/famapp.git +To https://gitea.ginnoir.com/ginnoir/famapp.git +``` + +## Task 10: Publish and Deploy Famapp From the Self-Hosted Registry + +**Files:** +- Modify: `stacks/famapp/docker-compose.yml` + +- [ ] **Step 1: Push a famapp release tag** + +Run from famapp: + +```powershell +git -C C:\Users\MattC\Documents\famapp tag v0.5.4 +git -C C:\Users\MattC\Documents\famapp push origin v0.5.4 +``` + +Expected: + +```text +To https://github.com/ginnoir/famapp.git +To https://gitea.ginnoir.com/ginnoir/famapp.git +``` + +- [ ] **Step 2: Verify image exists in registry** + +Run: + +```powershell +$envContent = Get-Content C:\Users\MattC\Documents\homelabstack\.env +$pullUser = ($envContent | Where-Object { $_ -match '^REGISTRY_PULL_USERNAME=' }) -replace '^REGISTRY_PULL_USERNAME=', '' +$pullPass = ($envContent | Where-Object { $_ -match '^REGISTRY_PULL_PASSWORD=' }) -replace '^REGISTRY_PULL_PASSWORD=', '' +ssh -o BatchMode=yes ginnoir@valhalla "echo '$pullPass' | docker login registry.ginnoir.com --username '$pullUser' --password-stdin && docker pull registry.ginnoir.com/ginnoir/famapp:latest" +``` + +Expected: + +```text +Login Succeeded +latest: Pulling from ginnoir/famapp +Status: Downloaded newer image for registry.ginnoir.com/ginnoir/famapp:latest +``` + +- [ ] **Step 3: Point homelab famapp compose at local registry** + +In `stacks/famapp/docker-compose.yml`, replace: + +```yaml + image: ghcr.io/ginnoir/famapp:latest +``` + +with: + +```yaml + image: registry.ginnoir.com/ginnoir/famapp:latest +``` + +- [ ] **Step 4: Commit famapp compose switch** + +Run: + +```powershell +git add stacks/famapp/docker-compose.yml +git commit -m "fix(famapp): pull image from valhalla registry" +``` + +Expected: commit exits `0` and prints a one-line summary for `fix(famapp): pull image from valhalla registry`. + +- [ ] **Step 5: Push and wait for Portainer redeploy** + +Run: + +```powershell +git push origin main +``` + +Expected: + +```text +To https://github.com/ginnoir/homelabstack.git +To https://gitea.ginnoir.com/ginnoir/homelabstack.git +``` + +- [ ] **Step 6: Verify famapp is using registry image** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "docker inspect famapp --format '{{.Config.Image}}'; curl -skI https://fam.ginnoir.com/ | head -5" +``` + +Expected: + +```text +registry.ginnoir.com/ginnoir/famapp:latest +HTTP/2 200 +``` + +## Task 11: Update Docs and Retire GitHub Runner After Staged Verification + +**Files:** +- Modify: `CLAUDE.md` +- Modify: `AGENTS.md` +- Modify: `apply-compose.ps1` +- Modify: `portainer-compose.yml` +- Modify: `stacks/proxy/docker-compose.yml` +- Modify: `.github/workflows/deploy.yml` + +- [ ] **Step 1: Update homelab docs** + +In `CLAUDE.md` and `AGENTS.md`, update deployment channel language: + +```text +stacks/domain-name/* -> git push to Gitea main -> Portainer polls every 5 minutes +Caddyfile -> git push to Gitea main -> Gitea Actions copies Caddyfile and reloads Caddy +GitHub -> temporary mirror only during staged migration +Portainer app stacks -> no Portainer UI env vars; use stack.env via env_file +``` + +- [ ] **Step 2: Update proxy comment** + +In `stacks/proxy/docker-compose.yml`, replace: + +```text +hot-reloaded by the GitHub Actions runner on push +``` + +with: + +```text +hot-reloaded by the Gitea Actions runner on push +``` + +- [ ] **Step 3: Update apply script management text** + +In `apply-compose.ps1`, replace: + +```powershell +Write-Host "Applying management plane (portainer, vault, github-runner, watchtower) ..." +``` + +with: + +```powershell +Write-Host "Applying management plane (portainer, vault, gitea, registry, watchtower) ..." +``` + +- [ ] **Step 4: Remove GitHub runner after two successful Gitea deploy cycles** + +Remove the `github-runner` service block from `portainer-compose.yml` only after: + +```text +Gitea Actions has reloaded Caddy successfully at least twice. +Portainer has redeployed at least one stack from Gitea polling. +Famapp has been built by Gitea Actions and pulled from registry.ginnoir.com. +``` + +- [ ] **Step 5: Keep `.github/workflows/deploy.yml` until final GitHub mirror retirement** + +During the temporary mirror period, disable rather than delete the GitHub workflow by changing triggers to manual only: + +```yaml +name: Deploy to valhalla + +on: + workflow_dispatch: + +jobs: + deploy: + runs-on: self-hosted + steps: + - name: Disabled + run: echo "Deployment moved to Gitea Actions." +``` + +- [ ] **Step 6: Commit documentation and runner retirement** + +Run: + +```powershell +git add CLAUDE.md AGENTS.md apply-compose.ps1 portainer-compose.yml stacks/proxy/docker-compose.yml .github/workflows/deploy.yml +git commit -m "docs: document gitea-first homelab deploys" +``` + +Expected: commit exits `0` and prints a one-line summary for `docs: document gitea-first homelab deploys`. + +## Task 12: Final Verification and Vault Write-Back + +**Files:** +- Read: `CLAUDE.md` +- Read: `AGENTS.md` +- Use: Obsidian note `Homelab/_Claude.md` when vault MCP is available + +- [ ] **Step 1: Run repo verification** + +Run: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\check-portainer-stack-env.ps1 +docker compose -f .\portainer-compose.yml --env-file .\.env config | Out-Null +git status --short --branch +``` + +Expected: + +```text +Portainer stack env checks passed. +docker compose config exits 0 +## main...origin/main +``` + +- [ ] **Step 2: Verify live management plane** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "docker compose -f ~/valhalla-lab/portainer-compose.yml --env-file ~/valhalla-lab/.env ps" +``` + +Expected: + +```text +portainer Up +vault Up +gitea Up +postgres_gitea Up +gitea_runner Up +registry Up +registry_ui Up +watchtower Up +``` + +- [ ] **Step 3: Verify public/internal endpoints** + +Run: + +```powershell +ssh -o BatchMode=yes ginnoir@valhalla "curl -skI https://portainer.ginnoir.com/ | head -5; curl -skI https://gitea.ginnoir.com/ | head -5; curl -skI https://registry.ginnoir.com/v2/ | head -8; curl -skI https://fam.ginnoir.com/ | head -5" +``` + +Expected: + +```text +HTTP/2 200 +HTTP/2 200 +HTTP/2 401 +HTTP/2 200 +``` + +- [ ] **Step 4: Verify no app stack relies on Portainer UI env vars** + +For each recreated Portainer stack, open Portainer UI and confirm: + +```text +Stack > Environment variables: empty +Stack > GitOps updates: enabled +Repository URL: https://gitea.ginnoir.com/ginnoir/homelabstack.git +``` + +Record any stack with non-empty UI env vars as a blocker and recreate it before continuing. + +- [ ] **Step 5: Write durable findings to Obsidian** + +When `mcp__obsidian__vault_patch` or the Obsidian MCP is available, append to `Homelab/_Claude.md`: + +```markdown +## Gitea-first deploy architecture + +- Homelab app stacks now poll `https://gitea.ginnoir.com/ginnoir/homelabstack.git` from Portainer. +- GitHub is only a temporary mirror during the cutover period. +- Gitea, `postgres_gitea`, `gitea_runner`, `registry`, and `registry_ui` are management-plane services, not Portainer app-stack services. +- Portainer app stacks must not define UI environment variables; stack config comes from repo-local `stack.env` via `env_file: stack.env`. +- Famapp images publish to `registry.ginnoir.com/ginnoir/famapp` from Gitea Actions. +- Registry auth uses separate push and pull credentials; Portainer/Docker pulls use the pull credential. +``` + +- [ ] **Step 6: Final commit if vault/docs changed** + +If only the vault was patched through MCP, no repo commit is needed. If repo docs changed during final verification, run: + +```powershell +git add CLAUDE.md AGENTS.md +git commit -m "docs: record gitea-first deploy verification" +``` + +Expected only if docs changed: commit exits `0` and prints a one-line summary for `docs: record gitea-first deploy verification`. + +## Rollback Plan + +- If Gitea management-plane apply fails: + - Re-apply the previous `portainer-compose.yml` commit with `apply-compose.ps1 -Portainer`. + - Gitea data is bind-mounted and remains under `/config/gitea` and `/storage1/labdata/gitea`. +- If registry auth blocks famapp pulls: + - Temporarily set `stacks/famapp/docker-compose.yml` back to `ghcr.io/ginnoir/famapp:latest`. + - Push to Gitea/GitHub mirror. + - Recreate Portainer registry credentials before switching back. +- If Portainer cannot poll Gitea: + - Keep GitHub mirror stack definitions active. + - Recreate only the affected stack against GitHub until Gitea auth/TLS is fixed. +- If `dev` stack recreation fails after removing Gitea/registry: + - Management-plane Gitea and registry remain independent. + - Restore the previous `stacks/dev/docker-compose.yml` only for non-moved services if needed. + +## Self-Review + +- Spec coverage: + - Gitea-first migration is covered by Tasks 4, 7, 8, 11, and 12. + - Temporary GitHub mirror is covered by Tasks 7, 8, 9, 10, and 11. + - Docker registry setup and famapp image publishing are covered by Tasks 3, 4, 9, and 10. + - Portainer `stack.env` enforcement is covered by Tasks 2, 5, 8, and 12. + - Work-repo remote protection is covered by Task 1. +- Red-flag scan: + - Secret values are intentionally generated at execution time by `scripts/new-registry-auth.ps1`; plan text names exact keys and commands without embedding secrets. + - No implementation step depends on an unnamed file or undefined command. +- Type and name consistency: + - Registry image path is consistently `registry.ginnoir.com/ginnoir/famapp`. + - Gitea repository path is consistently `https://gitea.ginnoir.com/ginnoir/homelabstack.git`. + - Guard script and exemption file paths are consistent across tasks. diff --git a/scripts/migration-backup.sh b/scripts/migration-backup.sh new file mode 100644 index 0000000..5d13de3 --- /dev/null +++ b/scripts/migration-backup.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +# +# Phase 0 migration backup — valhalla (Dell R510 -> R730XD / Proxmox move). +# +# Captures everything that does NOT travel on the /storage1 ZFS pool disks: +# * logical dumps of all Postgres + MariaDB databases (clean, version-portable) +# * the /config app-state tree (app configs, vault, portainer, couchdb, raw datadirs) +# * native Plex "Application Support" (watch history, collections, metadata) +# +# NOT copied here (they live on /storage1/labdata and ride along on the disks): +# MinIO blobs (famapp/resume/plane), gitea repos+LFS, docker registry, media. +# +# Best-effort by design: a failed individual DB dump logs a WARNING and is skipped +# (its raw datadir is still inside config.tar) — it never blocks the /config/Plex tars. +# +# Run ON valhalla as ginnoir: +# bash migration-backup.sh # live insurance backup (no downtime) +# bash migration-backup.sh --stop # consistent backup: stop app containers first, +# # then restart them when done (rehearsal-safe) +# +# Output: /storage1/migration-backup// (+ SHA256SUMS) +# +set -uo pipefail # NOTE: deliberately no `set -e` — we want best-effort completion. + +DEST="/storage1/migration-backup" +STAMP="$(date +%Y%m%d-%H%M%S)" +OUT="$DEST/$STAMP" +STOP=0 +[[ "${1:-}" == "--stop" ]] && STOP=1 + +mkdir -p "$OUT/db" "$OUT/config" "$OUT/plex" +LOG="$OUT/backup.log" +exec > >(tee -a "$LOG") 2>&1 +log(){ printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; } +running(){ docker ps --format '{{.Names}}' | grep -qx "$1"; } + +# Resolve a working Postgres superuser for a container (env POSTGRES_USER, then +# common fallbacks). Echoes the user on success; non-zero if none connect. +pg_super(){ + local c="$1" cand + for cand in "$(docker exec "$c" printenv POSTGRES_USER 2>/dev/null)" \ + "$(docker exec "$c" printenv POSTGRES_DB 2>/dev/null)" \ + immich postgres; do + [ -z "$cand" ] && continue + if docker exec "$c" psql -U "$cand" -tAc 'select 1' >/dev/null 2>&1; then + echo "$cand"; return 0 + fi + done + return 1 +} + +log "migration backup -> $OUT (stop=$STOP)" + +PG_CONTAINERS="postgres_authentik postgres_famapp postgres_gitea postgres_plane postgres_resume nextcloud-postgres immich-postgres" +MARIA_CONTAINERS="owncloud_mariadb romm-db mariadb_bookstack" + +# --- 1. logical DB dumps (best-effort; authoritative restore path) --- +for c in $PG_CONTAINERS; do + running "$c" || { log "SKIP $c (not running)"; continue; } + if u="$(pg_super "$c")"; then + log "pg_dumpall $c (user=$u)" + if docker exec "$c" pg_dumpall --clean --if-exists -U "$u" | gzip > "$OUT/db/${c}.sql.gz"; then + : + else + log "WARN: pg_dumpall $c FAILED — raw pgdata remains in config.tar"; rm -f "$OUT/db/${c}.sql.gz" + fi + else + log "WARN: no working superuser found for $c — raw pgdata remains in config.tar" + fi +done + +for c in $MARIA_CONTAINERS; do + running "$c" || { log "SKIP $c (not running)"; continue; } + log "mariadb-dump $c" + pw="$(docker exec "$c" sh -c 'printf %s "${MYSQL_ROOT_PASSWORD:-${MARIADB_ROOT_PASSWORD:-}}"' 2>/dev/null)" + if docker exec -e MYSQL_PWD="$pw" "$c" sh -c \ + 'mariadb-dump -uroot --all-databases --single-transaction --routines --triggers --events 2>/dev/null \ + || mysqldump -uroot --all-databases --single-transaction --routines --triggers --events' \ + | gzip > "$OUT/db/${c}.sql.gz"; then + : + else + log "WARN: mariadb-dump $c FAILED — raw datadir remains in config.tar"; rm -f "$OUT/db/${c}.sql.gz" + fi +done + +# --- 2. optionally stop app containers for a fully-consistent /config snapshot --- +STOPPED="" +if [[ "$STOP" == "1" ]]; then + log "stopping app containers for a consistent /config snapshot (portainer left up)" + KEEP="portainer" + for c in $(docker ps --format '{{.Names}}'); do + case " $KEEP " in *" $c "*) continue;; esac + docker stop "$c" >/dev/null && STOPPED="$STOPPED $c" + done + log "stopped:$STOPPED" +fi + +# --- 3. /config tree (app state, vault, couchdb, raw datadirs) --- +log "tar /config (the big one, ~100GB)" +sudo tar --warning=no-file-changed -cf "$OUT/config/config.tar" -C / config || log "WARN: /config tar returned non-zero" + +# --- 4. native Plex Application Support --- +log "tar Plex Application Support" +tar --warning=no-file-changed -cf "$OUT/plex/plex-appsupport.tar" \ + -C "/var/lib/plexmediaserver/Library/Application Support" "Plex Media Server" || log "WARN: Plex tar returned non-zero" + +# --- 5. restart anything we stopped (leave the box as we found it) --- +if [[ -n "$STOPPED" ]]; then + log "restarting stopped containers" + # shellcheck disable=SC2086 + docker start $STOPPED >/dev/null +fi + +# --- 6. inventory + checksums --- +log "writing inventory + checksums" +docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Label "com.docker.compose.project"}}' > "$OUT/inventory-containers.txt" +docker network ls > "$OUT/inventory-networks.txt" +cp /config/caddy/Caddyfile "$OUT/Caddyfile" 2>/dev/null || true +cp /home/ginnoir/valhalla-lab/portainer-compose.yml "$OUT/portainer-compose.yml" 2>/dev/null || true +cp /home/ginnoir/valhalla-lab/.env "$OUT/valhalla-lab.env" 2>/dev/null || true +( cd "$OUT" && find . -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS ) + +log "sizes:" +du -sh "$OUT"/db "$OUT"/config "$OUT"/plex 2>/dev/null || true +du -sh "$OUT" 2>/dev/null || true +echo "=== db dumps ==="; ls -lh "$OUT/db" +log "DONE -> $OUT" +echo +echo "NOTE: MinIO blobs, gitea repos/LFS, the registry, and all media are on /storage1" +echo " and travel with the pool disks; they are intentionally NOT in this backup." diff --git a/scripts/migration-cutover.sh b/scripts/migration-cutover.sh new file mode 100644 index 0000000..3640428 --- /dev/null +++ b/scripts/migration-cutover.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# +# CUTOVER final backup — valhalla R510 (run as ginnoir, right before zpool export). +# +# Takes the FINAL consistent backup of everything not already on the pool, lands it on +# /storage1 (which travels with the disks), then leaves all services STOPPED. +# Does NOT restart services and does NOT export the pool (that's a separate confirmed step). +# +# Order matters: dump DBs while their containers are still up, THEN stop everything, THEN +# tar /config (consistent), THEN stop Plex + delta-rsync it. +# +set -uo pipefail +DEST=/storage1/migration-backup +STAMP=$(date +%Y%m%d-%H%M%S) +OUT="$DEST/cutover-$STAMP" +mkdir -p "$OUT/db" "$OUT/config" +LOG="$OUT/cutover.log"; exec > >(tee -a "$LOG") 2>&1 +log(){ printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; } +running(){ docker ps --format '{{.Names}}' | grep -qx "$1"; } +pg_super(){ local c="$1" cand; for cand in "$(docker exec "$c" printenv POSTGRES_USER 2>/dev/null)" immich postgres; do [ -z "$cand" ] && continue; docker exec "$c" psql -U "$cand" -tAc 'select 1' >/dev/null 2>&1 && { echo "$cand"; return 0; }; done; return 1; } + +log "===== CUTOVER FINAL BACKUP -> $OUT =====" + +# --- 1. dump all databases (containers still running) --- +for c in postgres_authentik postgres_famapp postgres_gitea postgres_plane postgres_resume nextcloud-postgres immich-postgres; do + running "$c" || { log "skip $c (down)"; continue; } + if u=$(pg_super "$c"); then + docker exec "$c" pg_dumpall --clean --if-exists -U "$u" | gzip > "$OUT/db/$c.sql.gz" && log "pg $c (u=$u) ok" || { log "WARN pg $c failed"; rm -f "$OUT/db/$c.sql.gz"; } + else log "WARN no superuser for $c"; fi +done +# owncloud + romm: root w/ env password, named DB +dump_maria_pw(){ local c="$1" db="$2" pw; pw=$(docker exec "$c" sh -c 'printf %s "${MARIADB_ROOT_PASSWORD:-${MYSQL_ROOT_PASSWORD:-}}"' 2>/dev/null); docker exec -e MYSQL_PWD="$pw" "$c" mariadb-dump -uroot --single-transaction --routines --triggers "$db" 2>/dev/null | gzip > "$OUT/db/$c.sql.gz" && log "maria $c ok" || { log "WARN maria $c failed"; rm -f "$OUT/db/$c.sql.gz"; }; } +running owncloud_mariadb && dump_maria_pw owncloud_mariadb owncloud +running romm-db && dump_maria_pw romm-db romm +# bookstack: root via unix_socket, all DBs +running mariadb_bookstack && { docker exec mariadb_bookstack mariadb-dump -uroot --single-transaction --all-databases 2>/dev/null | gzip > "$OUT/db/mariadb_bookstack.sql.gz" && log "maria bookstack ok" || log "WARN bookstack failed"; } +log "db dumps:"; ls -lh "$OUT/db" + +# --- 2. stop ALL containers (homelab goes down) --- +log "stopping all containers..." +docker stop $(docker ps -q) >/dev/null 2>&1; sleep 2 +log "running containers now: $(docker ps -q | wc -l) (expect 0)" + +# --- 3. tar /config (now fully consistent) --- +log "tar /config (~112G, the long step)" +sudo tar --warning=no-file-changed -cf "$OUT/config/config.tar" -C / config && log "config.tar done" || log "WARN config tar non-zero" + +# --- 4. Plex: stop + final delta rsync --- +log "stopping Plex + final delta rsync" +sudo systemctl stop plexmediaserver +sudo rsync -aHX "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server" /storage1/labdata/plex/ && log "plex delta synced" || log "WARN plex rsync" + +# --- 5. checksums + sizes --- +log "checksums" +( cd "$OUT" && find . -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS ) +du -sh "$OUT"/db "$OUT"/config 2>/dev/null +log "===== CUTOVER BACKUP COMPLETE. Services stopped. Ready to release /storage1 + export. =====" diff --git a/scripts/refresh-b2-snapshot.sh b/scripts/refresh-b2-snapshot.sh new file mode 100644 index 0000000..e8e2c53 --- /dev/null +++ b/scripts/refresh-b2-snapshot.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# +# Pre-migration: make the Kopia -> B2 snapshots current AND complete. +# +# 1) waits for the running migration-backup.sh to finish (avoid /config read contention) +# 2) supplements the logical DB dumps the nightly job omits (nextcloud, immich) and +# replaces the empty bookstack dump, using the complete set from migration-backup.sh +# 3) runs the existing Kopia snapshot pipeline -> fresh B2 snapshot of /config + dumps + labdata +# +# Run ON valhalla as ginnoir: nohup bash ~/refresh-b2-snapshot.sh > ~/refresh-b2-snapshot.out 2>&1 & +# +set -uo pipefail +log(){ printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; } + +# 1. wait for the migration backup to complete +log "waiting for migration-backup.sh to finish..." +while pgrep -f 'migration-backup.sh' >/dev/null 2>&1; do sleep 15; done +log "migration backup finished" + +# locate newest migration backup db dir and newest official dump dir +MIG_DB="$(ls -1dt /storage1/migration-backup/*/db 2>/dev/null | head -1)" +DUMPS="$(ls -1dt /config/backup/dumps/*/ 2>/dev/null | head -1)" +log "migration db dir: ${MIG_DB:-}" +log "official dump dir: ${DUMPS:-}" + +# 2. supplement the dumps the nightly job misses (raw datadirs already cover them in /config, +# but this gives B2 a complete logical-dump layer too) +if [ -n "${MIG_DB:-}" ] && [ -n "${DUMPS:-}" ]; then + for f in nextcloud-postgres.sql.gz immich-postgres.sql.gz mariadb_bookstack.sql.gz; do + if [ -s "$MIG_DB/$f" ]; then + sudo cp -f "$MIG_DB/$f" "$DUMPS/$f" && log "supplemented $f ($(du -h "$DUMPS/$f" | cut -f1))" + else + log "WARN: $MIG_DB/$f missing/empty — not supplemented" + fi + done + log "dump dir now:"; ls -lh "$DUMPS" +fi + +# 3. fresh Kopia -> B2 snapshot (same pipeline cron uses) +log "running Kopia snapshot to B2..." +docker exec backup_scheduler sh /scripts/backup-run-snapshots.sh 2>&1 + +log "DONE — verify with: docker exec kopia sh -c 'KOPIA_PASSWORD=\$KOPIA_REPOSITORY_PASSWORD kopia snapshot list --all' | tail"