famapp/docs/RUNBOOK.md

Runbook

Operational reference for famapp in production. See deploy/README.md for the one-time host setup and docs/tasks/09-pre-deploy-checklist.md for the pre-deploy checklist.

Cutting a release

From main on the dev machine, with a clean working tree:

pnpm release:patch   # or :minor / :major
# — bumps package.json version
# — prepends to CHANGELOG.md
# — creates and pushes a signed git tag (v0.x.y)
# — creates a GitHub Release with generated notes
# Requires GITHUB_TOKEN in .env

Gitea Actions (.gitea/workflows/release.yml) builds and pushes the Docker image to the self-hosted registry (production source of truth):

• registry.ginnoir.com/ginnoir/famapp:v0.x.y
• registry.ginnoir.com/ginnoir/famapp:0.x (minor alias)
• registry.ginnoir.com/ginnoir/famapp:latest

GitHub Actions (.github/workflows/release.yml) mirrors the same tag to GHCR as a temporary backup only — prod must not pull from ghcr.io.

Wait for the Gitea Actions run to finish (or confirm the new digest on the registry) before redeploying.

Deploying a release

On the home server, in /srv/famapp/deploy/:

# Pin the new tag
sed -i 's|FAMAPP_IMAGE=.*|FAMAPP_IMAGE=registry.ginnoir.com/ginnoir/famapp:v0.x.y|' .env

# Pull and restart only the app container
docker compose pull famapp
docker compose up -d famapp

# Watch the boot log — migrations run before the server starts
docker compose logs -f famapp

The container entrypoint runs node scripts/migrate.mjs automatically. A healthy boot ends with a log line like ready on http://0.0.0.0:3000.

Health check

# Container status
docker compose ps

# App response (200 = healthy)
curl -sf https://fam.yourdomain.com/ -o /dev/null -w "%{http_code}\n"

# Recent app logs
docker compose logs --tail=100 famapp

# Database connectivity
docker compose exec famapp-db pg_isready -U famapp -d famapp

Rollback

1. Find the previous working tag in CHANGELOG.md or docker images.
2. Pin it in deploy/.env:

   sed -i 's|FAMAPP_IMAGE=.*|FAMAPP_IMAGE=registry.ginnoir.com/ginnoir/famapp:v0.x.y|' .env
   

3. Restart the container:

   docker compose up -d famapp
   

If the rollback target predates a migration that has already been applied to the database, restore from backup first — see Backups below. To skip auto-migration on a given start (rarely needed):

RUN_MIGRATIONS=false docker compose up -d famapp

Backups

The famapp-backup container runs nightly pg_dump for both famapp and authentik databases into the backups volume.

# Check last backup run
docker compose logs famapp-backup | tail -20

# List backup files
docker compose exec famapp-backup ls -lh /backups

# Manual backup now
docker compose exec famapp-backup sh /scripts/backup.sh

# Restore from a backup file
docker compose exec famapp-backup sh /scripts/restore.sh famapp_2026-06-01.sql.gz

See deploy/backups/README.md for retention policy and full restore details.

Common issues

App container exits immediately

docker compose logs famapp

Likely causes:

• Missing required env var — the app throws on startup if AUTH_SECRET, DATABASE_URL, or AUTH_OIDC_* are absent.
• ENABLE_DEV_LOGIN=true in production — src/lib/dev-login-config.ts throws an assertion error on import. Remove the variable from the production env.
• Database not ready — Postgres healthcheck should prevent this, but if the DB is slow to start, increase start_period in compose.yaml.

Migrations fail on boot

docker compose logs famapp | grep -i migration

• Schema is ahead of code: roll back the image or forward-migrate manually.
• Database unreachable: confirm famapp-db is healthy (docker compose ps).

OIDC login fails

1. Confirm AUTH_OIDC_ISSUER matches the Authentik provider URL exactly (trailing slash matters).
2. Confirm the OIDC client redirect URI in Authentik includes https://fam.yourdomain.com/api/auth/callback/oidc.
3. Check Authentik logs: docker compose logs authentik-server | tail -50.

Push notifications not arriving

1. Confirm VAPID_PUBLIC_KEY, VAPID_PRIVATE_KEY, and VAPID_SUBJECT are all set in the production env.
2. Verify the browser's push subscription is still valid (Settings → notifications → re-enable).
3. If using ntfy as a secondary channel, confirm NTFY_URL and NTFY_TOPIC are set.

MinIO / garden image uploads failing

docker compose logs famapp-minio | tail -30

• Confirm MINIO_ENDPOINT is http://famapp-minio:9000 in the compose env (not localhost).
• Confirm the garden bucket exists — create it manually via the MinIO console at port 9001 if it is missing.

Out of disk space

df -h /var/lib/docker
docker system prune --volumes   # removes stopped containers, dangling images, unused volumes

Old backup files accumulate in the backups volume. The retention script (deploy/backups/retain.sh) runs nightly — check its logs if the volume keeps growing.

Authentik admin access

The Authentik admin UI is at https://auth.yourdomain.com. Log in with the superuser credentials set during bootstrap — see deploy/authentik/README.md.

To reset the Authentik admin password from the CLI:

docker compose exec authentik-server ak create_recovery_key 1 akadmin
# prints a one-time recovery URL