163 lines
5.3 KiB
Markdown
163 lines
5.3 KiB
Markdown
# Runbook
|
|
|
|
Operational reference for famapp in production. See `deploy/README.md` for the one-time host setup and `docs/tasks/09-pre-deploy-checklist.md` for the pre-deploy checklist.
|
|
|
|
## Cutting a release
|
|
|
|
From `main` on the dev machine, with a clean working tree:
|
|
|
|
```bash
|
|
pnpm release:patch # or :minor / :major
|
|
# — bumps package.json version
|
|
# — prepends to CHANGELOG.md
|
|
# — creates and pushes a signed git tag (v0.x.y)
|
|
# — creates a GitHub Release with generated notes
|
|
# Requires GITHUB_TOKEN in .env
|
|
```
|
|
|
|
Gitea Actions (`.gitea/workflows/release.yml`) builds and pushes the Docker image to the **self-hosted registry** (production source of truth):
|
|
|
|
- `registry.ginnoir.com/ginnoir/famapp:v0.x.y`
|
|
- `registry.ginnoir.com/ginnoir/famapp:0.x` (minor alias)
|
|
- `registry.ginnoir.com/ginnoir/famapp:latest`
|
|
|
|
GitHub Actions (`.github/workflows/release.yml`) mirrors the same tag to **GHCR** as a temporary backup only — prod must not pull from `ghcr.io`.
|
|
|
|
Wait for the Gitea Actions run to finish (or confirm the new digest on the registry) before redeploying.
|
|
|
|
## Deploying a release
|
|
|
|
On the home server, in `/srv/famapp/deploy/`:
|
|
|
|
```bash
|
|
# Pin the new tag
|
|
sed -i 's|FAMAPP_IMAGE=.*|FAMAPP_IMAGE=registry.ginnoir.com/ginnoir/famapp:v0.x.y|' .env
|
|
|
|
# Pull and restart only the app container
|
|
docker compose pull famapp
|
|
docker compose up -d famapp
|
|
|
|
# Watch the boot log — migrations run before the server starts
|
|
docker compose logs -f famapp
|
|
```
|
|
|
|
The container entrypoint runs `node scripts/migrate.mjs` automatically. A healthy boot ends with a log line like `ready on http://0.0.0.0:3000`.
|
|
|
|
## Health check
|
|
|
|
```bash
|
|
# Container status
|
|
docker compose ps
|
|
|
|
# App response (200 = healthy)
|
|
curl -sf https://fam.yourdomain.com/ -o /dev/null -w "%{http_code}\n"
|
|
|
|
# Recent app logs
|
|
docker compose logs --tail=100 famapp
|
|
|
|
# Database connectivity
|
|
docker compose exec famapp-db pg_isready -U famapp -d famapp
|
|
```
|
|
|
|
## Rollback
|
|
|
|
1. Find the previous working tag in `CHANGELOG.md` or `docker images`.
|
|
2. Pin it in `deploy/.env`:
|
|
```bash
|
|
sed -i 's|FAMAPP_IMAGE=.*|FAMAPP_IMAGE=registry.ginnoir.com/ginnoir/famapp:v0.x.y|' .env
|
|
```
|
|
3. Restart the container:
|
|
```bash
|
|
docker compose up -d famapp
|
|
```
|
|
|
|
If the rollback target predates a migration that has already been applied to the database, restore from backup first — see **Backups** below. To skip auto-migration on a given start (rarely needed):
|
|
|
|
```bash
|
|
RUN_MIGRATIONS=false docker compose up -d famapp
|
|
```
|
|
|
|
## Backups
|
|
|
|
The `famapp-backup` container runs nightly `pg_dump` for both `famapp` and `authentik` databases into the `backups` volume.
|
|
|
|
```bash
|
|
# Check last backup run
|
|
docker compose logs famapp-backup | tail -20
|
|
|
|
# List backup files
|
|
docker compose exec famapp-backup ls -lh /backups
|
|
|
|
# Manual backup now
|
|
docker compose exec famapp-backup sh /scripts/backup.sh
|
|
|
|
# Restore from a backup file
|
|
docker compose exec famapp-backup sh /scripts/restore.sh famapp_2026-06-01.sql.gz
|
|
```
|
|
|
|
See `deploy/backups/README.md` for retention policy and full restore details.
|
|
|
|
## Common issues
|
|
|
|
### App container exits immediately
|
|
|
|
```bash
|
|
docker compose logs famapp
|
|
```
|
|
|
|
Likely causes:
|
|
|
|
- **Missing required env var** — the app throws on startup if `AUTH_SECRET`, `DATABASE_URL`, or `AUTH_OIDC_*` are absent.
|
|
- **`ENABLE_DEV_LOGIN=true` in production** — `src/lib/dev-login-config.ts` throws an assertion error on import. Remove the variable from the production env.
|
|
- **Database not ready** — Postgres healthcheck should prevent this, but if the DB is slow to start, increase `start_period` in `compose.yaml`.
|
|
|
|
### Migrations fail on boot
|
|
|
|
```bash
|
|
docker compose logs famapp | grep -i migration
|
|
```
|
|
|
|
- Schema is ahead of code: roll back the image or forward-migrate manually.
|
|
- Database unreachable: confirm `famapp-db` is healthy (`docker compose ps`).
|
|
|
|
### OIDC login fails
|
|
|
|
1. Confirm `AUTH_OIDC_ISSUER` matches the Authentik provider URL exactly (trailing slash matters).
|
|
2. Confirm the OIDC client redirect URI in Authentik includes `https://fam.yourdomain.com/api/auth/callback/oidc`.
|
|
3. Check Authentik logs: `docker compose logs authentik-server | tail -50`.
|
|
|
|
### Push notifications not arriving
|
|
|
|
1. Confirm `VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY`, and `VAPID_SUBJECT` are all set in the production env.
|
|
2. Verify the browser's push subscription is still valid (Settings → notifications → re-enable).
|
|
3. If using ntfy as a secondary channel, confirm `NTFY_URL` and `NTFY_TOPIC` are set.
|
|
|
|
### MinIO / garden image uploads failing
|
|
|
|
```bash
|
|
docker compose logs famapp-minio | tail -30
|
|
```
|
|
|
|
- Confirm `MINIO_ENDPOINT` is `http://famapp-minio:9000` in the compose env (not `localhost`).
|
|
- Confirm the `garden` bucket exists — create it manually via the MinIO console at port `9001` if it is missing.
|
|
|
|
### Out of disk space
|
|
|
|
```bash
|
|
df -h /var/lib/docker
|
|
docker system prune --volumes # removes stopped containers, dangling images, unused volumes
|
|
```
|
|
|
|
Old backup files accumulate in the `backups` volume. The retention script (`deploy/backups/retain.sh`) runs nightly — check its logs if the volume keeps growing.
|
|
|
|
## Authentik admin access
|
|
|
|
The Authentik admin UI is at `https://auth.yourdomain.com`. Log in with the superuser credentials set during bootstrap — see `deploy/authentik/README.md`.
|
|
|
|
To reset the Authentik admin password from the CLI:
|
|
|
|
```bash
|
|
docker compose exec authentik-server ak create_recovery_key 1 akadmin
|
|
# prints a one-time recovery URL
|
|
```
|