Implement tasks 60, 61, 62: backups, rate limiting, structured logging
Task 60 — Postgres backups: - deploy/backups/: backup.sh (pg_dump -Fc nightly), retain.sh (14/8/6 tiers), restore.sh, entrypoint.sh, crontab - famapp-backup Alpine service + backups volume added to deploy/compose.yaml - Restore procedure in deploy/backups/README.md Task 61 — Rate limiting on share links: - src/lib/rate-limit.ts: Edge-compatible sliding-window counter (50/min, LRU eviction) with consume(), isRateLimited(), recordFailure() exports - middleware.ts: enforces 429 with Retry-After: 60 for /s/[token] (IP + token prefix) - /s/[token]/page.tsx: tracks only failed resolveShareToken calls via recordFailure() Task 62 — Structured logging: - pino + pino-pretty installed; serverExternalPackages added to next.config.ts - src/lib/logger.ts: JSON in production, pretty in dev, level from LOG_LEVEL env - middleware.ts: structured JSON request log (method, path, status, ms, authenticated) - _core/push.ts, notify.ts, reminders.ts: console.error/log → logger.error/info Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
ginnoir
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
b6abe052c9
commit
285a460eb8
@@ -0,0 +1,102 @@
|
||||
# Backups
|
||||
|
||||
The `famapp-backup` service performs nightly compressed `pg_dump` of both
|
||||
`famapp-db` and `authentik-db` at **02:00 server time**.
|
||||
|
||||
## Storage layout
|
||||
|
||||
```
|
||||
/backups/ (named Docker volume: famapp_backups)
|
||||
famapp/
|
||||
daily/ ← last 14 days
|
||||
weekly/ ← last 8 Sundays
|
||||
monthly/ ← last 6 first-of-month dumps
|
||||
authentik/
|
||||
daily/
|
||||
weekly/
|
||||
monthly/
|
||||
```
|
||||
|
||||
Dump files are named `YYYY-MM-DD.dump` in custom (`-Fc`) format (internal
|
||||
compression, ~3–5× smaller than plain SQL).
|
||||
|
||||
## Retention
|
||||
|
||||
| Tier | Kept | Trigger |
|
||||
| ------- | ---- | --------------------- |
|
||||
| daily | 14 | every night |
|
||||
| weekly | 8 | Sunday night |
|
||||
| monthly | 6 | 1st of the month |
|
||||
|
||||
Retention is enforced by `retain.sh` at the end of each `backup.sh` run.
|
||||
|
||||
## Restore procedure
|
||||
|
||||
### 1. Identify the dump
|
||||
|
||||
```sh
|
||||
# List available dumps
|
||||
docker exec famapp-backup-1 ls /backups/famapp/daily/
|
||||
```
|
||||
|
||||
### 2a. Restore inside the backup container (recommended)
|
||||
|
||||
```sh
|
||||
docker exec famapp-backup-1 /scripts/restore.sh \
|
||||
/backups/famapp/daily/2024-06-01.dump \
|
||||
postgres://famapp:SECRET@famapp-db:5432/famapp
|
||||
```
|
||||
|
||||
Replace `SECRET` with the value of `FAMAPP_DB_PASSWORD` in your `.env` file.
|
||||
For authentik:
|
||||
|
||||
```sh
|
||||
docker exec famapp-backup-1 /scripts/restore.sh \
|
||||
/backups/authentik/daily/2024-06-01.dump \
|
||||
postgres://authentik:SECRET@authentik-db:5432/authentik
|
||||
```
|
||||
|
||||
### 2b. Restore to a separate database (safe — non-destructive)
|
||||
|
||||
Create a fresh target database first, then restore into it:
|
||||
|
||||
```sh
|
||||
# Create the target DB
|
||||
docker exec famapp-db-1 createdb \
|
||||
-U "$FAMAPP_DB_USER" famapp_restore
|
||||
|
||||
# Restore
|
||||
docker exec famapp-backup-1 /scripts/restore.sh \
|
||||
/backups/famapp/daily/2024-06-01.dump \
|
||||
postgres://famapp:SECRET@famapp-db:5432/famapp_restore
|
||||
```
|
||||
|
||||
### 2c. Restore on a fresh host (disaster recovery)
|
||||
|
||||
```sh
|
||||
# Copy the dump file out of the volume
|
||||
docker cp famapp-backup-1:/backups/famapp/daily/2024-06-01.dump ./
|
||||
|
||||
# Spin up a temporary Postgres container and restore
|
||||
docker run --rm \
|
||||
-e PGPASSWORD=SECRET \
|
||||
-v "$(pwd)/2024-06-01.dump:/dump.dump:ro" \
|
||||
postgres:16-alpine \
|
||||
pg_restore -h <new-db-host> -U famapp -d famapp \
|
||||
--no-owner --no-acl /dump.dump
|
||||
```
|
||||
|
||||
## Off-site replication
|
||||
|
||||
The backups live in the `famapp_backups` Docker named volume. To copy them
|
||||
to another host, rsync the volume's data directory periodically (e.g. from a
|
||||
host cron job):
|
||||
|
||||
```sh
|
||||
# On the Docker host, add to /etc/cron.d/famapp-rsync:
|
||||
30 3 * * * root rsync -a --delete \
|
||||
/var/lib/docker/volumes/famapp_backups/_data/ \
|
||||
user@offsite-server:/opt/famapp-backups/
|
||||
```
|
||||
|
||||
Encryption at rest is handled at the disk/filesystem layer (e.g. LUKS).
|
||||
@@ -0,0 +1,58 @@
|
||||
#!/bin/sh
|
||||
# Nightly pg_dump for famapp-db and authentik-db.
|
||||
# Called by crond. Outputs to /backups/<db>/daily/<YYYY-MM-DD>.dump
|
||||
# and copies into weekly/ (Sundays) and monthly/ (1st of month).
|
||||
set -eu
|
||||
|
||||
TODAY=$(date +%Y-%m-%d)
|
||||
DOW=$(date +%u) # 1=Mon … 7=Sun
|
||||
DOM=$(date +%d | sed 's/^0*//') # day-of-month without leading zero
|
||||
|
||||
dump_db() {
|
||||
local name="$1"
|
||||
local host="$2"
|
||||
local port="$3"
|
||||
local user="$4"
|
||||
local pass="$5"
|
||||
local dbname="$6"
|
||||
|
||||
local daily_dir="/backups/$name/daily"
|
||||
local dest="$daily_dir/$TODAY.dump"
|
||||
|
||||
mkdir -p "$daily_dir" \
|
||||
"/backups/$name/weekly" \
|
||||
"/backups/$name/monthly"
|
||||
|
||||
echo "[backup] dumping $name ..."
|
||||
PGPASSWORD="$pass" pg_dump \
|
||||
-h "$host" -p "$port" -U "$user" -d "$dbname" \
|
||||
-Fc -f "$dest"
|
||||
echo "[backup] $name → $dest"
|
||||
|
||||
if [ "$DOW" = "7" ]; then
|
||||
cp "$dest" "/backups/$name/weekly/$TODAY.dump"
|
||||
echo "[backup] weekly copy saved for $name"
|
||||
fi
|
||||
|
||||
if [ "$DOM" = "1" ]; then
|
||||
cp "$dest" "/backups/$name/monthly/$TODAY.dump"
|
||||
echo "[backup] monthly copy saved for $name"
|
||||
fi
|
||||
}
|
||||
|
||||
dump_db famapp \
|
||||
"${FAMAPP_DB_HOST:-famapp-db}" \
|
||||
"${FAMAPP_DB_PORT:-5432}" \
|
||||
"$FAMAPP_DB_USER" \
|
||||
"$FAMAPP_DB_PASSWORD" \
|
||||
"$FAMAPP_DB_NAME"
|
||||
|
||||
dump_db authentik \
|
||||
"${AUTHENTIK_DB_HOST:-authentik-db}" \
|
||||
"${AUTHENTIK_DB_PORT:-5432}" \
|
||||
"$AUTHENTIK_DB_USER" \
|
||||
"$AUTHENTIK_DB_PASSWORD" \
|
||||
"$AUTHENTIK_DB_NAME"
|
||||
|
||||
/scripts/retain.sh
|
||||
echo "[backup] complete"
|
||||
@@ -0,0 +1 @@
|
||||
0 2 * * * /scripts/backup.sh >> /var/log/backup.log 2>&1
|
||||
@@ -0,0 +1,21 @@
|
||||
#!/bin/sh
|
||||
# Container entrypoint: install postgresql-client, create backup directories,
|
||||
# install crontab, and start crond.
|
||||
set -eu
|
||||
|
||||
apk add --no-cache postgresql-client >/dev/null
|
||||
|
||||
mkdir -p \
|
||||
/backups/famapp/daily /backups/famapp/weekly /backups/famapp/monthly \
|
||||
/backups/authentik/daily /backups/authentik/weekly /backups/authentik/monthly
|
||||
|
||||
# Make scripts executable (volume mount may strip +x).
|
||||
chmod +x /scripts/backup.sh /scripts/retain.sh /scripts/restore.sh
|
||||
|
||||
# Install root crontab from the mounted file.
|
||||
mkdir -p /var/spool/cron/crontabs
|
||||
cp /scripts/crontab /var/spool/cron/crontabs/root
|
||||
chmod 600 /var/spool/cron/crontabs/root
|
||||
|
||||
echo "[entrypoint] backup service started — first run at 02:00"
|
||||
exec crond -f -l 2
|
||||
@@ -0,0 +1,26 @@
|
||||
#!/bin/sh
|
||||
# Restore a pg_dump (-Fc format) to a target Postgres database.
|
||||
# The target DB must already exist and be empty (or you accept overwriting data).
|
||||
#
|
||||
# Usage:
|
||||
# restore.sh <dump-file> <target-db-url>
|
||||
#
|
||||
# Example (inside the backup container):
|
||||
# /scripts/restore.sh \
|
||||
# /backups/famapp/daily/2024-06-01.dump \
|
||||
# postgres://famapp:secret@famapp-db:5432/famapp
|
||||
#
|
||||
# Example (from the host via docker exec):
|
||||
# docker exec famapp-backup-1 /scripts/restore.sh \
|
||||
# /backups/famapp/daily/2024-06-01.dump \
|
||||
# postgres://famapp:secret@famapp-db:5432/famapp_restore
|
||||
set -eu
|
||||
|
||||
DUMP_FILE="${1:?Usage: restore.sh <dump-file> <target-db-url>}"
|
||||
TARGET_URL="${2:?Usage: restore.sh <dump-file> <target-db-url>}"
|
||||
|
||||
[ -f "$DUMP_FILE" ] || { echo "[restore] ERROR: $DUMP_FILE not found"; exit 1; }
|
||||
|
||||
echo "[restore] $DUMP_FILE → $TARGET_URL"
|
||||
pg_restore -d "$TARGET_URL" --no-owner --no-acl --exit-on-error "$DUMP_FILE"
|
||||
echo "[restore] done"
|
||||
@@ -0,0 +1,24 @@
|
||||
#!/bin/sh
|
||||
# Prune old backup files according to retention policy.
|
||||
# daily: keep 14 weekly: keep 8 monthly: keep 6
|
||||
set -eu
|
||||
|
||||
prune() {
|
||||
local dir="$1"
|
||||
local keep="$2"
|
||||
[ -d "$dir" ] || return 0
|
||||
# Files are YYYY-MM-DD.dump; lexicographic sort = chronological.
|
||||
# tail skips the N newest; xargs deletes the rest.
|
||||
ls -1 "$dir"/*.dump 2>/dev/null \
|
||||
| sort \
|
||||
| head -n "-$keep" \
|
||||
| xargs -r rm -f --
|
||||
}
|
||||
|
||||
for db in famapp authentik; do
|
||||
prune "/backups/$db/daily" 14
|
||||
prune "/backups/$db/weekly" 8
|
||||
prune "/backups/$db/monthly" 6
|
||||
done
|
||||
|
||||
echo "[retain] done"
|
||||
Reference in New Issue
Block a user