Add Kopia backup stack with Backblaze B2 offsite target.
Protects /config, selective labdata, and nightly DB dumps; documents homelab improvement plan briefs.
This commit is contained in:
ginnoir
@@ -0,0 +1,39 @@
|
||||
# backup stack
|
||||
|
||||
Kopia → Backblaze B2. Backs up `/config`, **selected** `labdata` paths (see `SCOPE.md`), and nightly SQL dumps.
|
||||
|
||||
## Before first deploy
|
||||
|
||||
1. **Backblaze B2** (recommended offsite — ~$6/TB/mo)
|
||||
- Create a bucket (e.g. `valhalla-homelab-backup`)
|
||||
- Create an **Application Key** scoped to that bucket
|
||||
- Note the S3 endpoint for your region
|
||||
|
||||
2. **Fill `stack.env`**
|
||||
- `KOPIA_REPOSITORY_PASSWORD` — `openssl rand -hex 32`
|
||||
- `B2_KEY_ID`, `B2_APPLICATION_KEY`, `B2_BUCKET`, `B2_ENDPOINT`
|
||||
|
||||
3. **Host paths** (auto-created on first start if missing)
|
||||
- `/config/kopia/{config,cache,logs}`
|
||||
- `/config/backup/dumps`
|
||||
|
||||
4. **Register** new Portainer git stack → `stacks/backup` (one-time)
|
||||
|
||||
5. **Push** → Portainer redeploys within 5 min
|
||||
|
||||
## After deploy
|
||||
|
||||
- UI: `http://valhalla:51515` (or add `backup.ginnoir.com` internal-only in Caddy)
|
||||
- Subscribe phone to ntfy topic `backup` (or path in `NTFY_URL`)
|
||||
- **Restore test** (do once): pick a small dump from `/config/backup/dumps/`, restore to a throwaway container
|
||||
|
||||
## Schedule
|
||||
|
||||
| Time | What |
|
||||
|------|------|
|
||||
| 02:50 | `backup-dump-databases.sh` — SQL dumps |
|
||||
| 03:00 | Kopia snapshots (paths in `snapshot-paths.txt`) |
|
||||
|
||||
## If B2 creds are empty
|
||||
|
||||
Kopia still starts with UI — configure repository manually, then set policies for each path in `snapshot-paths.txt`.
|
||||
@@ -0,0 +1,49 @@
|
||||
# What we back up (and skip)
|
||||
|
||||
**Rule:** everything irreplaceable or slow to rebuild — **not** ROM files, **not** provider cover art.
|
||||
|
||||
## Always
|
||||
|
||||
| Path | Why |
|
||||
|------|-----|
|
||||
| `/config` | App state, DBs, CouchDB vault — **includes RomM MariaDB** (`/config/romm/mariadb`) |
|
||||
| `/config/backup/dumps` | Nightly SQL dumps — **includes `romm-db`** |
|
||||
|
||||
## RomM
|
||||
|
||||
Per [RomM docs](https://docs.romm.app/): **`assets` = saves/states**; **`resources` = fetched covers**.
|
||||
|
||||
| Included | Path | What |
|
||||
|----------|------|------|
|
||||
| Database | `/config/romm/mariadb` + nightly dump | Scans, hashes, matches (weeks of work) |
|
||||
| Saves & states | `labdata/romm/assets` | RomM-managed save files |
|
||||
| Config/cache | `/config/romm/config`, `/config/romm/redis` | Via `/config` snapshot |
|
||||
|
||||
| Skipped | Path | What |
|
||||
|---------|------|------|
|
||||
| Cover art | `labdata/romm/resources` | IGDB/SteamGridDB — minutes to re-fetch |
|
||||
|
||||
## Emulation (`/storage1/Emulation`) — excluded
|
||||
|
||||
Legacy pre-RomM tree (EmuDeck, ES-DE, 13T `roms/`, symlinks). **Not backed up offsite.** RomM is the source of truth for saves via `labdata/romm/assets` + DB.
|
||||
|
||||
## labdata — included
|
||||
|
||||
| Path | Why |
|
||||
|------|-----|
|
||||
| `gitea/repositories`, `gitea/lfs` | Git + LFS |
|
||||
| `famapp/minio`, `resume/minio`, `plane/minio` | User uploads |
|
||||
| `romhacks/metadata`, `romhacks/library` | Catalog work + guides/spreadsheets |
|
||||
| `romm/assets` | Saves (see RomM above) |
|
||||
|
||||
## labdata — skipped
|
||||
|
||||
| Path | Why |
|
||||
|------|-----|
|
||||
| `romm/resources` | Provider cover art |
|
||||
| `registry` | Re-pull images |
|
||||
| `romhacks/incoming` | Transient download queue |
|
||||
|
||||
## Change scope
|
||||
|
||||
Edit `snapshot-paths.txt`, bind mounts in `docker-compose.yml`, and `SCOPE.md`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# min hour day month weekday command
|
||||
50 2 * * * /bin/sh /scripts/backup-dump-databases.sh
|
||||
0 3 * * * /bin/sh /scripts/backup-run-snapshots.sh
|
||||
@@ -0,0 +1,85 @@
|
||||
# backup stack — Kopia snapshots of /config, selected /storage1/labdata paths, and DB dumps.
|
||||
# Scope: stacks/backup/SCOPE.md + snapshot-paths.txt (excludes RomM art, registry, transient dirs).
|
||||
#
|
||||
# Offsite target: Backblaze B2 (S3-compatible). One-time repo setup — see README.md.
|
||||
# Schedule: backup-scheduler dumps databases ~02:50, Kopia snapshots ~03:00 (policy).
|
||||
#
|
||||
# Env convention: stack.env only (no Portainer UI vars). Kopia UI is internal-only
|
||||
# via Caddy after you add backup.ginnoir.com (optional, README).
|
||||
|
||||
services:
|
||||
kopia:
|
||||
container_name: kopia
|
||||
image: kopia/kopia:0.17
|
||||
hostname: kopia
|
||||
restart: unless-stopped
|
||||
networks: [backup, edge]
|
||||
env_file:
|
||||
- stack.env
|
||||
environment:
|
||||
- KOPIA_PASSWORD=${KOPIA_REPOSITORY_PASSWORD}
|
||||
- TZ=${TZ}
|
||||
- B2_BUCKET=${B2_BUCKET}
|
||||
- B2_KEY_ID=${B2_KEY_ID}
|
||||
- B2_APPLICATION_KEY=${B2_APPLICATION_KEY}
|
||||
- B2_ENDPOINT=${B2_ENDPOINT}
|
||||
cap_add:
|
||||
- SYS_ADMIN
|
||||
security_opt:
|
||||
- apparmor:unconfined
|
||||
devices:
|
||||
- /dev/fuse:/dev/fuse
|
||||
volumes:
|
||||
- /config/kopia/config:/app/config
|
||||
- /config/kopia/cache:/app/cache
|
||||
- /config/kopia/logs:/app/logs
|
||||
- /config:/backup/config:ro
|
||||
- /config/backup/dumps:/backup/dumps:ro
|
||||
- /storage1/labdata/gitea/repositories:/backup/labdata/gitea/repositories:ro
|
||||
- /storage1/labdata/gitea/lfs:/backup/labdata/gitea/lfs:ro
|
||||
- /storage1/labdata/famapp/minio:/backup/labdata/famapp/minio:ro
|
||||
- /storage1/labdata/resume/minio:/backup/labdata/resume/minio:ro
|
||||
- /storage1/labdata/plane/minio:/backup/labdata/plane/minio:ro
|
||||
- /storage1/labdata/romhacks/metadata:/backup/labdata/romhacks/metadata:ro
|
||||
- /storage1/labdata/romhacks/library:/backup/labdata/romhacks/library:ro
|
||||
- /storage1/labdata/romm/assets:/backup/labdata/romm/assets:ro
|
||||
- ./init-kopia.sh:/init-kopia.sh:ro
|
||||
- ./snapshot-paths.txt:/snapshot-paths.txt:ro
|
||||
ports:
|
||||
- "51515:51515"
|
||||
entrypoint: ["/bin/sh", "/init-kopia.sh"]
|
||||
labels:
|
||||
- "com.centurylabs.watchtower.enable=false"
|
||||
|
||||
backup-scheduler:
|
||||
container_name: backup_scheduler
|
||||
image: alpine:3.20
|
||||
restart: unless-stopped
|
||||
networks: [backup]
|
||||
env_file:
|
||||
- stack.env
|
||||
environment:
|
||||
- TZ=${TZ}
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /config/backup/dumps:/dumps
|
||||
- ./scripts/backup-dump-databases.sh:/scripts/backup-dump-databases.sh:ro
|
||||
- ./scripts/backup-run-snapshots.sh:/scripts/backup-run-snapshots.sh:ro
|
||||
- ./snapshot-paths.txt:/snapshot-paths.txt:ro
|
||||
- ./crontab:/etc/crontab:ro
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
- |
|
||||
apk add --no-cache docker-cli curl tzdata >/dev/null
|
||||
printenv | grep -E '^(NTFY_|TZ=)' > /etc/backup.env || true
|
||||
crond -f -l 8 -c /etc
|
||||
depends_on:
|
||||
- kopia
|
||||
|
||||
networks:
|
||||
backup:
|
||||
name: backup
|
||||
driver: bridge
|
||||
edge:
|
||||
name: edge
|
||||
external: true
|
||||
@@ -0,0 +1,46 @@
|
||||
#!/bin/sh
|
||||
# One-time (or idempotent) Kopia repo + policy setup, then start server UI.
|
||||
set -e
|
||||
|
||||
CONFIG_DIR=/app/config
|
||||
export KOPIA_CONFIG_PATH="${CONFIG_DIR}/repository.config"
|
||||
|
||||
mkdir -p "${CONFIG_DIR}" /app/cache /app/logs
|
||||
|
||||
# Connect or create B2 repository when credentials are present.
|
||||
if [ -n "${B2_KEY_ID}" ] && [ -n "${B2_APPLICATION_KEY}" ] && [ -n "${B2_BUCKET}" ]; then
|
||||
if [ ! -f "${KOPIA_CONFIG_PATH}" ]; then
|
||||
echo "init-kopia: creating S3 repository on bucket ${B2_BUCKET}"
|
||||
kopia repository create s3 \
|
||||
--bucket="${B2_BUCKET}" \
|
||||
--access-key="${B2_KEY_ID}" \
|
||||
--secret-access-key="${B2_APPLICATION_KEY}" \
|
||||
--endpoint="${B2_ENDPOINT}" \
|
||||
--password="${KOPIA_PASSWORD}"
|
||||
else
|
||||
echo "init-kopia: repository config exists, skipping create"
|
||||
fi
|
||||
|
||||
while IFS= read -r path || [ -n "${path}" ]; do
|
||||
case "${path}" in ''|\#*) continue ;; esac
|
||||
kopia policy set "${path}" \
|
||||
--keep-latest 14 \
|
||||
--keep-daily 30 \
|
||||
--keep-weekly 12 \
|
||||
--keep-monthly 24 \
|
||||
--compression=zstd \
|
||||
--snapshot-time=03:00 \
|
||||
--enable-actions=false \
|
||||
2>/dev/null || kopia policy set "${path}" \
|
||||
--keep-latest 14 \
|
||||
--keep-daily 30 \
|
||||
--keep-weekly 12 \
|
||||
--keep-monthly 24 \
|
||||
--compression=zstd \
|
||||
--snapshot-time=03:00
|
||||
done < /snapshot-paths.txt
|
||||
else
|
||||
echo "init-kopia: B2 credentials empty — start UI only; configure repo manually (see README)"
|
||||
fi
|
||||
|
||||
exec kopia server start --ui --insecure --address=0.0.0.0:51515
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/bin/sh
|
||||
# Dump all homelab databases to /config/backup/dumps (gzip SQL).
|
||||
# Runs inside backup_scheduler; needs docker.sock.
|
||||
set -eu
|
||||
|
||||
DUMP_ROOT=/dumps
|
||||
STAMP=$(date +%Y%m%d-%H%M%S)
|
||||
DIR="${DUMP_ROOT}/${STAMP}"
|
||||
mkdir -p "${DIR}"
|
||||
|
||||
log() { echo "[backup-dump ${STAMP}] $*"; }
|
||||
|
||||
dump_postgres() {
|
||||
name="$1"
|
||||
user="$2"
|
||||
db="$3"
|
||||
if ! docker ps --format '{{.Names}}' | grep -qx "${name}"; then
|
||||
log "skip ${name} (not running)"
|
||||
return 0
|
||||
fi
|
||||
log "dump ${name} → ${db}"
|
||||
docker exec "${name}" pg_dump -U "${user}" "${db}" | gzip -c > "${DIR}/${name}.sql.gz"
|
||||
}
|
||||
|
||||
dump_mariadb() {
|
||||
name="$1"
|
||||
db="$2"
|
||||
if ! docker ps --format '{{.Names}}' | grep -qx "${name}"; then
|
||||
log "skip ${name} (not running)"
|
||||
return 0
|
||||
fi
|
||||
log "dump ${name} → ${db}"
|
||||
docker exec "${name}" sh -c "mariadb-dump -u root -p\"\$MYSQL_ROOT_PASSWORD\" \"${db}\"" \
|
||||
| gzip -c > "${DIR}/${name}.sql.gz"
|
||||
}
|
||||
|
||||
log "starting database dumps → ${DIR}"
|
||||
|
||||
dump_postgres postgres_gitea gitea gitea
|
||||
dump_postgres postgres_plane plane plane
|
||||
dump_postgres postgres_authentik authentik authentik
|
||||
dump_postgres postgres_famapp famapp famapp
|
||||
dump_postgres postgres_resume postgres postgres
|
||||
|
||||
dump_mariadb mariadb_bookstack bookstack
|
||||
dump_mariadb owncloud_mariadb owncloud
|
||||
dump_mariadb romm-db romm
|
||||
|
||||
find "${DUMP_ROOT}" -mindepth 1 -maxdepth 1 -type d -mtime +14 -exec rm -rf {} + 2>/dev/null || true
|
||||
|
||||
log "done"
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
[ -f /etc/backup.env ] && . /etc/backup.env
|
||||
|
||||
NTFY_URL="${NTFY_URL:-https://ntfy.ginnoir.com/backup}"
|
||||
NTFY_TITLE="${NTFY_TITLE:-ValhallaBackup}"
|
||||
|
||||
notify() {
|
||||
priority="$1"
|
||||
message="$2"
|
||||
curl -sf -H "Title: ${NTFY_TITLE}" -H "Priority: ${priority}" -d "${message}" "${NTFY_URL}" >/dev/null 2>&1 || true
|
||||
}
|
||||
|
||||
if ! docker ps --format '{{.Names}}' | grep -qx kopia; then
|
||||
notify high "Kopia container not running — snapshots skipped"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
FAILED=0
|
||||
while IFS= read -r path || [ -n "${path}" ]; do
|
||||
case "${path}" in ''|\#*) continue ;; esac
|
||||
if ! docker exec kopia kopia snapshot create "${path}"; then
|
||||
FAILED=1
|
||||
fi
|
||||
done < /snapshot-paths.txt
|
||||
|
||||
if [ "${FAILED}" -eq 0 ]; then
|
||||
notify default "Snapshots OK ($(date +%Y-%m-%d\ %H:%M))"
|
||||
else
|
||||
notify high "One or more snapshots FAILED — check docker logs kopia"
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,11 @@
|
||||
# Kopia snapshot roots (one path per line). Edit when scope changes.
|
||||
/backup/config
|
||||
/backup/dumps
|
||||
/backup/labdata/gitea/repositories
|
||||
/backup/labdata/gitea/lfs
|
||||
/backup/labdata/famapp/minio
|
||||
/backup/labdata/resume/minio
|
||||
/backup/labdata/plane/minio
|
||||
/backup/labdata/romhacks/metadata
|
||||
/backup/labdata/romhacks/library
|
||||
/backup/labdata/romm/assets
|
||||
@@ -0,0 +1,15 @@
|
||||
# backup stack — Kopia repository + Backblaze B2 offsite.
|
||||
TZ=America/Chicago
|
||||
|
||||
# Encrypts the Kopia repository (NOT your B2 login).
|
||||
KOPIA_REPOSITORY_PASSWORD=8f4c2a9e1b7d6035f8e2c4a6910d3b7e5f9a8c2d6b4e1f0a3c7d9e2b5f8a1c4
|
||||
|
||||
# Backblaze B2 (S3-compatible API)
|
||||
B2_BUCKET=ginnoir-backup
|
||||
B2_KEY_ID=00452ab80ae44e10000000001
|
||||
B2_APPLICATION_KEY=K004P2lGTtRiCe430O5dVZJiTn7Kwr8
|
||||
B2_ENDPOINT=https://s3.us-west-004.backblazeb2.com
|
||||
|
||||
# ntfy alerts (same pattern as watchtower)
|
||||
NTFY_URL=https://ntfy.ginnoir.com/backup
|
||||
NTFY_TITLE=ValhallaBackup
|
||||
Reference in New Issue
Block a user