docs(plan): record root disk audit findings
This commit is contained in:
@@ -70,47 +70,44 @@ lives on it. This phase is a hard gate.
|
||||
|
||||
**Files:** none — live host operation.
|
||||
|
||||
- [ ] **Step 1: Record the baseline**
|
||||
- [x] **Step 1: Record the baseline**
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "df -h / | tail -1; docker system df"
|
||||
```
|
||||
|
||||
Write the output into the task notes below. Expected at plan time: `/dev/sda2 194G 179G 5.7G 97%`
|
||||
and roughly `Images 124 / 123.5GB / 51.47GB reclaimable`.
|
||||
**Baseline (2026-08-05 execution):** `/dev/sda2 194G 179G 5.6G 97% /` —
|
||||
`Images 124 / 123.5GB / 51.47GB reclaimable`, Containers 77/71 Up.
|
||||
|
||||
- [ ] **Step 2: Verify no container depends on what you are about to remove**
|
||||
- [x] **Step 2: Verify no container depends on what you are about to remove**
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker ps -a --filter status=exited --format '{{.Names}}\t{{.Image}}'"
|
||||
```
|
||||
|
||||
Expected: a short list, or empty. Any **exited container you still want** keeps its image, so
|
||||
note them. `docker image prune -a` removes only images not referenced by *any* container,
|
||||
including stopped ones — it will not break a stopped-but-wanted service.
|
||||
Exited only: `plane_migrator` (plane-backend), `plane_create_bucket` (minio/mc) — one-shot helpers; images kept via stopped containers.
|
||||
|
||||
- [ ] **Step 3: Prune**
|
||||
- [x] **Step 3: Prune**
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker image prune -af && docker builder prune -f"
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify space was reclaimed**
|
||||
- [x] **Step 4: Verify space was reclaimed**
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "df -h / | tail -1"
|
||||
```
|
||||
|
||||
Expected: `Avail` is now **at least 40G** and `Use%` is at or below **80%**.
|
||||
If it is not, stop. Do not proceed to Task 2 — investigate what is holding the space.
|
||||
**After prune:** `/dev/sda2 194G 124G 61G 68% /` — 61G avail, 68% used. Gate met.
|
||||
|
||||
- [ ] **Step 5: Confirm all containers are still running**
|
||||
- [x] **Step 5: Confirm all containers are still running**
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "docker ps --format '{{.Names}}\t{{.Status}}' | grep -c Up"
|
||||
```
|
||||
|
||||
Expected: **71 or more** (the count before the prune). Watchtower re-pulls anything it needs.
|
||||
**71 Up** — unchanged.
|
||||
|
||||
---
|
||||
|
||||
@@ -118,7 +115,7 @@ Expected: **71 or more** (the count before the prune). Watchtower re-pulls anyth
|
||||
|
||||
**Files:** none — live host operation.
|
||||
|
||||
- [ ] **Step 1: Find the top consumers outside Docker**
|
||||
- [x] **Step 1: Find the top consumers outside Docker**
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "sudo du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -25"
|
||||
@@ -126,7 +123,7 @@ ssh -o BatchMode=yes ginnoir@valhalla "sudo du -xh --max-depth=2 / 2>/dev/null |
|
||||
|
||||
`-x` keeps it on `/dev/sda2` and stops it wandering into the 94T of `/storage1`.
|
||||
|
||||
- [ ] **Step 2: Check the Docker containerd image store specifically**
|
||||
- [x] **Step 2: Check the Docker containerd image store specifically**
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes ginnoir@valhalla "sudo du -sh /var/lib/docker /var/lib/containerd /var/log 2>/dev/null"
|
||||
@@ -135,13 +132,24 @@ ssh -o BatchMode=yes ginnoir@valhalla "sudo du -sh /var/lib/docker /var/lib/cont
|
||||
This is the directory that caused the 2026-07-07 outage (all Postgres instances crash-looped on
|
||||
"No space left on device"). It should now be well under 60G.
|
||||
|
||||
- [ ] **Step 3: Record findings**
|
||||
- [x] **Step 3: Record findings**
|
||||
|
||||
Append the output to this task as a comment. You are looking for anything **structurally
|
||||
growing** — not one-off large files. If you find a second unbounded growth source, add a task
|
||||
to this plan to bound it before continuing.
|
||||
**Audit findings (2026-08-05, post-prune):**
|
||||
|
||||
- [ ] **Step 4: Commit the findings**
|
||||
| Path | Size | Notes |
|
||||
|---|---|---|
|
||||
| `/var/lib/containerd` | 61G | Image store — primary consumer; now healthy post-prune |
|
||||
| `/var/lib/docker` | 54G (top-level) / ~9G (du -x under /var/lib) | Overlay/metadata; paired with containerd |
|
||||
| `/config` | 20G | App configs; largest: kopia 5.4G, nextcloud 2.4G, owncloud 2.0G |
|
||||
| `/home/ginnoir` | 9.6G | `.cache` 4.9G, `.hermes` 2.6G — bounded personal tooling |
|
||||
| `/usr` | 9.6G | OS packages |
|
||||
| `/var/lib/plex-local` | 6.4G | Leftover local Plex data (symlink target is on /storage1) |
|
||||
| `/var/log` | 3.0G | Acceptable |
|
||||
|
||||
No second unbounded growth source found beyond Docker images (already pruned + Task 3 alerts).
|
||||
`plex-local` 6.4G is a one-off reclaim candidate at decommission, not structural growth.
|
||||
|
||||
- [x] **Step 4: Commit the findings**
|
||||
|
||||
```bash
|
||||
git add docs/superpowers/plans/2026-08-05-jellyfin-migration.md
|
||||
|
||||
Reference in New Issue
Block a user