ginnoir/homelabstack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add R730XD post-migration task plan

Browse Source
This commit is contained in:
ginnoir
2026-06-27 05:05:30 -05:00
parent 6cef600d25
commit b7ff9b2411
1 changed files with 380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-06-26-r730xd-post-migration-tasks.md
+380
View File
@@ -0,0 +1,380 @@
# R730XD Post-Migration Host Tasks — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Close out the five remaining post-migration items on the Proxmox host `valhalla-dev`: a permanent fix for virtiofsd FD exhaustion, the hookscript bug, GPU thermal safety, container GPU access, and retiring the last failing pool disk.
**Architecture:** These are five **independent** host-ops tasks against a live single-host Proxmox box (no application code, no test suite). Each is self-contained and can be executed in any order, with two dependencies noted below. "Tests" here are verification commands with expected output; "rollback" replaces "revert commit." Most changes are on the Proxmox host (`.68`) or the `valhalla` VM (`.69`), neither of which is tracked in this git repo — persistence is via on-host files + the vault migration note `[[Server Migration (Proxmox R730XD)]]`.
**Tech Stack:** Proxmox VE (Debian) host, `virtiofsd` 1.13.2, ZFS (`storage1` stripe), EndeavourOS guest VM (id 100), NVIDIA Tesla P100 + driver 580.159.04 + nvidia-ctk 1.19.1, systemd, ipmitool fan control.
## Access & conventions (read first)
- **Host (`valhalla-dev`, Proxmox):** `wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "<cmd>"'` — the **WSL** `id_ed25519` (`ginnoir@TELLUS`) key is authorized for root.
- **Guest (`valhalla` VM):** `wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 ginnoir@192.168.1.69 "<cmd>"'` (sudo is NOPASSWD).
- For multi-line remote scripts use the heredoc form: `wsl.exe -- bash -lc 'ssh ... root@192.168.1.68 bash -s' <<'EOF' … EOF` (avoids shell-variable mangling through the WSL→ssh layers).
- **GateGuard:** the first `Bash` and every `Write`/`Edit` requires stating the user request + what the op does/affects before it runs.
## Recommended sequence & dependencies
1. **Task 1** (virtiofsd `--inode-file-handles`) and **Task 2** (hookscript fix) are both the FD-exhaustion fix — do them together. Task 1 is the real fix; Task 2 repairs the fallback. Both require a VM restart, so batch them into one VM bounce.
2. **Task 3** (GPU fan safety) is a **hard prerequisite** for **Task 4** (container GPU) being *used* under load — never run a sustained CUDA job until Task 3 is live (passive P100 + Dell-auto fans that can't see GPU temp = overheat risk).
3. **Task 5** (replace `sdb`) is fully independent and starts a ~1–1.5 day resilver — run it when power is stable.
---
## Task 1: Permanent virtiofsd FD fix — `--inode-file-handles=prefer`
**Problem:** virtiofsd holds one `O_PATH` file descriptor per inode the guest touches on `/storage1`. With ~78 TB of media scanned by *arr/Plex it climbed to ~1,000,000 open FDs and the guest got `Too many open files` on new `/storage1` access. `--inode-file-handles=prefer` makes virtiofsd use `name_to_handle_at`/`open_by_handle_at` instead, keeping near-zero FDs. Proxmox 9 has **no config option** to pass this flag, so we inject it via a `dpkg-divert` wrapper around the virtiofsd binary (survives package updates).
**Targets (host `.68`):**
- Divert: `/usr/libexec/virtiofsd` → `/usr/libexec/virtiofsd.real`
- Create: `/usr/libexec/virtiofsd` (wrapper script)
- Restart: VM 100 (cold)
- [ ] **Step 1: Baseline — capture the current launch args and FD count**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
echo "=== current virtiofsd cmdline ==="
for p in $(pgrep virtiofsd); do tr '\0' ' ' </proc/$p/cmdline; echo; done
echo "=== worker FD count (the one that climbs) ==="
for p in $(pgrep virtiofsd); do echo "pid $p fds=$(ls /proc/$p/fd 2>/dev/null | wc -l)"; done
EOF
```
Expected: two virtiofsd processes, args include `--shared-dir=/storage1 --xattr`; worker FD count is whatever it has grown to (could be tens of thousands+). Record it.
- [ ] **Step 2: Confirm the binary supports the flag**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "/usr/libexec/virtiofsd --help 2>&1 | grep -- --inode-file-handles"'
```
Expected: a line containing `--inode-file-handles=<INODE_FILE_HANDLES>`. (Confirmed present on 1.13.2.)
- [ ] **Step 3: Create the diversion and the wrapper**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
set -e
# Move the real binary aside, persistently across apt upgrades
dpkg-divert --add --rename --divert /usr/libexec/virtiofsd.real /usr/libexec/virtiofsd
# Write a wrapper that injects the flag, then execs the real binary with all original args
cat > /usr/libexec/virtiofsd <<'WRAP'
#!/bin/bash
# Proxmox-launched virtiofsd wrapper: force inode-file-handles to stop FD growth.
# See docs/superpowers/plans/2026-06-26-r730xd-post-migration-tasks.md Task 1.
exec /usr/libexec/virtiofsd.real --inode-file-handles=prefer "$@"
WRAP
chmod 0755 /usr/libexec/virtiofsd
echo "=== verify ==="
dpkg-divert --list /usr/libexec/virtiofsd
ls -l /usr/libexec/virtiofsd /usr/libexec/virtiofsd.real
EOF
```
Expected: diversion listed (`diversion of /usr/libexec/virtiofsd to /usr/libexec/virtiofsd.real`), wrapper is a 0755 regular file, `.real` is the original ELF binary.
- [ ] **Step 4: Cold-restart VM 100 so a new virtiofsd launches via the wrapper**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
qm shutdown 100 --timeout 180 && qm status 100
qm start 100 && qm status 100
EOF
```
Expected: `status: stopped` then `status: running`.
- [ ] **Step 5: VERIFY — new virtiofsd carries the flag and FDs stay low**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
echo "=== flag present in launch args? ==="
for p in $(pgrep virtiofsd); do tr '\0' ' ' </proc/$p/cmdline | grep -o -- '--inode-file-handles=prefer' && echo " (pid $p OK)"; done
echo "=== /storage1 still works in guest ==="
ssh -o BatchMode=yes -o ConnectTimeout=8 -i /root/.ssh/id_ed25519 ginnoir@192.168.1.69 'ls /storage1 >/dev/null 2>&1 && echo storage1-OK || echo storage1-FAIL' 2>/dev/null || echo "(set up host->guest key per Task 3, or check from the admin box)"
EOF
```
Expected: `--inode-file-handles=prefer (pid … OK)` for the worker. If the host→guest key isn't set yet, verify `/storage1` from the admin box instead: `wsl.exe -- bash -lc 'ssh ... ginnoir@192.168.1.69 "ls /storage1 >/dev/null && echo OK"'`.
- [ ] **Step 6: Soak check — FD count after the library gets re-scanned**
Re-run Step 1's FD-count command after a Plex/*arr scan cycle (or ~an hour of normal use). Expected: worker FD count stays in the **hundreds/low-thousands**, not climbing toward 1M. This confirms the fix.
**Rollback:**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
rm -f /usr/libexec/virtiofsd
dpkg-divert --remove --rename /usr/libexec/virtiofsd
qm shutdown 100 --timeout 180 && qm start 100
EOF
```
**Persist:** copy the wrapper to this repo under `docs/host/usr-libexec-virtiofsd.wrapper` (reference only) and note the diversion in the vault migration page.
---
## Task 2: Fix the `virtiofsd-limits.sh` hookscript bug
**Problem:** `qm start 100` logs `hookscript error for 100 on post-start: /var/lib/vz/snippets/virtiofsd-limits.sh: line 19: exit: 0: numeric argument required`. Line 19 is `exit 0` — the "0 not numeric" error means a stray carriage return (`\r`) is attached (`exit $'0\r'`). The script is the FD-limit fallback (raises virtiofsd nofile to 10M); after Task 1 it's a belt-and-suspenders safety net, but it should run cleanly.
**Targets (host `.68`):**
- Rewrite: `/var/lib/vz/snippets/virtiofsd-limits.sh` (clean LF, correct content)
- [ ] **Step 1: Confirm the CRLF/stray-char hypothesis**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "file /var/lib/vz/snippets/virtiofsd-limits.sh; grep -c $'"'"'\r'"'"' /var/lib/vz/snippets/virtiofsd-limits.sh"'
```
Expected: `file` reports `... with CRLF line terminators` and/or a non-zero `\r` count. (If zero, the bug is a different stray char — proceed to Step 2 anyway; the rewrite fixes it regardless.)
- [ ] **Step 2: Rewrite the script cleanly (LF only)**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
cp /var/lib/vz/snippets/virtiofsd-limits.sh /var/lib/vz/snippets/virtiofsd-limits.sh.bak
cat > /var/lib/vz/snippets/virtiofsd-limits.sh <<'SCRIPT'
#!/bin/bash
# Fallback: raise virtiofsd open-file limit after VM 100 starts.
# Primary fix is the --inode-file-handles=prefer wrapper (Task 1); this is a safety net.
VMID="$1"
PHASE="$2"
[ "$VMID" = "100" ] && [ "$PHASE" = "post-start" ] || exit 0
sleep 3
pgrep virtiofsd | while read -r pid; do
prlimit --pid "$pid" --nofile=10000000:10000000 \
&& logger -t virtiofsd-limits "raised nofile for PID $pid to 10M"
done
exit 0
SCRIPT
chmod 0755 /var/lib/vz/snippets/virtiofsd-limits.sh
# guarantee no CRLF crept back in
sed -i 's/\r$//' /var/lib/vz/snippets/virtiofsd-limits.sh
echo "=== syntax check ==="
bash -n /var/lib/vz/snippets/virtiofsd-limits.sh && echo "syntax OK"
file /var/lib/vz/snippets/virtiofsd-limits.sh
EOF
```
Expected: `syntax OK` and `file` reports a plain `Bourne-Again shell script, ASCII text executable` (no CRLF).
- [ ] **Step 3: VERIFY — dry-run the hook and confirm no error**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "/var/lib/vz/snippets/virtiofsd-limits.sh 100 post-start; echo exit=$?"'
```
Expected: `exit=0`, no `numeric argument required` error. (It will also bump the live virtiofsd nofile to 10M — harmless.)
- [ ] **Step 4: VERIFY at next real start (do this opportunistically with Task 1's restart)**
After any `qm start 100`, check the journal:
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "journalctl -u qemu-server@100 -n 30 --no-pager | grep -iE \"hookscript|virtiofsd-limits\" | tail"'
```
Expected: no `hookscript error`; a `virtiofsd-limits: raised nofile …` logger line.
**Rollback:** `cp /var/lib/vz/snippets/virtiofsd-limits.sh.bak /var/lib/vz/snippets/virtiofsd-limits.sh`
---
## Task 3: GPU thermal safety — wire P100 temp into the fan daemon
**Problem:** The P100 is passively cooled. Fans are on **Dell-auto, which cannot read the GPU's temperature** (non-Dell card), and the custom `valhalla-thermal-monitor.service` (the CPU/GPU-reactive controller with a watchdog→Dell-auto failsafe) is currently **disabled** (shut off during the June panics). The daemon already supports GPU temp via a `VM_SSH` env var (`gpu_max()` runs `$VM_SSH 'nvidia-smi --query-gpu=temperature.gpu …'`), but `VM_SSH` is unset → `gpu=off`. We set up host→guest SSH, point `VM_SSH` at it, and re-enable the daemon so GPU temp drives the fans. **This must be live before any sustained GPU workload.**
**Targets:**
- Create (host): `/root/.ssh/id_ed25519` keypair (if absent) for host→guest SSH
- Modify (guest `.69`): `~ginnoir/.ssh/authorized_keys` (append host pubkey)
- Modify (host): `valhalla-thermal-monitor.service` drop-in with `Environment=VM_SSH=…`
- Enable + start: `valhalla-thermal-monitor.service`
- [ ] **Step 1: Generate a host→guest SSH key (host side)**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
[ -f /root/.ssh/id_ed25519 ] || ssh-keygen -t ed25519 -N "" -C "valhalla-dev-host->vm" -f /root/.ssh/id_ed25519
echo "=== host pubkey ==="
cat /root/.ssh/id_ed25519.pub
EOF
```
Expected: prints an `ssh-ed25519 … valhalla-dev-host->vm` pubkey. Copy this line.
- [ ] **Step 2: Authorize that key on the guest (bridge via the admin/WSL key)**
Substitute `<HOST_PUBKEY>` with the line from Step 1:
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 ginnoir@192.168.1.69 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && grep -qxF \"<HOST_PUBKEY>\" ~/.ssh/authorized_keys 2>/dev/null || echo \"<HOST_PUBKEY>\" >> ~/.ssh/authorized_keys; chmod 600 ~/.ssh/authorized_keys; echo done"'
```
Expected: `done`.
- [ ] **Step 3: VERIFY host→guest SSH + nvidia-smi works non-interactively**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "ssh -o BatchMode=yes -o ConnectTimeout=5 -i /root/.ssh/id_ed25519 ginnoir@192.168.1.69 \"nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits\""'
```
Expected: a bare integer like `56`. If it errors, fix the key before continuing — do not enable the daemon without a working GPU temp source.
- [ ] **Step 4: Set `VM_SSH` on the service via a drop-in**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
mkdir -p /etc/systemd/system/valhalla-thermal-monitor.service.d
cat > /etc/systemd/system/valhalla-thermal-monitor.service.d/gpu.conf <<'DROP'
[Service]
Environment=VM_SSH=ssh -o BatchMode=yes -o ConnectTimeout=5 -i /root/.ssh/id_ed25519 ginnoir@192.168.1.69
DROP
systemctl daemon-reload
echo "=== drop-in ==="; systemctl cat valhalla-thermal-monitor.service | grep -i VM_SSH
EOF
```
Expected: the `Environment=VM_SSH=…` line echoed back.
- [ ] **Step 5: Enable + start the daemon**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "systemctl enable --now valhalla-thermal-monitor.service; systemctl is-active valhalla-thermal-monitor.service"'
```
Expected: `active`.
- [ ] **Step 6: VERIFY — GPU temp is in the curve, and the failsafe is intact**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "journalctl -t thermal-monitor -n 8 --no-pager"'
```
Expected: lines like `cpu=NNC gpu=NNC -> fan NN%` with a **real number** for `gpu=` (not `na`), and the startup line shows `gpu=on`.
- [ ] **Step 7: VERIFY the stop/crash failsafe returns fans to Dell-auto**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "systemctl cat valhalla-thermal-monitor.service | grep -iE \"ExecStopPost|trap\"; echo --- ; grep -nE \"0x30 0x30 0x01 0x01|trap\" /usr/local/sbin/valhalla-thermal-monitor.sh"'
```
Expected: confirms an `ExecStopPost`/`trap` that issues `ipmitool raw 0x30 0x30 0x01 0x01` (return to Dell auto). This was previously proven; just re-confirm it's present.
**Rollback:**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "systemctl disable --now valhalla-thermal-monitor.service; rm -f /etc/systemd/system/valhalla-thermal-monitor.service.d/gpu.conf; systemctl daemon-reload; ipmitool raw 0x30 0x30 0x01 0x01"'
```
(The final `ipmitool` call forces fans back to Dell auto.)
---
## Task 4: Container GPU access (nvidia-ctk + CDI) in the guest
**Problem:** The guest has `nvidia-ctk` 1.19.1 and a working driver, but Docker has only the `runc` runtime and no CDI spec, so containers can't use the P100. Generate a CDI spec and enable CDI in Docker so workloads request the GPU with `--device nvidia.com/gpu=all`. **Prerequisite: Task 3 must be live before running any non-trivial GPU container.** Restarting Docker bounces all ~73 containers — do it in a maintenance window.
**Targets (guest `.69`):**
- Create: `/etc/cdi/nvidia.yaml` (CDI spec)
- Modify: `/etc/docker/daemon.json` (enable CDI feature)
- Restart: `docker` service
- [ ] **Step 1: Baseline — Docker version (CDI needs Engine ≥ 25) and current runtimes**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 ginnoir@192.168.1.69 "docker version --format \"{{.Server.Version}}\"; docker info 2>/dev/null | grep -iE \"Runtimes|Default Runtime\""'
```
Expected: a server version. If **≥ 25.0**, use CDI (Steps 2–5). If older, use the classic nvidia runtime instead (see Alternative below).
- [ ] **Step 2: Generate the CDI spec**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 ginnoir@192.168.1.69 "sudo mkdir -p /etc/cdi && sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml && sudo nvidia-ctk cdi list"'
```
Expected: `nvidia-ctk cdi list` shows `nvidia.com/gpu=all` (and `nvidia.com/gpu=0`).
- [ ] **Step 3: Enable the CDI feature in Docker**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 ginnoir@192.168.1.69 bash -s' <<'EOF'
sudo install -d /etc/docker
# merge "features.cdi=true" into daemon.json (create if absent)
sudo python3 - <<'PY'
import json,os
p="/etc/docker/daemon.json"
d=json.load(open(p)) if os.path.exists(p) and os.path.getsize(p) else {}
d.setdefault("features",{})["cdi"]=True
json.dump(d,open(p,"w"),indent=2)
print(open(p).read())
PY
EOF
```
Expected: prints a `daemon.json` containing `"features": { "cdi": true }`.
- [ ] **Step 4: Restart Docker (MAINTENANCE WINDOW — bounces all containers)**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 ginnoir@192.168.1.69 "sudo systemctl restart docker && sleep 20 && echo running:$(docker ps -q|wc -l)"'
```
Expected: Docker restarts; running container count climbs back toward ~72 over the next minute.
- [ ] **Step 5: VERIFY — a CUDA container sees the P100**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 ginnoir@192.168.1.69 "docker run --rm --device nvidia.com/gpu=all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi --query-gpu=name --format=csv,noheader"'
```
Expected: `Tesla P100-PCIE-16GB`. Watch the host thermal log (Task 3) during any longer test to confirm fans respond to GPU temp.
**Alternative (Docker < 25, classic runtime):**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 ginnoir@192.168.1.69 "sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker && docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -L"'
```
Expected: `GPU 0: Tesla P100-PCIE-16GB (UUID: …)`.
**Rollback:** remove `/etc/cdi/nvidia.yaml`, revert `daemon.json` (drop `features.cdi`), `sudo systemctl restart docker`.
---
## Task 5: Replace the last failing disk `sdb` with the staged 4 TB
**Problem:** `storage1` is still `DEGRADED` because `sdb` (4 TB WD Red, serial `WD-WCC4E4TSK1S2`, "too many errors") is failing. A new 4 TB disk (serial `V6HXW23W`) is already installed in the old Slot 2 (currently `sdc`, not in the pool). `zpool replace` swaps it in. This starts a **~1–1.5 day resilver** over a no-redundancy stripe — run only when power is stable, and keep `sdb` seated during the resilver (pulling it first loses its data).
**Targets (host `.68`):** `storage1` pool — no files.
- [ ] **Step 1: Resolve the new disk's stable by-id (do NOT use bare `sdc` — letters shuffle on reboot)**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "ls -l /dev/disk/by-id/ | grep -iE \"V6HXW23W\" | grep -vE \"part\""'
```
Expected: one or more by-id symlinks for serial `V6HXW23W` (e.g. `wwn-0x…` / `ata-…V6HXW23W`). Pick the `wwn-…` or `ata-…` whole-disk link → call it `$NEWDISK` (full path `/dev/disk/by-id/<that>`).
- [ ] **Step 2: Pre-flight — confirm target is `sdb` and new disk is unused**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 bash -s' <<'EOF'
echo "=== sdb serial (must be WD-WCC4E4TSK1S2) ==="; lsblk -dno NAME,SERIAL | awk '$1=="sdb"'
echo "=== new disk NOT in pool ==="; zpool status -P storage1 | grep -q "V6HXW23W" && echo "ALREADY IN POOL?!" || echo "new disk not in pool (good)"
echo "=== pool baseline ==="; zpool status storage1 | grep -E "state:|scan:"
EOF
```
Expected: `sdb` serial is `WD-WCC4E4TSK1S2`; new disk not in pool; no active scan.
- [ ] **Step 3: Kick off the replace** (substitute `$NEWDISK` from Step 1)
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "zpool replace storage1 sdb /dev/disk/by-id/<NEWDISK>; echo exit=$?; zpool status storage1 | grep -A4 replacing"'
```
Expected: `exit=0`; status now shows a `replacing-N` vdev with `sdb` (old) and the new disk `(resilvering)`.
- [ ] **Step 4: VERIFY at completion** (poll periodically — reuse the existing resilver-watch cadence)
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "zpool status storage1"'
```
Expected at completion: `scan: resilvered … with N errors`, the `replacing` vdev is gone, `sdb` is **evicted**, and the new disk is a normal member. Pool state should move toward `ONLINE` (it may still report the 7 pre-existing data errors — see Step 5).
- [ ] **Step 5: After resilver — clear stale errors if no live counters remain**
```bash
wsl.exe -- bash -lc 'ssh -o BatchMode=yes -i ~/.ssh/id_ed25519 root@192.168.1.68 "zpool status -v storage1"'
```
If only the 3 known media files remain and all device READ/WRITE/CKSUM are 0, optionally delete + re-grab those files via the *arr stack, then `zpool clear storage1` to reset the error state. Expected after clear + a scrub: `state: ONLINE`, `errors: No known data errors`.
- [ ] **Step 6: Physically remove the evicted `sdb`** (optional, later)
Use the SES locate-LED method to confirm the bay before pulling (host must be up): match `sdb`'s SAS address to its `sg_ses --dev-slot-num` and `sg_ses --dev-slot-num=<N> --set=ident /dev/sg12`. Verify the drive label reads `WD-WCC4E4TSK1S2` before full removal.
**Rollback (only before/early in resilver):** `zpool detach storage1 <NEWDISK>` cancels the replace and keeps the original `sdb` in service.
---
## Self-review notes
- **Coverage:** all five requested items have a task (fan safety = Task 3, container GPU = Task 4, hookscript = Task 2, `sdb` = Task 5, virtiofs permanent fix = Task 1). ✅
- **Dependencies flagged:** Task 3 before Task 4 under load; Tasks 1+2 share a VM restart.
- **Stable identifiers:** disk ops use serial/by-id, never bare `sdX` (letters shuffled across the week's reboots).
- **Each task has explicit verification + rollback.** No `zpool`/Docker/systemd change lacks a confirmation command.