ginnoir
dd26aafc42
Enrich the Obsidian ROM-hack vault from the valhalla Discord scrape
(/storage1/labdata/romhacks/metadata) plus web research, growing the catalog
from 285 to 368 notes and preparing a data export for a wiki site.
Tooling (all dry-run by default, --apply to write; .scrape/ is gitignored):
- scrape_match.py reconcile scrape entries vs notes (43 overlap / 211 match / 83 new)
- enrich_vault.py backfill frontmatter, merge scrape into curated notes,
create 83 new notes, rebuild import bodies to one layout
- enrich_web.py apply hand/web-verified facts from .scrape/web_facts*.json
- build-wiki-data.py export wiki/catalog.json (the site data source)
Results: 368 notes, 327 with served banner art, 332 rich summaries, 200 with
feature lists; new frontmatter developer/release_date/banner/homepage/scrape_dir;
12 flagship hacks web-verified. wiki/SPEC.md describes the site build.
build-vault-mocs.py Index callout updated for the new counts.
4.1 KiB
Pokémon ROM Hacks — tooling
Acquisition + patching pipeline for the Pokémon ROM-hack collection in the
valhalla homelab library. Spun off from the homelabstack repo on 2026-06-06.
The catalog (the notes) lives in the self-hosted Obsidian vault, not in this repo — under
Pokémon ROM Hacks/: one note per hack, plusIndex,Types,Play Queue,Bases/,Platforms/, aWantedacquisition list, and a_Claude.mdops reference. Access it via the Obsidian MCP (mcp__obsidian__*) or thevaultskill. New hacks are added there manually (see the vault's_Claude.mdchecklist). This repo is just the tooling.
The ROMs live on valhalla under /storage1/Emulation/roms/<platform>/Hacks/,
served by the RomM stack (roms.ginnoir.com) — deployed from homelabstack
(stacks/roms/).
Layout
| Path | What |
|---|---|
scripts/romhack-import.py |
Import a drop folder of mixed ROMs/patches/docs into the valhalla library (validate, clean-name, apply patches, archive). Runs on valhalla. |
scripts/romhack-fetch.py |
Acquire + patch + place worthwhile hacks from direct (non-gated) sources. Manifest-driven. Runs on valhalla. |
scripts/romhack-apply.py |
Apply a single IPS/BPS/UPS patch to a base ROM with checksum verification. |
scripts/build-vault-mocs.py |
Regenerate the vault's static MOC tables + header counts (Index/Platforms/Bases) from the notes. Read-only on Hacks/*.md. Re-run after any note change. |
scripts/scrape_match.py |
Match the Discord-scrape metadata.json entries to vault notes → .scrape/_match_report.json (curated-overlap / discord-match / unmatched). |
scripts/enrich_vault.py |
Synthesize the scrape into notes: backfill (frontmatter for imports), merge (scrape into curated), create (new notes), rebuild (normalize import bodies). --apply writes; default dry-runs to .scrape/_preview/. |
scripts/enrich_web.py |
Apply hand/web-researched facts from .scrape/web_facts*.json onto notes (summary, features, dev, version, notability), idempotently. |
scripts/build-wiki-data.py |
Export all notes → wiki/catalog.json, the data source for the wiki site (see wiki/SPEC.md). |
scripts/build-romhack-vault.py |
Archived. Original 75-hack bootstrap generator — do not re-run against the live vault. Kept only for its structured HACKS dataset. |
wiki/ |
catalog.json (exported 368-hack dataset) + SPEC.md (build spec for the future static wiki site). |
drop/ |
gitignored staging area (~1.3 GB) of ROM/patch binaries pending import. |
.scrape/ |
gitignored per-machine scratch: Discord-scrape metadata.json pulled from valhalla + match report + web_facts*.json + note backups. |
Enrichment pipeline
The vault catalog (368 notes) was enriched from the Discord scrape on valhalla
(/storage1/labdata/romhacks/metadata/*/metadata.json) plus web research. To
re-run or extend:
# 1. pull the scrape metadata locally (one-time / refresh)
ssh valhalla "cd /storage1/labdata/romhacks/metadata && tar cf - */metadata.json" | tar xf - -C .scrape
# 2. match scrape -> notes
python scripts/scrape_match.py --report .scrape/_match_report.json
# 3. synthesize (drop --apply first to preview into .scrape/_preview/)
python scripts/enrich_vault.py backfill --apply
python scripts/enrich_vault.py rebuild --apply
python scripts/enrich_vault.py merge --apply
python scripts/enrich_vault.py create --apply
# 4. web-enrich more hacks: add a .scrape/web_factsN.json batch, then
python scripts/enrich_web.py --apply
# 5. refresh static surfaces + the site data
python scripts/build-vault-mocs.py
python scripts/build-wiki-data.py
Art for every scraped hack is web-served at
https://romhacks-files.ginnoir.com/_meta/<scrape_dir>/<file> (copied into the
Caddy-served library/_meta/ tree on valhalla).
Acquisition policy
Pre-3DS handhelds (gb/gbc/gba/nds) + pre-Wii consoles. Direct, non-anti-bot
sources only for auto-fetch; gated hacks are tracked in the vault's
Pokémon ROM Hacks/Wanted.md for manual hand-off. No offensive/"shitpost"
content (e.g. Clover excluded).