feat(vault): synthesize Discord scrape + web research into the ROM-hack catalog
Enrich the Obsidian ROM-hack vault from the valhalla Discord scrape
(/storage1/labdata/romhacks/metadata) plus web research, growing the catalog
from 285 to 368 notes and preparing a data export for a wiki site.
Tooling (all dry-run by default, --apply to write; .scrape/ is gitignored):
- scrape_match.py reconcile scrape entries vs notes (43 overlap / 211 match / 83 new)
- enrich_vault.py backfill frontmatter, merge scrape into curated notes,
create 83 new notes, rebuild import bodies to one layout
- enrich_web.py apply hand/web-verified facts from .scrape/web_facts*.json
- build-wiki-data.py export wiki/catalog.json (the site data source)
Results: 368 notes, 327 with served banner art, 332 rich summaries, 200 with
feature lists; new frontmatter developer/release_date/banner/homepage/scrape_dir;
12 flagship hacks web-verified. wiki/SPEC.md describes the site build.
build-vault-mocs.py Index callout updated for the new counts.
This commit is contained in:
@@ -21,8 +21,46 @@ served by the RomM stack (`roms.ginnoir.com`) — deployed from `homelabstack`
|
||||
| `scripts/romhack-import.py` | Import a drop folder of mixed ROMs/patches/docs into the valhalla library (validate, clean-name, apply patches, archive). Runs on valhalla. |
|
||||
| `scripts/romhack-fetch.py` | Acquire + patch + place worthwhile hacks from direct (non-gated) sources. Manifest-driven. Runs on valhalla. |
|
||||
| `scripts/romhack-apply.py` | Apply a single IPS/BPS/UPS patch to a base ROM with checksum verification. |
|
||||
| `scripts/build-romhack-vault.py` | **Archived.** Original generator that bootstrapped the vault notes from an in-script dataset. The vault has since been enriched and is now hand-maintained via the Obsidian MCP — do **not** re-run it against the live vault (its output is poorer: no `play_status`/`rating`/`Bases`/`Play Queue`). Kept only for the structured `HACKS` dataset; its output dir is gitignored. |
|
||||
| `scripts/build-vault-mocs.py` | Regenerate the vault's static MOC tables + header counts (Index/Platforms/Bases) from the notes. Read-only on `Hacks/*.md`. Re-run after any note change. |
|
||||
| `scripts/scrape_match.py` | Match the Discord-scrape `metadata.json` entries to vault notes → `.scrape/_match_report.json` (curated-overlap / discord-match / unmatched). |
|
||||
| `scripts/enrich_vault.py` | Synthesize the scrape into notes: `backfill` (frontmatter for imports), `merge` (scrape into curated), `create` (new notes), `rebuild` (normalize import bodies). `--apply` writes; default dry-runs to `.scrape/_preview/`. |
|
||||
| `scripts/enrich_web.py` | Apply hand/web-researched facts from `.scrape/web_facts*.json` onto notes (summary, features, dev, version, notability), idempotently. |
|
||||
| `scripts/build-wiki-data.py` | Export all notes → `wiki/catalog.json`, the data source for the wiki site (see `wiki/SPEC.md`). |
|
||||
| `scripts/build-romhack-vault.py` | **Archived.** Original 75-hack bootstrap generator — do **not** re-run against the live vault. Kept only for its structured `HACKS` dataset. |
|
||||
| `wiki/` | `catalog.json` (exported 368-hack dataset) + `SPEC.md` (build spec for the future static wiki site). |
|
||||
| `drop/` | **gitignored** staging area (~1.3 GB) of ROM/patch binaries pending import. |
|
||||
| `.scrape/` | **gitignored** per-machine scratch: Discord-scrape `metadata.json` pulled from valhalla + match report + `web_facts*.json` + note backups. |
|
||||
|
||||
## Enrichment pipeline
|
||||
|
||||
The vault catalog (368 notes) was enriched from the Discord scrape on valhalla
|
||||
(`/storage1/labdata/romhacks/metadata/*/metadata.json`) plus web research. To
|
||||
re-run or extend:
|
||||
|
||||
```bash
|
||||
# 1. pull the scrape metadata locally (one-time / refresh)
|
||||
ssh valhalla "cd /storage1/labdata/romhacks/metadata && tar cf - */metadata.json" | tar xf - -C .scrape
|
||||
|
||||
# 2. match scrape -> notes
|
||||
python scripts/scrape_match.py --report .scrape/_match_report.json
|
||||
|
||||
# 3. synthesize (drop --apply first to preview into .scrape/_preview/)
|
||||
python scripts/enrich_vault.py backfill --apply
|
||||
python scripts/enrich_vault.py rebuild --apply
|
||||
python scripts/enrich_vault.py merge --apply
|
||||
python scripts/enrich_vault.py create --apply
|
||||
|
||||
# 4. web-enrich more hacks: add a .scrape/web_factsN.json batch, then
|
||||
python scripts/enrich_web.py --apply
|
||||
|
||||
# 5. refresh static surfaces + the site data
|
||||
python scripts/build-vault-mocs.py
|
||||
python scripts/build-wiki-data.py
|
||||
```
|
||||
|
||||
Art for every scraped hack is web-served at
|
||||
`https://romhacks-files.ginnoir.com/_meta/<scrape_dir>/<file>` (copied into the
|
||||
Caddy-served `library/_meta/` tree on valhalla).
|
||||
|
||||
## Acquisition policy
|
||||
|
||||
|
||||
Reference in New Issue
Block a user