homelabstack/stacks/romhacks/README.md

romhacks stack

Automated ingestion of Pokémon rom-hacks from a Discord server's per-generation threads into the RomM library.

romhacks (orchestrator)              jdownloader (jlesage)        RomM (roms stack)
  DiscordChatExporter  ── JSON ──▶   FolderWatch reads .crawljob,   scans
  parse links + art    ── crawljob ▶ resolves ~all hosts,   ──────▶ /storage1/Emulation
  stage metadata       ── art ─────▶ downloads to /output

State: INERT

Ships idle and safe. It will not export or download until armed:

• DISCORD_TOKEN / GUILD_ID empty → orchestrator loop idles (see run.sh).
• DRY_RUN=true → even configured, it parses + stages metadata but writes no .crawljob, so JDownloader fetches nothing.

Why this design

• Member-only Discord access → no official bot possible. DiscordChatExporter (DCE) does a read-only export with a user token. That violates Discord ToS, so this is wired for an alt account (joined to the server purely for this) to keep any ban risk off the main account.
• Links span many file hosts → JDownloader2 auto-resolves hundreds of hosts and handles waits/captcha via its noVNC UI; far more robust than per-host glue.
• Art + blurb → DCE --media downloads inline thread images; the orchestrator stages them with the thread name/description for a later RomM custom-cover hand-off (pairs with the deck-sgdb art pipeline).

Arming checklist

1. Alt account: create it, join the server, grab its user token → DISCORD_TOKEN in stack.env. Set GUILD_ID (right-click server → Copy Server ID; Developer Mode on).
2. Channels: fill orchestrator/channels.json with the per-generation channel (or forum) IDs. Add any new file hosts you see to hosts.
3. Register the stack in Portainer once (new git stacks need a one-time registration — see memory portainer-new-stack-registration). Portainer does not build anything: the orchestrator image is built by Gitea Actions and pulled from registry.ginnoir.com/ginnoir/romhacks-orchestrator:latest.
4. JDownloader FolderWatch: open the noVNC UI at http://valhalla:8998 (set VNC_PASSWORD first), Settings → Extensions → Folder Watch → enable, add watch folder /watch. This is what consumes the orchestrator's crawljobs.
5. Dry run: leave DRY_RUN=true, let one cycle run, inspect /storage1/labdata/romhacks/metadata/* — confirm names, blurbs, art, and that the extracted links look like real ROM links (not random URLs).
6. Arm: flip DRY_RUN=false. Downloads land in /storage1/labdata/romhacks/incoming/<game>/.
7. Caddy (optional, internal-only) — expose the JD UI on the LAN:

   jd.ginnoir.com {
       import internal_only
       reverse_proxy jdownloader:5800
   }
   

   then ./scripts/gen-bookmarks.ps1 and push (runner reloads Caddy).
8. RomM hand-off (next phase, not built yet): sort incoming/<game> into the right /storage1/Emulation/<platform> folder and push the staged art to RomM's custom-cover API.

Editing config

channels.json and the scripts are baked into the image (Portainer's git-stack checkout doesn't reliably bind sibling repo files — see the compose header). So a change under orchestrator/ needs a push, which fires .gitea/workflows/build-romhacks-orchestrator.yml: it builds the image and pushes registry.ginnoir.com/ginnoir/romhacks-orchestrator as :latest + :sha-<short>. Portainer then pulls :latest on its next 5-min poll (pull_policy: always).

Compose has no build: on purpose — Portainer's auto-update runs docker compose pull and never build, so a build-only image name deadlocks the stack. Don't add one back.

To rebuild without a source change, run the workflow manually (Gitea → the repo → Actions → Build romhacks orchestrator image → Run workflow), then Portainer → the stack → Re-pull and redeploy.

Paths

Host path	Purpose
/config/romhacks/exports	DCE JSON + downloaded media (SSD)
/config/romhacks/state	processed-message-id dedupe (SSD)
/config/romhacks/crawljobs	orchestrator → JDownloader FolderWatch (/watch)
/config/romhacks/jd	JDownloader config (SSD)
/storage1/labdata/romhacks/incoming	JDownloader downloads (ZFS)
/storage1/labdata/romhacks/metadata	staged name/blurb/art per game (ZFS)

Gotchas

• ToS / ban risk is real but low for read-only export. Use the alt account.
• Pre-patched ROMs vs patches: bundled base-game ROMs are piracy; IPS/BPS patches are broadly tolerated. Prefer patch links where the thread offers both.
• DCE flags / dll path are pinned in orchestrate.py; verify on first run (the code globs for the dll if the pinned path moves between image versions).