feat(romhacks): add Discord rom-hack ingestion stack (DCE -> JDownloader -> RomM)
Inert by default: empty token keeps the orchestrator idle and DRY_RUN=true suppresses downloads. Orchestrator exports per-generation forum threads via DiscordChatExporter, extracts download links + art, stages metadata, and writes JDownloader crawljobs. Armed here with an alt account token and all 11 romhack forums, still in DRY_RUN pending first-cycle validation.
This commit is contained in:
ginnoir
@@ -0,0 +1,86 @@
|
||||
# romhacks stack
|
||||
|
||||
Automated ingestion of Pokémon rom-hacks from a Discord server's per-generation
|
||||
threads into the RomM library.
|
||||
|
||||
```
|
||||
romhacks (orchestrator) jdownloader (jlesage) RomM (roms stack)
|
||||
DiscordChatExporter ── JSON ──▶ FolderWatch reads .crawljob, scans
|
||||
parse links + art ── crawljob ▶ resolves ~all hosts, ──────▶ /storage1/Emulation
|
||||
stage metadata ── art ─────▶ downloads to /output
|
||||
```
|
||||
|
||||
## State: INERT
|
||||
|
||||
Ships idle and safe. It will not export or download until armed:
|
||||
- `DISCORD_TOKEN` / `GUILD_ID` empty → orchestrator loop idles (see `run.sh`).
|
||||
- `DRY_RUN=true` → even configured, it parses + stages metadata but writes **no**
|
||||
`.crawljob`, so JDownloader fetches nothing.
|
||||
|
||||
## Why this design
|
||||
|
||||
- **Member-only Discord access** → no official bot possible. DiscordChatExporter
|
||||
(DCE) does a **read-only** export with a **user token**. That violates Discord
|
||||
ToS, so this is wired for an **alt account** (joined to the server purely for
|
||||
this) to keep any ban risk off the main account.
|
||||
- **Links span many file hosts** → JDownloader2 auto-resolves hundreds of hosts
|
||||
and handles waits/captcha via its noVNC UI; far more robust than per-host glue.
|
||||
- **Art + blurb** → DCE `--media` downloads inline thread images; the orchestrator
|
||||
stages them with the thread name/description for a later RomM custom-cover
|
||||
hand-off (pairs with the `deck-sgdb` art pipeline).
|
||||
|
||||
## Arming checklist
|
||||
|
||||
1. **Alt account**: create it, join the server, grab its user token → `DISCORD_TOKEN`
|
||||
in `stack.env`. Set `GUILD_ID` (right-click server → Copy Server ID; Developer
|
||||
Mode on).
|
||||
2. **Channels**: fill `orchestrator/channels.json` with the per-generation channel
|
||||
(or forum) IDs. Add any new file hosts you see to `hosts`.
|
||||
3. **Register the stack** in Portainer once (new git stacks need a one-time
|
||||
registration — see memory `portainer-new-stack-registration`). Build is on:
|
||||
Portainer builds `homelab/romhacks-orchestrator` from `./orchestrator`.
|
||||
4. **JDownloader FolderWatch**: open the noVNC UI at `http://valhalla:8998`
|
||||
(set `VNC_PASSWORD` first), Settings → Extensions → **Folder Watch** → enable,
|
||||
add watch folder `/watch`. This is what consumes the orchestrator's crawljobs.
|
||||
5. **Dry run**: leave `DRY_RUN=true`, let one cycle run, inspect
|
||||
`/storage1/labdata/romhacks/metadata/*` — confirm names, blurbs, art, and that
|
||||
the extracted `links` look like real ROM links (not random URLs).
|
||||
6. **Arm**: flip `DRY_RUN=false`. Downloads land in
|
||||
`/storage1/labdata/romhacks/incoming/<game>/`.
|
||||
7. **Caddy (optional, internal-only)** — expose the JD UI on the LAN:
|
||||
```
|
||||
jd.ginnoir.com {
|
||||
import internal_only
|
||||
reverse_proxy jdownloader:5800
|
||||
}
|
||||
```
|
||||
then `./scripts/gen-bookmarks.ps1` and push (runner reloads Caddy).
|
||||
8. **RomM hand-off** (next phase, not built yet): sort `incoming/<game>` into the
|
||||
right `/storage1/Emulation/<platform>` folder and push the staged art to RomM's
|
||||
custom-cover API.
|
||||
|
||||
## Editing config
|
||||
|
||||
`channels.json` and the scripts are **baked into the image** (Portainer's git-stack
|
||||
checkout doesn't reliably bind sibling repo files — see the compose header). So a
|
||||
change there needs a **push + rebuild**; if a content-only edit doesn't trigger a
|
||||
rebuild, use Portainer → the stack → **Re-pull and redeploy**.
|
||||
|
||||
## Paths
|
||||
|
||||
| Host path | Purpose |
|
||||
|---|---|
|
||||
| `/config/romhacks/exports` | DCE JSON + downloaded media (SSD) |
|
||||
| `/config/romhacks/state` | processed-message-id dedupe (SSD) |
|
||||
| `/config/romhacks/crawljobs` | orchestrator → JDownloader FolderWatch (`/watch`) |
|
||||
| `/config/romhacks/jd` | JDownloader config (SSD) |
|
||||
| `/storage1/labdata/romhacks/incoming` | JDownloader downloads (ZFS) |
|
||||
| `/storage1/labdata/romhacks/metadata` | staged name/blurb/art per game (ZFS) |
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **ToS / ban risk** is real but low for read-only export. Use the alt account.
|
||||
- **Pre-patched ROMs vs patches**: bundled base-game ROMs are piracy; IPS/BPS
|
||||
patches are broadly tolerated. Prefer patch links where the thread offers both.
|
||||
- DCE flags / dll path are pinned in `orchestrate.py`; verify on first run (the
|
||||
code globs for the dll if the pinned path moves between image versions).
|
||||
Reference in New Issue
Block a user