feat(romhacks): add Discord rom-hack ingestion stack (DCE -> JDownloader -> RomM)

Inert by default: empty token keeps the orchestrator idle and DRY_RUN=true
suppresses downloads. Orchestrator exports per-generation forum threads via
DiscordChatExporter, extracts download links + art, stages metadata, and writes
JDownloader crawljobs. Armed here with an alt account token and all 11 romhack
forums, still in DRY_RUN pending first-cycle validation.
This commit is contained in:
ginnoir
2026-06-07 18:53:23 -05:00
parent 91d6724175
commit ba207a0fcd
8 changed files with 460 additions and 0 deletions
+83
View File
@@ -0,0 +1,83 @@
# romhacks stack — automated Pokémon rom-hack ingestion from a Discord server's
# per-generation threads into the RomM library.
#
# FLOW
# romhacks (orchestrator) jdownloader (jlesage) RomM (roms stack)
# ┌───────────────────────┐ ┌──────────────────┐ ┌──────────────┐
# │ DiscordChatExporter ──▶│ JSON │ FolderWatch picks│ files │ scans │
# │ parse links + art ───▶│ crawljob│ up .crawljob, │ ───────▶ │ /storage1/ │
# │ stage metadata ───▶│ ──────▶ │ resolves ~all │ │ Emulation │
# └───────────────────────┘ │ hosts, downloads │ └──────────────┘
# └──────────────────┘
#
# WHY THIS SHAPE
# - Member-only access => no official Discord bot is possible. DiscordChatExporter
# (DCE) does a READ-ONLY export with a USER token. Self-token use violates
# Discord ToS, so this is wired for an ALT account (see stack.env) to isolate
# ban risk from the main account.
# - The download links span many file hosts (Mega/Drive/MediaFire/Pixeldrain/...),
# so JDownloader2 (jlesage image) does the actual fetching — it auto-resolves
# hundreds of hosts and handles waits/captcha via its noVNC UI.
# - DCE `--media` downloads the inline thread images (box art / screenshots); the
# orchestrator stages those + the thread blurb as metadata for a later RomM
# custom-cover hand-off (dovetails with the deck-sgdb art pipeline).
#
# INERT BY DEFAULT. Does nothing until armed: DISCORD_TOKEN/GUILD_ID are empty and
# DRY_RUN=true, so even if deployed it idles and never enqueues a download. See
# README.md for the arming checklist.
#
# BUILD (not bind) for the orchestrator: Portainer git-stack checkouts do NOT
# reliably materialize sibling repo files at relative FILE bind paths (Docker then
# auto-creates a directory — see the `share` stack note). So the orchestrator's
# scripts + channels.json are baked into a built image from ./orchestrator instead
# of bind-mounted. Editing channels.json => push => Portainer rebuild (use
# "Re-pull and redeploy" if a content-only change doesn't trigger a rebuild).
#
# Tiered binds: working state (exports/state/crawljobs + JD config) -> /config
# (SSD); bulk output + art -> /storage1/labdata/romhacks (ZFS, root-auto-created).
#
# Only jdownloader joins `edge` (Caddy proxies its noVNC UI, internal-only).
services:
romhacks:
build:
context: ./orchestrator
image: homelab/romhacks-orchestrator:latest
container_name: romhacks
restart: unless-stopped
networks: [romhacks]
env_file:
- stack.env
volumes:
- /config/romhacks/exports:/exports
- /config/romhacks/state:/state
- /config/romhacks/crawljobs:/crawljobs
- /storage1/labdata/romhacks/metadata:/metadata
jdownloader:
image: jlesage/jdownloader-2:latest
container_name: jdownloader
restart: unless-stopped
networks: [romhacks, edge]
env_file:
- stack.env
environment:
# root so it can write to the ZFS-tier /output bind (dirs auto-created as root)
- USER_ID=0
- GROUP_ID=0
volumes:
- /config/romhacks/jd:/config
# shared with the orchestrator: it drops .crawljob files here; the JD
# FolderWatch extension must be enabled + pointed at /watch (one-time, README).
- /config/romhacks/crawljobs:/watch
- /storage1/labdata/romhacks/incoming:/output
ports:
- "8998:5800"
networks:
romhacks:
name: romhacks
driver: bridge
edge:
name: edge
external: true