feat(romhacks): add Discord rom-hack ingestion stack (DCE -> JDownloader -> RomM)

Inert by default: empty token keeps the orchestrator idle and DRY_RUN=true
suppresses downloads. Orchestrator exports per-generation forum threads via
DiscordChatExporter, extracts download links + art, stages metadata, and writes
JDownloader crawljobs. Armed here with an alt account token and all 11 romhack
forums, still in DRY_RUN pending first-cycle validation.

stacks/romhacks/README.md

@@ -0,0 +1,86 @@
+# romhacks stack
+
+Automated ingestion of Pokémon rom-hacks from a Discord server's per-generation
+threads into the RomM library.
+
+```
+romhacks (orchestrator)              jdownloader (jlesage)        RomM (roms stack)
+  DiscordChatExporter  ── JSON ──▶   FolderWatch reads .crawljob,   scans
+  parse links + art    ── crawljob ▶ resolves ~all hosts,   ──────▶ /storage1/Emulation
+  stage metadata       ── art ─────▶ downloads to /output
+```
+
+## State: INERT
+
+Ships idle and safe. It will not export or download until armed:
+- `DISCORD_TOKEN` / `GUILD_ID` empty → orchestrator loop idles (see `run.sh`).
+- `DRY_RUN=true` → even configured, it parses + stages metadata but writes **no**
+  `.crawljob`, so JDownloader fetches nothing.
+
+## Why this design
+
+- **Member-only Discord access** → no official bot possible. DiscordChatExporter
+  (DCE) does a **read-only** export with a **user token**. That violates Discord
+  ToS, so this is wired for an **alt account** (joined to the server purely for
+  this) to keep any ban risk off the main account.
+- **Links span many file hosts** → JDownloader2 auto-resolves hundreds of hosts
+  and handles waits/captcha via its noVNC UI; far more robust than per-host glue.
+- **Art + blurb** → DCE `--media` downloads inline thread images; the orchestrator
+  stages them with the thread name/description for a later RomM custom-cover
+  hand-off (pairs with the `deck-sgdb` art pipeline).
+
+## Arming checklist
+
+1. **Alt account**: create it, join the server, grab its user token → `DISCORD_TOKEN`
+   in `stack.env`. Set `GUILD_ID` (right-click server → Copy Server ID; Developer
+   Mode on).
+2. **Channels**: fill `orchestrator/channels.json` with the per-generation channel
+   (or forum) IDs. Add any new file hosts you see to `hosts`.
+3. **Register the stack** in Portainer once (new git stacks need a one-time
+   registration — see memory `portainer-new-stack-registration`). Build is on:
+   Portainer builds `homelab/romhacks-orchestrator` from `./orchestrator`.
+4. **JDownloader FolderWatch**: open the noVNC UI at `http://valhalla:8998`
+   (set `VNC_PASSWORD` first), Settings → Extensions → **Folder Watch** → enable,
+   add watch folder `/watch`. This is what consumes the orchestrator's crawljobs.
+5. **Dry run**: leave `DRY_RUN=true`, let one cycle run, inspect
+   `/storage1/labdata/romhacks/metadata/*` — confirm names, blurbs, art, and that
+   the extracted `links` look like real ROM links (not random URLs).
+6. **Arm**: flip `DRY_RUN=false`. Downloads land in
+   `/storage1/labdata/romhacks/incoming/<game>/`.
+7. **Caddy (optional, internal-only)** — expose the JD UI on the LAN:
+   ```
+   jd.ginnoir.com {
+       import internal_only
+       reverse_proxy jdownloader:5800
+   }
+   ```
+   then `./scripts/gen-bookmarks.ps1` and push (runner reloads Caddy).
+8. **RomM hand-off** (next phase, not built yet): sort `incoming/<game>` into the
+   right `/storage1/Emulation/<platform>` folder and push the staged art to RomM's
+   custom-cover API.
+
+## Editing config
+
+`channels.json` and the scripts are **baked into the image** (Portainer's git-stack
+checkout doesn't reliably bind sibling repo files — see the compose header). So a
+change there needs a **push + rebuild**; if a content-only edit doesn't trigger a
+rebuild, use Portainer → the stack → **Re-pull and redeploy**.
+
+## Paths
+
+| Host path | Purpose |
+|---|---|
+| `/config/romhacks/exports` | DCE JSON + downloaded media (SSD) |
+| `/config/romhacks/state` | processed-message-id dedupe (SSD) |
+| `/config/romhacks/crawljobs` | orchestrator → JDownloader FolderWatch (`/watch`) |
+| `/config/romhacks/jd` | JDownloader config (SSD) |
+| `/storage1/labdata/romhacks/incoming` | JDownloader downloads (ZFS) |
+| `/storage1/labdata/romhacks/metadata` | staged name/blurb/art per game (ZFS) |
+
+## Gotchas
+
+- **ToS / ban risk** is real but low for read-only export. Use the alt account.
+- **Pre-patched ROMs vs patches**: bundled base-game ROMs are piracy; IPS/BPS
+  patches are broadly tolerated. Prefer patch links where the thread offers both.
+- DCE flags / dll path are pinned in `orchestrate.py`; verify on first run (the
+  code globs for the dll if the pinned path moves between image versions).

stacks/romhacks/docker-compose.yml

@@ -0,0 +1,83 @@
+# romhacks stack — automated Pokémon rom-hack ingestion from a Discord server's
+# per-generation threads into the RomM library.
+#
+# FLOW
+#   romhacks (orchestrator)            jdownloader (jlesage)         RomM (roms stack)
+#   ┌───────────────────────┐         ┌──────────────────┐          ┌──────────────┐
+#   │ DiscordChatExporter ──▶│ JSON    │ FolderWatch picks│ files    │ scans         │
+#   │ parse links + art  ───▶│ crawljob│ up .crawljob,    │ ───────▶ │ /storage1/    │
+#   │ stage metadata     ───▶│ ──────▶ │ resolves ~all    │          │ Emulation     │
+#   └───────────────────────┘         │ hosts, downloads │          └──────────────┘
+#                                      └──────────────────┘
+#
+# WHY THIS SHAPE
+#   - Member-only access => no official Discord bot is possible. DiscordChatExporter
+#     (DCE) does a READ-ONLY export with a USER token. Self-token use violates
+#     Discord ToS, so this is wired for an ALT account (see stack.env) to isolate
+#     ban risk from the main account.
+#   - The download links span many file hosts (Mega/Drive/MediaFire/Pixeldrain/...),
+#     so JDownloader2 (jlesage image) does the actual fetching — it auto-resolves
+#     hundreds of hosts and handles waits/captcha via its noVNC UI.
+#   - DCE `--media` downloads the inline thread images (box art / screenshots); the
+#     orchestrator stages those + the thread blurb as metadata for a later RomM
+#     custom-cover hand-off (dovetails with the deck-sgdb art pipeline).
+#
+# INERT BY DEFAULT. Does nothing until armed: DISCORD_TOKEN/GUILD_ID are empty and
+# DRY_RUN=true, so even if deployed it idles and never enqueues a download. See
+# README.md for the arming checklist.
+#
+# BUILD (not bind) for the orchestrator: Portainer git-stack checkouts do NOT
+# reliably materialize sibling repo files at relative FILE bind paths (Docker then
+# auto-creates a directory — see the `share` stack note). So the orchestrator's
+# scripts + channels.json are baked into a built image from ./orchestrator instead
+# of bind-mounted. Editing channels.json => push => Portainer rebuild (use
+# "Re-pull and redeploy" if a content-only change doesn't trigger a rebuild).
+#
+# Tiered binds: working state (exports/state/crawljobs + JD config) -> /config
+# (SSD); bulk output + art -> /storage1/labdata/romhacks (ZFS, root-auto-created).
+#
+# Only jdownloader joins `edge` (Caddy proxies its noVNC UI, internal-only).
+
+services:
+  romhacks:
+    build:
+      context: ./orchestrator
+    image: homelab/romhacks-orchestrator:latest
+    container_name: romhacks
+    restart: unless-stopped
+    networks: [romhacks]
+    env_file:
+      - stack.env
+    volumes:
+      - /config/romhacks/exports:/exports
+      - /config/romhacks/state:/state
+      - /config/romhacks/crawljobs:/crawljobs
+      - /storage1/labdata/romhacks/metadata:/metadata
+
+  jdownloader:
+    image: jlesage/jdownloader-2:latest
+    container_name: jdownloader
+    restart: unless-stopped
+    networks: [romhacks, edge]
+    env_file:
+      - stack.env
+    environment:
+      # root so it can write to the ZFS-tier /output bind (dirs auto-created as root)
+      - USER_ID=0
+      - GROUP_ID=0
+    volumes:
+      - /config/romhacks/jd:/config
+      # shared with the orchestrator: it drops .crawljob files here; the JD
+      # FolderWatch extension must be enabled + pointed at /watch (one-time, README).
+      - /config/romhacks/crawljobs:/watch
+      - /storage1/labdata/romhacks/incoming:/output
+    ports:
+      - "8998:5800"
+
+networks:
+  romhacks:
+    name: romhacks
+    driver: bridge
+  edge:
+    name: edge
+    external: true

stacks/romhacks/orchestrator/.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc

stacks/romhacks/orchestrator/Dockerfile

@@ -0,0 +1,15 @@
+# Orchestrator = DiscordChatExporter (has .NET + the DCE CLI) + Python for parsing.
+# Baking scripts + channels.json into the image avoids Portainer's relative
+# FILE-bind quirk (see docker-compose.yml header).
+FROM tyrrrz/discordchatexporter:stable
+
+USER root
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends python3 python3-requests \
+ && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+COPY orchestrate.py run.sh channels.json ./
+RUN chmod +x run.sh
+
+ENTRYPOINT ["./run.sh"]

stacks/romhacks/orchestrator/channels.json

@@ -0,0 +1,26 @@
+{
+  "_comment": "Forum channels to harvest (type 15 — each thread is one rom-hack). DCE's --include-threads all captures the threads beneath each forum. 'platform' is a hint for the later RomM hand-off and is not used yet. Editing this file requires a push + Portainer rebuild (it is baked into the orchestrator image). Cheat-code forums are intentionally omitted.",
+  "hosts": [
+    "mega.nz",
+    "mediafire.com",
+    "drive.google.com",
+    "pixeldrain.com",
+    "gofile.io",
+    "1fichier.com",
+    "workupload.com",
+    "krakenfiles.com"
+  ],
+  "channels": [
+    { "id": "1474174360926290105", "label": "gen-1-romhacks", "platform": "gb" },
+    { "id": "1474174532196761802", "label": "gen-2-romhacks", "platform": "gbc" },
+    { "id": "1474174608587489416", "label": "gen-3-romhacks", "platform": "gba" },
+    { "id": "1474174672210755714", "label": "gen-4-romhacks", "platform": "nds" },
+    { "id": "1474174736438132756", "label": "gen-5-romhacks", "platform": "nds" },
+    { "id": "1474174789680632084", "label": "gen-6-romhacks", "platform": "3ds" },
+    { "id": "1474174859671240898", "label": "gen-7-romhacks", "platform": "3ds" },
+    { "id": "1474174950138183880", "label": "gen-8-romhacks", "platform": "switch" },
+    { "id": "1474175004248903935", "label": "gen-9-romhacks", "platform": "switch" },
+    { "id": "1474175244209098986", "label": "joiplay-rpgxp-games", "platform": "pc" },
+    { "id": "1475605095805747351", "label": "unique-romhacks", "platform": "" }
+  ]
+}

stacks/romhacks/orchestrator/orchestrate.py

@@ -0,0 +1,205 @@
+#!/usr/bin/env python3
+"""romhacks orchestrator.
+
+One cycle:
+  1. For each configured channel, run DiscordChatExporter (DCE) to dump the
+     channel + all its threads to JSON, downloading inline media (box art).
+  2. Parse every exported JSON. For each NOT-yet-seen message, pull download
+     links (filtered to known file hosts) and any image attachments.
+  3. Stage thread name + blurb + art + links as metadata (for a later RomM
+     custom-cover hand-off), and — unless DRY_RUN — write a JDownloader
+     .crawljob so JD fetches the links.
+
+Idempotent: the full thread JSON is re-parsed each cycle; a message-id state file
+ensures each post is only enqueued once. INERT guards live in run.sh (token/guild)
+and here (DRY_RUN).
+"""
+import glob
+import hashlib
+import json
+import os
+import pathlib
+import re
+import shutil
+import subprocess
+import sys
+
+TOKEN = os.environ.get("DISCORD_TOKEN", "").strip()
+DRY_RUN = os.environ.get("DRY_RUN", "true").lower() != "false"
+JD_DOWNLOAD_ROOT = os.environ.get("JD_DOWNLOAD_ROOT", "/output")
+
+# Paths default to the in-container mounts; overridable via env for local testing.
+EXPORTS = pathlib.Path(os.environ.get("EXPORTS_DIR", "/exports"))
+STATE = pathlib.Path(os.environ.get("STATE_FILE", "/state/processed.json"))
+CRAWLJOBS = pathlib.Path(os.environ.get("CRAWLJOBS_DIR", "/crawljobs"))
+METADATA = pathlib.Path(os.environ.get("METADATA_DIR", "/metadata"))
+CHANNELS_FILE = pathlib.Path(os.environ.get("CHANNELS_FILE", "/app/channels.json"))
+
+# Fallback host list; overridden by channels.json -> "hosts".
+DEFAULT_HOSTS = [
+    "mega.nz", "mediafire.com", "drive.google.com", "pixeldrain.com",
+    "gofile.io", "1fichier.com", "workupload.com", "krakenfiles.com",
+    "bunkr", "anonfiles",
+]
+URL_RE = re.compile(r"https?://[^\s<>()\[\]]+", re.I)
+IMG_EXTS = (".png", ".jpg", ".jpeg", ".webp", ".gif")
+
+
+def load_json(p):
+    try:
+        return json.loads(pathlib.Path(p).read_text(encoding="utf-8"))
+    except Exception:
+        return None
+
+
+def slug(s):
+    return re.sub(r"[^A-Za-z0-9._-]+", "_", s or "").strip("_")[:120] or "unknown"
+
+
+def find_dce():
+    direct = "/opt/app/DiscordChatExporter.Cli.dll"
+    if os.path.exists(direct):
+        return direct
+    hits = glob.glob("/opt/**/DiscordChatExporter.Cli.dll", recursive=True)
+    return hits[0] if hits else None
+
+
+def run_dce(channel_id):
+    dll = find_dce()
+    if not dll:
+        print("[romhacks] DiscordChatExporter.Cli.dll not found in image", file=sys.stderr)
+        return False
+    out = EXPORTS / channel_id
+    out.mkdir(parents=True, exist_ok=True)
+    cmd = [
+        "dotnet", dll, "export",
+        "-t", TOKEN,
+        "-c", channel_id,
+        "-f", "Json",
+        "--include-threads", "all",
+        "--media", "True",
+        "--reuse-media", "True",
+        "-o", str(out / "%c.json"),
+    ]
+    print(f"[romhacks] exporting channel {channel_id}")
+    return subprocess.run(cmd).returncode == 0
+
+
+def load_state():
+    return set((load_json(STATE) or {}).get("processed", []))
+
+
+def save_state(processed):
+    STATE.parent.mkdir(parents=True, exist_ok=True)
+    STATE.write_text(json.dumps({"processed": sorted(processed)}), encoding="utf-8")
+
+
+def host_match(url, hosts):
+    u = url.lower()
+    return any(h in u for h in hosts)
+
+
+def write_crawljob(game, links):
+    CRAWLJOBS.mkdir(parents=True, exist_ok=True)
+    job = [{
+        "text": "\n".join(links),
+        "packageName": game,
+        "downloadFolder": f"{JD_DOWNLOAD_ROOT}/{slug(game)}",
+        "enabled": "TRUE",
+        "autoStart": "TRUE",
+        "autoConfirm": "TRUE",
+    }]
+    digest = hashlib.sha1("|".join(links).encode()).hexdigest()[:8]
+    f = CRAWLJOBS / f"{slug(game)}-{digest}.crawljob"
+    f.write_text(json.dumps(job, indent=2), encoding="utf-8")
+    print(f"[romhacks] enqueued {len(links)} link(s) for '{game}' -> {f.name}")
+
+
+def stage_metadata(game, description, links, art_files):
+    d = METADATA / slug(game)
+    d.mkdir(parents=True, exist_ok=True)
+    meta = {
+        "name": game,
+        "description": description,
+        "links": links,
+        "art": [a.name for a in art_files],
+    }
+    (d / "metadata.json").write_text(json.dumps(meta, indent=2), encoding="utf-8")
+    for a in art_files:
+        try:
+            shutil.copy2(a, d / a.name)
+        except Exception as e:
+            print(f"[romhacks] art copy failed {a}: {e}", file=sys.stderr)
+
+
+def process_file(path, hosts, processed):
+    data = load_json(path)
+    if not data:
+        return 0
+    channel = data.get("channel", {})
+    game = channel.get("name") or "unknown"
+    msgs = data.get("messages", [])
+    description = next((m.get("content", "") for m in msgs if (m.get("content") or "").strip()), "")
+
+    links, art, new = [], [], 0
+    for m in msgs:
+        mid = m.get("id")
+        if not mid or mid in processed:
+            continue
+        # Links usually live in the message body as markdown links, but some posts
+        # only surface them via an auto-embed — scan both.
+        blobs = [m.get("content") or ""]
+        for emb in m.get("embeds", []):
+            blobs.append(emb.get("url") or "")
+            blobs.append(emb.get("description") or "")
+            for fld in emb.get("fields", []):
+                blobs.append(fld.get("value") or "")
+        for blob in blobs:
+            for u in URL_RE.findall(blob):
+                if host_match(u, hosts):
+                    links.append(u)
+        for att in m.get("attachments", []):
+            if att.get("fileName", "").lower().endswith(IMG_EXTS):
+                # DCE --media rewrites attachment "url" to a path relative to the export
+                local = pathlib.Path(path).parent / att.get("url", "")
+                if local.exists():
+                    art.append(local)
+        processed.add(mid)
+        new += 1
+
+    links = list(dict.fromkeys(links))
+    if links:
+        if DRY_RUN:
+            print(f"[romhacks][DRY_RUN] would enqueue {len(links)} link(s) for '{game}'")
+        else:
+            write_crawljob(game, links)
+    if links or art:
+        stage_metadata(game, description, links, art)
+    return new
+
+
+def main():
+    cfg = load_json(CHANNELS_FILE) or {}
+    channels = cfg.get("channels", [])
+    hosts = cfg.get("hosts", DEFAULT_HOSTS)
+    if not channels:
+        print("[romhacks] no channels configured in channels.json — nothing to do")
+        return
+
+    processed = load_state()
+    for ch in channels:
+        cid = str(ch.get("id", "")).strip()
+        if not cid:
+            continue
+        if not run_dce(cid):
+            print(f"[romhacks] export failed for {cid}, skipping", file=sys.stderr)
+
+    total_new = 0
+    for jf in EXPORTS.glob("**/*.json"):
+        total_new += process_file(jf, hosts, processed)
+    save_state(processed)
+    print(f"[romhacks] cycle done — {total_new} new message(s); DRY_RUN={DRY_RUN}")
+
+
+if __name__ == "__main__":
+    main()

stacks/romhacks/orchestrator/run.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+# Harvest loop. Stays idle (no export, no enqueue) until the alt-account token and
+# guild id are configured, so deploying the stack unconfigured is harmless.
+set -u
+
+POLL_INTERVAL="${POLL_INTERVAL:-21600}"
+
+while true; do
+  if [ -z "${DISCORD_TOKEN:-}" ] || [ -z "${GUILD_ID:-}" ]; then
+    echo "[romhacks] DISCORD_TOKEN/GUILD_ID not set — idle. Sleeping ${POLL_INTERVAL}s."
+  else
+    python3 /app/orchestrate.py || echo "[romhacks] cycle failed (exit $?)"
+  fi
+  sleep "${POLL_INTERVAL}"
+done

stacks/romhacks/stack.env

@@ -0,0 +1,28 @@
+# romhacks stack — INERT until armed.
+#
+# Use an ALT Discord account's token. Self-token use violates Discord ToS; read-only
+# export is the low-risk end of that gray zone, but an alt account that has joined
+# the server purely for this isolates any ban risk from your main account.
+TZ=America/Chicago
+
+# ── Discord (ALT account) ────────────────────────────────────────────────────
+# User token from the ALT account. Read-only export only. Rotating the alt's
+# password / full logout invalidates this — re-grab via DevTools Network tab.
+DISCORD_TOKEN=MTUxMzMyNDc1MTYxMzg1Mzc5Ng.GjT6qU.A-VRmDoca1iT2ys54IGQJhTbudqteb69Klhjbk
+# Guild (server) ID.
+GUILD_ID=1474050123347525652
+
+# ── Behavior ─────────────────────────────────────────────────────────────────
+# DRY_RUN=true  => export, parse, and stage metadata, but DO NOT enqueue any
+#                  downloads. Inspect /storage1/labdata/romhacks/metadata first.
+# DRY_RUN=false => write .crawljob files so JDownloader starts fetching.
+DRY_RUN=true
+# Seconds between harvest cycles (21600 = 6h).
+POLL_INTERVAL=21600
+# JDownloader in-container download root (maps to /storage1/labdata/romhacks/incoming).
+JD_DOWNLOAD_ROOT=/output
+
+# ── JDownloader noVNC UI (jlesage) ───────────────────────────────────────────
+# Password for the web UI (reachable at http://valhalla:8998, or via Caddy
+# internal-only once you add a site block). Set this before exposing it.
+VNC_PASSWORD=