From ba207a0fcd79d5b02730f010877b05c370844455 Mon Sep 17 00:00:00 2001 From: ginnoir Date: Sun, 7 Jun 2026 18:53:23 -0500 Subject: [PATCH] feat(romhacks): add Discord rom-hack ingestion stack (DCE -> JDownloader -> RomM) Inert by default: empty token keeps the orchestrator idle and DRY_RUN=true suppresses downloads. Orchestrator exports per-generation forum threads via DiscordChatExporter, extracts download links + art, stages metadata, and writes JDownloader crawljobs. Armed here with an alt account token and all 11 romhack forums, still in DRY_RUN pending first-cycle validation. --- stacks/romhacks/README.md | 86 ++++++++ stacks/romhacks/docker-compose.yml | 83 ++++++++ stacks/romhacks/orchestrator/.gitignore | 2 + stacks/romhacks/orchestrator/Dockerfile | 15 ++ stacks/romhacks/orchestrator/channels.json | 26 +++ stacks/romhacks/orchestrator/orchestrate.py | 205 ++++++++++++++++++++ stacks/romhacks/orchestrator/run.sh | 15 ++ stacks/romhacks/stack.env | 28 +++ 8 files changed, 460 insertions(+) create mode 100644 stacks/romhacks/README.md create mode 100644 stacks/romhacks/docker-compose.yml create mode 100644 stacks/romhacks/orchestrator/.gitignore create mode 100644 stacks/romhacks/orchestrator/Dockerfile create mode 100644 stacks/romhacks/orchestrator/channels.json create mode 100644 stacks/romhacks/orchestrator/orchestrate.py create mode 100644 stacks/romhacks/orchestrator/run.sh create mode 100644 stacks/romhacks/stack.env diff --git a/stacks/romhacks/README.md b/stacks/romhacks/README.md new file mode 100644 index 0000000..8b878be --- /dev/null +++ b/stacks/romhacks/README.md @@ -0,0 +1,86 @@ +# romhacks stack + +Automated ingestion of Pokémon rom-hacks from a Discord server's per-generation +threads into the RomM library. + +``` +romhacks (orchestrator) jdownloader (jlesage) RomM (roms stack) + DiscordChatExporter ── JSON ──▶ FolderWatch reads .crawljob, scans + parse links + art ── crawljob ▶ resolves ~all hosts, ──────▶ /storage1/Emulation + stage metadata ── art ─────▶ downloads to /output +``` + +## State: INERT + +Ships idle and safe. It will not export or download until armed: +- `DISCORD_TOKEN` / `GUILD_ID` empty → orchestrator loop idles (see `run.sh`). +- `DRY_RUN=true` → even configured, it parses + stages metadata but writes **no** + `.crawljob`, so JDownloader fetches nothing. + +## Why this design + +- **Member-only Discord access** → no official bot possible. DiscordChatExporter + (DCE) does a **read-only** export with a **user token**. That violates Discord + ToS, so this is wired for an **alt account** (joined to the server purely for + this) to keep any ban risk off the main account. +- **Links span many file hosts** → JDownloader2 auto-resolves hundreds of hosts + and handles waits/captcha via its noVNC UI; far more robust than per-host glue. +- **Art + blurb** → DCE `--media` downloads inline thread images; the orchestrator + stages them with the thread name/description for a later RomM custom-cover + hand-off (pairs with the `deck-sgdb` art pipeline). + +## Arming checklist + +1. **Alt account**: create it, join the server, grab its user token → `DISCORD_TOKEN` + in `stack.env`. Set `GUILD_ID` (right-click server → Copy Server ID; Developer + Mode on). +2. **Channels**: fill `orchestrator/channels.json` with the per-generation channel + (or forum) IDs. Add any new file hosts you see to `hosts`. +3. **Register the stack** in Portainer once (new git stacks need a one-time + registration — see memory `portainer-new-stack-registration`). Build is on: + Portainer builds `homelab/romhacks-orchestrator` from `./orchestrator`. +4. **JDownloader FolderWatch**: open the noVNC UI at `http://valhalla:8998` + (set `VNC_PASSWORD` first), Settings → Extensions → **Folder Watch** → enable, + add watch folder `/watch`. This is what consumes the orchestrator's crawljobs. +5. **Dry run**: leave `DRY_RUN=true`, let one cycle run, inspect + `/storage1/labdata/romhacks/metadata/*` — confirm names, blurbs, art, and that + the extracted `links` look like real ROM links (not random URLs). +6. **Arm**: flip `DRY_RUN=false`. Downloads land in + `/storage1/labdata/romhacks/incoming//`. +7. **Caddy (optional, internal-only)** — expose the JD UI on the LAN: + ``` + jd.ginnoir.com { + import internal_only + reverse_proxy jdownloader:5800 + } + ``` + then `./scripts/gen-bookmarks.ps1` and push (runner reloads Caddy). +8. **RomM hand-off** (next phase, not built yet): sort `incoming/` into the + right `/storage1/Emulation/` folder and push the staged art to RomM's + custom-cover API. + +## Editing config + +`channels.json` and the scripts are **baked into the image** (Portainer's git-stack +checkout doesn't reliably bind sibling repo files — see the compose header). So a +change there needs a **push + rebuild**; if a content-only edit doesn't trigger a +rebuild, use Portainer → the stack → **Re-pull and redeploy**. + +## Paths + +| Host path | Purpose | +|---|---| +| `/config/romhacks/exports` | DCE JSON + downloaded media (SSD) | +| `/config/romhacks/state` | processed-message-id dedupe (SSD) | +| `/config/romhacks/crawljobs` | orchestrator → JDownloader FolderWatch (`/watch`) | +| `/config/romhacks/jd` | JDownloader config (SSD) | +| `/storage1/labdata/romhacks/incoming` | JDownloader downloads (ZFS) | +| `/storage1/labdata/romhacks/metadata` | staged name/blurb/art per game (ZFS) | + +## Gotchas + +- **ToS / ban risk** is real but low for read-only export. Use the alt account. +- **Pre-patched ROMs vs patches**: bundled base-game ROMs are piracy; IPS/BPS + patches are broadly tolerated. Prefer patch links where the thread offers both. +- DCE flags / dll path are pinned in `orchestrate.py`; verify on first run (the + code globs for the dll if the pinned path moves between image versions). diff --git a/stacks/romhacks/docker-compose.yml b/stacks/romhacks/docker-compose.yml new file mode 100644 index 0000000..b9cf031 --- /dev/null +++ b/stacks/romhacks/docker-compose.yml @@ -0,0 +1,83 @@ +# romhacks stack — automated Pokémon rom-hack ingestion from a Discord server's +# per-generation threads into the RomM library. +# +# FLOW +# romhacks (orchestrator) jdownloader (jlesage) RomM (roms stack) +# ┌───────────────────────┐ ┌──────────────────┐ ┌──────────────┐ +# │ DiscordChatExporter ──▶│ JSON │ FolderWatch picks│ files │ scans │ +# │ parse links + art ───▶│ crawljob│ up .crawljob, │ ───────▶ │ /storage1/ │ +# │ stage metadata ───▶│ ──────▶ │ resolves ~all │ │ Emulation │ +# └───────────────────────┘ │ hosts, downloads │ └──────────────┘ +# └──────────────────┘ +# +# WHY THIS SHAPE +# - Member-only access => no official Discord bot is possible. DiscordChatExporter +# (DCE) does a READ-ONLY export with a USER token. Self-token use violates +# Discord ToS, so this is wired for an ALT account (see stack.env) to isolate +# ban risk from the main account. +# - The download links span many file hosts (Mega/Drive/MediaFire/Pixeldrain/...), +# so JDownloader2 (jlesage image) does the actual fetching — it auto-resolves +# hundreds of hosts and handles waits/captcha via its noVNC UI. +# - DCE `--media` downloads the inline thread images (box art / screenshots); the +# orchestrator stages those + the thread blurb as metadata for a later RomM +# custom-cover hand-off (dovetails with the deck-sgdb art pipeline). +# +# INERT BY DEFAULT. Does nothing until armed: DISCORD_TOKEN/GUILD_ID are empty and +# DRY_RUN=true, so even if deployed it idles and never enqueues a download. See +# README.md for the arming checklist. +# +# BUILD (not bind) for the orchestrator: Portainer git-stack checkouts do NOT +# reliably materialize sibling repo files at relative FILE bind paths (Docker then +# auto-creates a directory — see the `share` stack note). So the orchestrator's +# scripts + channels.json are baked into a built image from ./orchestrator instead +# of bind-mounted. Editing channels.json => push => Portainer rebuild (use +# "Re-pull and redeploy" if a content-only change doesn't trigger a rebuild). +# +# Tiered binds: working state (exports/state/crawljobs + JD config) -> /config +# (SSD); bulk output + art -> /storage1/labdata/romhacks (ZFS, root-auto-created). +# +# Only jdownloader joins `edge` (Caddy proxies its noVNC UI, internal-only). + +services: + romhacks: + build: + context: ./orchestrator + image: homelab/romhacks-orchestrator:latest + container_name: romhacks + restart: unless-stopped + networks: [romhacks] + env_file: + - stack.env + volumes: + - /config/romhacks/exports:/exports + - /config/romhacks/state:/state + - /config/romhacks/crawljobs:/crawljobs + - /storage1/labdata/romhacks/metadata:/metadata + + jdownloader: + image: jlesage/jdownloader-2:latest + container_name: jdownloader + restart: unless-stopped + networks: [romhacks, edge] + env_file: + - stack.env + environment: + # root so it can write to the ZFS-tier /output bind (dirs auto-created as root) + - USER_ID=0 + - GROUP_ID=0 + volumes: + - /config/romhacks/jd:/config + # shared with the orchestrator: it drops .crawljob files here; the JD + # FolderWatch extension must be enabled + pointed at /watch (one-time, README). + - /config/romhacks/crawljobs:/watch + - /storage1/labdata/romhacks/incoming:/output + ports: + - "8998:5800" + +networks: + romhacks: + name: romhacks + driver: bridge + edge: + name: edge + external: true diff --git a/stacks/romhacks/orchestrator/.gitignore b/stacks/romhacks/orchestrator/.gitignore new file mode 100644 index 0000000..7a60b85 --- /dev/null +++ b/stacks/romhacks/orchestrator/.gitignore @@ -0,0 +1,2 @@ +__pycache__/ +*.pyc diff --git a/stacks/romhacks/orchestrator/Dockerfile b/stacks/romhacks/orchestrator/Dockerfile new file mode 100644 index 0000000..4bf0981 --- /dev/null +++ b/stacks/romhacks/orchestrator/Dockerfile @@ -0,0 +1,15 @@ +# Orchestrator = DiscordChatExporter (has .NET + the DCE CLI) + Python for parsing. +# Baking scripts + channels.json into the image avoids Portainer's relative +# FILE-bind quirk (see docker-compose.yml header). +FROM tyrrrz/discordchatexporter:stable + +USER root +RUN apt-get update \ + && apt-get install -y --no-install-recommends python3 python3-requests \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app +COPY orchestrate.py run.sh channels.json ./ +RUN chmod +x run.sh + +ENTRYPOINT ["./run.sh"] diff --git a/stacks/romhacks/orchestrator/channels.json b/stacks/romhacks/orchestrator/channels.json new file mode 100644 index 0000000..65537ce --- /dev/null +++ b/stacks/romhacks/orchestrator/channels.json @@ -0,0 +1,26 @@ +{ + "_comment": "Forum channels to harvest (type 15 — each thread is one rom-hack). DCE's --include-threads all captures the threads beneath each forum. 'platform' is a hint for the later RomM hand-off and is not used yet. Editing this file requires a push + Portainer rebuild (it is baked into the orchestrator image). Cheat-code forums are intentionally omitted.", + "hosts": [ + "mega.nz", + "mediafire.com", + "drive.google.com", + "pixeldrain.com", + "gofile.io", + "1fichier.com", + "workupload.com", + "krakenfiles.com" + ], + "channels": [ + { "id": "1474174360926290105", "label": "gen-1-romhacks", "platform": "gb" }, + { "id": "1474174532196761802", "label": "gen-2-romhacks", "platform": "gbc" }, + { "id": "1474174608587489416", "label": "gen-3-romhacks", "platform": "gba" }, + { "id": "1474174672210755714", "label": "gen-4-romhacks", "platform": "nds" }, + { "id": "1474174736438132756", "label": "gen-5-romhacks", "platform": "nds" }, + { "id": "1474174789680632084", "label": "gen-6-romhacks", "platform": "3ds" }, + { "id": "1474174859671240898", "label": "gen-7-romhacks", "platform": "3ds" }, + { "id": "1474174950138183880", "label": "gen-8-romhacks", "platform": "switch" }, + { "id": "1474175004248903935", "label": "gen-9-romhacks", "platform": "switch" }, + { "id": "1474175244209098986", "label": "joiplay-rpgxp-games", "platform": "pc" }, + { "id": "1475605095805747351", "label": "unique-romhacks", "platform": "" } + ] +} diff --git a/stacks/romhacks/orchestrator/orchestrate.py b/stacks/romhacks/orchestrator/orchestrate.py new file mode 100644 index 0000000..41e2836 --- /dev/null +++ b/stacks/romhacks/orchestrator/orchestrate.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""romhacks orchestrator. + +One cycle: + 1. For each configured channel, run DiscordChatExporter (DCE) to dump the + channel + all its threads to JSON, downloading inline media (box art). + 2. Parse every exported JSON. For each NOT-yet-seen message, pull download + links (filtered to known file hosts) and any image attachments. + 3. Stage thread name + blurb + art + links as metadata (for a later RomM + custom-cover hand-off), and — unless DRY_RUN — write a JDownloader + .crawljob so JD fetches the links. + +Idempotent: the full thread JSON is re-parsed each cycle; a message-id state file +ensures each post is only enqueued once. INERT guards live in run.sh (token/guild) +and here (DRY_RUN). +""" +import glob +import hashlib +import json +import os +import pathlib +import re +import shutil +import subprocess +import sys + +TOKEN = os.environ.get("DISCORD_TOKEN", "").strip() +DRY_RUN = os.environ.get("DRY_RUN", "true").lower() != "false" +JD_DOWNLOAD_ROOT = os.environ.get("JD_DOWNLOAD_ROOT", "/output") + +# Paths default to the in-container mounts; overridable via env for local testing. +EXPORTS = pathlib.Path(os.environ.get("EXPORTS_DIR", "/exports")) +STATE = pathlib.Path(os.environ.get("STATE_FILE", "/state/processed.json")) +CRAWLJOBS = pathlib.Path(os.environ.get("CRAWLJOBS_DIR", "/crawljobs")) +METADATA = pathlib.Path(os.environ.get("METADATA_DIR", "/metadata")) +CHANNELS_FILE = pathlib.Path(os.environ.get("CHANNELS_FILE", "/app/channels.json")) + +# Fallback host list; overridden by channels.json -> "hosts". +DEFAULT_HOSTS = [ + "mega.nz", "mediafire.com", "drive.google.com", "pixeldrain.com", + "gofile.io", "1fichier.com", "workupload.com", "krakenfiles.com", + "bunkr", "anonfiles", +] +URL_RE = re.compile(r"https?://[^\s<>()\[\]]+", re.I) +IMG_EXTS = (".png", ".jpg", ".jpeg", ".webp", ".gif") + + +def load_json(p): + try: + return json.loads(pathlib.Path(p).read_text(encoding="utf-8")) + except Exception: + return None + + +def slug(s): + return re.sub(r"[^A-Za-z0-9._-]+", "_", s or "").strip("_")[:120] or "unknown" + + +def find_dce(): + direct = "/opt/app/DiscordChatExporter.Cli.dll" + if os.path.exists(direct): + return direct + hits = glob.glob("/opt/**/DiscordChatExporter.Cli.dll", recursive=True) + return hits[0] if hits else None + + +def run_dce(channel_id): + dll = find_dce() + if not dll: + print("[romhacks] DiscordChatExporter.Cli.dll not found in image", file=sys.stderr) + return False + out = EXPORTS / channel_id + out.mkdir(parents=True, exist_ok=True) + cmd = [ + "dotnet", dll, "export", + "-t", TOKEN, + "-c", channel_id, + "-f", "Json", + "--include-threads", "all", + "--media", "True", + "--reuse-media", "True", + "-o", str(out / "%c.json"), + ] + print(f"[romhacks] exporting channel {channel_id}") + return subprocess.run(cmd).returncode == 0 + + +def load_state(): + return set((load_json(STATE) or {}).get("processed", [])) + + +def save_state(processed): + STATE.parent.mkdir(parents=True, exist_ok=True) + STATE.write_text(json.dumps({"processed": sorted(processed)}), encoding="utf-8") + + +def host_match(url, hosts): + u = url.lower() + return any(h in u for h in hosts) + + +def write_crawljob(game, links): + CRAWLJOBS.mkdir(parents=True, exist_ok=True) + job = [{ + "text": "\n".join(links), + "packageName": game, + "downloadFolder": f"{JD_DOWNLOAD_ROOT}/{slug(game)}", + "enabled": "TRUE", + "autoStart": "TRUE", + "autoConfirm": "TRUE", + }] + digest = hashlib.sha1("|".join(links).encode()).hexdigest()[:8] + f = CRAWLJOBS / f"{slug(game)}-{digest}.crawljob" + f.write_text(json.dumps(job, indent=2), encoding="utf-8") + print(f"[romhacks] enqueued {len(links)} link(s) for '{game}' -> {f.name}") + + +def stage_metadata(game, description, links, art_files): + d = METADATA / slug(game) + d.mkdir(parents=True, exist_ok=True) + meta = { + "name": game, + "description": description, + "links": links, + "art": [a.name for a in art_files], + } + (d / "metadata.json").write_text(json.dumps(meta, indent=2), encoding="utf-8") + for a in art_files: + try: + shutil.copy2(a, d / a.name) + except Exception as e: + print(f"[romhacks] art copy failed {a}: {e}", file=sys.stderr) + + +def process_file(path, hosts, processed): + data = load_json(path) + if not data: + return 0 + channel = data.get("channel", {}) + game = channel.get("name") or "unknown" + msgs = data.get("messages", []) + description = next((m.get("content", "") for m in msgs if (m.get("content") or "").strip()), "") + + links, art, new = [], [], 0 + for m in msgs: + mid = m.get("id") + if not mid or mid in processed: + continue + # Links usually live in the message body as markdown links, but some posts + # only surface them via an auto-embed — scan both. + blobs = [m.get("content") or ""] + for emb in m.get("embeds", []): + blobs.append(emb.get("url") or "") + blobs.append(emb.get("description") or "") + for fld in emb.get("fields", []): + blobs.append(fld.get("value") or "") + for blob in blobs: + for u in URL_RE.findall(blob): + if host_match(u, hosts): + links.append(u) + for att in m.get("attachments", []): + if att.get("fileName", "").lower().endswith(IMG_EXTS): + # DCE --media rewrites attachment "url" to a path relative to the export + local = pathlib.Path(path).parent / att.get("url", "") + if local.exists(): + art.append(local) + processed.add(mid) + new += 1 + + links = list(dict.fromkeys(links)) + if links: + if DRY_RUN: + print(f"[romhacks][DRY_RUN] would enqueue {len(links)} link(s) for '{game}'") + else: + write_crawljob(game, links) + if links or art: + stage_metadata(game, description, links, art) + return new + + +def main(): + cfg = load_json(CHANNELS_FILE) or {} + channels = cfg.get("channels", []) + hosts = cfg.get("hosts", DEFAULT_HOSTS) + if not channels: + print("[romhacks] no channels configured in channels.json — nothing to do") + return + + processed = load_state() + for ch in channels: + cid = str(ch.get("id", "")).strip() + if not cid: + continue + if not run_dce(cid): + print(f"[romhacks] export failed for {cid}, skipping", file=sys.stderr) + + total_new = 0 + for jf in EXPORTS.glob("**/*.json"): + total_new += process_file(jf, hosts, processed) + save_state(processed) + print(f"[romhacks] cycle done — {total_new} new message(s); DRY_RUN={DRY_RUN}") + + +if __name__ == "__main__": + main() diff --git a/stacks/romhacks/orchestrator/run.sh b/stacks/romhacks/orchestrator/run.sh new file mode 100644 index 0000000..1a6bb4b --- /dev/null +++ b/stacks/romhacks/orchestrator/run.sh @@ -0,0 +1,15 @@ +#!/bin/sh +# Harvest loop. Stays idle (no export, no enqueue) until the alt-account token and +# guild id are configured, so deploying the stack unconfigured is harmless. +set -u + +POLL_INTERVAL="${POLL_INTERVAL:-21600}" + +while true; do + if [ -z "${DISCORD_TOKEN:-}" ] || [ -z "${GUILD_ID:-}" ]; then + echo "[romhacks] DISCORD_TOKEN/GUILD_ID not set — idle. Sleeping ${POLL_INTERVAL}s." + else + python3 /app/orchestrate.py || echo "[romhacks] cycle failed (exit $?)" + fi + sleep "${POLL_INTERVAL}" +done diff --git a/stacks/romhacks/stack.env b/stacks/romhacks/stack.env new file mode 100644 index 0000000..69ab9ed --- /dev/null +++ b/stacks/romhacks/stack.env @@ -0,0 +1,28 @@ +# romhacks stack — INERT until armed. +# +# Use an ALT Discord account's token. Self-token use violates Discord ToS; read-only +# export is the low-risk end of that gray zone, but an alt account that has joined +# the server purely for this isolates any ban risk from your main account. +TZ=America/Chicago + +# ── Discord (ALT account) ──────────────────────────────────────────────────── +# User token from the ALT account. Read-only export only. Rotating the alt's +# password / full logout invalidates this — re-grab via DevTools Network tab. +DISCORD_TOKEN=MTUxMzMyNDc1MTYxMzg1Mzc5Ng.GjT6qU.A-VRmDoca1iT2ys54IGQJhTbudqteb69Klhjbk +# Guild (server) ID. +GUILD_ID=1474050123347525652 + +# ── Behavior ───────────────────────────────────────────────────────────────── +# DRY_RUN=true => export, parse, and stage metadata, but DO NOT enqueue any +# downloads. Inspect /storage1/labdata/romhacks/metadata first. +# DRY_RUN=false => write .crawljob files so JDownloader starts fetching. +DRY_RUN=true +# Seconds between harvest cycles (21600 = 6h). +POLL_INTERVAL=21600 +# JDownloader in-container download root (maps to /storage1/labdata/romhacks/incoming). +JD_DOWNLOAD_ROOT=/output + +# ── JDownloader noVNC UI (jlesage) ─────────────────────────────────────────── +# Password for the web UI (reachable at http://valhalla:8998, or via Caddy +# internal-only once you add a site block). Set this before exposing it. +VNC_PASSWORD=