feat(romhacks): add Discord rom-hack ingestion stack (DCE -> JDownloader -> RomM)
Inert by default: empty token keeps the orchestrator idle and DRY_RUN=true suppresses downloads. Orchestrator exports per-generation forum threads via DiscordChatExporter, extracts download links + art, stages metadata, and writes JDownloader crawljobs. Armed here with an alt account token and all 11 romhack forums, still in DRY_RUN pending first-cycle validation.
This commit is contained in:
@@ -0,0 +1,83 @@
|
||||
# romhacks stack — automated Pokémon rom-hack ingestion from a Discord server's
|
||||
# per-generation threads into the RomM library.
|
||||
#
|
||||
# FLOW
|
||||
# romhacks (orchestrator) jdownloader (jlesage) RomM (roms stack)
|
||||
# ┌───────────────────────┐ ┌──────────────────┐ ┌──────────────┐
|
||||
# │ DiscordChatExporter ──▶│ JSON │ FolderWatch picks│ files │ scans │
|
||||
# │ parse links + art ───▶│ crawljob│ up .crawljob, │ ───────▶ │ /storage1/ │
|
||||
# │ stage metadata ───▶│ ──────▶ │ resolves ~all │ │ Emulation │
|
||||
# └───────────────────────┘ │ hosts, downloads │ └──────────────┘
|
||||
# └──────────────────┘
|
||||
#
|
||||
# WHY THIS SHAPE
|
||||
# - Member-only access => no official Discord bot is possible. DiscordChatExporter
|
||||
# (DCE) does a READ-ONLY export with a USER token. Self-token use violates
|
||||
# Discord ToS, so this is wired for an ALT account (see stack.env) to isolate
|
||||
# ban risk from the main account.
|
||||
# - The download links span many file hosts (Mega/Drive/MediaFire/Pixeldrain/...),
|
||||
# so JDownloader2 (jlesage image) does the actual fetching — it auto-resolves
|
||||
# hundreds of hosts and handles waits/captcha via its noVNC UI.
|
||||
# - DCE `--media` downloads the inline thread images (box art / screenshots); the
|
||||
# orchestrator stages those + the thread blurb as metadata for a later RomM
|
||||
# custom-cover hand-off (dovetails with the deck-sgdb art pipeline).
|
||||
#
|
||||
# INERT BY DEFAULT. Does nothing until armed: DISCORD_TOKEN/GUILD_ID are empty and
|
||||
# DRY_RUN=true, so even if deployed it idles and never enqueues a download. See
|
||||
# README.md for the arming checklist.
|
||||
#
|
||||
# BUILD (not bind) for the orchestrator: Portainer git-stack checkouts do NOT
|
||||
# reliably materialize sibling repo files at relative FILE bind paths (Docker then
|
||||
# auto-creates a directory — see the `share` stack note). So the orchestrator's
|
||||
# scripts + channels.json are baked into a built image from ./orchestrator instead
|
||||
# of bind-mounted. Editing channels.json => push => Portainer rebuild (use
|
||||
# "Re-pull and redeploy" if a content-only change doesn't trigger a rebuild).
|
||||
#
|
||||
# Tiered binds: working state (exports/state/crawljobs + JD config) -> /config
|
||||
# (SSD); bulk output + art -> /storage1/labdata/romhacks (ZFS, root-auto-created).
|
||||
#
|
||||
# Only jdownloader joins `edge` (Caddy proxies its noVNC UI, internal-only).
|
||||
|
||||
services:
|
||||
romhacks:
|
||||
build:
|
||||
context: ./orchestrator
|
||||
image: homelab/romhacks-orchestrator:latest
|
||||
container_name: romhacks
|
||||
restart: unless-stopped
|
||||
networks: [romhacks]
|
||||
env_file:
|
||||
- stack.env
|
||||
volumes:
|
||||
- /config/romhacks/exports:/exports
|
||||
- /config/romhacks/state:/state
|
||||
- /config/romhacks/crawljobs:/crawljobs
|
||||
- /storage1/labdata/romhacks/metadata:/metadata
|
||||
|
||||
jdownloader:
|
||||
image: jlesage/jdownloader-2:latest
|
||||
container_name: jdownloader
|
||||
restart: unless-stopped
|
||||
networks: [romhacks, edge]
|
||||
env_file:
|
||||
- stack.env
|
||||
environment:
|
||||
# root so it can write to the ZFS-tier /output bind (dirs auto-created as root)
|
||||
- USER_ID=0
|
||||
- GROUP_ID=0
|
||||
volumes:
|
||||
- /config/romhacks/jd:/config
|
||||
# shared with the orchestrator: it drops .crawljob files here; the JD
|
||||
# FolderWatch extension must be enabled + pointed at /watch (one-time, README).
|
||||
- /config/romhacks/crawljobs:/watch
|
||||
- /storage1/labdata/romhacks/incoming:/output
|
||||
ports:
|
||||
- "8998:5800"
|
||||
|
||||
networks:
|
||||
romhacks:
|
||||
name: romhacks
|
||||
driver: bridge
|
||||
edge:
|
||||
name: edge
|
||||
external: true
|
||||
Reference in New Issue
Block a user