ginnoirandClaude Opus 5
8455178aaa
Point the service at registry.ginnoir.com/ginnoir/romhacks-orchestrator:latest (built by the workflow added in60121a5, verified pullable on valhalla) and drop the `build:` context that Portainer never honoured. This unfreezes stack 28, which has been stuck at847edffsince 2026-06-24: the containers kept running, so nothing alerted, but no commit had deployed in two months. README: document that Portainer builds nothing here, and that re-adding a `build:` would deadlock the stack again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
98 lines
4.8 KiB
Markdown
98 lines
4.8 KiB
Markdown
# romhacks stack
|
|
|
|
Automated ingestion of Pokémon rom-hacks from a Discord server's per-generation
|
|
threads into the RomM library.
|
|
|
|
```
|
|
romhacks (orchestrator) jdownloader (jlesage) RomM (roms stack)
|
|
DiscordChatExporter ── JSON ──▶ FolderWatch reads .crawljob, scans
|
|
parse links + art ── crawljob ▶ resolves ~all hosts, ──────▶ /storage1/Emulation
|
|
stage metadata ── art ─────▶ downloads to /output
|
|
```
|
|
|
|
## State: INERT
|
|
|
|
Ships idle and safe. It will not export or download until armed:
|
|
- `DISCORD_TOKEN` / `GUILD_ID` empty → orchestrator loop idles (see `run.sh`).
|
|
- `DRY_RUN=true` → even configured, it parses + stages metadata but writes **no**
|
|
`.crawljob`, so JDownloader fetches nothing.
|
|
|
|
## Why this design
|
|
|
|
- **Member-only Discord access** → no official bot possible. DiscordChatExporter
|
|
(DCE) does a **read-only** export with a **user token**. That violates Discord
|
|
ToS, so this is wired for an **alt account** (joined to the server purely for
|
|
this) to keep any ban risk off the main account.
|
|
- **Links span many file hosts** → JDownloader2 auto-resolves hundreds of hosts
|
|
and handles waits/captcha via its noVNC UI; far more robust than per-host glue.
|
|
- **Art + blurb** → DCE `--media` downloads inline thread images; the orchestrator
|
|
stages them with the thread name/description for a later RomM custom-cover
|
|
hand-off (pairs with the `deck-sgdb` art pipeline).
|
|
|
|
## Arming checklist
|
|
|
|
1. **Alt account**: create it, join the server, grab its user token → `DISCORD_TOKEN`
|
|
in `stack.env`. Set `GUILD_ID` (right-click server → Copy Server ID; Developer
|
|
Mode on).
|
|
2. **Channels**: fill `orchestrator/channels.json` with the per-generation channel
|
|
(or forum) IDs. Add any new file hosts you see to `hosts`.
|
|
3. **Register the stack** in Portainer once (new git stacks need a one-time
|
|
registration — see memory `portainer-new-stack-registration`). Portainer does
|
|
**not** build anything: the orchestrator image is built by Gitea Actions and
|
|
pulled from `registry.ginnoir.com/ginnoir/romhacks-orchestrator:latest`.
|
|
4. **JDownloader FolderWatch**: open the noVNC UI at `http://valhalla:8998`
|
|
(set `VNC_PASSWORD` first), Settings → Extensions → **Folder Watch** → enable,
|
|
add watch folder `/watch`. This is what consumes the orchestrator's crawljobs.
|
|
5. **Dry run**: leave `DRY_RUN=true`, let one cycle run, inspect
|
|
`/storage1/labdata/romhacks/metadata/*` — confirm names, blurbs, art, and that
|
|
the extracted `links` look like real ROM links (not random URLs).
|
|
6. **Arm**: flip `DRY_RUN=false`. Downloads land in
|
|
`/storage1/labdata/romhacks/incoming/<game>/`.
|
|
7. **Caddy (optional, internal-only)** — expose the JD UI on the LAN:
|
|
```
|
|
jd.ginnoir.com {
|
|
import internal_only
|
|
reverse_proxy jdownloader:5800
|
|
}
|
|
```
|
|
then `./scripts/gen-bookmarks.ps1` and push (runner reloads Caddy).
|
|
8. **RomM hand-off** (next phase, not built yet): sort `incoming/<game>` into the
|
|
right `/storage1/Emulation/<platform>` folder and push the staged art to RomM's
|
|
custom-cover API.
|
|
|
|
## Editing config
|
|
|
|
`channels.json` and the scripts are **baked into the image** (Portainer's git-stack
|
|
checkout doesn't reliably bind sibling repo files — see the compose header). So a
|
|
change under `orchestrator/` needs a **push**, which fires
|
|
`.gitea/workflows/build-romhacks-orchestrator.yml`: it builds the image and pushes
|
|
`registry.ginnoir.com/ginnoir/romhacks-orchestrator` as `:latest` + `:sha-<short>`.
|
|
Portainer then pulls `:latest` on its next 5-min poll (`pull_policy: always`).
|
|
|
|
Compose has **no `build:`** on purpose — Portainer's auto-update runs
|
|
`docker compose pull` and never `build`, so a build-only image name deadlocks the
|
|
stack. Don't add one back.
|
|
|
|
To rebuild without a source change, run the workflow manually (Gitea → the repo →
|
|
Actions → *Build romhacks orchestrator image* → **Run workflow**), then Portainer →
|
|
the stack → **Re-pull and redeploy**.
|
|
|
|
## Paths
|
|
|
|
| Host path | Purpose |
|
|
|---|---|
|
|
| `/config/romhacks/exports` | DCE JSON + downloaded media (SSD) |
|
|
| `/config/romhacks/state` | processed-message-id dedupe (SSD) |
|
|
| `/config/romhacks/crawljobs` | orchestrator → JDownloader FolderWatch (`/watch`) |
|
|
| `/config/romhacks/jd` | JDownloader config (SSD) |
|
|
| `/storage1/labdata/romhacks/incoming` | JDownloader downloads (ZFS) |
|
|
| `/storage1/labdata/romhacks/metadata` | staged name/blurb/art per game (ZFS) |
|
|
|
|
## Gotchas
|
|
|
|
- **ToS / ban risk** is real but low for read-only export. Use the alt account.
|
|
- **Pre-patched ROMs vs patches**: bundled base-game ROMs are piracy; IPS/BPS
|
|
patches are broadly tolerated. Prefer patch links where the thread offers both.
|
|
- DCE flags / dll path are pinned in `orchestrate.py`; verify on first run (the
|
|
code globs for the dll if the pinned path moves between image versions).
|