Loading-path shim fixes, host setjmp/longjmp, recompiler fixes, CI

.github/workflows/check.yml

@@ -0,0 +1,149 @@
+# Host-side checks for conquertron: everything that can be proved without a
+# game dump or a Switch.
+#
+#   recompiler  builds recomp and its two verification tools against the
+#               pinned Capstone, with warnings shown
+#   shims       runs hosttest/sync_harness.c -- the real event, mutex,
+#               semaphore and FS-completion shims, not a model of them --
+#               normally and under ThreadSanitizer
+#   verify      blaster's verify.sh: C programs compiled for PowerPC,
+#               recompiled, then built for the host and for ARM64 (run under
+#               QEMU) and compared with a native build of the same source
+#
+# None of this touches the retail RPX, and nothing here is uploaded as an
+# artifact. The Switch build stays on the self-hosted runner in jouster.
+#
+# verify checks out Arkchemy/blaster. That works while blaster is public; if
+# it is made private, add a read-only token as the BLASTER_TOKEN secret.
+name: check
+
+on:
+  push:
+    paths-ignore: ['**.md', 'findings/**', 'docs/**']
+  pull_request:
+    paths-ignore: ['**.md', 'findings/**', 'docs/**']
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+env:
+  CAPSTONE_VERSION: 5.0.3
+  ZIG_VERSION: 0.13.0
+
+jobs:
+  recompiler:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Cache Capstone
+        id: capstone
+        uses: actions/cache@v4
+        with:
+          path: ~/devtools/capstone-install
+          key: capstone-${{ env.CAPSTONE_VERSION }}-ppc-${{ runner.os }}
+
+      - name: Build Capstone ${{ env.CAPSTONE_VERSION }} (PowerPC only)
+        if: steps.capstone.outputs.cache-hit != 'true'
+        run: |
+          git clone -q --depth 1 --branch "$CAPSTONE_VERSION" https://github.com/capstone-engine/capstone.git /tmp/capstone
+          cmake -S /tmp/capstone -B /tmp/capstone/build -DCMAKE_BUILD_TYPE=Release \
+            -DCAPSTONE_ARCHITECTURE_DEFAULT=OFF -DCAPSTONE_PPC_SUPPORT=ON \
+            -DCAPSTONE_BUILD_TESTS=OFF -DCAPSTONE_BUILD_CSTOOL=OFF \
+            -DCMAKE_INSTALL_PREFIX="$HOME/devtools/capstone-install"
+          cmake --build /tmp/capstone/build -j"$(nproc)"
+          cmake --install /tmp/capstone/build
+
+      - name: Build recomp
+        run: |
+          cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCAPSTONE_PREFIX="$HOME/devtools/capstone-install"
+          cmake --build build -j"$(nproc)"
+
+  shims:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Headers compile clean
+        run: |
+          printf '#include "ppc_runtime.h"\n#include "cafeos_coreinit_fs.h"\n#include "cafeos_coreinit_sync.h"\nvoid ppc_dispatch(PpcContext *c, uint32_t a) { (void)c; (void)a; }\n' > /tmp/tu.c
+          for cc in gcc clang; do
+            $cc -std=gnu11 -fsyntax-only -Wall -Wextra -Werror \
+                -Wno-unused-function -Wno-unused-parameter -Wno-unused-variable \
+                -I include /tmp/tu.c
+          done
+
+      - name: sync_harness
+        run: |
+          gcc -O1 -g -w -I include hosttest/sync_harness.c include/cafeos_state.c \
+              -o /tmp/sync_harness -lpthread -lm
+          # several runs: the cases are about races, and one pass proves little
+          for i in 1 2 3 4 5; do timeout 120 /tmp/sync_harness; done
+
+      - name: sync_harness under ThreadSanitizer
+        run: |
+          gcc -O1 -g -w -fsanitize=thread -I include hosttest/sync_harness.c include/cafeos_state.c \
+              -o /tmp/sync_harness_tsan -lpthread -lm
+          # The diagnostic counters (g_ark_sev_*, g_ark_wev_*, g_arkchemy_event_*)
+          # are unlocked on purpose. Anything else racing fails the job.
+          TSAN_OPTIONS="halt_on_error=0 exitcode=0" setarch "$(uname -m)" -R timeout 600 \
+              /tmp/sync_harness_tsan 2> /tmp/tsan.txt
+          # Every report must be on one of those globals: count all reports,
+          # count the allowed ones, and fail on any difference -- a race on
+          # heap or struct memory has no "Location is global" line at all.
+          total=$(grep -c 'SUMMARY: ThreadSanitizer' /tmp/tsan.txt || true)
+          allowed=$(grep 'Location is global' /tmp/tsan.txt \
+                    | grep -cE "'(g_ark_sev_|g_ark_wev_|g_arkchemy_event_|g_case_done|limit)" || true)
+          echo "ThreadSanitizer: $total report(s), $allowed on diagnostic counters"
+          if [ "$total" != "$allowed" ]; then
+            echo "::error::ThreadSanitizer found a race outside the diagnostic counters"
+            cat /tmp/tsan.txt
+            exit 1
+          fi
+
+  verify:
+    runs-on: ubuntu-24.04
+    needs: recompiler
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          path: conquertron
+
+      - uses: actions/checkout@v4
+        with:
+          repository: Arkchemy/blaster
+          path: blaster
+          token: ${{ secrets.BLASTER_TOKEN || github.token }}
+
+      - name: Tools
+        run: |
+          sudo apt-get update -q
+          sudo apt-get install -y -q qemu-user-static
+          mkdir -p "$HOME/devtools"
+          curl -sSL "https://ziglang.org/download/${ZIG_VERSION}/zig-linux-x86_64-${ZIG_VERSION}.tar.xz" | tar -xJ -C "$HOME/devtools"
+          mv "$HOME/devtools/zig-linux-x86_64-${ZIG_VERSION}" "$HOME/devtools/zig"
+
+      - name: Cache Capstone
+        id: capstone
+        uses: actions/cache@v4
+        with:
+          path: ~/devtools/capstone-install
+          key: capstone-${{ env.CAPSTONE_VERSION }}-ppc-${{ runner.os }}
+
+      - name: Build Capstone ${{ env.CAPSTONE_VERSION }} (PowerPC only)
+        if: steps.capstone.outputs.cache-hit != 'true'
+        run: |
+          git clone -q --depth 1 --branch "$CAPSTONE_VERSION" https://github.com/capstone-engine/capstone.git /tmp/capstone
+          cmake -S /tmp/capstone -B /tmp/capstone/build -DCMAKE_BUILD_TYPE=Release \
+            -DCAPSTONE_ARCHITECTURE_DEFAULT=OFF -DCAPSTONE_PPC_SUPPORT=ON \
+            -DCAPSTONE_BUILD_TESTS=OFF -DCAPSTONE_BUILD_CSTOOL=OFF \
+            -DCMAKE_INSTALL_PREFIX="$HOME/devtools/capstone-install"
+          cmake --build /tmp/capstone/build -j"$(nproc)"
+          cmake --install /tmp/capstone/build
+
+      - name: verify.sh
+        env:
+          CONQUERTRON: ${{ github.workspace }}/conquertron
+          QEMU_AARCH64: /usr/bin/qemu-aarch64-static
+        run: sh blaster/verify.sh

ROADMAP.md

@@ -30,16 +30,28 @@ scrutiny than the translator because they are hand-written and look simple.
 
 - [x] guest thread stacks reserve an EABI linkage area — a missing 16 bytes
       corrupted a heap and stalled boot for four sessions
+- [x] OSEvent AUTO mode wakes exactly one waiter per signal, and a signal
+      landing mid-pump is kept rather than lost (2026-09-24, EVCREDIT;
+      pinned by seven sync_harness cases)
+- [x] the FS completion queue is locked and the pumps claim atomically; the
+      unlocked queue lost and duplicated completions under contention
+      (2026-09-24). Next hardware run should say whether this moves the
+      loading stall -- judge it on `served/back`, not bytes
 - [ ] **audit every shim against real Cafe OS semantics**, especially anything
       that sets up guest register state: thread creation, TLS, callbacks,
       anything that fabricates a stack frame
 - [ ] `memset`/`memcpy` bypass `ppc_store_u32`, so they are invisible to the
       store watch. Either route them through it under a debug flag, or make
       that limitation impossible to forget.
-- [ ] `setjmp`/`longjmp` are recompiled PowerPC. A `longjmp` restoring guest
-      registers cannot unwind the **host** call stack the recompiled code runs
-      on. Not yet implicated in a real bug, but it cannot work as written and
-      will matter to any code using it for error handling.
+- [x] `setjmp`/`longjmp` are done on the host (2026-09-24). A recompiled
+      `longjmp` restored guest registers but could not unwind the host call
+      stack. Calls to them are now recognised by name and replaced -- the
+      host `setjmp` runs inline in the caller's own C function, and `longjmp`
+      jumps back to it with the guest register file restored. XenonRecomp's
+      approach, adapted to C. Pinned by blaster's verify.sh at guest -O0 and
+      -O1. **Unconfirmed against the retail binary:** the default names are
+      `setjmp`/`_setjmp`/`__setjmp` and their `longjmp` twins; if GHS's
+      libc spells them differently, pass `--setjmp-name`/`--longjmp-name`
 - [x] GX2 is no longer "shims that mostly record and discard". As of
       2026-09-16 the path is real end to end: surfaces are sized and
       allocated, each one keeps its own deko3d image across re-binds, draws

docs/prior-art-2026-09.md

@@ -0,0 +1,147 @@
+# Prior art, September 2026 update
+
+A follow-up to the "External prior art — useful repos" note (last edited
+2026-09-12), which already covers igRewrite8, re_nsyshid, Texthead1,
+spyrosadventure, hYdos, NefariousTechSupport, LG-RZ, decaf-emu,
+kinnay/Nintendo-File-Formats, CafeGLSL and the GX2 shader samples. Nothing
+from that page is repeated here.
+
+Same rule as that page: **the licence decides whether a project can be built
+on or only read.** Arkchemy's own licence is permission-required, so anything
+GPL can be studied but not copied in.
+
+Researched 2026-09-24. Star counts and commit counts are as of that day.
+
+## The headline: other people are statically recompiling Wii U code now
+
+When the prior-art page was written, the only other Skylanders
+recompilation effort was an empty placeholder (`sky2015-recomp`). There are
+now at least three Wii U static recompilers in public, one of them
+running a real game.
+
+| Project | Licence | State | Worth |
+| --- | --- | --- | --- |
+| [BlackLineInteractive/nWiiURecomp](https://github.com/BlackLineInteractive/nWiiURecomp) | **GPL-3.0** | *Wind Waker HD* EU v0 "authenticates, maps its sections and relocations, initializes Cafe ABI state, and runs deterministically through cooperative startup". One validated title. 6★, 11 commits. | **Read, compare notes.** The closest project to conquertron that exists: RPX in, C++ out, an HLE Cafe OS runtime, and an Espresso interpreter as a fallback for code it did not compile. It says it runs "deterministically through cooperative startup" -- the same cooperative-scheduling territory the loading stall lives in. Spun out of [NWiiRecomp](https://github.com/BlackLineInteractive/NWiiRecomp) (GameCube/Wii, 25★, 226 commits). |
+| [ApfelTeeSaft/RebrewU](https://github.com/ApfelTeeSaft/RebrewU) | none stated | RPX/RPL → C++. Function discovery, CFGs, jump-table detection; tools to inspect sections, relocations, imports, exports. 5★, 5 commits. | **Read only** (no licence = all rights reserved). Its jump-table detection is the one piece worth comparing against. |
+| [chrissotraidis/DolRecomp](https://github.com/chrissotraidis/DolRecomp) | **GPL-3.0** | GameCube/Wii recompiler with an "experimental" RPX frontend its authors say is not maintained. 236 opcodes; *Luigi's Mansion* reaches its title screen. | **Read.** Its decoder, CPU-behaviour and RPX tests are a second opinion on instruction semantics -- useful for the instruction audit in ROADMAP.md, where every bug so far has been a silent one. |
+| [hardkiller2565123123/Wii-U-Recomp](https://github.com/hardkiller2565123123/Wii-U-Recomp) | not yet | Announced, no source. | Watch. |
+
+None of these ports a game to the Switch, and none targets Skylanders or
+Alchemy. They are peers, not overlap -- but the people behind nWiiURecomp in
+particular are the ones most likely to have already met whatever
+conquertron meets next.
+
+## The best permissively-licensed reference for conquertron itself
+
+### [hedge-dev/XenonRecomp](https://github.com/hedge-dev/XenonRecomp) — MIT
+
+The Xbox 360 recompiler behind the *Unleashed Recompiled* port. The Xbox
+360's CPU is PowerPC, so most of the hard problems are the same ones, and
+**it is MIT: it can be built on, with attribution.** Three things in it map
+directly onto open items:
+
+- **`setjmp`/`longjmp`.** ROADMAP.md: "a `longjmp` restoring guest registers
+  cannot unwind the host call stack the recompiled code runs on." XenonRecomp's
+  answer is to not recompile them at all: calls to the guest's `setjmp`/
+  `longjmp` are redirected to the host's own, with guest CPU state saved
+  alongside. That is the known-good shape for the fix here.
+- **Jump tables.** It detects `mtctr` … `bctr` patterns and turns them into C
+  `switch` statements, with per-game TOML for the cases the pattern misses.
+  Its README is candid that there is "no fully generic solution".
+- **Mid-assembly hooks.** Calls to host functions injected at chosen guest
+  addresses, with register arguments, and options to return or branch
+  afterwards. conquertron's probes (ark_blockprobe.h) are hand-rolled
+  versions of the same idea; a general mechanism would stop each probe
+  needing its own codegen touch.
+
+It also documents a subtlety conquertron should check: the FPU leaves
+denormals alone while the vector unit flushes them. The Espresso has no VMX,
+but it does have paired singles, which have their own rounding behaviour.
+
+## Cafe OS semantics: where to check a shim against
+
+### [cemu-project/Cemu](https://github.com/cemu-project/Cemu) — MPL-2.0
+
+Already cited in the shim comments. Two findings from reading it on
+2026-09-24:
+
+1. **The event fix is right.** `coreinit_Synchronization.cpp`: in AUTO mode,
+   `OSSignalEvent` latches if the wait queue is empty and otherwise calls
+   `wakeupSingleThreadWaitQueue` -- one thread. `OSSignalEventAll` latches if
+   empty and otherwise wakes the whole queue. `OSWaitEventWithTimeout`
+   returns false on timeout and true on a signal. That is exactly what the
+   EVCREDIT rewrite implements and what sync_harness now pins.
+2. **Real FS callbacks run on dedicated threads.** `coreinit_FS.cpp`
+   comments that on hardware async FS completion processing "delegates the
+   processing to the AppIO threads" (Cemu itself runs it on its IPC thread).
+   conquertron runs callbacks cooperatively, on whichever thread next calls
+   an import that pumps -- which is why OSWaitEvent, OSWaitEventWithTimeout,
+   OSLockMutex and now OSWaitSemaphore all have to pump, and why the archive
+   pump exists at all. **A dedicated host thread that delivers completions,
+   the way AppIO does, is the architecture that would retire the pumps.**
+   Not attempted here: it changes which thread runs guest callbacks, which
+   is a hardware question.
+
+   Cemu also queues FS commands by priority (`__FSQueueCmdByPriority`), so
+   completion order on hardware is not strictly submission order.
+   conquertron's FIFO queue is stricter than hardware, which is the safe
+   direction.
+
+MPL-2.0 is file-level copyleft: a file adapted from Cemu must stay MPL-2.0,
+but it can sit in a project under another licence. Reading it to confirm
+semantics, as the shims do, carries no obligation at all.
+
+### [devkitPro/wut](https://github.com/devkitPro/wut) — zlib
+
+The homebrew Wii U SDK. Its `include/coreinit/*.h` headers are the
+signatures and structure layouts the shims already cite, and its
+[generated docs](https://wut.devkitpro.org/group__coreinit__event.html) are
+the quickest place to check a function's documented contract. zlib licence:
+usable freely.
+
+### [WiiUBrew: Coreinit.rpl](https://wiiubrew.org/wiki/Coreinit.rpl) and [/dev/fsa](https://wiiubrew.org/wiki//dev/fsa)
+
+Community documentation of coreinit exports and the filesystem IPC device.
+
+## Tools for reading the binary
+
+| Project | Licence | Why |
+| --- | --- | --- |
+| [Maschell/GhidraRPXLoader](https://github.com/Maschell/GhidraRPXLoader) | GPL-3.0 (tool) | Opens `.rpx`/`.rpl` directly in Ghidra, with Espresso (paired-singles) processor definitions and a script that names imports. Using a GPL tool puts no obligation on what it is used to read. Every "read the instruction, not the name" lesson in the Loading Deadlock notes gets cheaper with a decompiler view next to the probes. |
+| [wiiu-env/RPXParserLib](https://github.com/wiiu-env/RPXParserLib) | Apache-2.0 | A Java library that parses RPX/RPL: symbols, imports, exports. Usable, and a cross-check for `elf_loader.cpp`, particularly compressed sections and import resolution. |
+| [BullyWiiPlaza/RPL-Studio](https://github.com/BullyWiiPlaza/RPL-Studio) | see repo | GUI pack/unpack for RPX/RPL. |
+
+## Paired singles: toolchain support is arriving
+
+- [llvm/llvm-project#211463](https://github.com/llvm/llvm-project/pull/211463)
+  adds a `ppc750cl` CPU and about 40 paired-single instructions to LLVM's
+  assembler and disassembler (not codegen yet). Open, approved in review,
+  last active 2026-09-24. **Once merged, clang -- and so the zig that
+  verify.sh uses -- can assemble paired-single test programs**, which would
+  let verify.sh check conquertron's `PPC_INS_ARKCHEMY_PS_*` handling the way
+  it already checks integer and float code. Today nothing does.
+- [capstone#476](https://github.com/capstone-engine/capstone/issues/476):
+  Capstone still lacks paired singles, which is why conquertron decodes them
+  itself.
+
+## The Switch side
+
+| Project | Licence | Why |
+| --- | --- | --- |
+| [devkitPro/uam](https://github.com/devkitPro/uam) | see repo (mesa/nouveau-derived) | The offline GLSL → DKSH compiler blaster's shader path targets. |
+| [averne/libuam](https://github.com/averne/libuam) | see repo | A library form of uam. If shaders ever have to be compiled when the game first binds them, rather than ahead of time by `build-shaders.sh`, this is the route. 1★. |
+
+## Suggested next steps, in order of payoff
+
+1. **Build and run the conquertron branch on hardware.** Judge it on
+   `served/back`, per the Loading Deadlock notes -- the FS queue race is the
+   first candidate cause for the stall that is a certain bug rather than a
+   theory.
+2. **Port XenonRecomp's `setjmp`/`longjmp` approach** (MIT, with
+   attribution). It is a ROADMAP item with a known-good answer.
+3. **Prototype an AppIO-style completion thread** behind a flag, and compare
+   it with the pumps on hardware.
+4. **Contact the nWiiURecomp author.** Two projects doing the same thing to
+   the same OS will keep finding the same bugs.
+5. **Watch LLVM #211463.** When it lands, add paired-single programs to
+   verify.sh.