[thehobbit85]

Reduce first-build work through ordered parallel shader discovery and function analysis, bounded IPA compression, requested-platform Vulkan/OpenGL compilation, dependency-ready NVN shader scheduling, adaptive rewrite CPU allocation and concurrent PSO preparation. The regenerated shader chain is f5b4fbea0450. Its inputs include main’s Ampere translator and updated shader ABI and the scheduler. The translator binaries and Metal lock match the integrated main revision.

Parallel RomFS extraction was withdrawn. The extractor uses the serial LocalStorage / WriteStream path and retains raw and RomFS-translated integrity-error handling. A second machine showed about a 2× slowdown on a 3.5 GB image with 16 GB RAM, alongside a smaller-image speedup. The cause remains open. The remaining stages have separate measurements below. SevenSpiders’ review

Revision fbdc8e03cb95c42b024eafe15c158b6e2f6e2346 includes main 4a2a81a64f029d0a75ade9523c8ffc6c89465375. The new jump-table guard and Ampere translation do not supply bounded shader scheduling, parallel function analysis, adaptive rewrite allocation or concurrent PSO preparation.

Validation on fbdc8e03cb95. DM_GAME=010015100B514000 python3 -m unittest dm.tools.tests.test_docs dm.tools.tests.test_jump_copies dm.tools.tests.test_rewrite_parallel dm.tools.tests.test_rewrite_jobs dm.tools.tests.test_function_parallel dm.tools.tests.test_nsp_incremental dm.tools.tests.test_nsp_chain dm.tools.tests.test_shader_toolchain_cache dm.tools.tests.test_pso_offline dm.gfx.tests.test_dmsh_schedule dm.gfx.tests.test_shader_discovery — 214 passed.

The rebased rewriter retains the new upstream jump-table bound check through the indirect load. The upstream function changes match main and the jump-table regressions passed.

scripts/regress.py check --no-smoke --games 04007EF01F8AE000,0400380020056000,0100BC0018138000,0100A83022A22000,0100F43008C44000 exited 2 at discovery. Fortune’s Weave and Splatoon Raiders exercise the newer shader pipeline, Super Mario RPG and Z-A cover other NVN profiles, and Absolum exercises executable rewriting. Only Z-A was installed. No game comparison ran.

Pinned Python lint passed with zero new or stale findings.

GitHub checks for this revision

Earlier integration tests on ac9d9d8531f6. DM_GAME=010015100B514000 python3 -m unittest dm.tools.tests.test_docs dm.tools.tests.test_nsp_chain dm.tools.tests.test_shader_toolchain_cache dm.gfx.tests.test_dmsh_pairs dm.gfx.tests.test_shader_discovery dm.gfx.tests.test_dmsh_schedule dm.gfx.tests.test_unity_compute dm.gfx.tests.test_unity_known_trees dm.tools.tests.test_rewrite_parallel dm.tools.tests.test_rewrite_jobs dm.tools.tests.test_pso_offline dm.tools.tests.test_nsp_incremental. All 161 tests passed. Pinned Python lint passed with zero new or stale findings.

Each rewrite analysis pass reads its current worker allocation and keeps the requested job chunk size. The pool inherits main’s SIGTERM masking around fork and teardown, default signal action in workers, and orderly close/join. The shared cancellation fixture imports the allocation helper in every subprocess. Gzip and Splatoon NDSR detection, profile promotion and observed-depth selection remain in the combined paths. PSO preparation feeds cached library digests into main’s content-based archive key and refreshes metadata on reuse.

Earlier validation on 7de907d2ad7d passed 186 focused tests and the pinned lint check. Historical stage measurements below come from their stated revisions.

Canonical developer documentation describes the concurrency, serial extraction and archive reuse behavior. The game note retains the timing tables, uncertainty and limitations, with subsystem references under “Found on this game”.

Historical validation: the prior integration passed 109 focused tests, host build/full native host suite, memory/heap ASan/UBSan, memory TSan, real GPU dirty tracking and memory-profile checks. The broader review-follow-up Python campaign reported 1,083 passed and 13 optional skips across 124 suites, and the native graphics suite passed using the installed compiler. Original extraction follow-up produced identical 5,989,103,860-byte images: three alternating serial/parallel medians were 7.997/5.322 s, with wide 6.329–10.278/2.175–5.882 s ranges; stack opening took 0.017 s in one diagnostic run. These local observations neither reproduce nor explain the reported large-image regression. The restored serial extractor matches the baseline image hash.

The latest historical regress.py check --games 0100F43008C44000 --baseline cd859c38 attempt on the scheduling/withdrawal implementation stopped at candidate compiler preflight: installed Mac/iOS Metal 32023.921.6 differs from locked 32023.922.4. No override was used for that check; no shader comparison or gameplay completed. An earlier 28,603-shader cached comparison predates the scheduler and does not validate this chain. Z-A was the only discovered local Mac build; its extraction, modules, shaders and PSO inputs cover the changed stages. Other games remain unchecked locally.

Measurements

Apple M2 Max, macOS 26.6, native ARM Python 3.14. These are individual stage timings, not a complete first-build or cold-disk speedup.

Historical scheduling measurements

Comparison: prior PR head 2a8ee2e5 versus the earlier scheduling implementation. Alternating runs use separate fresh output directories. Shader/Metal caches start empty with no prefill; filesystem caches are not purged. Shaders use eight workers, rewrite uses the automatic budget, and PSO preparation uses eight readers.

Real Legends Z-A v393216 stage	Runs per version	Before median	After median	Reduction
Shader sample, 982 shaders including dependencies	3	123.245 s	116.296 s	5.6%
Full main + sdk rewrite stage	3	35.835 s	23.384 s	34.7%
PSO preparation, 26,051 pairs / 26,029 bases	5	8.862 s	6.983 s	21.2%

Shader and rewrite timings were noisy under filesystem event processing and Spotlight indexing. Shader ranges: baseline 113.822–160.096 s, candidate 113.126–136.036 s. Rewrite ranges: baseline 21.969–37.801 s, candidate 22.808–48.868 s. These medians do not establish precise gains, especially the 5.6% shader result. PSO runs were baseline 24.118, 9.400, 8.862, 7.959, 8.582 s and candidate 7.215, 7.093, 6.983, 6.525, 6.627 s; the median includes all five, including the first baseline outlier.

• The shader sample requests 512 evenly spaced keys, expanding to 982 with dependencies. All 4,816 artifacts match after excluding the expected chain-version field and normalizing only verified filename-derived Metal initializer symbols and their covering MTLB HASH. Of 2,851 libraries, 75 variant libraries in the first comparison differed only in those symbols/hashes: scheduling can select a different first source filename for shared identical MSL. All other bytes matched; raw outputs are retained. Scratch runs explicitly used DM_ALLOW_OTHER_METAL=1 with installed Metal 32023.921.6, not pinned 32023.922.4.
• Rewritten code and tables match byte-for-byte, excluding only the expected build_src.sha256 source fingerprint.
• PSO descriptors, stamps and library digests match exactly. Timing includes planning, serialization, hashing and writing new descriptors/stamps; it excludes metal-tt compilation. Inputs were isolated from b-b7cd656abd7c. An earlier empty-input attempt was discarded and is not included in the results.

Earlier first-build improvements

These earlier stage benchmarks compare saved pre-change sources and were recorded before the publication rebase onto cd859c38. They have not been rerun or relabeled as timings of the latest commit. They use fresh outputs and warm filesystem caches on the same machine. Their baseline differs from the scheduling measurements above; the percentage reductions cannot be added.

Real Legends Z-A v393216 stage	Runs per version	Before median	After median	Reduction
Full main NSO rewrite	3	25.498 s	22.153 s	13.1%
Full shader split, RomFS + main	5	19.338 s	15.706 s	18.8%

Patched code and all six rewriter tables matched; build_src.sha256 intentionally changes with the source. All 16 shader packages/tables matched, covering 28,603 shaders. Shader timings were noisy: baseline 16.395–29.172 s, candidate 15.310–29.586 s, with system indexing active during slow rounds.

Synthetic stage	Runs per version	Before median	After median
SARC names, 8 MiB / 128 members	5	0.535652 s	0.000240 s
IPA, 288 MiB incompressible input	3	5.471 s	1.075 s
IPA, 288 MiB mixed input	3	3.010 s	0.505 s
IPA, 288 MiB compressible input	3	1.222 s	0.309 s
iOS-only Vulkan, 48 shaders	3	2.576 s	1.368 s
iOS-only OpenGL, 48 shaders	3	3.830 s	3.221 s

SARC members and decompressed ZIP payload hashes matched. IPA sizes were unchanged for incompressible input, 151,683,065 → 151,632,138 bytes for mixed input, and 880,171 → 880,514 bytes for compressible input. A 4 GiB + 17 byte Zip64 fixture passed Python CRC and /usr/bin/unzip -t checks. Synthetic Vulkan/GL iOS libraries and tables were byte-identical; these runs used installed Xcode 27.0, not the pinned 27.2 toolchain.

Independent coverage of the earlier published revision

SevenSpiders compared cd859c38 against 2a8ee2e5 on a 10-core / 16 GB Mac, macOS 26.1, Xcode 16.4 / Metal 32023.620, with DM_ALLOW_OTHER_METAL=1. This is attributed external evidence for that earlier head, not validation of the later scheduler or the current review changes.

Game	Successfully built shaders (both revisions)	New failures / changed MSL / new interfaces
Star Wars: The Force Unleashed	206	0 / 0 / 0
Metroid Dread	1,010	0 / 0 / 0
Mario Strikers: Battle League	2,884	0 / 0 / 0
Chants of Sennaar	3,726	0 / 0 / 0
Dodgeball Academia	7,996	0 / 0 / 0
Eastward	56	0 / 0 / 0

Total: 15,878 successfully built shaders; Strikers' three other shaders failed identically on both sides. The reviewer also reproduced SARC, rewrite and discovery improvements, byte-identical outputs and damaged-file reporting. Runtime-root and derived-profile setup are now documented, including the distinction between the harness's automatic profile selection and direct shader-tool invocation.

Remaining validation limitations

• The historical local inventory contained only Pokémon Legends: Z-A (0100F43008C44000, update 2.0.2 / v393216). Its large RomFS, executable modules and shader/PSO inputs exercise the affected paths; additional local games remain unchecked; the six-game external results above cover only the earlier head.
• Installed Xcode 27.0 / Metal 32023.921.6 differs from pinned Xcode 27.2 / Metal 32023.922.4. Full Mac/iOS shader rebuilds and regression comparison with the supported compiler remain outstanding. The scratch shader benchmark's override is explicitly unsupported-toolchain evidence; the repository lock stays unchanged.
• No new gameplay or device run; Z-A has no registered smoke.tsv route. Passing native host/graphics checks is not gameplay validation. No existing player build was replaced.
• No complete cold-disk or end-to-end first-build timing. Scheduling shader/rewrite timings need quieter repeated measurements to establish reliable gains; earlier first-build timings precede the publication rebase.
• The historical 1,083-test campaign reported 13 optional skips. The current 186-test selection completed with zero skips.

Shared geometry fixture CI fix

The fixture in this branch matches the #959 prerequisite in both patch and file contents. Its rebased commit is fbdc8e03cb95.

Prior published commit 128460af4393 applied #959 as a separate prerequisite. On ac9d9d8531f6, LLVM 23.1.4 reproduced both C lint findings in the geometry hazard fixture inherited from main. At that earlier head, the focused lint check passed and the Mac GPU fixture passed all 65 assertions in each of three cut configurations with Metal API validation. The prerequisite changes only the synthetic test. The feature validation and its limitations above still apply to the feature revision they name.

[SevenSpiders]

Verification on a second machine. Base cd859c38, candidate 2a8ee2e5. Apple M-class, 10 cores / 16 GB,
macOS 26.1, Xcode 16.4 / Metal 32023.620 (pinned is 32023.922.4, so everything ran with
DM_ALLOW_OTHER_METAL=1; the three stages changed here do not involve the Metal compiler, so the ratios are still
meaningful, but Metal bytes are not comparable to your runs).

What reproduces

Claim	Your number	Measured here
SARC names bounded (128 members / 7.3 MiB)	0.536 s → 0.00024 s	302.3 ms → 0.372 ms
Rewriter analysis parallel (TFU main)	−13.1 %	−12 % (1.71 → 1.51 s)
Shader split parallel (Dodgeball, 7,996 shaders)	−18.8 %	−7.5 % (2.12 → 1.96 s)
#810 damaged-file reporting	synthetic	real corrupted NCA: identical — exit 3 and the same hash check failed: /Content/Audio/SFXPack.pbn; with --allow-damaged, a byte-identical image
Byte-identical output	Z-A	split (TFU, Dodgeball), rewrite (TFU), extraction (TFU, Sennaar) all identical
Determinism	—	6/6 extraction runs one SHA-256; repeated splits identical

The one that does not

Extraction, nx_extract base+update → RomFS.img, interleaved runs, identical output bytes every time:

image	base	candidate
3.5 GB (TFU)	5.27 / 7.05 / 5.92 s	11.63 / 14.63 / 11.74 s
3.5 GB, NSPs copied to the internal disk	5.02 / 4.18 s	8.51 / 9.54 s
0.5 GB (Sennaar)	1.08 / 0.91 s	0.42 / 0.43 s

So on this machine the parallel extraction is 2.4× faster on a 500 MB image and 2× slower on a 3.5 GB one, and
system time roughly doubles (≈1.0 s → ≈2.2 s on the large image) while user time barely moves. The slowdown is not
the external volume — it reproduces with the NSPs on the internal disk.

Given your note that the timings predate the publication rebase, I suspect the re-open-per-worker path added there
(one LibHac stack per worker, integrity checks) is what costs the system time on a 16 GB machine, and that the
benefit only shows while the working set fits comfortably in cache. Worth re-measuring after the rebase, and worth
knowing that on 16 GB hardware the change can be a regression for large RomFS images.

Regression coverage you listed as missing

regress.py check --no-smoke (chain comparison; #896 changes no shader bytes) on base vs candidate, using each
build's own profile (DM_GAMES_DIR=<build>/profile) and DOTNET_ROOT set:

game	shaders (base → cand)	newly failing	changed MSL	interface
Star Wars: The Force Unleashed 0100153014544000	206 → 206	0	0	0 / 0
Metroid Dread 010093801237C000	1010 → 1010	0	0	0 / 0
Mario Strikers: Battle League 010019401051C000	2884 → 2884 (of 2887, both sides)	0	0	0 / 0
Chants of Sennaar 0100543019CB0000	3726 → 3726	0	0	0 / 0
Dodgeball Academia 010001F014D9A000	7996 → 7996	0	0	0 / 0
Eastward 010071B00F63A000	56 → 56	0	0	0 / 0

No regressions across six games and 15,878 shader slots. (Strikers' three missing shaders are Next Level Games
revision-2 archives that neither tested tree can read; identical before and after.)

Tests

Your five new tests fail on cd859c38 and pass on 2a8ee2e5 (verified by staging the test files into the base
tree). dm/tools/tests 856 passed / 16 skipped, dm/ios/tests 48 passed. dm/gfx/tests has one failure that is
identical on base — Apple metal-tt here rejects .mtl4-json, i.e. a Metal-620 limitation, not your change.

Two harness notes (not your PR)

Filling in for the regress.py runs you could not do, two things needed setting here and are worth documenting:
DOTNET_ROOT (else the translator apphost finds the system .NET 9 and the lock preflight fails), and
DM_GAMES_DIR=<build>/profile for derived-profile builds (otherwise dmsh_build raises IndexError indexing a
12k-record package with a 16k-row repo table). Both bite anyone packing an unregistered version.

No game files or keys are included in this report; all artifacts stay local.

[thehobbit85]

Addressed in 91a9e209.

The parallel extraction change is withdrawn. The original serial LocalStorage / WriteStream path is restored, including #810's repeated-damage recovery. The remaining extractor change recognizes both the raw integrity result and its RomFS translation (2002-4144); synthetic tests cover strict/lenient handling of both.

Fresh post-rebase Z-A runs still favored parallel extraction on this Mac but showed increased system CPU time and wide timing variation. Worker-stack setup took only 0.017 s in the instrumented run, and healthy input never enters the reopen-after-damage path. That does not identify the cause of your large-image regression, so the extraction speedup is removed from the PR's claims rather than presented as fixed. The restored serial path produces the same 5,989,103,860-byte image and SHA-256 as both tested revisions.

Your six-game results are now credited with their exact base/head and unsupported-compiler setting, separately from the later scheduling changes. The development guide now explains DOTNET_ROOT, derived-profile selection, and the DM_GAMES_DIR / DM_GAMES_FIRST settings for direct tool calls.

The follow-up preserves the scheduling work published in 86c4ece3 while this review was being handled. The final local-game regression stops on the new chain's pinned-Metal check (32023.921.6 installed versus 32023.922.4 required); no override was used, and no current-chain gameplay pass is claimed.

The integrated Python suite passes 1,083 tests with 13 optional skips across 124 suites; pinned lint has no new or stale findings. The independent code and integration reviews found no actionable issues.

The native graphics suite also passes on the integrated revision with the installed compiler. The supported-compiler game regression limitation above remains.

[thehobbit85]

CI follow-up is green: all eight checks pass on e309db89 (tests, lint-c, lint-python).

The host and sanitizer failures came from two test fixtures that still expected every guest-memory release to use mprotect. Upstream 8ea27784 now releases pages through MAP_FIXED mmap, with mprotect as its failure fallback. Commits a8375f90 and e309db89 update the fixtures to cover both paths, keeping their state-preservation and fatal-exit assertions. No production behavior or CI checks were disabled.

Both failures were reproduced before the fixes. Focused tests pass against the old and new release implementations; the full host and ASan/UBSan suites also pass locally against upstream with both fixes. The PR description now includes this follow-up and the current CI results.