Repository navigation
Yakuza Kiwami plays on the Mac and the iPhone; ADR jump tables with a far bounds check; PARC/SLLZ shader split - #912
Open
joshhhhyyyy wants to merge 5 commits into
Open
joshhhhyyyy wants to merge 5 commits into
joshhhhyyyy wants to merge 5 commits into
Conversation
…scheduled far before their branch Yakuza Kiwami 1.01's main stopped the rewrite with `AssertionError: jump table 91b24c`. The dispatch at main+0x91b24c (b.hi; mov w16, w21; adrp / add of the table; adr x18; ldrsw; add; br x18) checks its index with `cmp w21, seedds#3` 31 instructions before the br: the compiler scheduled 23 floating-point and bitfield instructions between the compare and its b.hi. jump_adr_offsets looks 16 instructions back for the compare and the other forms expect it right before the b.hi; a wider window alone would not prove that the flags the b.hi reads come from it. jump_adr_far_check, tried after every other form so the tables they recognize keep their targets, finds the check by the flags: back from the load through straight-line code the first control flow must be the b.hi, and before it the first instruction that writes the flags must be a cmp of the index with an immediate. Between that compare and the br nothing may call, branch, trap or write the index, the entry or x18, no branch or known table of the function may enter the run, and every target must be an instruction of the function outside it. The entry is in docs/reference/rewriter.md, "ADR-relative jump table whose bounds check is scheduled far before its branch". Checked: test_jump_copies.py's AdrFarCheck (the far and the near check, every kind of flag writer between the check and its branch, writes of each register involved, control flow into and out of the run, undecodable words, the real rewriter on a synthetic module). Yakuza Kiwami's main rewrites, the table resolving to four entries of which 1 and 3 share a case block; a census of its modules found no other unrecognized table. On main the rewrite of that main stops with the assertion; with this change its seven output files are byte for byte those of the build the game was played on. The new form runs only where every older one failed, which stopped the rewrite before, so a module that rewrote before rewrites the same: Tomodachi Life: Living the Dream 1.0.4's main and sdk, the other game on hand, do byte for byte. ruff and pyright find nothing new. Co-Authored-By: Claude <claude@users.noreply.github.com>
Yakuza Kiwami's NSP build stopped at `the shader split found no shaders in main or the RomFS`, after `[RED] main control section @... no code area` lines. Ryu Ga Gotoku Studio's engine keeps nearly all its files in PARC archives, most of them SLLZ-compressed, and its shaders are shader objects (GSVS4 .vso, GSPS4 .pso, GSFX4 .fxo) that hold whole GLSLC outputs, all SLLZ version 1, so a scan of the stored bytes finds no magic. The engine's own objects are SLLZ streams in main's data, whose literals copy the GLSLC and control magics verbatim: those were the control sections with no code area. shader_split.py lists a file that starts with PARC by its tables (par_entries) and decompresses only the files whose first 8 decoded bytes are a shader object, a GLSLC output or another PARC archive (par_shaders; sllz peeks), so the archives of models, textures and motions are only listed. Each GLSLC output is taken whole, one stage each, and the pixel-input cover pairs them. sllz_main_objects reads the objects in main, raw_shaders leaves out control magics inside SLLZ streams, and a file that is SLLZ as a whole is decompressed first. One damaged file is left out with a red line, not its archive. The layouts follow ParManager's ParLibrary (Kaplas80), whose SLLZ decoder credits Rick Gibbed. The entry is in docs/reference/shaders.md, "GLSLC outputs in Ryu Ga Gotoku's PARC archives and SLLZ streams". Yakuza Kiwami 1.01: 0 -> 2,462 shaders (1,259 vertex, 1,193 pixel, 9 geometry, 1 compute). 2,399 are in the four archives under data/shader (3,172 objects with 3,752 outputs) and 63 more in 59 of main's 60 SLLZ streams, among them the only geometry and compute shaders. Its other 2,969 archives (17 GB) hold no shader object; data/chara.par (1.5 GB) is listed in 1.6 s. The new paths run only on files that start with PARC or SLLZ and on SLLZ streams. Tomodachi Life: Living the Dream, the other game on hand, has none: no file of its RomFS (66,343) starts with either, and its main (54 MB) holds no SLLZ byte, so its split is unchanged. Checked: test_split_par.py, 11 tests (SLLZ versions 1 and 2 in both byte orders, overlapping copies, peeks and damaged streams; PARC listings in both byte orders with subfolders; stored and compressed nested archives; only shader objects decompressed; one damaged file left out alone; .par and SLLZ-wrapped files through scan_file_in; main's streams and raw_shaders' skip). Co-Authored-By: Claude <claude@users.noreply.github.com>
Update 1.01 (v65536), Ryu Ga Gotoku Studio's engine with CRIWARE, SDK 18.3.2, on NVN: a profile with its note and games table row. Played on the Mac (M3 Pro) and an iPhone19,4 (A20 Pro, iOS 27.2): boot, the title, the story with its cutscenes and fights. No game-specific rewriter cut or host change: it needed the two general fixes before this commit. The game maps all its memory itself. main reserves a 32 GB address region (main+0x379938) and its own allocators carve 13 GB of reservations from it (a 4 GB growable heap and pools of 1, 4, 2 and 2 GB), about 1.5 GB of them used at the title. The profile holds them unchanged in a 36 GB alias region, one 36.62 GB block (mem_reserve 0x928000000, mem_alias 0x900000000): the regions reached +33.72 GB in the NSP script's address region check, the highest mapped page +33.75 GB in play, with at most 2.25 GB mapped. A standalone mmap probe on the iPhone19,4 found one 64 GB range with the Increased Memory Limit entitlement the builds carry by default (6 GB with neither entitlement), so the block fits there; docs/reference/memory.md records it. A phone limited to about 6 GB would need the game's sizes cut (the note's Open). The other keys are the provisional profile's static guesses, kept because they boot and play; profile_guess.py --check differs only in mem_reserve, the measured layout. The shader tables are the provisional profile's, byte for byte (the split of the commit before: 2,462 shaders), and shader_groups / shader_note stay as it wrote them, as every shader_* key is in each shader's cache key. The version is v65536, the update's, which the NSP script reads from the [v65536] tag in an update's file name (a single update without one is taken for it). The Mac run (24,600 frames) and the iPhone run (4,500) printed no [RED] line, no unimplemented call and no shader key missing from the map, and neither crashed. Most frames take 33.3 ms (30 fps), with stretches at 60: the game paces itself under present interval 1. Peak memory: Mac 2.54 GB resident and 1.39 GB of GPU allocations, iPhone 2.04 GB and 1.36 GB, with 6.15 GB still below its memory limit at the lowest. Those runs had an older form of the sdk's cuts than this tree's SDK-wide rule (regions of 1 GB and more aligned to 32 MB, instead of 2 MB from 4 MB up); the note says so. Co-Authored-By: Claude <claude@users.noreply.github.com>
The gallery picture is a frame of a street fight from a Mac run of this branch's own build (DM_NVN_SHOT, frame 6000, 1280x720 scaled to 960x540). That run took the sdk's SDK-wide cuts (2 MB alignment from 4 MB up) where the earlier runs had the older per-build ones: 8,400 frames from a save into street fights, the regions' highest mapped page at +33.65 GB of the 36 GB alias region (+33.75 GB before), peak 2.40 GB resident and 1.38 GB of GPU allocations, no crash, no unimplemented call, no shader key missing from the map. Its one red line, a 3D texture bound in a shader's cube slot once in a fight, is in the note's Open. Co-Authored-By: Claude <claude@users.noreply.github.com>
joshhhhyyyy
force-pushed
the
yakuza-kiwami
branch
3 times, most recently
from
November 2, 2026 18:06
2bcfa14 to
fd279cb
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
joshhhhyyyy
force-pushed
the
yakuza-kiwami
branch
from
November 2, 2026 18:07
fd279cb to
c4665a3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Yakuza Kiwami (
0100C9801FEE6000, update 1.01 = v65536, SDK 18.3.2, Ryu Ga Gotoku Studio's engine with CRIWARE, NVN) plays on the Mac and on an iPhone: boot, the title, the story with its cutscenes and fights. Four commits, the first two standing on their own:Rewriter (
dm/tools/patch_nso.py,jump_adr_far_check): an ADR-relative jump table whosecmpis scheduled 31 instructions before itsbr(main+0x91b24c: 23 float and bitfield instructions between thecmpand itsb.hi) stops the rewrite onmainwithAssertionError: jump table 91b24c. The new form finds the check by the flags theb.hireads, not by a distance, and is tried only after every other form, that is only where the rewrite used to stop. Entry:docs/reference/rewriter.md, "ADR-relative jump table whose bounds check is scheduled far before its branch".Shader split (
dm/gfx/tools/shader_split.py): the game keeps its shaders as whole GLSLC outputs in SLLZ-compressed shader objects (.vso/.pso/.fxo) inside PARC archives, and the engine's own in SLLZ streams inmain, so the split found none and the NSP script stopped. It now lists PARC archives by their tables, decompresses only the shader objects (SLLZ versions 1 and 2, after ParManager's ParLibrary), and readsmain's streams: 0 → 2,462 shaders. Entry:docs/reference/shaders.md, "GLSLC outputs in Ryu Ga Gotoku's PARC archives and SLLZ streams".Profile and note (
dm/games/0100C9801FEE6000/, games table row): the provisional profile's keys, except the memory layout. The game reserves a 32 GB address region and carves 13 GB of reservations from it (a 4 GB heap and pools of 1, 4, 2 and 2 GB), so the profile gives it one 36.62 GB block with a 36 GB alias region (mem_reserve 0x928000000,mem_alias 0x900000000). An iPhone19,4 on iOS 27.2 reserves that with the Increased Memory Limit entitlement the builds carry by default: a standalonemmapprobe found a 64 GB largest block with it and 6 GB without (docs/reference/memory.md). The shader tables and theshader_*keys are the provisional profile's, unchanged, as everyshader_*key is in each shader's cache key (as in The Legend of Zelda: Echoes of Wisdom (01008CF01BAAC000, base game 1.0.0): game profile #152).Gallery (
docs/images/yakuza-kiwami.jpg,gallery.tsv, README regenerated): a frame of a street fight from a Mac run of this branch.No game-specific rewriter cut, no host change, no shader-chain change.
Checked
DM_GAME=010015100B514000as in CI): every file ofdm/tools/tests,dm/ios/testsanddm/gfx/tests.test_docs.py, the newtest_split_par.py(11 tests),test_jump_copies.pywith the newAdrFarCheckcases andtest_game_profile.pypass; 23 other files fail there with the same counts onmain(they need macOS tools or a newer Python). ruff 0.17.2 and pyright 1.1.414, the pinned versions: no new finding in the changed files againstmain. Host and GPU tests not run: no host or graphics code changed.main(all seven output files),subsdk0and the game's NRO rewrite byte for byte the same; onmain, the rewrite of thismainstops with the assertion. Reach: the new form runs only where every older one failed, so a module that rewrote before rewrites the same: Tomodachi Life: Living the Dream 1.0.4'smainandsdk, the other game on hand, do byte for byte.shader_split.pyhere is byte for byte the version that split the game (2,462 shaders: 1,259 vertex, 1,193 pixel, 9 geometry, 1 compute), and the profile's tables are that split's. Reach: the new paths run only on files that start withPARCorSLLZand on SLLZ streams; Tomodachi Life's 66,343 RomFS files and itsmainhave none.profile_guess.py --check: the guesses match the profile exceptmem_reserve, the measured layout (mem_aliasshows under "heap check / measured").mainwith other PRs merged (below):[RED]line, no unimplemented call, no shader key missing from the map. Most frames take 33.3 ms (30 fps), with stretches at 60, as the game paces itself under present interval 1. Peak 2.54 GB resident and 1.39 GB of GPU allocations; the regions' highest mapped page +33.75 GB of the 36 GB alias region, 2.25 GB mapped at most.0x7000000000. No[RED]line, no crash; peak 2.04 GB resident and 1.36 GB of GPU allocations, 6.15 GB below the memory limit at the lowest; thermal state 0.build_ipa_from_nsp.py --platform macfrom this tree): extraction reused;main,subsdk0and the NRO rewritten byte for byte as above, thesdkwithmain's 5 SDK-wide cuts; the split 2,462 shaders, its tables passing the check against this profile's; 2,462 Mac shaders and the host built. Then 8,400 frames with a window, from a save into street fights: no crash, no unimplemented call, no shader key missing from the map; the regions' highest mapped page +33.65 GB (+33.75 GB with the older cuts below); peak 2.40 GB resident, 1.38 GB of GPU allocations. One red line, once, at frame 2614 as a new pipeline was made:texture type mismatch: shader d303b2b1a0f401f7_b (stage 1) declares slot 134 as Metal type 5, texture ID 6493 is type 7 (NVN target 2, format 0x9a, Metal format 80); bound as is(a 3D texture in a cube slot; nothing wrong on screen; the note's Open). The gallery frame is from this run.regress.pyneeds macOS, and the reach of both tool changes is nil by construction (the two "Reach" checks above), so no game's run can change.Tested with
The earlier Mac and iPhone runs were made on a local
main(2026-11-02, before #855 and #689) with these PRs merged, none of them part of this PR or needed by it: #627 (Port Tomodachi Life: Living the Dream 1.0.4), #683 (iOS: adapt gameplay to Duo portrait and tabletop poses) and #812 (translate shaders the split missed at run time with maxmetal), plus a small local iOS build fix. #812 is on by default, and its host prints a[RED] shader … is not in the shader map: maxmetal translated it at run timeline for each shader it translates; neither run printed one, so every shader came from this profile's map.One difference follows from #627: its branch predates 17f670a, and merging it brought back the old per-build cut rows for this game's
sdk(f118ae1c…, Absolum's SDK 18.3.2 build). So the earlier runs had regions of 1 GB and more aligned to 32 MB, where this branch hasmain's SDK-wide rule (2 MB alignment from 4 MB up, as SDK 20), which only takes away alignment slack in front of the game's regions: the run of this branch above reached +33.65 GB where the older cuts reached +33.75. The game note records it.Not checked
sdkcuts; the new rule only lowers the regions' reach).Docs
docs/reference/rewriter.md,docs/reference/shaders.md,docs/reference/memory.mdand the game note, andpython3 dm/tools/docs_symptoms.pywas run (the symptom index)dm/gfx/nvn/dmsh_chain.huntouched)🤖 Generated with Claude Code