From 653a5bc637be043682b577ce90e9b614ee32fc62 Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 2 Nov 2026 01:39:46 -0800 Subject: [PATCH] Account for rewritten modules when a VA title sets a heap --- dm/games/README.md | 2 +- dm/tools/tests/test_nsp_incremental.py | 34 ++++++++++++++++++++++++++ docs/development.md | 2 +- docs/reference/memory.md | 2 +- scripts/build_ipa_from_nsp.py | 3 ++- 5 files changed, 39 insertions(+), 4 deletions(-) diff --git a/dm/games/README.md b/dm/games/README.md index ae7dd45af..e0e3037da 100644 --- a/dm/games/README.md +++ b/dm/games/README.md @@ -266,7 +266,7 @@ These are not profile keys: the tools read them from the game files each time, b `python3 dm/games/profile_guess.py --check` runs these rules on every profile in `dm/games` (with the extractions under `/Volumes/T7/nsp`) and lists the keys where the guess differs. - After the rewrite, `mem_used` is set when the rewritten modules exceed the default 64 MB (a game without a system resource); keep it when the profile moves into `dm/games` (Cuphead without it: a 0.03 GB heap and an abort at the first allocation). A game with a system resource that sets a heap then gets a **heap check**: it boots in the Mac host (headless, no shader libraries) until its first heap request, or 5 s into `nnMain` without one, at most 30 s (`HEAP_CHECK_SECS`). A request the heap region cannot hold (`SetHeapSize OOM`) sets the alias / heap split (Tears of the Kingdom asks for 0x86000000, which gives its hand-made 0x98000000 alias region; Wonder's 1.44 GB gives 0xc0000000), and a pool check that fails for good sets `mem_used`. A run with no heap set whose alias region is used up to within 0.25 GB of its end sets a 1 GB pool: `nninitStartup` handed pool − used to StandardAllocator as one address region and the game reserves regions of its own beside it (Unity 2020 and 2022: AI: THE SOMNIUM FILES - nirvanA Initiative reached +4.26 GB of 4.38 in 5 s with the hardware pool; Unity 2018.3's Raging Loop maps only inside StandardAllocator's region and keeps it). The 1 GB is a starting point, not a measurement: Advance Wars 1+2: Re-Boot Camp (Unity 2020.2) ran its malloc dry at its first loading screen with it and needs 2 GB. The profile gets a `# heap check:` line, and later runs keep its keys (delete the line to check again; `--no-heap-check` / `HEAP_CHECK = False` skips it). Other games skip the check, so it costs nothing for them and a few seconds plus a Mac host build for the rest. + After the rewrite, `mem_used` is set when the rewritten modules exceed the default 64 MB (a game without a system resource, or one whose main imports the heap setter); keep it when the profile moves into `dm/games` (Cuphead without it: a 0.03 GB heap and an abort at the first allocation). A game with a system resource that sets a heap then gets a **heap check**: it boots in the Mac host (headless, no shader libraries) until its first heap request, or 5 s into `nnMain` without one, at most 30 s (`HEAP_CHECK_SECS`). A request the heap region cannot hold (`SetHeapSize OOM`) sets the alias / heap split (Tears of the Kingdom asks for 0x86000000, which gives its hand-made 0x98000000 alias region; Wonder's 1.44 GB gives 0xc0000000), and a pool check that fails for good sets `mem_used`. A run with no heap set whose alias region is used up to within 0.25 GB of its end sets a 1 GB pool: `nninitStartup` handed pool − used to StandardAllocator as one address region and the game reserves regions of its own beside it (Unity 2020 and 2022: AI: THE SOMNIUM FILES - nirvanA Initiative reached +4.26 GB of 4.38 in 5 s with the hardware pool; Unity 2018.3's Raging Loop maps only inside StandardAllocator's region and keeps it). The 1 GB is a starting point, not a measurement: Advance Wars 1+2: Re-Boot Camp (Unity 2020.2) ran its malloc dry at its first loading screen with it and needs 2 GB. The profile gets a `# heap check:` line, and later runs keep its keys (delete the line to check again; `--no-heap-check` / `HEAP_CHECK = False` skips it). Other games skip the check, so it costs nothing for them and a few seconds plus a Mac host build for the rest. No short run decides `render_cond`, `tex_flush_upload`, `tex_dirty_upload`, `core_preempt`, `gpu_timestamps`, `vconv_range` or `present_interval`, nor `op_mode` / `pad_layout` outside the presets: they keep their defaults for a developer. A minute's boot without input counted Wonder's clears under render conditions (65,536 or more; it needs no `render_cond`) as readily as Tears of the Kingdom's, saw Breath of the Wild flush 32 CPU-filled textures with data (it needs no `tex_flush_upload`) and Age of Calamity none (its movie frames come later), and the old host's Metroid Dread movie crash showed in about half the runs (now resolved by [shared wake handoffs](../../docs/reference/threads.md#same-core-wake-handoffs)). On the iPhone, `Documents/env.txt` can still set `DM_OPMODE`, `DM_NVN_GPU_TS` and `DM_NVN_FB_SNAP`, and the pad layout switches in game. diff --git a/dm/tools/tests/test_nsp_incremental.py b/dm/tools/tests/test_nsp_incremental.py index 454798b99..20b3ae1c3 100644 --- a/dm/tools/tests/test_nsp_incremental.py +++ b/dm/tools/tests/test_nsp_incremental.py @@ -236,6 +236,40 @@ def test_provisional_computed_mem_used_grows_shrinks_and_stays_stable(self): nsp.refine_after_rewrite(g, self.ex, self.tmp / 'patched') self.assertEqual(g.table('game.tsv').read_bytes(), before) + def test_va_heap_accounts_for_rewritten_modules(self): + from dm.tools.tests.test_nso import make_nso + import struct + + self.derive() + exefs = self.ex / 'ExeFS' + meta = bytearray(0x80) + meta[:4] = b'META' + struct.pack_into('`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME=`. Profiles are named after the base game's title ID. There is no default game: without `GAME` / `DM_GAME` the tools stop and list the profiles. `python3 dm/games/game.py list` lists the profiles. Host code that relies on one build's fixed addresses checks that build's ID. To add a game: -1. Run `scripts/build_ipa_from_nsp.py` on its NSP folder, or on a Switch 2 game's `.nup` file or dump folder. With no matching profile, it writes a survey to `_survey/survey.md`. The survey covers the modules, SDK, engine, containers, rewrite sites, and the imports compared with the host's hooks. It then writes a provisional profile into `<title ID>_build/profile/<title ID>/` (with `--work-dir`, into that folder's `profile/`, so a front end that names a fresh build folder per build keeps its profile with it; a profile from an earlier run in the default `<title ID>_build/profile` is copied there once when the new folder has none, and stays; outside the repository; tools find it through `DM_GAMES_DIR`) with the keys `dm/games/profile_guess.py` reads off the game files: the graphics API, the modules to load, where the shaders come from, the memory partition, SDK and engine presets. After the rewrite it sets `mem_used` from the rewritten modules and, for a game with virtual address memory that sets a heap, boots it on the Mac until its heap request (the heap check, a few seconds) to size the heap region. A game with virtual address memory also gets the address region check, which says whether its reservations fit the iPhone's alias region and who makes those that do not. Then it generates the shader tables and chain header and packs a best-effort IPA that may not boot. `--survey-only` stops after the survey, `--no-heap-check` skips the heap check and the region check, and `--profile-out` puts the profile elsewhere (default `<build folder>/profile`). `--provisional` does all this for a game that has a profile, to test the guesses; `python3 dm/games/profile_guess.py --check` compares the static guesses with every profile. Details in [dm/games/README.md](../dm/games/README.md). +1. Run `scripts/build_ipa_from_nsp.py` on its NSP folder, or on a Switch 2 game's `.nup` file or dump folder. With no matching profile, it writes a survey to `<title ID>_survey/survey.md`. The survey covers the modules, SDK, engine, containers, rewrite sites, and the imports compared with the host's hooks. It then writes a provisional profile into `<title ID>_build/profile/<title ID>/` (with `--work-dir`, into that folder's `profile/`, so a front end that names a fresh build folder per build keeps its profile with it; a profile from an earlier run in the default `<title ID>_build/profile` is copied there once when the new folder has none, and stays; outside the repository; tools find it through `DM_GAMES_DIR`) with the keys `dm/games/profile_guess.py` reads off the game files: the graphics API, the modules to load, where the shaders come from, the memory partition, SDK and engine presets. After the rewrite it sizes `mem_used` from the loaded modules for games without virtual address memory and for games whose main imports the heap setter. For the latter with virtual address memory it boots on the Mac until the first heap request (the heap check, a few seconds) to size the heap region. A game with virtual address memory also gets the address region check, which says whether its reservations fit the iPhone's alias region and who makes those that do not. Then it generates the shader tables and chain header and packs a best-effort IPA that may not boot. `--survey-only` stops after the survey, `--no-heap-check` skips the heap check and the region check, and `--profile-out` puts the profile elsewhere (default `<build folder>/profile`). `--provisional` does all this for a game that has a profile, to test the guesses; `python3 dm/games/profile_guess.py --check` compares the static guesses with every profile. Details in [dm/games/README.md](../dm/games/README.md). 2. Add a profile as described in [dm/games/README.md](../dm/games/README.md). 3. Bring the game up following [docs/agents](agents/README.md): extraction, binary survey, a catalogue of host bring-up failures, graphics, and debugging recipes. diff --git a/docs/reference/memory.md b/docs/reference/memory.md index 41ff68d20..5f8dfde69 100644 --- a/docs/reference/memory.md +++ b/docs/reference/memory.md @@ -34,7 +34,7 @@ The guest address space, heaps, pools, virtual address memory and the allocators - **Symptom:** `nnMain` returns right after `nv::InitializeGraphics`, with no error (Hyrule Warriors: Age of Calamity); `[mem] SetHeapSize … + modules … > app pool` (Monster Hunter Rise), `[mem] SetHeapSize … + modules 0x5c00000 > app pool` (Catherine: Full Body); `SetHeapSize` asks for almost the whole application pool and fails at boot (Metroid Dread: 0xc9400000, larger than the heap region); the memory pool for the game's 96 MB GPU heap has no storage (`MemoryPoolInitialize` red line), then a null read at `main+0x2eb954` (Catherine: Full Body); heap allocations fail late in a busy scene. - **Cause:** the SDK sizes the heap as total − used memory (`QueryMemoryInfo`), so the reported application pool and used memory decide the game's heap size. Age of Calamity's `nninitStartup` does so, calls `SetMemoryHeapSize` and hands the block to its own malloc. The host reported 64 MB used beside the heap ("used" = heap + 64 MB), but `SetHeapSize` checks a request against the real size of the loaded modules (66.6 MB in Age of Calamity), so `SetHeapSize(pool − 64 MB)` failed with ResultLimitReached, `AllocateMemoryBlock` returned 0x1003, malloc was never set up, the 866 MB graphics heap allocation came back null and `nnMain` returned. Monster Hunter Rise's modules take 320 MB (`main` alone 314 MB), so its `SetHeapSize(pool − 64 MB − bss)` failed the same way. Catherine: Full Body's rewritten modules take 92 MB, so `SetHeapSize(pool − 64 MB)` failed and the game ran on a 32 MB heap. The 64 MB covers ordinary modules (Mario's two are about 74 MB, and its heap request matches hardware), but Metroid Dread's `main` carries a 1.8 GB bss that the Switch counts as used. -- **Fix:** report Switch-accurate numbers. The application pool was a host constant (`APP_POOL`, 3285 MB); it is now the profile key `app_pool` (`hle_svc.c`: `#define APP_POOL DM_GAME_APP_POOL`, Mario 0xCD500000, the same 3285 MB). The profile key `mem_used` sets the used-memory figure: 72 MB for Age of Calamity (the image plus stacks); 328 MB for Monster Hunter Rise, which gives a 2.88 GB heap; 96 MB for Catherine: Full Body, the NSP script's guess after the rewrite. Used memory now also adds the bss of modules in the front-patch layout (`big_bss_bytes` in `hle_svc.c`; the layout is in [Patch area before the module base for a large bss](rewriter.md#patch-area-before-the-module-base-for-a-large-bss)); Dread then asks for 1.38 GB. +- **Fix:** report Switch-accurate numbers. The application pool was a host constant (`APP_POOL`, 3285 MB); it is now the profile key `app_pool` (`hle_svc.c`: `#define APP_POOL DM_GAME_APP_POOL`, Mario 0xCD500000, the same 3285 MB). The profile key `mem_used` sets the used-memory figure: 72 MB for Age of Calamity (the image plus stacks); 328 MB for Monster Hunter Rise, which gives a 2.88 GB heap; 96 MB for Catherine: Full Body, the NSP script's guess after the rewrite. Used memory now also adds the bss of modules in the front-patch layout (`big_bss_bytes` in `hle_svc.c`; the layout is in [Patch area before the module base for a large bss](rewriter.md#patch-area-before-the-module-base-for-a-large-bss)); Dread then asks for 1.38 GB. The packer also measures the rewritten modules for a virtual-address title whose main imports the heap setter. Its first probe can hit the heap region limit before the host checks the application pool, so resizing the region alone can leave a second boot with too little reported used memory ([MIO, issue #941](https://github.com/seedds/nxnative/issues/941)). - **Found on:** [Hyrule Warriors: Age of Calamity](../../dm/games/01002B00111A2000/README.md); [Monster Hunter Rise](../../dm/games/0100B04011742000/README.md); [Metroid Dread](../../dm/games/010093801237C000/README.md); [Catherine: Full Body](../../dm/games/0100BF00112C0000/README.md) (SDK 8.2.1; one boot). The source names no game for the late failures in a busy scene. - **Check:** probe `DM_SVC=1`'s `svc 0x1` request and the `AllocateMemoryBlock` result. diff --git a/scripts/build_ipa_from_nsp.py b/scripts/build_ipa_from_nsp.py index bec2ce66a..0e579eed5 100755 --- a/scripts/build_ipa_from_nsp.py +++ b/scripts/build_ipa_from_nsp.py @@ -1919,9 +1919,10 @@ def refine_after_rewrite(g, ex_dir, patched, reg=None): computed = section == '# ---- After the rewrite ----' and reason.startswith(('# used memory for the rewritten modules', '# used memory for the current rewritten modules', '# the rewritten modules are larger than the default 64 MB')) if not computed: return g reason = line - if pg.npdm(ex_dir / 'ExeFS/main.npdm')[0]: return g used = pg.mem_used(patched, g.modules_in(patched)) or 0x4000000 if int(g.get('mem_used', '0x4000000'), 0) == used: return g + exefs = ex_dir / 'ExeFS' + if pg.has_va(exefs) and not pg.heap_check_needed(exefs): return g say(f'mem_used {used:#x}: from the current rewritten modules (64 MB when the default covers them)') return set_keys(g, {'mem_used': (f'{used:#x}', 'used memory for the current rewritten modules, at least the default 64 MB')}, 'After the rewrite')