Skip to content

Size used memory from rewritten modules when a virtual-address title sets a heap - #950

Open
thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/va-heap-used-memory
Open

thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/va-heap-used-memory

Conversation

@thehobbit85

@thehobbit85 thehobbit85 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Current revision and validation

76e18dfac53d2a87e67cd059fd914ed98486e1dc is one commit directly atop freshly fetched main 975bb4c14456d145eb3d969814534eecdd996ce9. The existing PR branch and canonical tracking are retained. Rebase range-diff and remaining changes were reviewed; original commit information is preserved below.

161 focused tests passed on this exact single-commit head: python3 -m unittest dm.tools.tests.test_docs dm.tools.tests.test_profile_guess dm.tools.tests.test_nsp_incremental dm.tools.tests.test_nsp_mac_app. The rebase preserves the measured-module/heap-import behavior and explicit profile overrides while retaining main's current module enumeration. The prior integration's source changes are retained; main's newer task/mesh ABI and game note are included. git diff --check passes.

No gameplay, device run, game build or shader comparison was performed. Fresh read-only discovery using the current regression harness found no usable Mac builds under /Users/thehobbit85/Games. MIO (reported heap abort), Wonder and TotK (virtual-address heap consumers), and Fire Emblem Engage (virtual-address non-heap control) remain the impacted comparison targets; that coverage is unavailable. The earlier shared-main Python lint B023 finding remains unresolved; its source is unchanged by this PR. Historical results below retain their tested revisions and toolchain limits.

Remote CI on this replacement head is pending Agent 3 verification; earlier checks do not validate it.

Prior proposal and validation — historical

The text below predates this refresh. Its revision labels and unresolved limitations remain applicable as stated.

Refs #941 (the heap abort at boot).

Current revision fcd16339aabb068eb8591146d80a54b173ad0c67, rebased onto upstream main bd98317eecfe832b40d401219c15207a398e78dd. Range-diff and the remaining diff were reviewed; main is an ancestor.

Fresh focused validation on this revision: 97 tests passed. The measured-module sizing change and its controls are preserved unchanged. A configured regression-harness attempt exited 2 before execution because Mac builds for MIO, Super Mario Bros. Wonder, Tears of the Kingdom and Fire Emblem Engage are unavailable. These cover the reported failure, virtual-address heap users and a virtual-address non-heap control. Only Legends Z-A was found locally. No gameplay was run.

Remote CI for this replacement head is pending verification by the CI agent; earlier successful checks do not validate it. Current checks.

Prior revision context and validation

The following evidence predates this rebase and retains its stated revisions and limitations.

MIO requests a heap from the available application pool after subtracting the memory the host reports as used. Its rewritten modules occupy 68 MiB, but the packer left the default at 64 MiB because the NPDM declares a system resource. The first heap probe can hit the virtual region limit before the host checks the application pool. Resizing the region then exposes the second failure on the next boot.

When the measured module size calls for a different mem_used, use the existing check for main’s heap-setter import to distinguish virtual-address titles that use the heap. This gives MIO’s reported layout a 72 MiB reservation. Explicit profile values still take precedence, and virtual-address titles without the import keep their former sizing. The change applies to provisional and derived profiles before the Mac heap probe, including reused profiles. It does not modify the reported draw-count limit or resolve the flat-shading question.

Revision 66c60fccb54bcbb94ea495b09711a837a26de4e0 includes main 65501fbbaf8526f7aab0a1edd028135888590f8e. Main still excludes virtual-address titles from measured module sizing without checking the heap-setter import.

Validation on 66c60fccb54b. python3 -m unittest dm.tools.tests.test_docs dm.tools.tests.test_profile_guess dm.tools.tests.test_nsp_incremental — 85 tests passed.

GitHub checks for this revision

Author validation on aaf0c9445481

  • A generated META header declares a 32 MiB system resource, a real synthetic NSO carries one dynamic import, and a sparse fixture file represents the 68 MiB rewritten image. On the unmodified packer, the heap-import case failed with 67108864 != 75497472. With the fix the calculated heap plus the rewritten modules fits the reported 0xcd500000 pool. The otherwise identical virtual-address control imports only NVN and keeps 64 MiB.
  • The 85 focused incremental-pack, profile-guess and documentation tests passed. Full suites passed: 888 tool tests (16 skipped), 190 graphics Python tests (18 skipped), and 42 iOS packaging tests. Pinned Python lint reported 391 baseline findings, zero new and zero stale. Generated symptoms and git diff --check passed.
  • Native graphics suite make -j2 -C dm/gfx/tests GAME=010015100B514000 run exited 0. Six optional translated-shader reflection samples were unavailable. Local environment: Apple M2 Max, macOS 26.6, Xcode 27.0. The build reported Apple Metal 32023.921. The supported .922 toolchain was not validated.
  • scripts/regress.py impact c646d43d..aaf0c944 marked four games under the broad packing rule, zero direct and 125 unknown. For the narrow profile change the intended checks were MIO (reported failure), Super Mario Bros. Wonder and Tears of the Kingdom (virtual address memory and a heap), and Fire Emblem Engage (virtual address memory without a startup heap). regress.py check --games 01000A001E978000,010015100B514000,0100F2C0115B6000,0100A6301214E000 exited 2 before execution because REGRESS_ROOT and REGRESS_GAMES are unset. No game assets were available, so neither a game build nor gameplay was verified.

Updated the profile-generation guidance and the existing heap-size reference. No game-specific profile or runtime changes.

Original commit context preserved through this squash

These messages retain historical intent and author credit. Validation mentioned in them belongs to the original revision, not automatically to the current head.

Original fcd16339aabb068eb8591146d80a54b173ad0c67 — seedds seedds@gmail.com

Account for rewritten modules when a VA title sets a heap

@thehobbit85
thehobbit85 force-pushed the codex/va-heap-used-memory branch 4 times, most recently from 20e079a to 66c60fc Compare November 2, 2026 17:44
@github-actions github-actions Bot added packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts) docs Documentation labels Nov 3, 2026
@thehobbit85
thehobbit85 force-pushed the codex/va-heap-used-memory branch from 66c60fc to fcd1633 Compare November 4, 2026 00:45
@thehobbit85
thehobbit85 force-pushed the codex/va-heap-used-memory branch from fcd1633 to 76e18df Compare November 4, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants