[thehobbit85]

Refs #1022. On a large offline build, Python formatted the entire pipeline description, converted it to bytes, and read the prior script in full to compare it. It also allocated another complete JSON representation for the archive key and retained the descriptors and script bytes while metal-tt was running. This addresses the Python portion of victorwads’s report.

Validation on 1efc4fdaa712

Current revision. Tested f839d61651d8abc58005dd67a0e5380f6e916345 on main c3e662455db727f0f4cf182a647a8382904faa28. The Git tree matches a clean integration of the feature and upstream changes. Documentation, Xcode fixture and Mac packaging checks, including the shader-cache pruning cases, ran on this revision (ran 75 tests, OK (skipped=1)).

Tested commit 1efc4fdaa71283dd2931778b6cd4e1813a534f20 against main bd688163afea8a7cf6d5e5f889e177ad1d589711. The archive key tracks the pipeline configuration and the compiled library contents, so moving a library leaves its key intact. The builder streams the pipeline script and frees its descriptor table before compilation. Of 87 tests covering script output, archive keys, LINKDATA inputs, toolchain caching and documentation, 86 passed and one was skipped. The impacted-game check (scripts/regress.py check --no-smoke --games 01006FE013472000,0100F43008C44000,01008CF01BAAC000,01007EF00011E000,0100F2C0115B6000) stopped at build discovery. Only Pokémon Legends: Z-A was available locally. No game comparison ran. CI checks.

Earlier results through 925bd92b1506

Earlier head 925bd92b1506d9e2c2b5049433ad1da47a838dab. python3 -m unittest dm.tools.tests.test_docs dm.tools.tests.test_pso_offline dm.tools.tests.test_split_unreadable passed on this revision (48 tests). The feature sources, tests and documentation match the previously validated revision b9196ee8692a byte for byte.

That earlier revision included main 6136418bcd564f086cf57cb0a59e8f997d36445e. Main still materializes the offline pipeline script. This PR streams it while preserving the script, archive key and memory regression coverage. The upstream generator now supplies the Pyright fix formerly carried from #1019. The distinct LINKDATA job-path behavior and its regression remain.

Local validation on b9196ee8692a. python3 -m unittest dm.tools.tests.test_docs dm.tools.tests.test_split_unreadable dm.tools.tests.test_split_nesting dm.tools.tests.test_pso_offline dm.tools.tests.test_shader_toolchain_cache dm.tools.tests.test_nsp_chain — 87 tests passed.

The focused suite covered streaming and memory bounds for the offline pipeline script, archive reuse and both missing-index layouts. Pinned Python lint reported zero new or stale findings.

scripts/regress.py check --no-smoke --games 01006FE013472000,0100F43008C44000,01008CF01BAAC000,01007EF00011E000,0100F2C0115B6000 stopped at discovery (exit 2). The earlier impact selection below covers large archives, archive-key reuse, depth-specific entries and bundled archive validation. Only Z-A was installed. No game comparison ran.

GitHub checks for this revision

Evidence from the author and earlier revisions

The builder streams the JSON script to a temporary file in the work directory, compares it in chunks and replaces the old script only when the complete file differs. Hashing streams the same canonical JSON used by earlier archive keys. Python releases the descriptor table before invoking the compiler. The directory needs room for the candidate script alongside an existing copy until the comparison finishes.

With 4,096 synthetic pipeline bases and an identical 15,398,151-byte script, peak Python allocations measured by tracemalloc fell from 52,124,624 to 13,086,166 bytes. At the mocked compiler invocation, live allocations fell from 27,446,661 to 213,172 bytes. The cache keys matched before and after. The traced run took 1.03 s before and 2.19 s after. These numbers measure the Python builder on this fixture. The reported 180.5 GB process-tree figure, the original compiler exit and large-archive compiler memory are still unverified.

The 512-base regression exceeded its memory budget on main and now passes. It checks the saved JSON, compatibility with the earlier key format, and live memory at compiler invocation. All 30 offline archive tests passed, including reuse across chain relocation, entry or library changes, and both target platforms. On Apple M2 Max, the real metal-tt integration built an archive and the host opened and probed it: 52 checks passed in check_t_pso_bundle. The full native graphics suite passed with six optional external reflection samples absent. The tools and graphics Python suites passed. Pinned Python lint on the original published head reported three upstream Pyright findings: one in the logic-op generator (covered by PR #1019) and two in the newly landed shader splitter at line 564. The archive builder and its tests introduced no lint findings.

Game regression targets were Mario Party Superstars (reported 140,036 bases), Pokémon Legends: Z-A (archive-key reuse), Echoes of Wisdom (depth-specific entries), Breath of the Wild (bundled archive validation), and Tears of the Kingdom (large NVN shader-pair set). scripts/regress.py check --games 01006FE013472000,0100F43008C44000,01008CF01BAAC000,01007EF00011E000,0100F2C0115B6000 stopped at preflight because no regression root or game builds are available. The reporter's 140,036-base build was not run here.