Repository navigation
Conversation
Fire Emblem Fortune's Weave's shader split took 12 minutes, 6 of them in one 9.8 MB SARC (Layout/Common.arc.zst). sarc_members took each member's name by splitting the whole rest of the archive at every NUL and keeping the first piece, so the work grew with the archive's size times its member count. It now finds the name's own NUL and decodes only that. The names are the same; the split of that game took 1:51. Checked: test_split_nesting.py's new test reads an 800-member archive and compares the names and data; the reading takes about 1.5 ms, against 47 s with the old code. The archive is built before the timing starts, so a slower runner does not count its construction. Donkey Kong Bananza's split gives the same 23,638 shaders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarc_members took each member's name by splitting the whole rest of the archive at every NUL and keeping the first piece, so the work grew with the archive's size times its member count. Fire Emblem Fortune's Weave's split took 12 minutes, 6 of them in one 9.8 MB SARC (Layout/Common.arc.zst). It now finds the name's own NUL and decodes only that, and only for a member that has a name. The author reports that game's split at 1:51 and the same 23,638 shaders on Donkey Kong Bananza. Checked: test_split_nesting.py's new test reads an 800-member archive and compares the names and data. The old and new sarc_members gave the same results on crafted archives (nameless members, unterminated names, offsets past the end, bad UTF-8, damaged archives) and on 20,000 random ones; an 800-member synthetic archive takes 1.2 ms against 47 s. No regression games were run: the output of this one function is unchanged. Co-authored-by: capibawa <12531452+capibawa@users.noreply.github.com>
|
Landed on One change from the PR: the NUL search now runs only for members that have a name. As submitted, it also ran for nameless members, whose result is never used. That made a contrived archive (5,000 nameless members, 64 MiB with no NUL) take 6 s instead of 1 ms. I don't expect a real archive to hit this. The test's I compared the old and new |
What and why
This is split out of #957, as offered in its closing comment.
shader_split.py'ssarc_memberstook each member's name by splitting the whole rest of the archive at every NUL and keeping the first piece. That makes the work grow with the archive's size times its member count. Fire Emblem Fortune's Weave's split took 12 minutes, 6 of them in one 9.8 MB SARC (Layout/Common.arc.zst). Each name is now read only up to its own NUL. The names come out the same, and that game's split takes 1:51.Checked
dm/tools/tests/test_split_nesting.py(its new test reads an 800-member archive and compares the names and data: about 1.5 ms, against 47 s with the old code; the archive is built outside the timing);dm/tools/tests/test_docs.py.04007EF01F8AE000, v131072): split from scratch, 12 minutes before, 1:51 after.sarc_membersfrommainand from this PR read Fortune's Weave'sLayout/Common.arc.zst: the same 1,156 members with the same data, in 408 s onmainand under 0.1 s here.regress.py: not run. This changes only how the split reads member names, and the output it gives is the same.Docs
🤖 Generated with Claude Code