Repository navigation
transform, cgen: speed up self-host transform, restore its memory, trim serial cgen work - #29358
Merged
Merged
Conversation
Recent changes made `v -building-v v.v` transform ~60% slower and raised its stage peak by ~250 MB. On an M5 Max, `-prod` v3, `-nocache -building-v v.v`: transform 220-229 ms -> 151-157 ms stage peak 1280-1297 MB -> 864-875 MB - call_param_types_from_decl recorded every unindexed name as a miss, which detached (cloned) the shared prepared index/cache/miss maps once per helper batch. Unindexed names are no longer recorded: -2.1 GB of transform allocations. - The escape-analysis precheck sent almost every function through the full escape walk: any call with a pointer parameter or a pointer receiver matched. It now mirrors the walk's conditions: receivers rooted at a `mut`/`&` parameter cannot move a local of this frame, and non-void reference parameters only matter when an argument holds inline fixed-array storage. The walk itself checks those cheap argument conditions before resolving callee signatures. - driver.run (the largest function) and restore_v3_cache_external_inputs contained function literals, so they were lowered serially before the workers started. The callbacks are now small named functions. - collect_selected_file_called_fns walks every reachable body, which is the whole compiler for a self build, and left ~120 MB of scratch in the compilation arena that every later stage carried. Scoped checks run the walk on a disposable fork. - Large helper chunks use smaller scratch batches and a 1 MB arena floor, the master takes 85% of a helper's share (it also publishes its batches), and scope save/restore, alias-chain normalization and escape alias lookups avoid needless map clones and allocations. Self-build C output is identical (modulo string-literal numbering) and an 859-program corpus produces byte-identical C.
Self host (`-prod` v3, `-nocache -building-v v.v`, M5 Max), best of 9: transform 222-225 ms on master -> 143-145 ms cgen postamble 5.3 ms -> 1.5 ms; typedecls task ~8 ms less CPU transform: - Compute the scope-owned base node list only when the driver will not keep the whole transform arena (the self-host path keeps it): -2.5 ms. - Find set ignored `$for` flags and helper-scope text flags with memchr instead of walking a flag per AST node: -3.5 ms and -1 ms. - Scan anonymous struct literal candidates on the worker pool: -2 ms. - Replace comparator literals in checker functions with named functions, so they no longer have to be transformed serially before the workers start. - Time the per-function expansion estimate only under -d v3_ttime. cgen: - Record `$embed_file` payload nodes during the collect scan; the blob writers no longer rescan the whole AST three times. - Find C extern declarations among the recorded declaration nodes instead of every transform-appended node. - Search all runtime const initializers at once per global instead of building four pattern strings for every (global, initializer) pair. - Skip the generic shared-type scan for declarations without shared fields. - Time each collected declaration only under -d v3_ttime. The generated C is unchanged: self-build output is byte-identical and an 859-program corpus differs only in the embedded compiler commit hash.
Member
Author
|
@codex review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speeds up the self-host transform and trims serial work in cgen, and restores transform memory, which had regressed recently.
-prodv3 fromvlib/v,./v3 -nocache -building-v -o v4 -v v.v, M5 Max, best of 9 alternating runs (load average 7–15):On this shared machine cgen moves by ±15 ms between sessions, so its wall-clock change is within noise. The cgen subphases do shrink: the serial postamble goes from 5.3 to 1.5 ms, and the type-declaration task uses ~8 ms less CPU.
Commit 1: restore transform speed and memory
Copy-on-write detach per batch (
fn.v).call_param_types_from_declrecorded every unindexed name as a miss. On a helper batch, that detached (cloned) the shared prepared index, cache and miss maps; transform: bound comptime monomorphization memory #29334 added the index to that clone. Unindexed names are no longer recorded, since the index lookup costs the same as a recorded miss. Transform allocations drop from 4.1 GB to 2.0 GB.Escape precheck matched almost everything. Since v3: fix the bugs that stopped C translated by c2v (SQLite, libpq) from compiling and running #29167 and transform: preserve fixed-array references and owned borrowed storage #29197, any call with a pointer parameter or a pointer receiver forced the full escape walk. The precheck now mirrors the walk's own conditions:
mut/¶meter can't move a local of this frame;The walk also checks those cheap argument conditions before resolving callee signatures.
driver.runlowered serially. v3: constraints on type parameters, editor answers in generic code, faster checks for editors #29254 added function literals todriver.run, the compiler's largest function, so it was transformed alone before the workers started. The callbacks are now small named functions.Check scratch retained into transform. Since checker: report hard errors in reachable dependency functions #29214,
collect_selected_file_called_fnswalks every reachable body, the whole compiler for a self build, and left ~120 MB of scratch in the compilation arena. Scoped checks now run the walk on a disposable fork.Worker balance and scratch. Large helper chunks use smaller scratch batches with a 1 MB arena floor. The master takes 85% of a helper's share. Scope save/restore, alias-chain normalization and escape alias lookups avoid needless map clones and allocations.
Commit 2: trim serial work around the parallel stages
transform:
$forflags and helper-scope text flags withmemchrinstead of a loop over one flag per AST node.cgen:
$embed_filepayload nodes in the collect scan instead of rescanning the AST three times.-vnow only runs with-d v3_ttime.Verification
V3_NO_ESCAPE_PRECHECK=1. The master-share change only renumbers_str_Nconstants.vlib/v/tests/*.v,examples/*.v,vlib/v/compiler_tests/*_test.v) produces the same C. The only difference is the embedded compiler commit hash.v test vlib/v/transform/passes 33/33,vlib/v/types/97/97 andvlib/v/gen/c/51/51.fn_test.vis updated: a miss on an unindexed name no longer detaches the worker's shared maps.Not addressed here: the eager called-function walk still costs ~190 ms serially in the check stage of a self build. It only matters when a dependency function has an error, so it could be deferred like the IError gating.