Skip to content

transform, cgen: speed up self-host transform, restore its memory, trim serial cgen work - #29358

Merged
medvednikov merged 2 commits into
vlang:masterfrom
medvednikov:fix-transform-perf
Oct 3, 2026
Merged

medvednikov merged 2 commits into
vlang:masterfrom
medvednikov:fix-transform-perf

Conversation

@medvednikov

@medvednikov medvednikov commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Speeds up the self-host transform and trims serial work in cgen, and restores transform memory, which had regressed recently.

-prod v3 from vlib/v, ./v3 -nocache -building-v -o v4 -v v.v, M5 Max, best of 9 alternating runs (load average 7–15):

master this PR
transform 222–225 ms 143–145 ms
transform stage peak ~1273 MB ~868 MB
cgen 154–157 ms 152–161 ms
cgen stage peak ~1086 MB ~977 MB

On this shared machine cgen moves by ±15 ms between sessions, so its wall-clock change is within noise. The cgen subphases do shrink: the serial postamble goes from 5.3 to 1.5 ms, and the type-declaration task uses ~8 ms less CPU.

Commit 1: restore transform speed and memory

Commit 2: trim serial work around the parallel stages

transform:

  • Builds the scope-owned base node list only when the driver won't keep the whole arena (the self-host path keeps it).
  • Finds ignored $for flags and helper-scope text flags with memchr instead of a loop over one flag per AST node.
  • Scans anonymous struct literal candidates on the pool.
  • Replaces the remaining comparator literals in checker functions with named functions.

cgen:

  • Records $embed_file payload nodes in the collect scan instead of rescanning the AST three times.
  • Finds C extern declarations among the recorded declaration nodes instead of every transform-appended node.
  • Searches runtime const initializers once per global instead of building four patterns per (global, initializer) pair.
  • Skips the generic shared-type scan for declarations without shared fields.
  • Per-declaration and per-function timing under -v now only runs with -d v3_ttime.

Verification

  • The generated C is unchanged. Self-build C for a fixed source tree is identical before and after, with and without V3_NO_ESCAPE_PRECHECK=1. The master-share change only renumbers _str_N constants.
  • An 859-program corpus (vlib/v/tests/*.v, examples/*.v, vlib/v/compiler_tests/*_test.v) produces the same C. The only difference is the embedded compiler commit hash.
  • v test vlib/v/transform/ passes 33/33, vlib/v/types/ 97/97 and vlib/v/gen/c/ 51/51. fn_test.v is updated: a miss on an unindexed name no longer detaches the worker's shared maps.
  • Self-host fixed point: v3 → v4 → v5 with v4 == v5 byte-identical.

Not addressed here: the eager called-function walk still costs ~190 ms serially in the check stage of a self build. It only matters when a dependency function has an error, so it could be deferred like the IError gating.

Recent changes made `v -building-v v.v` transform ~60% slower and raised its
stage peak by ~250 MB. On an M5 Max, `-prod` v3, `-nocache -building-v v.v`:

  transform   220-229 ms -> 151-157 ms
  stage peak  1280-1297 MB -> 864-875 MB

- call_param_types_from_decl recorded every unindexed name as a miss, which
  detached (cloned) the shared prepared index/cache/miss maps once per helper
  batch. Unindexed names are no longer recorded: -2.1 GB of transform
  allocations.
- The escape-analysis precheck sent almost every function through the full
  escape walk: any call with a pointer parameter or a pointer receiver
  matched. It now mirrors the walk's conditions: receivers rooted at a
  `mut`/`&` parameter cannot move a local of this frame, and non-void
  reference parameters only matter when an argument holds inline fixed-array
  storage. The walk itself checks those cheap argument conditions before
  resolving callee signatures.
- driver.run (the largest function) and restore_v3_cache_external_inputs
  contained function literals, so they were lowered serially before the
  workers started. The callbacks are now small named functions.
- collect_selected_file_called_fns walks every reachable body, which is the
  whole compiler for a self build, and left ~120 MB of scratch in the
  compilation arena that every later stage carried. Scoped checks run the
  walk on a disposable fork.
- Large helper chunks use smaller scratch batches and a 1 MB arena floor, the
  master takes 85% of a helper's share (it also publishes its batches), and
  scope save/restore, alias-chain normalization and escape alias lookups
  avoid needless map clones and allocations.

Self-build C output is identical (modulo string-literal numbering) and an
859-program corpus produces byte-identical C.
Self host (`-prod` v3, `-nocache -building-v v.v`, M5 Max), best of 9:

  transform  222-225 ms on master -> 143-145 ms
  cgen       postamble 5.3 ms -> 1.5 ms; typedecls task ~8 ms less CPU

transform:
- Compute the scope-owned base node list only when the driver will not keep
  the whole transform arena (the self-host path keeps it): -2.5 ms.
- Find set ignored `$for` flags and helper-scope text flags with memchr
  instead of walking a flag per AST node: -3.5 ms and -1 ms.
- Scan anonymous struct literal candidates on the worker pool: -2 ms.
- Replace comparator literals in checker functions with named functions, so
  they no longer have to be transformed serially before the workers start.
- Time the per-function expansion estimate only under -d v3_ttime.

cgen:
- Record `$embed_file` payload nodes during the collect scan; the blob writers
  no longer rescan the whole AST three times.
- Find C extern declarations among the recorded declaration nodes instead of
  every transform-appended node.
- Search all runtime const initializers at once per global instead of
  building four pattern strings for every (global, initializer) pair.
- Skip the generic shared-type scan for declarations without shared fields.
- Time each collected declaration only under -d v3_ttime.

The generated C is unchanged: self-build output is byte-identical and an
859-program corpus differs only in the embedded compiler commit hash.
@medvednikov medvednikov changed the title transform: restore self-host transform speed and memory transform, cgen: speed up self-host transform, restore its memory, trim serial cgen work Oct 3, 2026
@medvednikov

Copy link
Copy Markdown
Member Author

@codex review

@medvednikov
medvednikov merged commit 1c17fe0 into vlang:master Oct 3, 2026
@medvednikov
medvednikov deleted the fix-transform-perf branch October 3, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant