Repository navigation
Old-gen growth from large-object births never arms a full collection: RSS climbs unbounded on repeated wide-object parse #10123
Description
Activity
Root cause found: the RETAINING veto is fed by the survival its own pinning creates
A full collection reclaims all of it.
PERRY_GEN_GC=0onwide_1m:parse, same binary:iterations generational full-sweep-only 8 56 MiB 54 MiB 64 222 MiB 55 MiB (flat) So nothing is unreclaimable and no allocation needs to move. The generational regime simply never runs the collection that would free it. At 55 MiB Perry would be below Node (121) and Bun (95) on this row.
The veto
gc::policy::old_reclaim_pressure_due:let crossed_absolute_threshold = old_in_use >= threshold && baseline < threshold && !GC_MAJOR_PACING_RETAINING.with(|c| c.get());
GC_MAJOR_PACING_RETAININGis armed from measured young-survival, which on this workload readssurvival_permille=997on every minor. #7937's reasoning for the exemption is that "a heap whose young generation is not dying is retaining live data, and a full mark-sweep cannot lower the number being watched."That reasoning inverts here. The young generation is not dying because large old-gen containers pin it through the remembered set — every parse's 400 KB field storage is born tenured, and a minor never sweeps old-gen, so it holds its 50 000 key strings live. High survival is the symptom of the condition a full collection would clear, and the veto reads it as proof that one would be futile. The measurement above is the counter-example: the full sweep lowers exactly the number being watched, by 4x.
The proportional arm does still bound the exposure, but the RETAINING multiplier widens the band to 128 MB, and
old_in_usereaches ~112 MB over 64 iterations — so the bound is real but sits above where this workload ends.The width cliff confirms the container
RSS growth from 8 to 64 iterations, by object width — the step is exactly at
LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES(128 KB), i.e. 16 384 keys at 8 bytes each:keys keys array growth 8 000 62 KiB 16 MiB 14 000 109 KiB 21 MiB 17 000 132 KiB 67 MiB 24 000 187 KiB 85 MiB 50 000 390 KiB 166 MiB Under the line the storage is nursery-resident and a minor reclaims it. Over the line it is born tenured and only a full collection can, which is the one that never comes.
Suggested shape
Young-survival cannot distinguish "live data" from "dead data pinned by a born-tenured container", so it should not be the sole veto on the arm that would tell them apart. Two options, either of which is testable against this fixture:
- Exempt old-gen growth that came from births rather than promotion.
credit_promoted_bytes_to_old_baselinecredits the baseline only from promotion, so a born-tenured object movesold_in_usewithout movingbaseline— the two quantities the arm compares already disagree in exactly the informative way. Tracking born-tenured bytes since the last full collection and letting them bypass the RETAINING veto is a small change with a direct signal. - Make the veto self-correcting: apply it only while the previous full collection actually failed to reclaim. Here the first one would succeed and disarm it.
Either needs the gc-ratchet corpus run before and after, which is why this is filed rather than attempted inside a JSON pass.
Note this is the same root cause as the residual
records_array_8m:scan(187 vs 134 MiB) andrecords_array_20m:scan(256 vs 226), whose clusters are also genuinely over the pointer-bearing line. #10098 fixed the two rows whose clusters fit under it by making them nursery-resident; that approach cannot reach these.- Exempt old-gen growth that came from births rather than promotion.
Attempted the trigger fix. It does not work, and the measurements say why: the trigger is only half the chain.
Three attempts, each measured on
wide_1m:parse(64 iterations, Node 121 MiB, Bun 96 MiB):change full collections peak RSS main 0 222 MiB exempt births from the RETAININGveto1 222 MiB + recurring births arm, shared predicate 2 284 MiB + births arm split off the shared predicate 2 284 MiB Making the collection fire is achievable — a born-tenured byte counter reset at each reclaim, exempted from the
RETAININGveto, produces collections where main has none. It does not lower peak RSS, and two fulls make it worse than zero.Why: the arena never returns capacity
[gc-arena-right-size] episodes=0 starts=0 released_capacity_bytes=0 low_collections=0 fulls_remaining=0 disarmed=false arena_live=81407888 arena_capacity=90177536 [gc-arena-rebaseline] total=141557760 step=1073741824 -> 1073741824Right-sizing is armed (
disarmed=false) and never starts an episode. So the chain is:- Old-gen fills with born-tenured storage that no minor sweeps.
- The arena grows to hold it —
totalreaches 141 MB, rebaselining its own trigger 536 MB → 1 GB along the way. - A full collection, when it finally runs, frees objects but returns no capacity.
- Peak RSS is set by arena capacity, which only ever grows.
That is why
PERRY_GEN_GC=0holds 55 MiB flat: with full sweeps throughout, old-gen never accumulates, so the arena never grows in the first place. It is not that fulls shrink the heap — it is that they stop it expanding.Two late fulls are therefore strictly worse than none: the arena has already grown, and the collections add their own cost on top.
What this means for the fix
The trigger work above is necessary but not sufficient. Either:
- Fire much earlier, before the arena expands — the born counter makes that expressible, but the floor would have to be well below the current
threshold/2, and that trades CPU against a benchmark this issue has not measured. The gc-ratchet corpus is the right instrument. - Make
arena-right-sizeactually run after a productive full.episodes=0withdisarmed=falseis the specific thing to explain:arena_live=81 MBagainstarena_capacity=90 MBis only 10% slack, so a ratio-based gate may simply never see enough headroom — while the 141 MB the arena reached, versus the 55 MB the workload needs, is where the memory actually is.
Option 2 looks like the higher-value half, and it is independent of the trigger.
Also worth noting
old_reclaim_pressure_duehas three consumers, one of them
copied_minor_promotion_handoff_pressure_due. I split the births arm out of the
shared predicate on the theory that scheduling a full before a promotion was the
#7592 futile-handoff shape. It made no difference — 284 MiB either way — so the
handoff is not implicated here, and a future change need not carry that
complication.Everything above is reverted; nothing from this investigation is on a branch.
The complete chain: two independent heuristics, both fed by the pathology they would cure
Same fixture, same binary, only
PERRY_GEN_GCdiffers:regime arena_livearena_capacityutilization peak RSS generational 158 MB 175 MB 90% 222 MiB full-sweep-only 6.4 MB 14.7 MB 44% 55 MiB The workload's true live set is 6.4 MB. Everything above that in the generational column is dead storage that no collection has adjudicated: born tenured, over
LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES, and a minor never sweeps old-gen.That inflated number is then consumed by two separate gates, and both read it as "this heap is busy":
old_reclaim_pressure_due—GC_MAJOR_PACING_RETAININGis armed from young-survival, which reads 997permille because the born-tenured containers pin the young graph through the remembered set. The veto blocks the collection that would disprove it.arena_right_size— opens an episode only at or belowARENA_RIGHT_SIZE_TRIGGER_PCT(50%) live/capacity. It sees 90% and declines, soepisodes=0 starts=0 released_capacity_bytes=0withdisarmed=false. Capacity is never returned.
So the chain is: born-tenured storage accumulates ->
arena_livereads 25x the true live set -> the reclaim trigger declines AND right-sizing declines -> the arena grows to hold the garbage and never gives it back -> peak RSS is 4x the true requirement.PERRY_GEN_GC=0holds 55 MiB not because full sweeps shrink the heap, but because the arena never grows in the first place: live stays at 6.4 MB, so nothing ever asks for 175 MB of capacity.Why the trigger fix alone cannot work
This is the measured reason attempts 2 and 3 made things worse rather than better. By the time a births-driven trigger fires, the arena has already expanded, and right-sizing still will not open because
arena_liveis still inflated -- the two fulls left it at 81 MB against a 6.4 MB true live set. Two late fulls are strictly worse than none: no capacity comes back, and the collections cost their own time.Where a fix has to start
The shared defect is that both gates consult a liveness number that includes old-gen nothing has visited since it was born. Options, in the order I would try them:
- Give the right-size gate a birth-aware path. Born-tenured bytes since the last reclaim are already trackable (a counter at the two birth sites in
arena/allocators.rscosts one add). A heap whose capacity is dominated by un-adjudicated births should be able to open an episode regardless of apparent utilization, because apparent utilization is precisely what cannot be trusted there. - Fire the reclaim early enough that the arena never expands, which means a floor far below
threshold/2. That trades CPU, and the gc-ratchet corpus is the instrument.
Option 1 looks independent of the trigger work and addresses the half that actually holds the memory.
Dead ends, so nobody re-derives them
- Exempting births from the
RETAININGveto: produces collections (0 -> 1), no RSS change. - A recurring births arm in
old_reclaim_pressure_due: 2 collections, RSS 222 -> 284 MiB. - Splitting that arm off the shared predicate so
copied_minor_promotion_handoff_pressure_duekeeps the original: identical 284 MiB, so the handoff is not implicated and a future change need not carry that complication.
All reverted; nothing is on a branch.
Root cause found: promotion is structurally invisible to old-gen pressure
Correcting my three earlier comments on this issue. I framed this as a trigger-tuning
problem and measured three dead ends against that framing. It is not a tuning problem,
and that is why none of them moved.The mechanism
credit_promoted_bytes_to_old_baseline(gc/policy.rs) adds a minor's promoted bytes
straight onto the old-reclaim baseline:GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(|b| b.set(b.get().saturating_add(promoted_bytes)));
So
old_in_use - old_baselineis invariant under promotion, by construction. Promotion
raises both sides in lockstep. Only allocation born directly into old-gen can ever create
old-gen pressure.That is deliberate and I am not proposing to change it — #7902 and #7965 both settled it,
and the argument in the code is right: the baseline is the base of a growth measurement,
not a liveness claim, and withholding the credit pins that base at 0 on exactly the
workloads that reach it.But it leaves one question unanswerable: how much of old-gen arrived because a minor
could not tell garbage from survivors? Onwide_1m:parseevery minor promotes at
999 permille. Nothing dies. The entire nursery is laundered into old-gen, where no
minor will ever look at it again — and the only thing scheduling fulls was incidental
born-tenured allocation elsewhere in the parser.How that was proved
JSON.parsere-mints every key of a wide object as a born-tenured string on every call:
PARSE_KEY_CACHEholds 4096 entries, a 50k-field object overflows it on the first parse,
and the parse-boundary clear then wipes it before the second parse can hit it. So 50,000
js_string_from_bytes_longlivedallocations per iteration.Fixing that — raising the bound so the schema stays cached — makes the row much worse:
peak RSS @256 it fulls main 387 MiB 8 key-cache bound raised 833 MiB 0 Removing the garbage removed the only thing that was scheduling collections. That is the
cleanest possible demonstration of the mechanism, and it is a trap for anyone who fixes
the parser side first.The fix
Record a second, independent signal alongside the baseline credit rather than correcting
it. Bytes promoted by a minor at >= 950 permille survival accumulate in
GC_OLD_GARBAGE_SUSPECT_BYTES; 16 MB of it makes an old reclaim due on its own; a reclaim
resets it. The baseline is untouched, so #7902 and #7965 both still hold.Split off
old_reclaim_pressure_dueon purpose rather than folded into it — that predicate
is also whatcopied_minor_promotion_handoff_pressure_dueasks, and an arm there schedules
a full immediately before a promotion, over survivors it cannot reclaim, which is the
futile-handoff shape #7592 removed. (That was one of my earlier dead ends.)Floor is the measured knee of a 2/4/8/16/32/64/128 MB sweep at 256 iterations:
floor peak RSS CPU fulls 8 MB 243 MiB 3.14 s 14 16 MB 243 MiB 3.00 s 12 32 MB 274 MiB 3.00 s 11 64 MB 510 MiB 2.71 s 5 (unpatched) 387 MiB 3.03 s 8 wide_1m:parsepeak RSS 387 MiB -> 243 MiB (-37%) at baseline CPU. With the pacing arm
in place the key-cache fix is measurably redundant (243 MiB either way), so it is not
included.Branch
gc/old-born-bytes-arm-full, commit14ceaef3d6. Not opened as a PR — this is a
GC pacing change and needs the gc-ratchet corpus run before/after by someone with that setup.What still stands between this row and parity
Not pacing. RSS floors at 243 MiB however many fulls run — because freed pages are never
returned to the OS:arena_live=56678536 arena_capacity=67108864 <- 56.7 MB live in a 67 MB arena [gc-general-reclaim] examined=41 released=0 rejected: no_snapshot=40 ... [gc-arena-right-size] episodes=0 starts=0 released_capacity_bytes=0The process holds 243 MiB while its arena is 67 MB and its live set is 56.7 MB. Block
release rejects 40 of 41 blocks atsnapshot.data == 0(arena/reset.rs:700). Targets are
Node 121 MiB / Bun 96 MiB, and 56.7 MB of live data is comfortably inside both — so this is
the whole remaining gap on the row, and it is independent of everything above.Dead ends, so nobody re-derives them
- Exempting births from the
RETAININGveto — 1 full, 222 MiB, unchanged. - A recurring births arm on the shared predicate — 2 fulls, 284 MiB (worse; the
futile-handoff shape). - Splitting that arm off the shared predicate — identical 284 MiB, so the handoff was
never implicated. - A born-tenured-bytes floor — this does work (387 -> 154 MiB) but costs +7.6% CPU,
because it buys RSS with extra whole-heap marks. The suspect-promotion arm gets most of
the same RSS for free. Superseded. - Raising the parse-key cache bound alone — 833 MiB, as above.
- Exempting births from the
- added a commit that references this issue
on Sep 12, 2026 Correction, and the actual root cause
First, a correction to my previous comment. I said the suspect-promotion arm delivered
"387 MiB -> 243 MiB at baseline CPU". That CPU claim was measured onwide_1m:parseonly.
Measured across the rest of the JSON matrix against clean main, it is not baseline CPU:row CPU main -> arm RSS main -> arm records_array_1m:scan0.59s -> 1.30s (+120%) 410M -> 118M records_array_8m:scan1.73s -> 2.60s (+50%) 444M -> 178M records_array_20m:parse0.73s -> 1.05s (+44%) 267M -> 196M Worse, the regression does not respond to the floor — swept at 16/64/256 MB,
records_array_1m:scanstays at 1.26–1.30s even at 256 MB where the arm should be inert. So
the cost is the predicate, not the extra collections:old_reclaim_pressure_or_suspect_due
sits ingc_budgeted_due_trigger, which runs at every safepoint, and the arm adds a
thread-local read on the common path. Reverted in9e1c80e060. The diagnosis in that
commit stands (promotion credits the baseline, so only born-tenured allocation can create
old-gen pressure); the remedy does not.The real cause: wide objects are born tenured, and a minor never sweeps old-gen
arena/allocators.rsalready states the failure mode exactly:It is TYPE-DEPENDENT, because the price of crossing it is: this object is also stamped
GC_FLAG_TENURED, and a minor never sweeps old-gen — so for a POINTER-BEARING object the
cost is not its own bytes, it is every object it can reach, held live through the
remembered set by a container nothing refers to any more.A wide JSON document's shape-keys array is that container. Above 16,384 fields its keys
array crossesLARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES(128 KB), is born tenured, and
then holds its whole key set live after every instance has died.Confirmed to the byte —
PERRY_GC_CENSUS, 64 parses, fixtures straddling 131,072 bytes:fields keys array shapes retained live peak RSS 16,000 128,016 B 0 0.0 MB 29 MiB 16,300 130,416 B 0 0.0 MB 29 MiB 16,500 132,016 B 30 22.7 MB 98 MiB 50,000 400,016 B 15 34.3 MB 177 MiB The census names it directly:
shape_keys_arrays {bytes: 6000240, count: 15},
slot_tags {string: 750000}— 15 arrays × 50,000 keys = 750,000 live strings, on a document
whose live set is ~0. Same binary underPERRY_GEN_GC=0: live 336 bytes, 50 MiB RSS.Ruled out by direct experiment (each rebuilt and re-measured, all zero effect):
transition-cache size (raised 16,384 -> 65,536 on both the runtime constant and the mask
codegen bakes inproxy_reflect.rs); parse-key identity (PARSE_KEY_CACHEbound raised
4,096 -> 65,536); the shape table (table_len=12< 15 live arrays); registered mutable root
scanners (all reportslots=0 ptr_roots=0).What a fix is worth, and why I am not proposing one yet
Raising the threshold 128 KB -> 512 KB (still inside the copier's 1 MiB
MAX_YOUNG_MOVE_BYTES
and nursery-block ceilings) on wide_1m:parse:0.62s / 284 MiB -> 0.15s / 40 MiB — −76% CPU and −86% RSS, better on both axes, against
Node 121 MiB and Bun 96 MiB.perry-runtimeis green (3628 passed) after retargeting two
tests that hardcoded a field count and asserted old-gen residency — they now derive their
width from the constant, so the old-gen path stays covered at any threshold.But globally it is not shippable: the same change moves ordinary 128 KB–512 KB data arrays
into the nursery, and on rows that do not need it that costs RSS —
records_array_8m:scan444M -> 710M (+60%),records_array_20m:parse267M -> 314M (+18%)
(CPU on those two improves, −29% and −44%).So the widening has to be scoped to wide-object JSON construction, not global. I built
half of that: aWideShapeKeysBirthScopearoundallocate_parse_shape_keys_array, gated in
the cold large-object branch ofarena_alloc_gcso the hot path is untouched. It fires
correctly (born young past threshold: 400016 bytesevery parse) but only gets
wide_1m to 176 MiB, because the keys array is half the story — a 50,000-field object's own
property storage is also 400 KB and also born tenured. The scope needs to cover the
object-storage allocation on the JSON construction path too;records_array_*is unaffected
by that because its large allocation is array element storage, not wide-object storage.That is the next step, and it wants the quiet-mini matrix plus the gc-ratchet corpus rather
than more measurement on a box at load 42.Fixed — PR #10145
The scoped fix is finished and open as draft #10145 (gated on the gc-ratchet corpus, since it changes birth generation).
The half I built earlier didn't work because the scope was open in the wrong place. A mask probe at each large allocation showed the keys array was admitted (
mask=2) but the object's property storage sawmask=0. That storage is minted inobject/json_construction.rsviajs_object_alloc_class_inline_keys_stamped, not insidejs_json_parse_result. I found the site by tagging all nineGC_TYPE_OBJECTallocation sites inobject/alloc.rs. Scoping it there completes the fix.Measured on current
mainagainst a same-tree baseline (the commit reverted and rebuilt):row node bun baseline #10145 wide_1m:parse136 MiB 71 MiB 220 MiB / 0.20s 40 MiB / 0.15s records_array_8m:scan156 166 643 / 0.94s 643 / 0.93s records_array_20m:parse422 170 313 / 0.40s 313 / 0.41s records_array_1m:scan93 95 410 / 0.58s 410 / 0.59s records_array_16k:scan57 77 117 / 0.13s 117 / 0.13s One more correction to my earlier comments. The "main" figures I quoted before (283 MiB for
wide_1m, 444 MiB for8m:scan) came from a frozen binary whose runtime turned out to be a different source state. The compiler refused to cross-link it (library source 111389820894vsa511bd884a7f). The numbers above use a baseline built from the same tree as the fix. That also shrinks the global-widening regression I reported on8m:scanfrom "+60%" to 643 → 710 MiB. It's still a regression, and it's still why this is scoped.- added 4 commits that reference this issue
on Sep 12, 2026
Old-generation growth from large-object births does not arm a full collection
A workload that repeatedly allocates and drops a large pointer-bearing object grows old-gen without bound. Every minor promotes essentially the whole nursery, copies nothing and frees nothing, and the full collection that would reclaim it never arrives.
records_array_wide_1m:parse— a flat object of 50 000 integer keys, parsed in a loop:Peak RSS 228 MiB against Node's 121 and Bun's 95.
The same signature on
records_array_16k:scanbefore #10098's lazy-array work, with the trigger state visible:An old-gen threshold of 48 MB exists and
old_in_usereaches 48.3 MB withold_pending=false; meanwhile the arena trigger rebaselines itself upward exponentially. So the arm that would notice old-gen filling keeps stepping out of the way of the arm that is filling it.Why this is one cause, not two
Each object is individually reasonable.
arena_alloc_gcputs a pointer-bearing object overLARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES(128 KB) directly into non-moving old-gen, and the reasoning in its own comment is sound: for a pointer-bearing object the cost of tenuring "is not its own bytes, it is every object it can reach, held live through the remembered set by a container nothing refers to any more".That reasoning assumes a full collection eventually runs. When it does not, the policy inverts: the objects chosen for old-gen because they are expensive to hold are exactly the ones nothing reclaims.
Two of these are now fixed by moving the allocation rather than the policy — #10098 made the lazy JSON array cluster nursery-resident and took
records_array_16k:scanfrom 205 MiB to 51 MiB andrecords_array_1m:scanfrom 159 to 75, both now below Node and Bun. That works because a lazy cluster's cache is small enough to sit under the line. A 50 000-key object's field storage is ~400 KB and cannot.What to look at
old_in_useis climbing — the two arms currently move in opposite directions.old_pending=falseatold_in_use≈old_thresholdis the specific observation to explain first.Scope note
This is GC policy, which CLAUDE.md routes separately and warns about, so it wants its own change with the ratchet corpus run before and after rather than riding along with JSON work. Filing it with the evidence rather than attempting it inside a JSON performance pass.
Rows currently affected:
wide_1m:parseRSS 228 vs 95 MiB (2.4x), and the residualrecords_array_8m:scan187 vs 134 andrecords_array_20m:scan256 vs 226, whose clusters are genuinely over the pointer-bearing line.Refs #10098, #793.