go: the flat word codec — 712% to 248% of generated C/C++ - #198
Conversation
add0f16 to
be36b36
Compare
|
Adversarial review verdict: BLOCK — three CONFIRMED correctness regressions, one root cause. The performance claim is real and reproduced independently (2.06x write / 2.50x round_trip / 2.35x blended, i.e. better than the PR's 2.24x), wire identity holds on the current corpus (36 generate invocations re-run: zero diff; test/go byte-compares against the C++-pinned goldens and passes), and the lock and scope are clean. But:
ROOT CAUSE: #183's Rust template keeps a strict 1:1 item-to-piece mapping, so its fallback (re-emit each piece directly) is safe by construction. Levers 3-6 made TO UNBLOCK: fix the fallback to re-emit whole ITEMS against the correct base (or restore the invariant); move Also recorded: go's round_trip spread is 10.44% in the committed CSV against 0.40-1.73% elsewhere, sitting just under the 15% NOISY threshold so no note printed; on the median statistic the headline is 275%, not 255%. |
Two findings from re-proving the gate on a fully provisioned tree. dist/ is the Makefile's pinned toolchain drop — the Dart SDK, the JDK, OTP, Elixir. It is gitignored and absent on CI, which uses setup-dart/setup-java, so the shape-gate job never met it. On a developer machine that followed the Makefile's own dist/ instructions the gate refused with 42 findings, every one of them inside a downloaded toolchain: the Dart SDK ships lib/core/stopwatch.dart and Mix ships profile.fprof.ex. `make shape-gate` was unusable on exactly the trees that can run the whole bench. dist/ joins the skipDirs list. The package doc gains the limitation #198 demonstrated: this gate guards MEASUREMENT, not CORRECTNESS. The Go emitter wrote a fixed scalar array twice — 32 wire bytes where the other eight languages write 16 — and Go-to-Go round-trips passed clean because both ends shared the defect. A shape-blind runner driving a defective emitter is still shape-blind and still passes here. Issue #203 records the corpus blind spot. It also now says plainly that a shape name in a comment counts as a hit, which is deliberate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… point does not inline (#55) The doc comment said the flat writer state makes 'the per-field write path fit the compiler's inlining budget'. Measured at go1.27.0 (budget 80): tryWriteBits costs 64 and inlines, but WriteStream.writeBits — the entry point generated code calls once per field — costs 92 and does not, and ReadStream.readBits is 106. So the sentence claimed the opposite of what ships, and schema's Go emitter work (mas-bandwidth/schema#198) measured ~86% of generated-codec time sitting in exactly those calls. Corrected to state what is measured, with the go version beside the numbers since a cost without its toolchain version rots. Also records that BitReader.tryReadBits costs exactly 80 against a budget of 80 — one unit from silently outlining, with nothing to go red when it does. Comments only; no code, no wire, no behaviour change. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…out (#200) * The shape gate: a CI refusal for hand-coded shape measurement Owner concern, 2026-08-31: "if you have any other profiling per-language that is hand coded ... I don't trust this and it will drift. We have to lock this in." bench/tools/shapegate extracts the shape vocabulary from bench/corpus and refuses four things: a corpus identifier named under bench/, a timing primitive anywhere outside the sanctioned runner and tool directories, a bench-shaped source path outside them, and a shape's wire size written down as a literal. bench/SHAPE-GATE.allow is the complete register of what does not yet comply, with exact counts that ratchet: growth fails, and so does leaving a count too high once the debt is paid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * shapegate: satisfy errcheck and staticcheck ST1005 The rule text moves out of the error value and under it, which is where a reader wants it anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * shape gate: rebase onto main — the debt is paid, the register says so #199 deleted family rt and the §1.3 hand-coded shapes, #204 retired the §1.2 example-corpus rows from every runner, #201 deleted the decomp.cpp scratch harness. The ledger's ratchet caught all of it: nine stale counts, seven entries matching nothing. Name hits under bench/ drop from 2080 to 146, and of the 29 remaining in the nine runner legs NOT ONE is hand-coded shape measurement — every one is a comment, a generated-symbol callsite, or an English word that is also a corpus field name. The nine legs are shape-blind. The gate now guards a clean tree instead of a shrinking debt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * shapegate: skip dist/, and state the two limits the rebase exposed Two findings from re-proving the gate on a fully provisioned tree. dist/ is the Makefile's pinned toolchain drop — the Dart SDK, the JDK, OTP, Elixir. It is gitignored and absent on CI, which uses setup-dart/setup-java, so the shape-gate job never met it. On a developer machine that followed the Makefile's own dist/ instructions the gate refused with 42 findings, every one of them inside a downloaded toolchain: the Dart SDK ships lib/core/stopwatch.dart and Mix ships profile.fprof.ex. `make shape-gate` was unusable on exactly the trees that can run the whole bench. dist/ joins the skipDirs list. The package doc gains the limitation #198 demonstrated: this gate guards MEASUREMENT, not CORRECTNESS. The Go emitter wrote a fixed scalar array twice — 32 wire bytes where the other eight languages write 16 — and Go-to-Go round-trips passed clean because both ends shared the defect. A shape-blind runner driving a defective emitter is still shape-blind and still passes here. Issue #203 records the corpus blind spot. It also now says plainly that a shape name in a comment counts as a hit, which is deliberate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: keep the cpp-lock comment attached to the cpp-lock job The shape-gate job was appended after cpp-lock's explanatory comment and before cpp-lock itself, orphaning the comment above the wrong job. Same two jobs, comments back with their own. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…counts The write side already folded its bounds; reads still called the runtime's SerializeInt/SerializeInt64, which derive the bit count from min/max on EVERY call (a BitsRequired per field) and carry the bounds as arguments. Both are compile-time constants of the call site. Folded, reads drop onto SerializeBits — whose wrapper the Go compiler inlines, where the ranged entry points are 168/328 cost units against an 80-unit budget and never inline. The headroom refusal moves into generated code with the same error, ErrValueOutOfRange, and is elided where the range fills its bit width. Wire bytes unmoved; corpus_id 6b213fbfa1a03a99 unchanged. Measured, M2, bench_mixed, same sitting: write 1.37 -> 1.39 M msg/s (+1.5%), round_trip 0.61 -> 0.62 (+2.6%). The small size is the finding: the cost is the CALL COUNT, not the call's arguments. That is what lever 2, the flat word codec, attacks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Maximal runs of statically-sized pieces are gathered, field values computed into locals and OR'd into word-sized chunk locals at literal shifts, and one whole chunk handed to the stream per call. Reads fuse their bounds checks per chunk and take ONE sticky-error test per run instead of one per field. WriteMixedEntity goes from 14 stream calls to 3. Two deliberate deviations from the Rust template (#183), both Go physics: 1. CHUNKS ARE 64 BITS, not 32. SerializeBits64 places a whole word in one call with one bounds check (its two tryWriteBits are inlined into it), where two SerializeBits are two calls. Wire-identical: SerializeBits64 splits low-dword-then-remainder, which IS the 32-bit chunk order. Chunks of 32 bits or fewer still go through SerializeBits, which inlines. 2. EVERY PIECE IS MASKED TO ITS WIDTH. serialize.go's write path MASKS a too-wide value; the Rust runtime only debug_asserts, so #183 had to REFUSE. Masking is what keeps the Go form observably identical. And one coverage win the Rust template could not have: COMPRESSED FLOATS DO NOT BREAK A RUN. This emitter already folds quantization into generated arithmetic ending in a plain bit write, so quantized fields are ordinary pieces — where Rust's live in the runtime, which is why its compressed-float shapes barely moved. Wire bytes unmoved (testdata/wire clean, corpus_id 6b213fbfa1a03a99). Only Go outputs regenerated; every other language byte-identical. Green: schema_test, _random, _ludicrous, C and C-ludicrous, bench and bench_c (including the 6 bench_mixed refusal vectors), test/go, test/go-ludicrous, and the fuzz leg. Measured, M2, bench_mixed, same sitting, on top of lever 1: write 1.39 -> 2.08 M msg/s (1.50x) round_trip 0.62 -> 1.01 M msg/s (1.63x) Negative control: the hand-written rt rows, same source throughout, held at 1.22/1.20 against a 1.23/1.22 baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…element With the per-field calls gone, the profile's next item was the CALL boundary itself: bench_mixed carries up to 80 MixedStat and 8 MixedEntity elements, and each paid a function call plus its own sticky-error load. WriteMixedStat was 12.4% of samples for 18 bits of work, and at cost 134 against Go's fixed 80-unit budget it will never inline itself. An array whose element is a struct that flattens whole now has its element body placed directly in the loop. The element's own Write/Read functions are still emitted — the public surface is unchanged — they are simply no longer the path the generated caller takes. Measured, M2, bench_mixed, same sitting, on top of lever 2: write 2.08 -> 2.27 M msg/s (+9%) round_trip 1.01 -> 1.11 M msg/s (+10%) Control rows held at 1.17-1.20 write / 1.18-1.21 read against the 1.22/1.20 baseline. Wire unmoved, conformance green. Cumulative over the three levers: write 1.37 -> 2.27 (1.66x), round_trip 0.61 -> 1.11 (1.82x). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he run
Two coverage extensions on the same mechanism, both cheap once a run can be
built against an arbitrary base expression:
4. A nested struct field that flattens whole contributes its fields to the
PARENT's run, so they pack into the parent's chunks rather than paying a
call and starting fresh chunks of their own.
5. A small fixed array of scalars is a static run of bound*width bits like
any other, so it unrolls into the enclosing run instead of looping one
stream call per element. Capped at 128 bits so generated code stays
proportional to the schema, and byte-aligned [N]uint8 keeps the bulk-copy
path, which is faster than either form.
Measured, M2, bench_mixed, same sitting, on top of lever 3:
write 2.27 -> 2.32 M msg/s (+2%)
round_trip 1.11 -> 1.16 M msg/s (+4.5%)
Control rows 1.20/1.21 against the 1.22/1.20 baseline. Wire unmoved,
conformance green.
Cumulative over five levers: write 1.37 -> 2.32 (1.69x),
round_trip 0.61 -> 1.16 (1.90x), blended 1.83x.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full per-shape leg caught two rows going the wrong way: rigidbody_moving and rigidbody_at_rest both lost 9% on write. Mechanism, found in the emitted code — RigidBody is all float64, so every piece is exactly one whole chunk. The run packed into as many chunks as it had fields, removing no call, while ADDING a materialized local per field and an address-taken chunk local, where the per-field form had simply passed the struct field's own address. The policy is now the honest one: flatten when ceil(bits/chunk) is fewer calls than the run has bit-carrying pieces, and otherwise leave the per-field form alone. That makes the transform non-regressive on call count by construction rather than by luck. bench_mixed is unaffected (write 2.35, round_trip 1.16 — its runs all reduce calls); the rigidbody shapes return to the per-field form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The profile after the first five levers put the generated code's own packing arithmetic at 15% of samples and the runtime's bit placement at 30% — the per-field calls were gone, but bench_mixed still spent most of its calls on eighty 18-bit stat elements, each carrying 18 of a possible 64 bits. An array of a struct that flattens whole is now emitted as a K-at-a-time loop over a run spanning K elements, followed by a remainder loop. K is chosen as the element count that minimises chunks-per-element within the run cap and an eight-element unroll cap; for the 18-bit stat element that is 7, and eighty elements fall from 80 stream calls to 25. Measured, M2, bench_mixed, same sitting, on top of lever 5: write 2.34 -> 2.66 M msg/s (+14%) round_trip 1.15 -> 1.40 M msg/s (+21%) Cumulative over six levers, against current main in one sitting: write 1.38 -> 2.66 M msg/s (1.93x) round_trip 0.61 -> 1.40 M msg/s (2.30x) blended 1182 -> 545 ns/msg (2.17x) Named behaviour change, the same class the run form already carries: a group's refusals all run before the group packs, so a refused write has emitted fewer bits than before. Which values are refused, and with which error, is unchanged. Wire bytes unmoved, conformance green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uped loop Two things, found together when the cap was moved. THE BUG: the grouped array form declares its index outside the loop, so two grouped arrays in one function collided on i. It only surfaced once the cap let a second array group, but it was latent from lever 6. The grouped loop now carries its own block, as the runs already do. THE CAP: measured on the corpus shape rather than inherited from the Rust template's 256. bench_mixed write / round_trip, M msg/s, one sitting: 256 -> 2.66 / 1.40 384 -> 2.78 / 1.49 <- peak 512 -> 2.75 / 1.46 1024 -> 2.62 / 1.43 The curve peaks and falls: past the peak a run holds more field values live than the register file has room for and they spill, which is the cost the cap exists to bound. 384 also lets the 135-bit entity element group two at a time. Cumulative over seven levers, against current main in one sitting: write 1.38 -> 2.78 M msg/s (2.01x) round_trip 0.61 -> 1.49 M msg/s (2.44x) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six legs measured in one sitting (java, dart and elixir report ABSENT: this fresh clone has no pinned dist/ toolchains and the box has no JRE). The locked c/cpp reference reproduces at 100/104%, which is the control on the sitting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Emitter output unchanged — regenerated with no diff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR #183's Rust template maps one item to exactly one piece, so its fallback (re-emit each of a run's pieces) IS re-emitting each of its items. This emitter's levers made classification 1:N — an unrolled fixed array contributes one piece per element, a flattened nested struct contributes pieces naming ITS fields at a different base expression — and the fallback was inherited verbatim. Three defects followed from that one gap: - `type Pair { values [2]float64 }` emitted the element loop TWICE on both paths: 32 bytes where every other language writes 16. Go-to-Go round trips passed, so nothing went red. - a nested struct split by the run cap emitted `value.C` on the OUTER type — a compile error, or, under a name collision, the silent serialization of the wrong field. - a bare `type Vec2 { x float64; y float64 }` carried an unused "math" import: needsMath was set during SPECULATIVE classification and never cleared when the run fell back. A run now accumulates whole ITEMS as flatGroups and splits only on item boundaries, so the fallback re-emits items against the base each item owns. flatPiece no longer carries an ir.Item at all — a piece is a bit-placement recipe and nothing else, which makes re-emitting one structurally impossible rather than merely avoided. needsMath moves into the float pieces' emit/read closures, which run at emission. Wire is unmoved and generated/ is zero-diff on the corpus: none of the three shapes exists in it. That blind spot is issue #203, addressed next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#203) Three emitter defects — one of them silent wire corruption — reproduced on one-line schemas, and the entire conformance corpus caught none of them. It grew from real game shapes, and a real shape always carries enough neighbouring fields to mask the arrangements a run/chunk emitter keys on. The corpus's realism is precisely what made it blind. examples/Degenerate.schema carries twelve shapes that are deliberately unrealistic: a fixed scalar array as essentially the whole message ([2]float64, [2]uint64, [2]int64, [1]T, [N]T landing on a chunk boundary, an array plus a trailing field, two arrays back to back); a bare two-float unit with no other math consumer in the file; and a nested struct as the ONLY field, as the FIRST field, and straddling the Go emitter's run cap. They are pinned as one wire golden, testdata/wire/degenerate.bin, and every one of the nine legs is held to it: the stream legs (C++, C, Go, Rust, C#, JS) write the twelve into one stream, and the whole-message legs (Dart, Java, Elixir) concatenate their twelve buffers. That equality is why every type in the file is a whole number of bytes wide — the file's header says so, because it is load bearing. internal/codegen/golang states the same properties directly against maxRunBits, so re-tuning that constant cannot quietly retire the coverage. Verified red first: on the pre-fix emitter all four assertions fail, and generated/go/Degenerate.go does not compile ("math" imported and not used; value.C undefined on TrioStraddle). The new file caught a SECOND emitter defect on sight, the same class as the Go one: the Dart backend declared all four conversion-scratch views whenever any was needed, so a file converting only float64 carried an unreferenced _u32 and `dart analyze` refused it. The views are now emitted per conversion. Existing wire goldens are unmoved; the corpus protocol id moves, as adding a file must, and is re-pinned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stead of panicking The Go leg is not untouched: the flat word codec's conviction — ~86% of generated-codec time inside the runtime's per-field bit calls — came from this flag. It stays, because the next round will want it and a profiler kept in a scratch fork of the leg is a profiler that rots. It is an iteration instrument: it changes nothing about what is measured, and no timed row is taken under it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0ad563c to
0a0b86c
Compare
gcc's -Wmaybe-uninitialized (CI, -Werror) cannot see that a read function stores every field, and clang does not raise it, so the new block built clean here and refused there. The rest of the leg already dirties its read targets — a read that skips a field must be caught — so this block does the same, with the same 0xEF fill. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR carried a before/after table and a 0.9977x negative control with NO committed CSV, and #199 retired the shapes those numbers were taken over, so they could not be re-run on main. These two files are that evidence, re-taken on the shape that survives — bench_mixed, family gen — and committed before AND after, which is #183's precedent. One sitting, back to back, on the tree as it stands after #204 rewrote the runners. Nothing changed between the halves but generated/bench/go: main's Go emitter output for the before half, this branch's for the after. go write 1.35 -> 2.67 M msg/s (1.98x), round_trip 0.60 -> 1.41 (2.36x) headline 583% -> 248% of generated C++ on run.sh's own statistic (2.35x) controls geomean 0.9940x over 16 rows — every other leg, unchanged code, compiled identically in both halves (min 0.9426, max 1.0339) corpus_id is 6b213fbfa1a03a99 in every row of both files: the runner hashes the goldens it loads, and Degenerate.schema's golden is not one of them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0a0b86c to
0e930d5
Compare
…the per-field form Comment and shape only — generated output is byte-identical. flush() built the run twice, once to ask worthFlattening and once to emit it; the oversized-group branch had no note saying no classifier produces one today. The orphaned flatFieldPiece doc above flatArrayElemPiece now sits on the function it describes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Unblock response — every item in the verdict, with where it landed. The body is rewritten around it. 1. Silent wire corruption / 2. nested field on the outer type / 3. unused Corpus coverage (#203) — Baseline — corrected to 712% throughout, re-derived from main's committed
The unreceipted claims — the per-shape geomean and the 0.9977x control are withdrawn, not restated: #199 retired those shapes and no CSV was committed. Replaced by a paired same-sitting A/B on the shape that survives, both halves committed ( The 10.44% spread / 275% median — recorded in the body verbatim, and resolved by the re-run: go's round_trip rows now carry 0.36% and 1.89%, and the two statistics agree at 248% and 243%. Two things worth flagging for the reviewer:
Not merging — yours to call. |
|
Re-review verdict: MERGE. Every wire question answered by the reviewer's own commands in its own clone, not by reading claims — which is what a PR that already shipped one silent wire defect earns. THE REPAIR, verified structurally AND empirically. Structurally: RED-FIRST REPRODUCED VERBATIM at 997c553's parent — all four assertions fail, and compiling the pre-fix generated Go reproduced the PR's error text INCLUDING ITS LINE NUMBERS. The silent-corruption assertion is dispositive at source level: pre-fix WIRE: reviewer built bin/schema itself, THE DART FIX traced through all four reachable scratch states, correct and minimal. And a genuinely useful negative: C#/Java/Elixir use function-local scratch so they cannot carry this class, while JS-flat DOES emit NUMBERS recomputed from both committed CSVs: 583%→248% (2.35x), round_trip 2.3603x max / 2.3640x median (agrees both ways), controls geomean 0.9940x with min 0.9426 max 1.0339 over n=16 — exact. Baseline 712.4% max / 711.2% median; 712→248 = 2.871x. Three findings recorded, none blocking: (a) the headline sentence says '1.98x write' unqualified where that is the max statistic and median is 1.88x — disclosed at line 104 and the table row is labelled, so it wants one word, not a block; (b) both new CSVs stamp a schema commit that does not resolve on origin (it is the pre-rebase SHA of what is now 1f47f56, so the measurement was taken on this branch's content, but a re-runner cannot reconstruct the tree from the PR alone); (c) a 'denominators agree to 2.4%' line that did not reproduce on any statistic tried — immaterial, the conclusion routes through the paired sitting. Also noted mechanically: |
#198 landed `examples/Degenerate.schema` and pinned its Elixir source golden with the pre-round emitter, so this branch merging as-is turned main red: `TestGoldenSourceElixir` refused `testdata/golden/elixir/Degenerate.ex`. The re-pin is a SOURCE re-pin only. `make update-goldens` rewrote every wire golden under `SCHEMA_UPDATE_WIRE_GOLDENS=1` and every one came back byte-identical, `testdata/wire/` has no diff, and the nine-backend chain byte-compares `degenerate.bin` against the C++ pin in all nine legs and is green. The new text is the round's own shapes: one map destructure per scope (B) and literal-width flushes with `scratch_bits` gone (D).
… cannot reach Degenerate.schema's standing property is that every type in it is a whole number of bytes. That is load bearing for what it catches and it is also a ceiling: no clause boundary inside it ever lands mid-byte, so an emitter that groups array elements picks the same group size on the write and the read side of every type in the file. The Elixir round's write clause has a 52-bit budget and its read clause a 49-bit window, so on Degenerate the two always agree and the disagreement is untested. Clauses.schema picks element widths where they do not. At 13 bits the write clause takes four elements (52, the whole budget) and the read clause three (39, inside the window); at 17 it is three against two; at 26, two against one. Counts run 0, below a clause, exactly a clause, one past it, and the bound, so the remainder path is entered from every phase. It also carries a fixed mid-byte array, grouping across a nested struct boundary, a union of empty arms behind a tag, and string/bytes at zero, partial and full length behind a 5-bit lead so the align inside them is a real barrier. Joins.schema does the same to the static-offset state machine: arms that agree and disagree on width, a branch with no else, a branch inside a branch, an align that regains staticness on one path only, an array that gives it up on one path only, unions of unequal arms at mid-byte offsets, and a long static run after an align. Unlike Degenerate these shapes are NOT byte-aligned, so one shared stream would not equal a concatenation of the shapes written alone — and the Elixir emitter returns each message as its own binary from bit zero, so it cannot write a shared stream at all. Every shape is therefore written to its own stream and flushed, and the golden is those concatenated. Every leg can reproduce that, and each shape's bytes stay individually attributable. C++ pins, Go and Elixir byte-compare; the remaining six legs follow. Adding units to the corpus moves the example unit's protocol id, as #198 did when it added Degenerate. No existing wire golden moved.
The published figures were unverifiable: `git log origin/main..HEAD -- bench/results/` was empty, so no re-runner could check them. #198, #199 and #204 all committed their before/after data this week; this matches that. Both halves are a fresh paired A/B taken in one sitting on the M2 Air after the rebase, three minutes apart, corpus_id 6b213fbfa1a03a99 on every row. The BEFORE half stamps main at 52691a0 and the AFTER half elixir-swizzle at fef86ed — the last CODE commit on the branch, so both stamps resolve on origin and reconstruct the tree that was measured. Elixir, bench_mixed, family gen: write 135,696 -> 378,044 msg/s (2.79x; spread 2.08% -> 0.22%) round_trip 95,413 -> 239,723 msg/s (2.51x; spread 0.46% -> 0.38%) As a share of generated C++ (§2.9, max rates): 3743% -> 1457%. Both elixir rows are far inside §2.3's 15% noise gate, so neither is a row bench/tools/relative.go would exclude.
…#202) * elixir #174: the profile, before any lever is pulled Wall-clock decomposition of generated Elixir on the canonical shape (bench_mixed, 438 bytes: 8 entities x 135 bits, 80 stats x 18 bits, 4 loadout bytes), M2 Air, OTP 29.0.5 / Elixir 1.20.4, 100k ops per measurement, the same instances the bench rotates: write 7.344 us/op read 3.561 us/op (round_trip 10.748) write splits: stats loop, 80 elements 2.632 us 32.9 ns/stat (36%) entities, 8 elements 2.970 us 371.5 ns/ent (40%) header + footer + loadout 1.742 us (24%) The write path is 2x the read path, and 325 binary appends per message is what it is made of: the emitter's mergeW flushes every whole byte after EVERY field, so one bs_append BIF call rides each of the 131 static field sites plus 8x14 + 80x2 + 4x1 loop sites. eprof agrees on the ordering (stats 38.7%, entities 23.7% of write) and is not quoted for magnitudes: 162000 traced calls into the stats loop inflate exactly the function the wall clock says is hot. Nothing measured here lands in serialize.elixir. The generated Elixir codec has no runtime dependency at all, so every lever below is a schema-level emitter change and #170's routing question does not arise. * elixir: lever A — one binary append per group, not per field The emitter knew every field width statically and threw the knowledge away: mergeW flushed the scratch's whole bytes after EVERY field, so a 438-byte bench_mixed message cost 325 bs_append BIF calls. mergeW now carries a GROUP. It merges into the scratch and flushes only when the next field would pass the budget, and flushW closes the group at every barrier that observes data or scratch_bits — the write function's tail, an align, the bytes of a string, a loop helper's call and its own element tail, and the joins of a branch or a union case. flushW is a no-op when no group is open, which is what makes a barrier free where one is already closed. The budget is 52 bits and the number is not a taste. The BEAM's fixnum is 60-bit signed, so an intermediate at or above 2^59 costs a heap bignum; a flush leaves at most 7 bits behind, so 7 + 52 = 59 is the whole envelope. Measured on the read side of the same shape, a 72-bit window is SLOWER than the shipped 40-bit one (29.4 vs 27.6 ns/element) — boxing costs more than the flush it would save, so the group stops at the boundary. MEASURED, wall clock, canonical shape, 100k ops (before -> after): write 7.344 -> 5.208 us/op 1.41x stats loop 32.9 -> 22.3 ns/stat entities 371.5 -> 223.2 ns/entity round_trip 10.748 -> 8.886 us/op read 3.561 -> 3.561 us/op (untouched, as expected) bench --quick: write 0.13 -> 0.18 M msg/s, round_trip 0.09 -> 0.12 Static append sites in generated/bench/elixir: 131 -> 65; per entity 14 -> 3, per stat 2 -> 1; ~325 -> ~128 appends per message. Wire bytes unmoved: corpus_id 6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and test/elixir-ludicrous OK, mix format --check-formatted clean. * elixir: lever C — one window decode per group, not per field rd/3 opened a fresh match context for EVERY field: 14 per entity, 2 per stat. The generator knows every width statically, so readR now reads a GROUP into rv once and cuts each field out with a static shift and mask. The fused static run's own length sizes the group, so a short run keeps the cheap 40-bit window and a long one takes rdw's 56-bit window. Two windows, and the second one's width is the same fixnum argument the write group's budget rests on. A 56-bit window less the 7-bit worst-case offset is a 49-bit group and stays under 2^59; a 64-bit window would box. Measured on this shape a 72-bit window is SLOWER than the shipped 40-bit one it would replace — 29.4 vs 27.6 ns/element — so 49 is the ceiling and not an arbitrary stopping point. Reading a group wider than the fields it feeds is safe by construction: rd and rdw never raise (the tail falls back to the bytes that exist), and bits past the bounds-checked run are discarded, never observed. Every bounds check, range check, constant check and refusal is where it was. rdBreak closes the group at every barrier — the read surface, a loop helper's entry and every call to one, an align, the bytes of a string, and the arms of a branch or a union case. MEASURED, wall clock, canonical shape, 100k ops (after lever A -> after C): read 3.561 -> 2.384 us/op 1.49x stats 27.6 -> ~15 ns/stat (rd calls 2/elem -> 1) entities (rdw calls 14/elem -> 4) round_trip 8.886 -> 8.126 us/op write 5.208 -> 5.208 us/op (untouched, as expected) bench --quick: round_trip 0.12 -> 0.13 M msg/s Wire bytes unmoved: corpus_id 6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and test/elixir-ludicrous OK, mix format --check-formatted clean. * elixir: lever B — one map read per scope, not one per field Elixir's `.` on a struct is a get_map_elements of its own with its own raise branch. The writer spent one per field: 14 for an entity, 2 for a stat, ~30 at the top level. A scope now reads its fields ONCE — `%{stat_id: e_stat_id, delta: e_delta} = e` — and every reference to those fields is a local from there on. Scope, precisely: a scope binds the fields it reads UNCONDITIONALLY at its own level — its field items and the conditions of its branches. A branch arm's fields are NOT bound at the enclosing level; the arm is its own scope and binds them when it is taken, so a value the wire never asks for is still never demanded of the caller. Below two fields the pattern would not pay for itself and none is emitted. Refusals are unchanged. A struct always carries every key, so the bound locals are exactly as unconditional as the dotted accesses they replace, and every range, count, mask and length check is where it was. The one delta, named: a MAP missing a key now raises MatchError where it raised KeyError — a raise either way, on exactly the same inputs, never a wrong answer. The raise TEXT is unchanged: g.dsp resolves a local back to the dotted access it stands for, so the message still reads "e.delta is above the wire maximum" and never names a local the caller never wrote. MEASURED, wall clock, canonical shape, 100k ops (after lever C -> after B): write 5.208 -> 3.202 us/op 1.63x stats loop 23.4 -> 15.4 ns/stat entities 212.6 -> 96.4 ns/entity round_trip 8.126 -> 5.466 us/op read 2.384 -> 2.276 us/op (the read side binds locals already) bench --quick: write 0.18 -> 0.31 M msg/s, round_trip 0.13 -> 0.18 Cumulative over A + C + B, against the round's baseline: write 7.344 -> 3.202 us/op 2.29x read 3.561 -> 2.276 us/op 1.56x round_trip 10.748 -> 5.466 us/op 1.97x Wire bytes unmoved: corpus_id 6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and test/elixir-ludicrous OK, mix format --check-formatted clean. * elixir: lever E — the float32 step, and a declaration constant folded Three changes to the compressed-float helpers, all of them removing work that was provably redundant: fr/1 lets the REFUSAL be the test. A float segment does not match a non-finite pattern, so the finite path is one construction and one match and never touches the exponent field; the second clause reads the sign of exactly the patterns the first refused. NaN still maps by sign, as it did. cf_quantize takes miv32 — the float32 of the step count — as a generation-time literal instead of computing fr(miv * 1.0) on every call. It is a declaration constant; folding it removes one of the six float32 steps. cf_decode takes it directly in place of the integer count, which was only ever used to compute the same rounding. trunc(Float.floor(x)) becomes floor(x): one BIF returning an integer instead of a float floor and a truncation. The argument is finite by construction (normalized is clamped to [0, 1] and miv32 is finite), so the paths differ only in the class of an unreachable raise. MEASURED, wall clock, canonical shape, 100k ops, and reported honestly: write 3.202 -> 3.18 us/op AT OR BELOW the noise floor read 2.276 -> 2.19 us/op ~3% round_trip 5.466 -> 5.30 us/op The isolated micro said more (cf_quantize 203.6 -> 135.2 ns/call), and the micro was wrong to say it: it passed fr as a closure, so every fr call it removed was an inflated indirect call rather than the direct local call the generated module makes. The in-situ number is the ruling. The lever stays because it costs nothing and removes real work; it is recorded as small, not as what the micro promised. Wire bytes unmoved: corpus_id 6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and test/elixir-ludicrous OK, mix format --check-formatted clean. * elixir: re-pin the four source goldens the emitter moved Deliberate emitter change, and only the SOURCE goldens move: no file under testdata/wire is touched by this commit or by any of the four levers, and the full `make test` chain — every language's conformance suite, the wire goldens, the fuzzers, the format refuser — is green. * elixir: lever D — the static offsets the generator already knew The group model carried its widths statically and then computed with them at runtime anyway: every merge shifted by the scratch_bits VARIABLE and added to it, and every flush divided that variable by eight to size a binary segment whose width was therefore dynamic. A write function starts at a known empty scratch and stays known until the message's own data decides a length. Through that whole region the emitter now tracks the offset itself, so: - a merge is one statement, `scratch = scratch ||| v <<< 16`, with a literal shift and no scratch_bits arithmetic at all; the first merge of an empty group is a bare bind, `scratch = v` - a flush is `data = <<data::binary, scratch::little-size(4)-unit(8)>>` with a LITERAL segment width — the form the BEAM's binary construction is built for — plus a literal shift, in place of four statements - an align that lands on a boundary emits nothing, and one that does not appends the residual byte unconditionally instead of testing for it - the function's tail is `data` or `<<data::binary, scratch>>` outright, never `if scratch_bits != 0` - scratch_bits is not even bound where the surface never needs it Staticness is a property the emitter EARNS and gives up honestly. It is given up where a loop helper is called (how many elements rode is the message's business) and where a branch's arms end on offsets that disagree; in that case each arm publishes the offset it reached and the emitter goes back to maintaining the variable. It is REGAINED at every align, which lands the position on a byte whatever the data did. Where every arm of a branch or a union case does agree, the offset stays static past the join and scratch_bits does not ride the join's tuple at all. Behaviour is unchanged: the same statements in the same order over the same values, with the arithmetic the generator can do moved to generation time. Every range, count, mask and length check is where it was, and every raise carries the text it carried. Wire bytes unmoved: no file under testdata/wire is touched, corpus_id 6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and test/elixir-ludicrous OK, mix format --check-formatted clean, and the whole nine-backend `make test` chain green. Only the four SOURCE goldens move, re-pinned here with the emitter that moved them. MEASURED: in the sitting's A/B, recorded on the PR with the round's table. * elixir: lever J — a scalar array element sized its own read window Lever C sizes each window decode by the fused static run it sits in, and the run fuser is reached through emitReadItems. A scalar array element is not: readHelper hands it straight to the scalar read, so the run was zero — unknown — and readR fell back to its widest window. A one-byte element was opening the 56-bit window to take eight bits. That is not merely a wider mask. rdw needs seven bytes ahead of the position to match its window and rd needs five, so the wide window drops into the :binary.decode_unsigned tail fallback two bytes sooner — and a scalar array is very often the LAST thing in a message, which is exactly where that boundary lies. A scalar element's own width is the run, so readHelper sets it. The loadout loop now reads `rd(data, bits_read, 8)` where it read `rdw(data, bits_read, 49)`. Nothing else moves: a struct or union element still goes through emitReadItems and fuses its own runs, and readR's choice of window was already free to be any width that covers the field. Wire bytes unmoved: no file under testdata/wire is touched, corpus_id 6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and test/elixir-ludicrous OK, mix format --check-formatted clean, the whole nine-backend `make test` chain green, and the two source goldens the emitter moved re-pinned here. MEASURED: in the sitting's A/B, recorded on the PR with the round's table. * elixir: lever K — one append for a whole clause of elements, not one each The write loop appended once per ELEMENT: 80 bs_append calls for the stats array of one bench_mixed message, 4 for the loadout. Lever A closed the group across the fields of an element; it could not close it across the call boundary between elements, because a helper is entered at an offset the caller's element count decides. Measured in isolation, that boundary is where the write path's cost is. Twelve appends of one 32-bit segment each cost 103.6 ns; the same 48 bytes in three appends of four segments cost 79.0, and in one append of twelve, 73.3. The same twelve appends with a DYNAMIC segment width cost 103.7 — identical. The append is the expense; the arithmetic around it is not. So a clause takes SEVERAL elements off the list. k is chosen by the group budget the fixnum boundary already fixed — the most whole elements whose widths fit in 52 bits, capped at four so one array field cannot cost unbounded generated code — and the k element bodies merge into one group and flush once. The single-element clause behind the wide one is the remainder, so a list length never has to divide anything, and an element whose width the wire decides keeps one clause per element as before. stats (18 bits): 2 elements per clause, 80 appends -> 40 loadout (8 bits): 4 elements per clause, 4 appends -> 1 Nothing about an element's emission changes. The clause names its slots e1 and e2 where it used to have just e, and the raise TEXT is held: the display map resolves every slot back to "e", so a message still reads "e.delta is above the wire maximum" and never names a slot the caller never wrote. Every range, count, mask and length check is where it was. MEASURED, wall clock, the canonical shape from the committed variant corpus, 64 rotating instances, 150k ops, 4 runs x 4 interleaved passes, best per pass, median of passes: write 3.097 -> 2.65 us/op 1.17x round_trip 5.268 -> 4.74 us/op 1.11x read 2.132 -> 2.09 us/op (untouched, as expected) Wire bytes unmoved: no file under testdata/wire is touched, corpus_id 6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and test/elixir-ludicrous OK, every generated module compiles without a warning, mix format --check-formatted clean, the nine-backend `make test` chain green, and the source goldens the emitter moved re-pinned here. * elixir: lever L — one window decode for a whole clause of elements The read loop's other half of lever K. Lever C reads a GROUP into one window and cuts each field out of it with a static shift and mask, but a loop helper is a function boundary, so a group could never span two elements: 80 window decodes for the stats array, 4 for the loadout. A read clause now decodes k elements under ONE window. k is chosen by the wide window's usable width — the most whole elements whose widths fit in 49 bits, which is the same fixnum boundary the write budget rests on, capped at four — and the clause carries a guard on the remaining count, with the single-element clause behind it as the remainder. Two things had to give way for a window to span elements. The element emitters now take the variable they bind, so a clause can hold e1 and e2 without either shadowing the other. And the run fuser no longer overwrites an OUTER run: a named element's own scope would otherwise size the window to one element and undo the clause. Where no outer run is open the fuser behaves exactly as it did. stats (18 bits): 2 elements per rdw window of 36 bits, 80 decodes -> 40 loadout (8 bits): 4 elements per rd window of 32 bits, 4 decodes -> 1 Reading a window wider than one element is safe on exactly the grounds lever C established: rd and rdw never raise, the tail falls back to the bytes that exist, and bits past the bounds-checked run are discarded and never observed. The bounds check itself is unchanged — the call site proved count * elem for a counted array, and an unbounded run still checks its own span before it reads it. MEASURED, wall clock, the canonical shape from the committed variant corpus, 64 rotating instances, 150k ops, 4 runs x 4 interleaved passes, best per pass, median of passes, against lever K: read 2.089 -> 1.838 us/op 1.14x round_trip 4.736 -> 4.409 us/op 1.07x write 2.703 -> 2.691 us/op (untouched, as expected) Wire bytes unmoved: no file under testdata/wire is touched, corpus_id 6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and test/elixir-ludicrous OK, every generated module compiles without a warning, mix format --check-formatted clean, the nine-backend `make test` chain green, and the source goldens the emitter moved re-pinned here. * elixir: the emitter answers modernize and gofmt Four findings, all of them this branch's own, none of them a behaviour change: the two scope-binding map copies lever B introduced become maps.Copy, and the two unroll caps levers K and L introduced become min. The generator emits identical bytes for every schema in the tree. CI's lint job runs modernize at @latest, so a check that ships tomorrow lands on the next branch the same way; this is the branch paying for its own four. * elixir: re-pin Degenerate's source golden after the rebase onto main #198 landed `examples/Degenerate.schema` and pinned its Elixir source golden with the pre-round emitter, so this branch merging as-is turned main red: `TestGoldenSourceElixir` refused `testdata/golden/elixir/Degenerate.ex`. The re-pin is a SOURCE re-pin only. `make update-goldens` rewrote every wire golden under `SCHEMA_UPDATE_WIRE_GOLDENS=1` and every one came back byte-identical, `testdata/wire/` has no diff, and the nine-backend chain byte-compares `degenerate.bin` against the C++ pin in all nine legs and is green. The new text is the round's own shapes: one map destructure per scope (B) and literal-width flushes with `scratch_bits` gone (D). * corpus: Clauses.schema and Joins.schema — the arrangements Degenerate cannot reach Degenerate.schema's standing property is that every type in it is a whole number of bytes. That is load bearing for what it catches and it is also a ceiling: no clause boundary inside it ever lands mid-byte, so an emitter that groups array elements picks the same group size on the write and the read side of every type in the file. The Elixir round's write clause has a 52-bit budget and its read clause a 49-bit window, so on Degenerate the two always agree and the disagreement is untested. Clauses.schema picks element widths where they do not. At 13 bits the write clause takes four elements (52, the whole budget) and the read clause three (39, inside the window); at 17 it is three against two; at 26, two against one. Counts run 0, below a clause, exactly a clause, one past it, and the bound, so the remainder path is entered from every phase. It also carries a fixed mid-byte array, grouping across a nested struct boundary, a union of empty arms behind a tag, and string/bytes at zero, partial and full length behind a 5-bit lead so the align inside them is a real barrier. Joins.schema does the same to the static-offset state machine: arms that agree and disagree on width, a branch with no else, a branch inside a branch, an align that regains staticness on one path only, an array that gives it up on one path only, unions of unequal arms at mid-byte offsets, and a long static run after an align. Unlike Degenerate these shapes are NOT byte-aligned, so one shared stream would not equal a concatenation of the shapes written alone — and the Elixir emitter returns each message as its own binary from bit zero, so it cannot write a shared stream at all. Every shape is therefore written to its own stream and flushed, and the golden is those concatenated. Every leg can reproduce that, and each shape's bytes stay individually attributable. C++ pins, Go and Elixir byte-compare; the remaining six legs follow. Adding units to the corpus moves the example unit's protocol id, as #198 did when it added Degenerate. No existing wire golden moved. * corpus: the Rust, C and JS legs byte-compare Clauses and Joins * corpus: the C#, Dart and Java legs byte-compare Clauses and Joins All nine legs now hold the two units to the C++ pin. The corpus README gains a row for each, stating what each reaches that Degenerate cannot. * bench: commit both halves of the sweep the round's numbers come from The published figures were unverifiable: `git log origin/main..HEAD -- bench/results/` was empty, so no re-runner could check them. #198, #199 and #204 all committed their before/after data this week; this matches that. Both halves are a fresh paired A/B taken in one sitting on the M2 Air after the rebase, three minutes apart, corpus_id 6b213fbfa1a03a99 on every row. The BEFORE half stamps main at 52691a0 and the AFTER half elixir-swizzle at fef86ed — the last CODE commit on the branch, so both stamps resolve on origin and reconstruct the tree that was measured. Elixir, bench_mixed, family gen: write 135,696 -> 378,044 msg/s (2.79x; spread 2.08% -> 0.22%) round_trip 95,413 -> 239,723 msg/s (2.51x; spread 0.46% -> 0.38%) As a share of generated C++ (§2.9, max rates): 3743% -> 1457%. Both elixir rows are far inside §2.3's 15% noise gate, so neither is a row bench/tools/relative.go would exclude.
Main at d00cf49, all of the night's emitter work landed (#198 go, #202 elixir), box idle, nine legs, corpus_id 6b213fbfa1a03a99. c 98 / cpp 100 / rust 154 / java 157 / go 239 / cs 361 / dart 381 / js 486 / elixir 1498 This is ledger point one for #194: the first sitting where the canonical shape, the data-driven harness, and the night's optimizations are all on main together, measured with nothing else running. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The third port of the template Rust (#183) and Go (#198) already carry: field values into locals, OR'd into 64-bit chunk words at literal shifts, one packer step per chunk instead of one per field. WHY IT PAYS HERE IS NOT WHY IT PAID IN GO. #198's finding was that Go's per-field entry points sit past the compiler's inlining budget, so every field cost a real call and ~86% of the codec's time was inside them. C# has no such wall: the JitDisasm audit found ZERO direct calls into serialize.cs from any generated method — every entry point inlines. What C# pays instead is what those inlined bodies DO. BitWriter's packer step masks, shifts, ORs into the scratch word, tests `newScratchBits >= 64`, conditionally stores a qword and recovers the spill: a data-dependent branch and a conditional store PER FIELD. MixedEntity paid fourteen of them for 135 bits. It now pays three. THREE INVARIANTS, RE-DERIVED RATHER THAN INHERITED — #198's review record is why they are spelled out in flat.go's header rather than assumed: 1. 64-bit chunks are wire-identical because serialize.cs's SerializeBits64 splits low-dword-first-then-remainder, which IS the 32-bit chunk order. 2. Every piece is masked to its width, because BitWriter.WriteBitsUnchecked TRUNCATES a too-wide value rather than refusing it — the flat form must truncate identically or a wide value corrupts its neighbours in the chunk instead of losing its own high bits. This is the invariant the Go port lost by inheriting Rust's fallback; here it is enforced at the one place a piece's value expression is built. 3. Write refusals come first, in declaration order — same values refused, same non-latching false. What moves is how many bits reached the stream before the refusal, and a refused write leaves a partial message either way. A run breaks at align, strings/bytes, arrays, branches, nested struct and union calls, and the float, fixed-point, compressed-float and 128-bit families. A run that would not REDUCE the packer-step count is not flattened. A run that does not pay falls back ITEM BY ITEM — the item rides inside every piece for exactly that reason, because the item is the only unit the per-field path can re-emit. THE READ HALF IS NOT IN THIS COMMIT, and the blocker is named rather than worked around: folding a ranged read replaces SerializeInt with a generated comparison, and serialize.cs's ranged read LATCHES ValueOutOfRange where a generated guard returns false without latching. cs publishes checks=always and test/cs pins that latch, so the read fold waits on a public refusal-latch on ReadStream/ReadBatch — a serialize.cs change, its own PR. The write path has no such dependence: its folded range refusals already return false without latching. Measured ceiling for the read half, from a hand-written flat prototype of the two hot cores: round_trip 1.08 -> 1.55 M msg/s, a 42% cut in read time. MEASURED, against the previous commit, same box, median of 3 sittings: bench_mixed write 2.82 -> 3.30 M msg/s (+17%) bench_mixed round_trip 1.08 -> 1.18 M msg/s (+9%) and against main: bench_mixed write 2.10 -> 3.30 M msg/s (+57%) bench_mixed round_trip 0.97 -> 1.18 M msg/s (+22%) flatMaxRunBits is Go's 384 carried unmeasured: swept at 128/256/512 the C# numbers do not move, because this corpus's longest run is 135 bits and no cap in that range binds it. Said out loud in the file rather than presented as a C# measurement. Wire bytes unmoved: full `make test` green across nine backends, the degenerate corpus included, and the bench's golden + per-variant round-trip gates pass at corpus_id 6b213fbfa1a03a99. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…word codec (write 1.59x, round_trip 1.60x; 361% -> 219% of generated C++) (#208) * C# emitter: union batch cores, and a scoped batch per composition site The C# backend was the only one never audited for generation quality, and the profile of the bench shape found the cost in one place: composition. MEASURED FIRST (DOTNET_JitDisasm + DOTNET_JitDisasmSummary, FullOpts, tiering off, on generated/bench/cs/Bench.cs). Everything the runtime offers already inlines — zero direct calls into serialize.cs from any generated method. What did not inline was the generated code's own composition: the entity array (8 elements) and the stat array (80 elements) each reached their element type through its STREAM entry, and each of those entries opened a batch and closed it again. 88 BeginBatch captures and 88 End restores per 438-byte message, plus 88 real calls the JIT will not inline through, on a message where those two arrays carry 72% of the wire bits. They reached the stream entry because BenchMixed reaches a union, and the batch plan excluded any type that reached one — a union's arm dispatch had no core to compose with. That exclusion was landed as land-simple with the note "a follow-on if a profile ever convicts the plain path". It has. TWO CHANGES, one law each. 1. Unions carry batch cores. A union's arm dispatch is a switch over calls, and inside a core those calls go core-to-core by ref like any other composition — nothing about the inline-only rule ever stood in the way, only the missing emitter half. densityUnion weighs the tag plus the heaviest arm; the core closure walks arms as well as struct fields. 2. A composition site inside a stream body opens a SCOPED batch — one capture and one restore for the whole site, an array's entire loop included. Only the composition call rides inside the scope (an array's count site stays on the stream ahead of it), so the scope has exactly one way to fail and End always runs before the refusal leaves the function. AND ONE MEASURED REFUSAL, recorded because it is a result. The obvious third change — let the whole message body run on one batch now that the union no longer blocks it — is SLOWER, and materially: bench_mixed write 2.10 -> 2.04 M msg/s and round_trip 0.97 -> 0.76 M msg/s against the same baseline the two changes above improve. The JIT did inline the 12.7 KB core (IL size 23 at the entry), so this is not the address-exposure law firing; it is the delegated sites. A body with any bulk, 128-bit or fixed-point site must Sync the batch down and Recapture it around that site, and between them it hands the register allocator a ref struct that has to survive the whole body. batchEntry states it: density is necessary, b == 0 is the rest. MEASURED, this branch vs main, same box, bench --quick, median of 3 sittings (iteration instrument, not certification): bench_mixed write 2.10 -> 2.82 M msg/s (+34%) bench_mixed round_trip 0.97 -> 1.08 M msg/s (+11%) Wire bytes unmoved: full `make test` green across nine backends, the degenerate corpus included, and the bench's own golden + per-variant round-trip gates pass at corpus_id 6b213fbfa1a03a99. Checks semantics are untouched — the same inputs are refused with the same errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C# emitter: the flat word codec, write half The third port of the template Rust (#183) and Go (#198) already carry: field values into locals, OR'd into 64-bit chunk words at literal shifts, one packer step per chunk instead of one per field. WHY IT PAYS HERE IS NOT WHY IT PAID IN GO. #198's finding was that Go's per-field entry points sit past the compiler's inlining budget, so every field cost a real call and ~86% of the codec's time was inside them. C# has no such wall: the JitDisasm audit found ZERO direct calls into serialize.cs from any generated method — every entry point inlines. What C# pays instead is what those inlined bodies DO. BitWriter's packer step masks, shifts, ORs into the scratch word, tests `newScratchBits >= 64`, conditionally stores a qword and recovers the spill: a data-dependent branch and a conditional store PER FIELD. MixedEntity paid fourteen of them for 135 bits. It now pays three. THREE INVARIANTS, RE-DERIVED RATHER THAN INHERITED — #198's review record is why they are spelled out in flat.go's header rather than assumed: 1. 64-bit chunks are wire-identical because serialize.cs's SerializeBits64 splits low-dword-first-then-remainder, which IS the 32-bit chunk order. 2. Every piece is masked to its width, because BitWriter.WriteBitsUnchecked TRUNCATES a too-wide value rather than refusing it — the flat form must truncate identically or a wide value corrupts its neighbours in the chunk instead of losing its own high bits. This is the invariant the Go port lost by inheriting Rust's fallback; here it is enforced at the one place a piece's value expression is built. 3. Write refusals come first, in declaration order — same values refused, same non-latching false. What moves is how many bits reached the stream before the refusal, and a refused write leaves a partial message either way. A run breaks at align, strings/bytes, arrays, branches, nested struct and union calls, and the float, fixed-point, compressed-float and 128-bit families. A run that would not REDUCE the packer-step count is not flattened. A run that does not pay falls back ITEM BY ITEM — the item rides inside every piece for exactly that reason, because the item is the only unit the per-field path can re-emit. THE READ HALF IS NOT IN THIS COMMIT, and the blocker is named rather than worked around: folding a ranged read replaces SerializeInt with a generated comparison, and serialize.cs's ranged read LATCHES ValueOutOfRange where a generated guard returns false without latching. cs publishes checks=always and test/cs pins that latch, so the read fold waits on a public refusal-latch on ReadStream/ReadBatch — a serialize.cs change, its own PR. The write path has no such dependence: its folded range refusals already return false without latching. Measured ceiling for the read half, from a hand-written flat prototype of the two hot cores: round_trip 1.08 -> 1.55 M msg/s, a 42% cut in read time. MEASURED, against the previous commit, same box, median of 3 sittings: bench_mixed write 2.82 -> 3.30 M msg/s (+17%) bench_mixed round_trip 1.08 -> 1.18 M msg/s (+9%) and against main: bench_mixed write 2.10 -> 3.30 M msg/s (+57%) bench_mixed round_trip 0.97 -> 1.18 M msg/s (+22%) flatMaxRunBits is Go's 384 carried unmeasured: swept at 128/256/512 the C# numbers do not move, because this corpus's longest run is 135 bits and no cap in that range binds it. Said out loud in the file rather than presented as a C# measurement. Wire bytes unmoved: full `make test` green across nine backends, the degenerate corpus included, and the bench's golden + per-variant round-trip gates pass at corpus_id 6b213fbfa1a03a99. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C# emitter: the flat word codec, read half The read half of the same form: chunks read first, ONE sticky-error test for the whole run, then every value unpacked, validated and stored with literal shifts and literal masks. Where the per-field form paid a bounds check and an error test per field, MixedEntity now pays one of each per chunk. WHAT IT WILL NOT ABSORB, AND WHY. A ranged read whose refusal can actually fire goes through serialize.cs's SerializeInt/SerializeInt64, which LATCH SerializeError.ValueOutOfRange; a generated comparison returns false without latching. cs publishes checks=always and test/cs pins that latch, so a ranged piece joins a run ONLY where its check is VACUOUS — max - min is exactly 2^bits - 1, so no value the bits can hold is out of range — or where this emitter's own read path ALREADY refuses without latching (the full-range unsigned bits64 path, which has carried a generated refusal all along). Everything else closes the run and keeps the runtime call. Same inputs refused, same errors, on every path. AND THAT RESTRICTION COSTS ALMOST NOTHING — which retires a lever rather than deferring it. The hand-written prototype that folded EVERY ranged read of the two hot cores, latch semantics and all, measured round_trip 1.55 M msg/s. This commit, which leaves four of MixedEntity's fourteen fields on the runtime call because their range checks can fire, measures 1.54. The public refusal-latch on ReadStream/ReadBatch that the full fold would need is worth about 1% here, and is NOT worth a serialize.cs API addition on this evidence. Recorded as ranked-and-refused, not as a follow-on. TWO CONSEQUENCES OF FUSION, named in the file rather than found later: - A stream BOTH truncated inside a run AND carrying an out-of-range value before the truncation now surfaces the stream's overflow error where the per-field form surfaced the range refusal. Both refuse the packet; the set of ACCEPTED streams is unchanged, which is the property that matters at the trust boundary. Rust, Go, Java, Dart and JS-flat already carry it. - A failed run leaves the destination object untouched where the per-field form left the fields before the failure updated. Both are partial states a failed read gives no contract over (SPEC §5). NEGATIVE CONTROL, run rather than assumed: with flatMaskLit forced to emit no mask — invariant 2 deliberately broken — `make test` produces 65 failure lines across the backends. The gate that proves this form does see it. MEASURED, against the previous commit, same box, median of 3 sittings: bench_mixed write 3.30 -> 3.30 M msg/s (unchanged, as expected) bench_mixed round_trip 1.18 -> 1.54 M msg/s (+30%) and against main: bench_mixed write 2.10 -> 3.30 M msg/s (+57%) bench_mixed round_trip 0.97 -> 1.54 M msg/s (+59%) Wire bytes unmoved: full `make test` green across nine backends, the degenerate corpus included, test/cs and test/cs-ludicrous (which pin the corrupt-wire refusals) among them, and the bench's golden + per-variant round-trip gates pass at corpus_id 6b213fbfa1a03a99. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * bench/run.sh: the schema commit line carries its branch Every runtime line in the preamble already records `<sha> (<branch>)`; the schema commit line recorded a bare sha. Two failures that closes, both of which this repo has already had: a sweep that silently measured the wrong tree and a CSV that could not have said so, and a published result whose sha stopped resolving the moment the branch was rebased. The branch name survives the rebase. One line in run.sh (it reuses commit_of, which the runtime lines already use) and its law-home in BENCH-STANDARD.md §3.5, beside the provenance clause the same class of defect earned in August. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * bench/results: the cs-word-codec before/after pair, both halves, two sittings Both halves committed, --only cs, taken back to back in the same window, twice. Preambles carry the schema commit AND its branch on the after half; the before half runs main's own run.sh, which predates that line, so the file says so in its own note. bench_mixed, cs, family gen, checks=always (M msg/s): path before max after max before median after median write 2,104,563 3,342,814 2,103,454 3,334,630 1.59x round_trip 992,155 1,594,245 990,865 1,586,844 1.60x Against the quiet-box reference sweep of the same day and the same corpus_id (bench/results/2026-09-01-arm64-macbook.csv, cpp round_trip median 3,467,551; cpp write median 7,287,687), on the round_trip statistic the published ledger reads at: cs before 350% of generated C++ (the published ledger row: 361%) cs after 219% and on write, 347% -> 219%. The `bits` family rows in this pair are NOT attributable to the change and the files say so where a reader will meet them: BitsBench.cs exercises serialize.cs's raw BitWriter/BitReader and no generated code at all, yet its rows still move 11% across the halves at sub-3% spread. That can only be process-level — JIT method placement and cache state after a differently shaped bench_mixed pass in the same process. An open question, not a result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * flat.go: the bare-integer width test stops pretending to be a switch staticcheck QF1002 on CI: a two-arm `switch { case w == 64: default: }` is an if. Generated output is byte-identical — the regenerated tree is unchanged and `make test` stays green across nine backends. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* cs: the 1:1 piece↔item gate lands before lever F — closes #212 The flat codec re-derives #198-safety from every classifier returning one piece per item or false; nothing asserted it and the package had no tests. TestFlatFallbackReEmitsItemsNotPieces now pins the output properties the invariant protects: a fixed array emits exactly one element loop per direction, a single-scalar fallback (the one fallback path that runs today) emits exactly one per-field serialize per direction, and a nested struct at the run cap never names inner fields on the outer base. Sabotage control: doubling the fallback's item emission goes red (3 serializes detected); restored, green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * modernize: min() in the straddle builder --------- Co-authored-by: Rowan Claude <rowan@mas-bandwidth.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…3% of generated C++ (#237) * js flat: single-word 32-bit write merge replaces the two-lane staging pair Profile (node --prof, bench_mixed --quick): 51% of ticks inside writeBenchMixedFlatProduction itself, 27% in V8 runtime calls it makes. The per-field merge carried two data-dependent branches (the s<32 lane split and the 64-bit flush) plus an s>0 guard; the single 32-bit staging word does one shift-or, one literal add, and one flush branch storing one word. Micro (interleaved, node 26, M2): 1.44x on a mixed-width group of ten. On the leg (quick, production): write 1.35 -> 1.49 M msg/s (+10%), round_trip 0.77 -> 0.82 (+6.5%). Wire bytes unchanged: an LSB-first packer into consecutive little-endian 32-bit words emits the identical byte stream; golden + 64-variant re-encode gates and test/js (both modes, degenerate corpus included) green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * js flat: write-side BigInt lane discipline — the DataView scratch consumes, nothing allocates The profile's 27% V8-runtime bucket was BigInt: every BigInt '&', '>>', asUintN-that-truncates and Number() on the write path allocates a BigInt and calls the runtime, where a DataView setBigUint64 consumes the value into plain memory without allocating. Micro (node 26, M2): 128-bit split 15.0 -> 84.3 M ops/s via two scratch stores + one shift (5.6x); sub-32 truncation 116.9 -> 718.9 M ops/s via scratch wrap (6.1x); the existing <=64 scratch form measured already optimal (674 M ops/s) and keeps its shape minus the redundant asUintN(64) wrap (ToBigUint64 IS that wrap; for the >64 half, shift-then-wrap equals the asUintN(128) high half). Reworked: emitWriteWideOffset takes the raw BigInt offset and routes all four width classes through the scratch; flags <=32 wire mask through the scratch instead of Number(asUintN); the >32-storage int32-range truncation becomes SC.getInt32 (ToBigUint64 wrap + signed lane == asIntN(32) for every BigInt). wideOffsetWidth deleted — nothing reduces ahead of the store anymore. Checked-mode guards unchanged. Leg (quick, production): write 1.49 -> 2.22 M msg/s (+49%), round_trip 0.82 -> 1.01 (+23%). Golden + 64-variant gates and test/js{,-ludicrous} green in both modes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * js flat: wide (>64-bit) read assembly through the scratch — two halves, one shift-or The read twin of the write-side lane discipline: the 96/128-bit assembly built its BigInt from four BigInt(v) constructions chained with three shift-ors (seven allocating steps); it now assembles each 64-bit half in the number domain through SC.setUint32 pairs and joins two getBigUint64 loads with a single shift-or. Micro: 12.5 -> 40.3 M ops/s on the 128-bit assembly (3.2x, node 26, M2). Leg (quick, production): round_trip 1.01 -> 1.06 M msg/s, derived read 1.86 -> 2.07; write untouched within spread. Gates green both modes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * js flat: chunked write merges — adjacent constant-width fields pack into one staged word The family's grouping lever (go #198's runs, cs #208's word codec) in the form this backend can hold: adjacent <=32-bit fields whose value expressions are pure (field loads, hoisted element refs, literals — never a mutable temp) pack into one 'v' with literal RELATIVE shifts, then merge once. Relative offsets inside a chunk are static even where the absolute cursor is dynamic, so the lever reaches the loop bodies carrying most of the corpus wire (the 80-element stat array drops from two merges per element to one; the entity body from ~14 to ~6). Chunks never cross a scope boundary or a statement-form merge — wire order is emission order, enforced by chunkFlush at every loop, branch, switch, align, and scratch-based field; a temp-based piece (the >32-storage int32-range 'n') flushes around itself so no piece outlives its inputs. Leg (quick, production): write 2.22 -> 2.48 M msg/s (+12%), round_trip 1.06 -> 1.11 (+5%). Wire identical: golden + 64-variant gates, test/js{,-ludicrous} both modes, degenerate corpus — green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * js flat: string/bytes bodies merge four bytes per staged word Measured first as a hand-patched ceiling on the generated leg (+3.6% write), then landed in the emitter: the byte-copy loop for string(N)/ bytes(N) takes 4 bytes per 32-bit merge with a byte-at-a-time tail. The counter lives in a brace scope so two byte fields can share a nesting depth. Leg (quick, production): write 2.48 -> 2.55 M msg/s, round_trip 1.11 -> 1.12. Gates green both modes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * js flat: re-pin the SOURCE goldens for the restructured emission Source goldens only — testdata/wire is untouched by this round (the stop-the-line invariant), and the full nine-language make test is green against the unchanged wire pins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * bench/results: the js writes round's certification pair (family precedent: the pair rides with the round) before 1dba683 (main) / after 3b3e6f6 (the round), one sitting 39s apart, spreads 0.22-0.82%. The before leg reproduces the standing ledger row within 0.3% — the pair's validity check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Rowan Claude <rowan@mas-bandwidth.com>
…k-bytes path; 363% -> 221% of generated C++ (#238) * dart: chunked write merges — 64-bit lanes, single-merge wide fields Adjacent constant-width pure-expression fields pack into one staged word with literal relative shifts (the family lever, #198/#208/#237), capped at Dart's native 64-bit lane where the js number domain held it to 32. Wide 33..64-bit fields merge once instead of twice — their wire bits are contiguous, so the bytes are identical by the concatenation argument. Leg (quick, arm64 M2): write 1.894 -> 2.380 M msg/s (1.26x), round_trip 0.974 -> 1.088 (1.12x). Golden gate + 64-variant re-encode green; dart conformance suites green with asserts on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * dart: bulk-bytes write path — the #165 lead String/bytes bodies and bare [N]uint8 arrays stop paying one merge per byte: static runs of 8 or fewer bytes queue as chunk pieces (joining neighboring fields in one merge), longer and dynamic runs take 4 bytes per merge with a byte tail. No alignment requirement: LSB-first merging of the concatenation equals merging the bytes in sequence at any bit position, so the fix reaches the unaligned loadout too. Micro (AOT, M2): 4-byte grouping 2.33x, per-byte baseline 1.00x. Leg (quick): write 2.380 -> 2.499 M msg/s (+5.0%), round_trip 1.088 -> 1.104 (+1.4%) — bulk is only 3.65% of bench_mixed's wire by construction; the lever is for the real packets the issue names. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * dart: chunked read windows — one load and one tail branch per run, not per field The reader kept reloading its 64-bit window at every field. A load now folds the byte-interior shift in and guarantees 57 valid bits (in the tail arm, every bounds-proven remaining bit), so consecutive constant-width fields extract at literal relative shifts from one window through a compile-time ledger. Wide 33..57-bit fields extract in one masked read — their groups are contiguous on the wire. Short bare byte arrays unroll into the same windows (#165's read twin). The ledger invalidates at every dynamic bitsRead move and scope boundary. Micro (AOT, M2): chunked window 1.35x over per-field reload. Leg (quick): round_trip 1.104 -> 1.367 M msg/s (+23.8%), write flat; derived read 2.01 -> 3.03 M msg/s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * dart: grouped counted loops — k elements share one lane per iteration A counted array of static-width elements unrolls k = lane/elemBits elements per iteration (64-bit chunk lane on the write side, 57-bit window on the read side), with a remainder loop. The chunk accumulator and window ledger span the unrolled elements, so short elements merge together and extract from one load — bench_mixed's 80-element stat array drops from one merge and one window load per element to one per three. Hoisted element refs get unique names (e0, e0g1, e0g2) so the unroll shares one scope; correctness rides the same ledger/flush machinery as straight-line code. Bounded first by a hand-patched generated leg (+10% round_trip on the read side alone). Leg (quick): write 2.493 -> 2.856 M msg/s (+14.6%), round_trip 1.367 -> 1.592 (+16.5%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * dart: re-pin source goldens for the chunked emission Wire goldens untouched — testdata/wire byte pins hold in every leg; the re-pin is the SOURCE form only (SPEC §3.1's deliberate-change path). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * dart: range-over-int in the emitter's unroll loops (modernize) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * bench/results: the dart round's certification pair before 1,900,191/972,818 (reproducing the standing ledger row within 0.6%), after 2,857,586/1,591,336; spreads 0.07-2.28%; one sitting, corpus 6b213fbfa1a03a99. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Rowan Claude <rowan@mas-bandwidth.com>
Laggard 1 of the re-ranked queue (#170): generated Go, the biggest prize on the trusted table.
Go moves from 712% to 248% of generated C/C++. Measured as a paired same-sitting A/B — main's Go emitter output against this branch's, nothing else changed between the halves — that is 2.35x on run.sh's own statistic, with 1.98x write and 2.36x round_trip on the leg's own rates and a 0.9940x negative control over 16 unchanged rows. Both CSVs are committed. Wire bytes never move;
checks=alwaysis preserved;bench/LOCKcarries no active prefixes today (the #189 carve window) and nothing under its suspended prefixes moved except a protocol id.What changed since the block
The adversarial review found three CONFIRMED correctness regressions with one root cause. All three are fixed, all three now have corpus coverage in nine languages, and every number in this body is re-derived against the trusted baseline the review named.
The root cause, and the fix
PR #183's Rust template maps one item to exactly one piece, so its fallback — re-emit each of a run's pieces — IS re-emitting each of its items, safe by construction. This emitter's levers 3–6 made
flatPiecesOfreturn 1..N pieces per item, with pieces carrying items that belong to a NESTED struct at a different base expression, and the fallback was inherited verbatim. The count-based "flatten only where it reduces calls" policy is correctly implemented and is exactly what routes execution into that fallback.type Pair { values [2]float64 }emitted the element loop TWICE on write and read — 32 bytes where Rust and main emit 16. Go-to-Go round trips passed, so nothing went red.value.Con the OUTER type. Under a name collision it would have serialized the WRONG field silently.type Vec2 { x float64; y float64 }carried an unusedmathimport —g.needsMathwas set during SPECULATIVE classification and never cleared when the run fell back.A run now accumulates whole ITEMS as
flatGroups and splits only on item boundaries, so a fallback re-emits each item against the base that item owns. AndflatPieceno longer carries anir.Itemat all: a piece is a bit-placement recipe and nothing else, which makes re-emitting one structurally impossible rather than merely avoided. That is the choice between the two repairs the review offered — restoring the 1:1 invariant would have given up levers 4–6, so the invariant is restored one level up, where the fallback actually operates, and the 1:N classification inside a run is kept.needsMathmoves into the float pieces'emit/readclosures, which run at emission.Generated output for the existing corpus is byte-identical before and after this fix — none of the three shapes exists in it. That is the finding, not the reassurance, and it is issue #203. It also means the measured win is untouched by the correctness repair:
generated/bench/go/{Bench.go,realworld/RealWorld.go}are byte-for-byte what the blocked branch emitted, so the numbers below measure the same codec the review reproduced at 2.35x blended.The corpus blind spot (#203) — mandatory, and it caught a second defect
examples/Degenerate.schemacarries twelve shapes the corpus's realism masked: a fixed scalar array as essentially the whole message ([2]float64,[2]uint64,[2]int64,[1]T,[N]Tlanding exactly on a chunk boundary, an array plus a trailing field, two arrays back to back); a bare two-float unit with no other math consumer in the file; and a nested struct as the ONLY field, as the FIRST field, and straddling the run cap.They are pinned as one C++-authored wire golden,
testdata/wire/degenerate.bin(209 bytes), and all nine legs are held to it on every push: the stream legs (C++, C, Go, Rust, C#, JS) write the twelve into one stream; the whole-message legs (Dart, Java, Elixir) concatenate their twelve buffers. That equality is why every type in the file is a whole number of bytes wide — the file's header states it, because it is load bearing.internal/codegen/golangstates the same three properties directly againstmaxRunBits, so re-tuning that constant cannot quietly retire the coverage.RED FIRST. On the pre-fix emitter (997c553), against the new corpus file:
Defect 1 does not fail a compile — that is the whole problem — so it is measured directly. A
SpanF64-only unit, written through the generated Go writer:The emitter test goes red on all four assertions at 997c553 and green after:
The new file caught a SECOND emitter defect on sight, the same class as the Go one. The Dart backend declared all four conversion-scratch views whenever any was needed, so a file converting only float64 carried an unreferenced
_u32anddart analyzerefused it (exit 2). No corpus file had ever been float64-only. The views are now emitted per conversion. That is the corpus paying for itself before it was even committed.Existing wire goldens are unmoved —
git status testdata/wireshows only the newdegenerate.bin. The corpus protocol id moves, as adding a file must (0x0bde7acdd36abc6a→0xa89dd9e036603208), and is re-pinned; every other generated-source diff in this PR is that id and one JS tier banner relocating to the unit's new alphabetically-first flat module.Bench
corpus_idis unchanged at6b213fbfa1a03a99in every row of both new CSVs: the runner hashes the goldens it loads, and it does not load the conformance corpus.The measurement, receipted
bench/run.sh --quick, one sitting, back to back, M2 MacBook Air, generated subject, run.sh's own printed statistic (family gen,bench_mixed, round_trip over max rates). Nothing changed between the halves butgenerated/bench/go. Taken on the tree as it stands, after #204 rewrote the runners.bench/results/2026-09-01-gocodec-before-arm64-macbook.csvbench/results/2026-09-01-gocodec-after-arm64-macbook.csvThe negative control, in this sitting, from the committed files. Every other leg is unchanged code compiled identically in both halves:
Geomean 0.9940x over 16 control rows, min 0.9426, max 1.0339 — no row over §2.3's 40% INVALID threshold, so all sixteen count. The generated Go rows moved because the generated Go changed; nothing else did.
The box was shared throughout (other Claude sessions live on a fanless M2 Air); the noise line in both preambles says so, and
--quickis the iteration instrument, never certification (§2.8). The one row to judge against the noise rather than the digit is go's after-half write, at 7.36% spread — its max rate gives 1.98x and its median 1.88x. The round_trip rows, which are the published statistic, carry 0.36% and 1.89% and agree on both statistics.Replication, named but not offered as a receipt: the same paired A/B run before this branch was rebased onto #204 gave 602% → 251% (2.40x) with a 1.0050x control over 14 rows. Its CSVs are not committed, because #204 replaced the runner they were taken with, and a receipt should name an instrument that still exists. The committed pair above is the one taken on this tree.
The two-column table, after
Go was the worst laggard but elixir. It is now fifth of nine, below cs, dart and js, and within 1.56x of rust.
The baseline, corrected
The PR said 592%. That number appears nowhere in the repo. The trusted table — #170's re-ranking, from main's committed
bench/results/2026-08-31-arm64-macbook-datadriven-quick.csv— puts generated Go at 712% (max statistic, which is the one run.sh prints) and 711% (median). Against 712%, this branch's 248% is a 2.87x improvement: the direction is favourable and the true win is larger than the PR claimed.The two baselines do not agree, and the disagreement belongs in the open:
The denominators agree to 2.4%; the go leg alone ran 19% slower in the trusted sitting, whose own noise line reads "NOISY: shared laptop, one node process at 100% and Safari live". So the causal claim is the paired one — 2.35x, one sitting, controls at 0.9940x — and 712% → 248% is the same movement read off the published statistic across sittings, which is what the re-ranked queue tracks. The PR's earlier "580% and 592% agree to 2%" reconciliation is withdrawn; it reconciled a number that does not exist against a sitting whose CSV was not committed.
The spread the earlier CSV carried
Recording what the review recorded. In
bench/results/2026-08-31-gocodec-quick-arm64-macbook.csv(committed on this branch earlier), the go round_trip row has 10.44% spread against 0.40–1.73% for every other row in that file. That sits just under run.sh's 15% NOISY threshold, so no note printed. On the median statistic that CSV's headline is 275%, not 255% — a 20-point gap between the two statistics on one row.The re-run resolves it rather than arguing about it: in the new sitting the go round_trip rows carry 0.36% (before) and 1.89% (after) spread, and the two statistics agree to within 2% — 248% on max, 243% on median. The earlier CSV stays in the tree as the record of the sitting it came from; the paired files are the evidence.
Claims withdrawn as UNRECEIPTED
bench_intsread 2.57x andshipcreateread 2.05x, no cell below 0.99x" — the per-shape leg. No CSV was committed, and One bench, one shape: retire family rt and every hand-coded shape in the Bench-corpus leg (#196) #199 retired those shapes, so it cannot be re-run on main. Withdrawn. The paired bench_mixed files above are what stands.bench/gois untouched" — false, and dropped. See below.bench/goIS touchedA
--cpuprofileflag and aruntime/pprofimport were added tobench/go/main.go. The flag stays, and this PR now states it in the runner's usage line and makes it refuse instead of panicking. It is how this round's conviction was obtained —go tool pprofon the baseline put ~86% of generated-codec time inside the runtime's per-field bit calls against ~2% in the generated code itself — and the next round will want it. It is an iteration instrument: it changes nothing about what is measured, and no timed row in this PR was taken under it.Where the time went, measured first
-gcflags=-m -mgave the mechanism (go1.27, inline budget 80):SerializeBitsis inlinable, but the bodies underneath it are not — so every field cost one real call into the runtime, on both paths. Issue fewer calls, and the rest follows.The levers (development record — see the withdrawal note above)
BitsRequiredper field and drops off the non-inlinable ranged entry pointsWriteMixedEntity: 14 calls → 3Two results worth naming because they went against prediction:
SerializeBits64is itself non-inlinable at cost 278, so one bigger call replaced two small ones). Re-measured after the later levers, 64 wins clearly on reads — the balance shifted once call count fell. Kept at 64, on the second measurement, not the first.rigidbody_*write went 0.91x:RigidBodyis all float64, so every piece is exactly one whole chunk — flattening removed no call while adding a local per field, where the per-field form passed the struct field's own address. The policy became flatten only where it reduces the call count. That policy is correct and is kept — and it is also precisely what routes execution into the fallback the review found broken, which is why the fallback had to be made safe rather than the policy relaxed.Two deliberate deviations from the Rust template (#183)
debug_asserts, so rust: the self-contained flat word codec — 594% to 444% of generated C/C++ #183 had to refuse. Masking is what keeps the Go form observably identical to what it replaces.And one coverage win Rust could not have: compressed floats do not break a run here. This emitter already folds quantization into generated arithmetic ending in a plain bit write, so quantized fields are ordinary pieces — where Rust's live in the runtime, which is why #183's compressed-float shapes barely moved.
The floor, with its mechanism measured
The remaining overhead is Go's fixed inline budget at the runtime boundary. Building the leg with Go PGO (a labeled diagnostic, not shipped) reached 3.28 M/s write and 1.72 round_trip, a further 1.18x/1.15x;
-gcflags=-mconfirms why: under PGO,writeBits,WriteStream.SerializeBits64andReadStream.SerializeBits64all become inlinable; without it, none are. (Development-record number, same status as the lever table.)Two independent lifts, both outside this PR's remit:
writeBits(cost 92) andreadBits(106) under the 80-unit budget in serialize.go. They miss by 12 and 26. A routing question, not a workaround: The generated-code optimization round: forensics-corrected baselines, then per-emitter profiling passes #170 routes this round to schema-level changes only. The runtime's own comments already claim these bodies stay within the budget; at go1.27 that claim is false.B3 —
unsafepokes at the stream's private scratch from generated code was enumerated and rejected on sight: it reimplements the runtime inside generated output against unexported fields.Wire and semantics
testdata/wireclean apart from the newdegenerate.bin); benchcorpus_id 6b213fbfa1a03a99unchanged in both new CSVs.bin/schemabuild over a cleanedgenerated/is zero-diff against what is committed.go run ./bench/tools/shapegatereports clean, 19 files on the ledger, no new debt.make testis green in full on this box, including the java, dart and elixir conformance legs — the pinneddist/toolchains were available here, so this PR is not relying on CI for them:schema_test,_random,_ludicrous, the C and C-ludicrous legs,schema_test_bench,test/{go,rust,cs,js,dart,java,elixir}and their ludicrous twins,dart analyze+dart format --set-exit-if-changed,mix format --check-formatted, the goldens, the fuzz leg, andgo test ./....make checkpasses.cpp-lockis inactive (bench/LOCK carries no active prefixes during the C/C++ emitter defect: string helpers' include guard is TU-wide, definition namespace-local — two generated units in one TU break #189 carve).-gcflags=-mshows no chunk local escaping, and the after-half run's own alloc note readsbench_mixed one pass (256 ops/path): write 0 allocs, round_trip 0 allocs/write 0 bytes, round_trip 0 bytes.checks=alwayspreserved: the same values are refused, with the same error.Two behaviours move, both named rather than smoothed, and both the same class the Java/Dart/JS-flat/Rust fusion already carries:
One further delta, from folding the ranged read: the refusal is returned rather than latched on the stream. The write folds already shipped with exactly this property, and generated callers return immediately on error.
Named follow-ons, not held for
SerializeFixed64) still breaks every run it sits in — foldable in principle, the same shape as the compressed-float fold.bench_mixedis the one shape, by ruling.Degenerate.schemanow guards all of them on the wire; the structural guarantee — a piece that cannot name an item — is Go's alone so far.🤖 Generated with Claude Code