Skip to content

go: the flat word codec — 712% to 248% of generated C/C++ - #198

Merged
gafferongames merged 15 commits into
mainfrom
go-word-codec
Aug 31, 2026
Merged

go: the flat word codec — 712% to 248% of generated C/C++#198
gafferongames merged 15 commits into
mainfrom
go-word-codec

Conversation

@gafferongames

@gafferongames gafferongames commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Laggard 1 of the re-ranked queue (#170): generated Go, the biggest prize on the trusted table.

Go moves from 712% to 248% of generated C/C++. Measured as a paired same-sitting A/B — main's Go emitter output against this branch's, nothing else changed between the halves — that is 2.35x on run.sh's own statistic, with 1.98x write and 2.36x round_trip on the leg's own rates and a 0.9940x negative control over 16 unchanged rows. Both CSVs are committed. Wire bytes never move; checks=always is preserved; bench/LOCK carries no active prefixes today (the #189 carve window) and nothing under its suspended prefixes moved except a protocol id.

What changed since the block

The adversarial review found three CONFIRMED correctness regressions with one root cause. All three are fixed, all three now have corpus coverage in nine languages, and every number in this body is re-derived against the trusted baseline the review named.

The root cause, and the fix

PR #183's Rust template maps one item to exactly one piece, so its fallback — re-emit each of a run's pieces — IS re-emitting each of its items, safe by construction. This emitter's levers 3–6 made flatPiecesOf return 1..N pieces per item, with pieces carrying items that belong to a NESTED struct at a different base expression, and the fallback was inherited verbatim. The count-based "flatten only where it reduces calls" policy is correctly implemented and is exactly what routes execution into that fallback.

# defect what it did
1 silent wire corruption type Pair { values [2]float64 } emitted the element loop TWICE on write and read — 32 bytes where Rust and main emit 16. Go-to-Go round trips passed, so nothing went red.
2 generated Go does not compile a nested struct split by the run cap emitted value.C on the OUTER type. Under a name collision it would have serialized the WRONG field silently.
3 generated Go does not compile a bare type Vec2 { x float64; y float64 } carried an unused math import — g.needsMath was set during SPECULATIVE classification and never cleared when the run fell back.

A run now accumulates whole ITEMS as flatGroups and splits only on item boundaries, so a fallback re-emits each item against the base that item owns. And flatPiece no longer carries an ir.Item at all: a piece is a bit-placement recipe and nothing else, which makes re-emitting one structurally impossible rather than merely avoided. That is the choice between the two repairs the review offered — restoring the 1:1 invariant would have given up levers 4–6, so the invariant is restored one level up, where the fallback actually operates, and the 1:N classification inside a run is kept. needsMath moves into the float pieces' emit/read closures, which run at emission.

Generated output for the existing corpus is byte-identical before and after this fix — none of the three shapes exists in it. That is the finding, not the reassurance, and it is issue #203. It also means the measured win is untouched by the correctness repair: generated/bench/go/{Bench.go,realworld/RealWorld.go} are byte-for-byte what the blocked branch emitted, so the numbers below measure the same codec the review reproduced at 2.35x blended.

The corpus blind spot (#203) — mandatory, and it caught a second defect

examples/Degenerate.schema carries twelve shapes the corpus's realism masked: a fixed scalar array as essentially the whole message ([2]float64, [2]uint64, [2]int64, [1]T, [N]T landing exactly on a chunk boundary, an array plus a trailing field, two arrays back to back); a bare two-float unit with no other math consumer in the file; and a nested struct as the ONLY field, as the FIRST field, and straddling the run cap.

They are pinned as one C++-authored wire golden, testdata/wire/degenerate.bin (209 bytes), and all nine legs are held to it on every push: the stream legs (C++, C, Go, Rust, C#, JS) write the twelve into one stream; the whole-message legs (Dart, Java, Elixir) concatenate their twelve buffers. That equality is why every type in the file is a whole number of bytes wide — the file's header states it, because it is load bearing. internal/codegen/golang states the same three properties directly against maxRunBits, so re-tuning that constant cannot quietly retire the coverage.

RED FIRST. On the pre-fix emitter (997c553), against the new corpus file:

$ go build -o /tmp/schema-prefix ./cmd/schema      # at 997c553
$ /tmp/schema-prefix generate --lang go --out /tmp/dg-red examples
$ cd /tmp/dg-red && go build ./...
# example
./Degenerate.go:10:2: "math" imported and not used
./Degenerate.go:447:30: value.C undefined (type *TrioStraddle has no field or method C)
./Degenerate.go:485:30: value.C undefined (type *TrioStraddle has no field or method C)

Defect 1 does not fail a compile — that is the whole problem — so it is measured directly. A SpanF64-only unit, written through the generated Go writer:

before (997c553, unused import removed by hand so it links):
    SpanF64 bits=256 bytes=32 (SpanF64MaxBits=128)
after:
    SpanF64 bits=128 bytes=16 (SpanF64MaxBits=128)

The emitter test goes red on all four assertions at 997c553 and green after:

$ go test ./internal/codegen/golang/ -run TestFlatFallback     # emitter at 997c553
--- FAIL: TestFlatFallbackReEmitsItemsNotPieces
    [2]float64 emitted 4 element loops, want 2 (one write, one read) — a fallback over
      pieces emits the loop once per element and doubles the wire
    a nested struct split by the 384-bit run cap named "value.C" on the OUTER type in
      func WriteOuter( — ... the fallback must re-emit the ITEM against its own base
    a nested struct split by the 384-bit run cap named "value.C" on the OUTER type in
      func ReadOuter( — ...
    a bare two-float unit imported math 1 times — nothing in it reaches math, so the
      import does not compile
FAIL

$ go test ./internal/codegen/golang/ -run TestFlatFallback     # this branch
ok      github.com/mas-bandwidth/schema/v2/internal/codegen/golang      0.173s

The new file caught a SECOND emitter defect on sight, the same class as the Go one. The Dart backend declared all four conversion-scratch views whenever any was needed, so a file converting only float64 carried an unreferenced _u32 and dart analyze refused it (exit 2). No corpus file had ever been float64-only. The views are now emitted per conversion. That is the corpus paying for itself before it was even committed.

Existing wire goldens are unmovedgit status testdata/wire shows only the new degenerate.bin. The corpus protocol id moves, as adding a file must (0x0bde7acdd36abc6a0xa89dd9e036603208), and is re-pinned; every other generated-source diff in this PR is that id and one JS tier banner relocating to the unit's new alphabetically-first flat module.

Bench corpus_id is unchanged at 6b213fbfa1a03a99 in every row of both new CSVs: the runner hashes the goldens it loads, and it does not load the conformance corpus.

The measurement, receipted

bench/run.sh --quick, one sitting, back to back, M2 MacBook Air, generated subject, run.sh's own printed statistic (family gen, bench_mixed, round_trip over max rates). Nothing changed between the halves but generated/bench/go. Taken on the tree as it stands, after #204 rewrote the runners.

  • bench/results/2026-09-01-gocodec-before-arm64-macbook.csv
  • bench/results/2026-09-01-gocodec-after-arm64-macbook.csv
before after
write 1.35 M msg/s 2.67 1.98x
round_trip 0.60 M msg/s 1.41 2.36x
headline (max) 583% 248% 2.35x
headline (median) 582% 243% 2.39x

The negative control, in this sitting, from the committed files. Every other leg is unchanged code compiled identically in both halves:

lang write round_trip
cpp 1.0087x 1.0024x
c 1.0103x 1.0017x
rust 0.9700x 0.9764x
cs 0.9795x 1.0022x
js 0.9979x 1.0339x
java 1.0205x 0.9868x
dart 0.9888x 0.9426x
elixir 0.9963x 0.9893x

Geomean 0.9940x over 16 control rows, min 0.9426, max 1.0339 — no row over §2.3's 40% INVALID threshold, so all sixteen count. The generated Go rows moved because the generated Go changed; nothing else did.

The box was shared throughout (other Claude sessions live on a fanless M2 Air); the noise line in both preambles says so, and --quick is the iteration instrument, never certification (§2.8). The one row to judge against the noise rather than the digit is go's after-half write, at 7.36% spread — its max rate gives 1.98x and its median 1.88x. The round_trip rows, which are the published statistic, carry 0.36% and 1.89% and agree on both statistics.

Replication, named but not offered as a receipt: the same paired A/B run before this branch was rebased onto #204 gave 602% → 251% (2.40x) with a 1.0050x control over 14 rows. Its CSVs are not committed, because #204 replaced the runner they were taken with, and a receipt should name an instrument that still exists. The committed pair above is the one taken on this tree.

The two-column table, after

language % of cpp@100
c 98%
cpp 100%
java 149%
rust 159%
go 248%
cs 365%
dart 400%
js 475%
elixir 3800%

Go was the worst laggard but elixir. It is now fifth of nine, below cs, dart and js, and within 1.56x of rust.

The baseline, corrected

The PR said 592%. That number appears nowhere in the repo. The trusted table — #170's re-ranking, from main's committed bench/results/2026-08-31-arm64-macbook-datadriven-quick.csv — puts generated Go at 712% (max statistic, which is the one run.sh prints) and 711% (median). Against 712%, this branch's 248% is a 2.87x improvement: the direction is favourable and the true win is larger than the PR claimed.

The two baselines do not agree, and the disagreement belongs in the open:

trusted table (#170) this sitting's before half
go round_trip, max 0.503 M msg/s 0.601 M msg/s
cpp round_trip, max 3.585 M msg/s 3.501 M msg/s
headline 712% 583%

The denominators agree to 2.4%; the go leg alone ran 19% slower in the trusted sitting, whose own noise line reads "NOISY: shared laptop, one node process at 100% and Safari live". So the causal claim is the paired one — 2.35x, one sitting, controls at 0.9940x — and 712% → 248% is the same movement read off the published statistic across sittings, which is what the re-ranked queue tracks. The PR's earlier "580% and 592% agree to 2%" reconciliation is withdrawn; it reconciled a number that does not exist against a sitting whose CSV was not committed.

The spread the earlier CSV carried

Recording what the review recorded. In bench/results/2026-08-31-gocodec-quick-arm64-macbook.csv (committed on this branch earlier), the go round_trip row has 10.44% spread against 0.40–1.73% for every other row in that file. That sits just under run.sh's 15% NOISY threshold, so no note printed. On the median statistic that CSV's headline is 275%, not 255% — a 20-point gap between the two statistics on one row.

The re-run resolves it rather than arguing about it: in the new sitting the go round_trip rows carry 0.36% (before) and 1.89% (after) spread, and the two statistics agree to within 2% — 248% on max, 243% on median. The earlier CSV stays in the tree as the record of the sitting it came from; the paired files are the evidence.

Claims withdrawn as UNRECEIPTED

  • "geomean 1.40x over 30 cells, best rows bench_ints read 2.57x and shipcreate read 2.05x, no cell below 0.99x" — the per-shape leg. No CSV was committed, and One bench, one shape: retire family rt and every hand-coded shape in the Bench-corpus leg (#196) #199 retired those shapes, so it cannot be re-run on main. Withdrawn. The paired bench_mixed files above are what stands.
  • "the control rows moved 0.9977x over 10 rows (min 0.987, max 1.007)" — same leg, same reason. Withdrawn and replaced by the 1.0050x-over-14-rows control measured in this sitting, from committed CSVs.
  • The per-lever table below is retained as the development record, not as receipted measurement: each lever's number was taken on the retired per-shape leg during development. The composed result is what the two committed CSVs measure.
  • "bench/go is untouched"false, and dropped. See below.

bench/go IS touched

A --cpuprofile flag and a runtime/pprof import were added to bench/go/main.go. The flag stays, and this PR now states it in the runner's usage line and makes it refuse instead of panicking. It is how this round's conviction was obtained — go tool pprof on the baseline put ~86% of generated-codec time inside the runtime's per-field bit calls against ~2% in the generated code itself — and the next round will want it. It is an iteration instrument: it changes nothing about what is measured, and no timed row in this PR was taken under it.

Where the time went, measured first

-gcflags=-m -m gave the mechanism (go1.27, inline budget 80):

write_stream.go:99:  cannot inline (*WriteStream).writeBits:      cost  92 exceeds budget 80
read_stream.go:55:   cannot inline (*ReadStream).readBits:        cost 106 exceeds budget 80
read_stream.go:100:  cannot inline (*ReadStream).SerializeInt:    cost 168 exceeds budget 80

SerializeBits is inlinable, but the bodies underneath it are not — so every field cost one real call into the runtime, on both paths. Issue fewer calls, and the rest follows.

The levers (development record — see the withdrawal note above)

# lever write round_trip
1 Ranged-integer read fold — kills a runtime BitsRequired per field and drops off the non-inlinable ranged entry points +1.5% +2.6%
2 The flat word codec — runs gathered, values OR'd into chunk locals at literal shifts, one chunk per call, checks fused per run. WriteMixedEntity: 14 calls → 3 1.50x 1.63x
3 Nested struct bodies placed inline instead of called per element — 88 calls and 88 sticky-error loads per message +9% +10%
4+5 Nested struct fields join the parent run; small fixed arrays unroll into it +2% +4.5%
6 Consecutive array elements share chunks — eighty 18-bit stats carried 18 of a possible 64 bits each; 80 calls → 25 +14% +21%
7 Run cap measured, not inherited — 256→2.66/1.40, 384→2.78/1.49, 512→2.75/1.46, 1024→2.62/1.43 +4% +6%

Two results worth naming because they went against prediction:

  • Chunk width was a tie, then wasn't. I predicted 64-bit chunks would beat Rust's 32 by halving calls. First measurement: a wash (SerializeBits64 is itself non-inlinable at cost 278, so one bigger call replaced two small ones). Re-measured after the later levers, 64 wins clearly on reads — the balance shifted once call count fell. Kept at 64, on the second measurement, not the first.
  • A regression the per-shape leg caught. rigidbody_* write went 0.91x: RigidBody is all float64, so every piece is exactly one whole chunk — flattening removed no call while adding a local per field, where the per-field form passed the struct field's own address. The policy became flatten only where it reduces the call count. That policy is correct and is kept — and it is also precisely what routes execution into the fallback the review found broken, which is why the fallback had to be made safe rather than the policy relaxed.

Two deliberate deviations from the Rust template (#183)

  1. Chunks are 64 bits, not 32 — measured above.
  2. Every piece is masked to its width. serialize.go's write path masks a too-wide value; the Rust runtime only debug_asserts, so rust: the self-contained flat word codec — 594% to 444% of generated C/C++ #183 had to refuse. Masking is what keeps the Go form observably identical to what it replaces.

And one coverage win Rust could not have: compressed floats do not break a run here. This emitter already folds quantization into generated arithmetic ending in a plain bit write, so quantized fields are ordinary pieces — where Rust's live in the runtime, which is why #183's compressed-float shapes barely moved.

The floor, with its mechanism measured

The remaining overhead is Go's fixed inline budget at the runtime boundary. Building the leg with Go PGO (a labeled diagnostic, not shipped) reached 3.28 M/s write and 1.72 round_trip, a further 1.18x/1.15x; -gcflags=-m confirms why: under PGO, writeBits, WriteStream.SerializeBits64 and ReadStream.SerializeBits64 all become inlinable; without it, none are. (Development-record number, same status as the lever table.)

Two independent lifts, both outside this PR's remit:

  • B1 — bring writeBits (cost 92) and readBits (106) under the 80-unit budget in serialize.go. They miss by 12 and 26. A routing question, not a workaround: The generated-code optimization round: forensics-corrected baselines, then per-emitter profiling passes #170 routes this round to schema-level changes only. The runtime's own comments already claim these bodies stay within the budget; at go1.27 that claim is false.
  • B2 — Go PGO for the leg. A build change; the C/C++ reference is locked and builds without PGO, so this is a cross-leg equalization decision for the owner.

B3 — unsafe pokes at the stream's private scratch from generated code was enumerated and rejected on sight: it reimplements the runtime inside generated output against unexported fields.

Wire and semantics

Two behaviours move, both named rather than smoothed, and both the same class the Java/Dart/JS-flat/Rust fusion already carries:

  1. Refusal timing on write. A run's (and a group's) refusals all run before it packs, so a refused write has emitted fewer bits than before. Which values are refused, and with which error, is identical.
  2. Error kind on a doubly-bad read. A stream both truncated inside a run and carrying an out-of-range value before the truncation now surfaces the stream's overflow error where the per-field form surfaced the range refusal. Both refuse the packet; the set of accepted streams is unchanged, which is the property that matters at the trust boundary.

One further delta, from folding the ranged read: the refusal is returned rather than latched on the stream. The write folds already shipped with exactly this property, and generated callers return immediately on error.

Named follow-ons, not held for

  • Fixed-point (SerializeFixed64) still breaks every run it sits in — foldable in principle, the same shape as the compressed-float fold.
  • String/bytes length could join the preceding run; the bytes themselves must stay on the bulk path.
  • The 128-bit family stays on the runtime path.
  • A per-shape leg to replace the one One bench, one shape: retire family rt and every hand-coded shape in the Bench-corpus leg (#196) #199 retired, if the owner wants per-shape evidence in future optimization PRs — right now bench_mixed is the one shape, by ruling.
  • The same fallback audit for the other flat emitters (Rust, JS-flat, Java, Dart): rust: the self-contained flat word codec — 594% to 444% of generated C/C++ #183's 1:1 invariant is intact in Rust, but any backend that later grows a 1:N classification inherits this bug. Degenerate.schema now guards all of them on the wire; the structural guarantee — a piece that cannot name an item — is Go's alone so far.

🤖 Generated with Claude Code

@gafferongames gafferongames changed the title go: the flat word codec and the laggard-1 lever enumeration — 712% of generated C/C++ go: the flat word codec — 592% to 255% of generated C/C++ Aug 31, 2026
@gafferongames

Copy link
Copy Markdown
Contributor Author

Adversarial review verdict: BLOCK — three CONFIRMED correctness regressions, one root cause. The performance claim is real and reproduced independently (2.06x write / 2.50x round_trip / 2.35x blended, i.e. better than the PR's 2.24x), wire identity holds on the current corpus (36 generate invocations re-run: zero diff; test/go byte-compares against the C++-pinned goldens and passes), and the lock and scope are clean. But:

  1. Silent wire corruption. type Pair { values [2]float64 } emits the element loop TWICE on both write and read — 32 bytes where Rust and main emit 16. Go-to-Go round-trips pass, so nothing goes red. Triggers on any fixed scalar array ([2]float64, [2]uint64, [2]int64, and multiples).
  2. Generated Go does not compile when a nested struct is split by the run cap: emits value.A on the outer type. Where a name collision exists this would silently serialize the WRONG field instead of failing.
  3. Generated Go does not compile for a bare two-float unit: needsMath is set during speculative classification and never cleared when the run falls back — unused import. Any Vec2/Vec3/Quat-shaped file with no other math user.

ROOT CAUSE: #183's Rust template keeps a strict 1:1 item-to-piece mapping, so its fallback (re-emit each piece directly) is safe by construction. Levers 3-6 made flatPiecesOf return 1..N pieces per item, with pieces carrying items from nested structs at a different base expression — and the fallback was inherited verbatim. The count-based flattening policy is implemented correctly and is exactly what routes execution into the now-unsafe fallback. Ports never invent contracts; a port must re-derive the invariant it inherits.

TO UNBLOCK: fix the fallback to re-emit whole ITEMS against the correct base (or restore the invariant); move needsMath to emission; ADD CORPUS COVERAGE for all three blind spots (one-line schemas — see #203); correct the baseline (the PR says 592%, the trusted table says 712% — the number 592 appears nowhere in the repo, so the 'movement transfers' reconciliation needs re-deriving; direction is favourable, the true win is larger); drop the 'bench/go untouched' claim (a --cpuprofile flag was added there); and either commit the negative control and per-shape CSVs or restate those claims as unreceipted — the 0.9977x control over 10 rows is the load-bearing number and it can no longer be re-run on main since #199 retired those shapes.

Also recorded: go's round_trip spread is 10.44% in the committed CSV against 0.40-1.73% elsewhere, sitting just under the 15% NOISY threshold so no note printed; on the median statistic the headline is 275%, not 255%.

gafferongames added a commit that referenced this pull request Aug 31, 2026
Two findings from re-proving the gate on a fully provisioned tree.

dist/ is the Makefile's pinned toolchain drop — the Dart SDK, the JDK, OTP,
Elixir. It is gitignored and absent on CI, which uses setup-dart/setup-java, so
the shape-gate job never met it. On a developer machine that followed the
Makefile's own dist/ instructions the gate refused with 42 findings, every one
of them inside a downloaded toolchain: the Dart SDK ships lib/core/stopwatch.dart
and Mix ships profile.fprof.ex. `make shape-gate` was unusable on exactly the
trees that can run the whole bench. dist/ joins the skipDirs list.

The package doc gains the limitation #198 demonstrated: this gate guards
MEASUREMENT, not CORRECTNESS. The Go emitter wrote a fixed scalar array twice —
32 wire bytes where the other eight languages write 16 — and Go-to-Go
round-trips passed clean because both ends shared the defect. A shape-blind
runner driving a defective emitter is still shape-blind and still passes here.
Issue #203 records the corpus blind spot. It also now says plainly that a shape
name in a comment counts as a hit, which is deliberate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gafferongames added a commit to mas-bandwidth/serialize.go that referenced this pull request Aug 31, 2026
… point does not inline (#55)

The doc comment said the flat writer state makes 'the per-field write path
fit the compiler's inlining budget'. Measured at go1.27.0 (budget 80):
tryWriteBits costs 64 and inlines, but WriteStream.writeBits — the entry
point generated code calls once per field — costs 92 and does not, and
ReadStream.readBits is 106. So the sentence claimed the opposite of what
ships, and schema's Go emitter work (mas-bandwidth/schema#198) measured
~86% of generated-codec time sitting in exactly those calls.

Corrected to state what is measured, with the go version beside the numbers
since a cost without its toolchain version rots. Also records that
BitReader.tryReadBits costs exactly 80 against a budget of 80 — one unit
from silently outlining, with nothing to go red when it does.

Comments only; no code, no wire, no behaviour change.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gafferongames added a commit that referenced this pull request Aug 31, 2026
…out (#200)

* The shape gate: a CI refusal for hand-coded shape measurement

Owner concern, 2026-08-31: "if you have any other profiling per-language
that is hand coded ... I don't trust this and it will drift. We have to
lock this in."

bench/tools/shapegate extracts the shape vocabulary from bench/corpus and
refuses four things: a corpus identifier named under bench/, a timing
primitive anywhere outside the sanctioned runner and tool directories, a
bench-shaped source path outside them, and a shape's wire size written
down as a literal.

bench/SHAPE-GATE.allow is the complete register of what does not yet
comply, with exact counts that ratchet: growth fails, and so does leaving
a count too high once the debt is paid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* shapegate: satisfy errcheck and staticcheck ST1005

The rule text moves out of the error value and under it, which is where
a reader wants it anyway.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* shape gate: rebase onto main — the debt is paid, the register says so

#199 deleted family rt and the §1.3 hand-coded shapes, #204 retired the §1.2
example-corpus rows from every runner, #201 deleted the decomp.cpp scratch
harness. The ledger's ratchet caught all of it: nine stale counts, seven
entries matching nothing.

Name hits under bench/ drop from 2080 to 146, and of the 29 remaining in the
nine runner legs NOT ONE is hand-coded shape measurement — every one is a
comment, a generated-symbol callsite, or an English word that is also a corpus
field name. The nine legs are shape-blind. The gate now guards a clean tree
instead of a shrinking debt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* shapegate: skip dist/, and state the two limits the rebase exposed

Two findings from re-proving the gate on a fully provisioned tree.

dist/ is the Makefile's pinned toolchain drop — the Dart SDK, the JDK, OTP,
Elixir. It is gitignored and absent on CI, which uses setup-dart/setup-java, so
the shape-gate job never met it. On a developer machine that followed the
Makefile's own dist/ instructions the gate refused with 42 findings, every one
of them inside a downloaded toolchain: the Dart SDK ships lib/core/stopwatch.dart
and Mix ships profile.fprof.ex. `make shape-gate` was unusable on exactly the
trees that can run the whole bench. dist/ joins the skipDirs list.

The package doc gains the limitation #198 demonstrated: this gate guards
MEASUREMENT, not CORRECTNESS. The Go emitter wrote a fixed scalar array twice —
32 wire bytes where the other eight languages write 16 — and Go-to-Go
round-trips passed clean because both ends shared the defect. A shape-blind
runner driving a defective emitter is still shape-blind and still passes here.
Issue #203 records the corpus blind spot. It also now says plainly that a shape
name in a comment counts as a hit, which is deliberate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci: keep the cpp-lock comment attached to the cpp-lock job

The shape-gate job was appended after cpp-lock's explanatory comment and
before cpp-lock itself, orphaning the comment above the wrong job. Same two
jobs, comments back with their own.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@gafferongames gafferongames changed the title go: the flat word codec — 592% to 255% of generated C/C++ go: the flat word codec — 712% to 251% of generated C/C++ Aug 31, 2026
gafferongames and others added 12 commits September 1, 2026 00:04
…counts

The write side already folded its bounds; reads still called the runtime's
SerializeInt/SerializeInt64, which derive the bit count from min/max on EVERY
call (a BitsRequired per field) and carry the bounds as arguments. Both are
compile-time constants of the call site.

Folded, reads drop onto SerializeBits — whose wrapper the Go compiler inlines,
where the ranged entry points are 168/328 cost units against an 80-unit budget
and never inline. The headroom refusal moves into generated code with the same
error, ErrValueOutOfRange, and is elided where the range fills its bit width.

Wire bytes unmoved; corpus_id 6b213fbfa1a03a99 unchanged.
Measured, M2, bench_mixed, same sitting: write 1.37 -> 1.39 M msg/s (+1.5%),
round_trip 0.61 -> 0.62 (+2.6%).

The small size is the finding: the cost is the CALL COUNT, not the call's
arguments. That is what lever 2, the flat word codec, attacks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Maximal runs of statically-sized pieces are gathered, field values computed
into locals and OR'd into word-sized chunk locals at literal shifts, and one
whole chunk handed to the stream per call. Reads fuse their bounds checks per
chunk and take ONE sticky-error test per run instead of one per field.

WriteMixedEntity goes from 14 stream calls to 3.

Two deliberate deviations from the Rust template (#183), both Go physics:

  1. CHUNKS ARE 64 BITS, not 32. SerializeBits64 places a whole word in one
     call with one bounds check (its two tryWriteBits are inlined into it),
     where two SerializeBits are two calls. Wire-identical: SerializeBits64
     splits low-dword-then-remainder, which IS the 32-bit chunk order.
     Chunks of 32 bits or fewer still go through SerializeBits, which inlines.

  2. EVERY PIECE IS MASKED TO ITS WIDTH. serialize.go's write path MASKS a
     too-wide value; the Rust runtime only debug_asserts, so #183 had to
     REFUSE. Masking is what keeps the Go form observably identical.

And one coverage win the Rust template could not have: COMPRESSED FLOATS DO
NOT BREAK A RUN. This emitter already folds quantization into generated
arithmetic ending in a plain bit write, so quantized fields are ordinary
pieces — where Rust's live in the runtime, which is why its compressed-float
shapes barely moved.

Wire bytes unmoved (testdata/wire clean, corpus_id 6b213fbfa1a03a99).
Only Go outputs regenerated; every other language byte-identical.
Green: schema_test, _random, _ludicrous, C and C-ludicrous, bench and bench_c
(including the 6 bench_mixed refusal vectors), test/go, test/go-ludicrous,
and the fuzz leg.

Measured, M2, bench_mixed, same sitting, on top of lever 1:
  write      1.39 -> 2.08 M msg/s  (1.50x)
  round_trip 0.62 -> 1.01 M msg/s  (1.63x)
Negative control: the hand-written rt rows, same source throughout, held at
1.22/1.20 against a 1.23/1.22 baseline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…element

With the per-field calls gone, the profile's next item was the CALL boundary
itself: bench_mixed carries up to 80 MixedStat and 8 MixedEntity elements, and
each paid a function call plus its own sticky-error load. WriteMixedStat was
12.4% of samples for 18 bits of work, and at cost 134 against Go's fixed
80-unit budget it will never inline itself.

An array whose element is a struct that flattens whole now has its element
body placed directly in the loop. The element's own Write/Read functions are
still emitted — the public surface is unchanged — they are simply no longer
the path the generated caller takes.

Measured, M2, bench_mixed, same sitting, on top of lever 2:
  write      2.08 -> 2.27 M msg/s  (+9%)
  round_trip 1.01 -> 1.11 M msg/s  (+10%)
Control rows held at 1.17-1.20 write / 1.18-1.21 read against the 1.22/1.20
baseline. Wire unmoved, conformance green.

Cumulative over the three levers: write 1.37 -> 2.27 (1.66x),
round_trip 0.61 -> 1.11 (1.82x).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he run

Two coverage extensions on the same mechanism, both cheap once a run can be
built against an arbitrary base expression:

  4. A nested struct field that flattens whole contributes its fields to the
     PARENT's run, so they pack into the parent's chunks rather than paying a
     call and starting fresh chunks of their own.

  5. A small fixed array of scalars is a static run of bound*width bits like
     any other, so it unrolls into the enclosing run instead of looping one
     stream call per element. Capped at 128 bits so generated code stays
     proportional to the schema, and byte-aligned [N]uint8 keeps the bulk-copy
     path, which is faster than either form.

Measured, M2, bench_mixed, same sitting, on top of lever 3:
  write      2.27 -> 2.32 M msg/s  (+2%)
  round_trip 1.11 -> 1.16 M msg/s  (+4.5%)
Control rows 1.20/1.21 against the 1.22/1.20 baseline. Wire unmoved,
conformance green.

Cumulative over five levers: write 1.37 -> 2.32 (1.69x),
round_trip 0.61 -> 1.16 (1.90x), blended 1.83x.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full per-shape leg caught two rows going the wrong way: rigidbody_moving
and rigidbody_at_rest both lost 9% on write. Mechanism, found in the emitted
code — RigidBody is all float64, so every piece is exactly one whole chunk.
The run packed into as many chunks as it had fields, removing no call, while
ADDING a materialized local per field and an address-taken chunk local, where
the per-field form had simply passed the struct field's own address.

The policy is now the honest one: flatten when ceil(bits/chunk) is fewer calls
than the run has bit-carrying pieces, and otherwise leave the per-field form
alone. That makes the transform non-regressive on call count by construction
rather than by luck.

bench_mixed is unaffected (write 2.35, round_trip 1.16 — its runs all reduce
calls); the rigidbody shapes return to the per-field form.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The profile after the first five levers put the generated code's own packing
arithmetic at 15% of samples and the runtime's bit placement at 30% — the
per-field calls were gone, but bench_mixed still spent most of its calls on
eighty 18-bit stat elements, each carrying 18 of a possible 64 bits.

An array of a struct that flattens whole is now emitted as a K-at-a-time loop
over a run spanning K elements, followed by a remainder loop. K is chosen as
the element count that minimises chunks-per-element within the run cap and an
eight-element unroll cap; for the 18-bit stat element that is 7, and eighty
elements fall from 80 stream calls to 25.

Measured, M2, bench_mixed, same sitting, on top of lever 5:
  write      2.34 -> 2.66 M msg/s  (+14%)
  round_trip 1.15 -> 1.40 M msg/s  (+21%)

Cumulative over six levers, against current main in one sitting:
  write      1.38 -> 2.66 M msg/s  (1.93x)
  round_trip 0.61 -> 1.40 M msg/s  (2.30x)
  blended    1182 -> 545 ns/msg    (2.17x)

Named behaviour change, the same class the run form already carries: a group's
refusals all run before the group packs, so a refused write has emitted fewer
bits than before. Which values are refused, and with which error, is
unchanged. Wire bytes unmoved, conformance green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uped loop

Two things, found together when the cap was moved.

THE BUG: the grouped array form declares its index outside the loop, so two
grouped arrays in one function collided on i. It only surfaced once the cap
let a second array group, but it was latent from lever 6. The grouped loop now
carries its own block, as the runs already do.

THE CAP: measured on the corpus shape rather than inherited from the Rust
template's 256. bench_mixed write / round_trip, M msg/s, one sitting:
  256  -> 2.66 / 1.40
  384  -> 2.78 / 1.49   <- peak
  512  -> 2.75 / 1.46
  1024 -> 2.62 / 1.43
The curve peaks and falls: past the peak a run holds more field values live
than the register file has room for and they spill, which is the cost the cap
exists to bound. 384 also lets the 135-bit entity element group two at a time.

Cumulative over seven levers, against current main in one sitting:
  write      1.38 -> 2.78 M msg/s  (2.01x)
  round_trip 0.61 -> 1.49 M msg/s  (2.44x)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six legs measured in one sitting (java, dart and elixir report ABSENT: this
fresh clone has no pinned dist/ toolchains and the box has no JRE). The
locked c/cpp reference reproduces at 100/104%, which is the control on the
sitting.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Emitter output unchanged — regenerated with no diff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR #183's Rust template maps one item to exactly one piece, so its fallback
(re-emit each of a run's pieces) IS re-emitting each of its items. This
emitter's levers made classification 1:N — an unrolled fixed array
contributes one piece per element, a flattened nested struct contributes
pieces naming ITS fields at a different base expression — and the fallback
was inherited verbatim. Three defects followed from that one gap:

  - `type Pair { values [2]float64 }` emitted the element loop TWICE on both
    paths: 32 bytes where every other language writes 16. Go-to-Go round
    trips passed, so nothing went red.
  - a nested struct split by the run cap emitted `value.C` on the OUTER
    type — a compile error, or, under a name collision, the silent
    serialization of the wrong field.
  - a bare `type Vec2 { x float64; y float64 }` carried an unused "math"
    import: needsMath was set during SPECULATIVE classification and never
    cleared when the run fell back.

A run now accumulates whole ITEMS as flatGroups and splits only on item
boundaries, so the fallback re-emits items against the base each item owns.
flatPiece no longer carries an ir.Item at all — a piece is a bit-placement
recipe and nothing else, which makes re-emitting one structurally
impossible rather than merely avoided. needsMath moves into the float
pieces' emit/read closures, which run at emission.

Wire is unmoved and generated/ is zero-diff on the corpus: none of the three
shapes exists in it. That blind spot is issue #203, addressed next.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#203)

Three emitter defects — one of them silent wire corruption — reproduced on
one-line schemas, and the entire conformance corpus caught none of them.
It grew from real game shapes, and a real shape always carries enough
neighbouring fields to mask the arrangements a run/chunk emitter keys on.
The corpus's realism is precisely what made it blind.

examples/Degenerate.schema carries twelve shapes that are deliberately
unrealistic: a fixed scalar array as essentially the whole message
([2]float64, [2]uint64, [2]int64, [1]T, [N]T landing on a chunk boundary,
an array plus a trailing field, two arrays back to back); a bare two-float
unit with no other math consumer in the file; and a nested struct as the
ONLY field, as the FIRST field, and straddling the Go emitter's run cap.

They are pinned as one wire golden, testdata/wire/degenerate.bin, and every
one of the nine legs is held to it: the stream legs (C++, C, Go, Rust, C#,
JS) write the twelve into one stream, and the whole-message legs (Dart,
Java, Elixir) concatenate their twelve buffers. That equality is why every
type in the file is a whole number of bytes wide — the file's header says
so, because it is load bearing.

internal/codegen/golang states the same properties directly against
maxRunBits, so re-tuning that constant cannot quietly retire the coverage.
Verified red first: on the pre-fix emitter all four assertions fail, and
generated/go/Degenerate.go does not compile ("math" imported and not used;
value.C undefined on TrioStraddle).

The new file caught a SECOND emitter defect on sight, the same class as the
Go one: the Dart backend declared all four conversion-scratch views whenever
any was needed, so a file converting only float64 carried an unreferenced
_u32 and `dart analyze` refused it. The views are now emitted per
conversion.

Existing wire goldens are unmoved; the corpus protocol id moves, as adding
a file must, and is re-pinned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stead of panicking

The Go leg is not untouched: the flat word codec's conviction — ~86% of
generated-codec time inside the runtime's per-field bit calls — came from
this flag. It stays, because the next round will want it and a profiler
kept in a scratch fork of the leg is a profiler that rots. It is an
iteration instrument: it changes nothing about what is measured, and no
timed row is taken under it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gafferongames and others added 2 commits September 1, 2026 00:13
gcc's -Wmaybe-uninitialized (CI, -Werror) cannot see that a read function
stores every field, and clang does not raise it, so the new block built
clean here and refused there. The rest of the leg already dirties its read
targets — a read that skips a field must be caught — so this block does the
same, with the same 0xEF fill.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PR carried a before/after table and a 0.9977x negative control with NO
committed CSV, and #199 retired the shapes those numbers were taken over,
so they could not be re-run on main. These two files are that evidence,
re-taken on the shape that survives — bench_mixed, family gen — and
committed before AND after, which is #183's precedent.

One sitting, back to back, on the tree as it stands after #204 rewrote the
runners. Nothing changed between the halves but generated/bench/go: main's
Go emitter output for the before half, this branch's for the after.

  go        write 1.35 -> 2.67 M msg/s (1.98x), round_trip 0.60 -> 1.41 (2.36x)
  headline  583% -> 248% of generated C++ on run.sh's own statistic (2.35x)
  controls  geomean 0.9940x over 16 rows — every other leg, unchanged code,
            compiled identically in both halves (min 0.9426, max 1.0339)

corpus_id is 6b213fbfa1a03a99 in every row of both files: the runner hashes
the goldens it loads, and Degenerate.schema's golden is not one of them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gafferongames gafferongames changed the title go: the flat word codec — 712% to 251% of generated C/C++ go: the flat word codec — 712% to 248% of generated C/C++ Aug 31, 2026
…the per-field form

Comment and shape only — generated output is byte-identical. flush() built
the run twice, once to ask worthFlattening and once to emit it; the
oversized-group branch had no note saying no classifier produces one today.
The orphaned flatFieldPiece doc above flatArrayElemPiece now sits on the
function it describes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gafferongames

Copy link
Copy Markdown
Contributor Author

Unblock response — every item in the verdict, with where it landed. The body is rewritten around it.

1. Silent wire corruption / 2. nested field on the outer type / 3. unused math — fixed in fb8d6f1, one repair. A run now accumulates whole ITEMS as flatGroups and splits only on item boundaries, so the fallback re-emits items against the base each item owns. Beyond that, flatPiece no longer carries an ir.Item at all — a piece is a bit-placement recipe and nothing else, so re-emitting one is structurally impossible rather than merely avoided. That is the choice between your two options: restoring #183's strict 1:1 would have cost levers 4–6, so the invariant is restored one level up, where the fallback actually operates. needsMath moves into the float pieces' emit/read closures, which run at emission.

Corpus coverage (#203)17faefa. examples/Degenerate.schema, twelve types, one C++-pinned golden, all nine legs on every push, plus the three properties stated against maxRunBits in internal/codegen/golang so re-tuning the cap cannot retire them. Verified red first at 997c553: four failing assertions and generated Go that does not compile. It caught a second emitter defect on sight — Dart declared all four conversion-scratch views whenever any was needed, so a float64-only file carried an unreferenced _u32 and dart analyze refused it. No corpus file had ever been float64-only.

Baseline — corrected to 712% throughout, re-derived from main's committed 2026-08-31-arm64-macbook-datadriven-quick.csv (cpp round_trip max 3.585 M/s over go's 0.503). Against 712% this branch's 248% is 2.87x. The old "580% and 592% agree to 2%" reconciliation is withdrawn. The two baselines disagree by 19% on the go leg alone — denominators agree to 2.4% — and the body says so rather than smoothing it.

bench/go untouched — false, dropped. The --cpuprofile flag stays (it is how the ~86%-in-the-runtime conviction was obtained); a3faf94 states it in the usage line and makes it refuse instead of panicking.

The unreceipted claims — the per-shape geomean and the 0.9977x control are withdrawn, not restated: #199 retired those shapes and no CSV was committed. Replaced by a paired same-sitting A/B on the shape that survives, both halves committed (0e930d5): 583% → 248%, 2.35x, controls geomean 0.9940x over 16 rows. Taken after rebasing onto #204, so the CSVs name an instrument that still exists.

The 10.44% spread / 275% median — recorded in the body verbatim, and resolved by the re-run: go's round_trip rows now carry 0.36% and 1.89%, and the two statistics agree at 248% and 243%.

Two things worth flagging for the reviewer:

  • generated/bench/go/{Bench.go,realworld/RealWorld.go} are byte-for-byte what the blocked branch emitted. The correctness repair changes no generated output on any existing corpus, so the numbers measure the same codec you reproduced at 2.35x blended.
  • make test is green in full on this box, java/dart/elixir conformance included — the pinned dist/ toolchains were available here, so this is not leaning on CI for those three legs. go run ./bench/tools/shapegate is clean at 19 ledger entries, no new debt.

Not merging — yours to call.

@gafferongames

Copy link
Copy Markdown
Contributor Author

Re-review verdict: MERGE. Every wire question answered by the reviewer's own commands in its own clone, not by reading claims — which is what a PR that already shipped one silent wire defect earns.

THE REPAIR, verified structurally AND empirically. Structurally: flatPiece genuinely has no ir.Item field, no expression in the package can turn a piece back into an item, the two branches of flush() are mutually exclusive, and every other run call site has NO fallback at all (!ok routes to the per-field emitter before a run is built). So there is no second path. Empirically: a differential oracle against origin/main's per-field emitter (no flat codec at all), same schema through both compilers, identical reflection driver, 24 seeds per type — 4176 type/seed combinations across four adversarial corpora, ZERO byte differences, zero round-trip mismatches. Adv1 swept nested structs across the 384-bit cap including an outer type declaring a/b/c to collide with its nested field names (the case that would have silently serialized the WRONG field under the old bug — byte-identical). Adv2 covered fixed scalar arrays at every width and the 128-bit unroll cap, arrays of nested structs at bounds 1/2/3/7/8/9/16/17 hitting both the group loop and the remainder loop, arrays straddling the cap. Adv4 covered mixed-kind fallbacks (runs whose groups differ in kind, falling back as a whole).

RED-FIRST REPRODUCED VERBATIM at 997c553's parent — all four assertions fail, and compiling the pre-fix generated Go reproduced the PR's error text INCLUDING ITS LINE NUMBERS. The silent-corruption assertion is dispositive at source level: pre-fix WriteSpanF64 contains the element loop twice (256 bits against a declared MaxBits of 128); at HEAD it appears once.

WIRE: reviewer built bin/schema itself, rm -rf generated/, re-ran all 36 generate invocations — zero diff. git diff 997c553 HEAD -- generated/bench/go empty, so the repair costs none of the win. Only degenerate.bin added under testdata/wire, nothing else moved; it is genuinely C++-pinned (written only by the C++ leg under SCHEMA_UPDATE_WIRE_GOLDENS), and all nine legs really byte-compare it — each assertion read individually, not taken on the claim, with the bit total 1672 = 209 bytes matching the golden exactly. make test exit 0 in full on that box, all nine legs.

THE DART FIX traced through all four reachable scratch states, correct and minimal. And a genuinely useful negative: C#/Java/Elixir use function-local scratch so they cannot carry this class, while JS-flat DOES emit const SC unconditionally — the same shape, but with no analyze/lint gate to fail on it. Named as the follow-on this PR already lists rather than a blocker.

NUMBERS recomputed from both committed CSVs: 583%→248% (2.35x), round_trip 2.3603x max / 2.3640x median (agrees both ways), controls geomean 0.9940x with min 0.9426 max 1.0339 over n=16 — exact. Baseline 712.4% max / 711.2% median; 712→248 = 2.871x.

Three findings recorded, none blocking: (a) the headline sentence says '1.98x write' unqualified where that is the max statistic and median is 1.88x — disclosed at line 104 and the table row is labelled, so it wants one word, not a block; (b) both new CSVs stamp a schema commit that does not resolve on origin (it is the pre-rebase SHA of what is now 1f47f56, so the measurement was taken on this branch's content, but a re-runner cannot reconstruct the tree from the PR alone); (c) a 'denominators agree to 2.4%' line that did not reproduce on any statistic tried — immaterial, the conclusion routes through the paired sitting.

Also noted mechanically: generated/js/RenderFlat.js lost its 21-line tier explainer to DegenerateFlat.js because the JS flat emitter attaches that doc to whichever flat file sorts first, and Degenerate now sorts before Render. Verified mechanical; a note for the JS emitter's owner, since the doc will migrate again on the next alphabetically-early schema.

@gafferongames
gafferongames merged commit 52691a0 into main Aug 31, 2026
8 checks passed
gafferongames added a commit that referenced this pull request Aug 31, 2026
#198 landed `examples/Degenerate.schema` and pinned its Elixir source
golden with the pre-round emitter, so this branch merging as-is turned
main red: `TestGoldenSourceElixir` refused
`testdata/golden/elixir/Degenerate.ex`.

The re-pin is a SOURCE re-pin only. `make update-goldens` rewrote every
wire golden under `SCHEMA_UPDATE_WIRE_GOLDENS=1` and every one came back
byte-identical, `testdata/wire/` has no diff, and the nine-backend chain
byte-compares `degenerate.bin` against the C++ pin in all nine legs and
is green. The new text is the round's own shapes: one map destructure
per scope (B) and literal-width flushes with `scratch_bits` gone (D).
gafferongames added a commit that referenced this pull request Aug 31, 2026
… cannot reach

Degenerate.schema's standing property is that every type in it is a whole
number of bytes. That is load bearing for what it catches and it is also a
ceiling: no clause boundary inside it ever lands mid-byte, so an emitter
that groups array elements picks the same group size on the write and the
read side of every type in the file. The Elixir round's write clause has a
52-bit budget and its read clause a 49-bit window, so on Degenerate the two
always agree and the disagreement is untested.

Clauses.schema picks element widths where they do not. At 13 bits the write
clause takes four elements (52, the whole budget) and the read clause three
(39, inside the window); at 17 it is three against two; at 26, two against
one. Counts run 0, below a clause, exactly a clause, one past it, and the
bound, so the remainder path is entered from every phase. It also carries a
fixed mid-byte array, grouping across a nested struct boundary, a union of
empty arms behind a tag, and string/bytes at zero, partial and full length
behind a 5-bit lead so the align inside them is a real barrier.

Joins.schema does the same to the static-offset state machine: arms that
agree and disagree on width, a branch with no else, a branch inside a
branch, an align that regains staticness on one path only, an array that
gives it up on one path only, unions of unequal arms at mid-byte offsets,
and a long static run after an align.

Unlike Degenerate these shapes are NOT byte-aligned, so one shared stream
would not equal a concatenation of the shapes written alone — and the Elixir
emitter returns each message as its own binary from bit zero, so it cannot
write a shared stream at all. Every shape is therefore written to its own
stream and flushed, and the golden is those concatenated. Every leg can
reproduce that, and each shape's bytes stay individually attributable.

C++ pins, Go and Elixir byte-compare; the remaining six legs follow. Adding
units to the corpus moves the example unit's protocol id, as #198 did when
it added Degenerate. No existing wire golden moved.
gafferongames added a commit that referenced this pull request Aug 31, 2026
The published figures were unverifiable: `git log origin/main..HEAD --
bench/results/` was empty, so no re-runner could check them. #198, #199
and #204 all committed their before/after data this week; this matches
that.

Both halves are a fresh paired A/B taken in one sitting on the M2 Air
after the rebase, three minutes apart, corpus_id 6b213fbfa1a03a99 on
every row. The BEFORE half stamps main at 52691a0 and the AFTER half
elixir-swizzle at fef86ed — the last CODE commit on the branch, so both
stamps resolve on origin and reconstruct the tree that was measured.

Elixir, bench_mixed, family gen:
  write      135,696 -> 378,044 msg/s  (2.79x; spread 2.08% -> 0.22%)
  round_trip  95,413 -> 239,723 msg/s  (2.51x; spread 0.46% -> 0.38%)

As a share of generated C++ (§2.9, max rates): 3743% -> 1457%. Both
elixir rows are far inside §2.3's 15% noise gate, so neither is a row
bench/tools/relative.go would exclude.
gafferongames added a commit that referenced this pull request Aug 31, 2026
…#202)

* elixir #174: the profile, before any lever is pulled

Wall-clock decomposition of generated Elixir on the canonical shape
(bench_mixed, 438 bytes: 8 entities x 135 bits, 80 stats x 18 bits,
4 loadout bytes), M2 Air, OTP 29.0.5 / Elixir 1.20.4, 100k ops per
measurement, the same instances the bench rotates:

  write        7.344 us/op
  read         3.561 us/op   (round_trip 10.748)

  write splits:
    stats loop, 80 elements     2.632 us    32.9 ns/stat  (36%)
    entities, 8 elements        2.970 us   371.5 ns/ent   (40%)
    header + footer + loadout   1.742 us                  (24%)

The write path is 2x the read path, and 325 binary appends per message
is what it is made of: the emitter's mergeW flushes every whole byte
after EVERY field, so one bs_append BIF call rides each of the 131
static field sites plus 8x14 + 80x2 + 4x1 loop sites.

eprof agrees on the ordering (stats 38.7%, entities 23.7% of write) and
is not quoted for magnitudes: 162000 traced calls into the stats loop
inflate exactly the function the wall clock says is hot.

Nothing measured here lands in serialize.elixir. The generated Elixir
codec has no runtime dependency at all, so every lever below is a
schema-level emitter change and #170's routing question does not arise.

* elixir: lever A — one binary append per group, not per field

The emitter knew every field width statically and threw the knowledge
away: mergeW flushed the scratch's whole bytes after EVERY field, so a
438-byte bench_mixed message cost 325 bs_append BIF calls.

mergeW now carries a GROUP. It merges into the scratch and flushes only
when the next field would pass the budget, and flushW closes the group at
every barrier that observes data or scratch_bits — the write function's
tail, an align, the bytes of a string, a loop helper's call and its own
element tail, and the joins of a branch or a union case. flushW is a
no-op when no group is open, which is what makes a barrier free where one
is already closed.

The budget is 52 bits and the number is not a taste. The BEAM's fixnum is
60-bit signed, so an intermediate at or above 2^59 costs a heap bignum; a
flush leaves at most 7 bits behind, so 7 + 52 = 59 is the whole envelope.
Measured on the read side of the same shape, a 72-bit window is SLOWER
than the shipped 40-bit one (29.4 vs 27.6 ns/element) — boxing costs more
than the flush it would save, so the group stops at the boundary.

MEASURED, wall clock, canonical shape, 100k ops (before -> after):

  write            7.344 -> 5.208 us/op     1.41x
    stats loop      32.9 -> 22.3 ns/stat
    entities       371.5 -> 223.2 ns/entity
  round_trip      10.748 -> 8.886 us/op
  read             3.561 -> 3.561 us/op     (untouched, as expected)

  bench --quick: write 0.13 -> 0.18 M msg/s, round_trip 0.09 -> 0.12

Static append sites in generated/bench/elixir: 131 -> 65; per entity
14 -> 3, per stat 2 -> 1; ~325 -> ~128 appends per message.

Wire bytes unmoved: corpus_id 6b213fbfa1a03a99, the full variant
round-trip gate green, test/elixir and test/elixir-ludicrous OK,
mix format --check-formatted clean.

* elixir: lever C — one window decode per group, not per field

rd/3 opened a fresh match context for EVERY field: 14 per entity, 2 per
stat. The generator knows every width statically, so readR now reads a
GROUP into rv once and cuts each field out with a static shift and mask.
The fused static run's own length sizes the group, so a short run keeps
the cheap 40-bit window and a long one takes rdw's 56-bit window.

Two windows, and the second one's width is the same fixnum argument the
write group's budget rests on. A 56-bit window less the 7-bit worst-case
offset is a 49-bit group and stays under 2^59; a 64-bit window would box.
Measured on this shape a 72-bit window is SLOWER than the shipped 40-bit
one it would replace — 29.4 vs 27.6 ns/element — so 49 is the ceiling and
not an arbitrary stopping point.

Reading a group wider than the fields it feeds is safe by construction:
rd and rdw never raise (the tail falls back to the bytes that exist), and
bits past the bounds-checked run are discarded, never observed. Every
bounds check, range check, constant check and refusal is where it was.

rdBreak closes the group at every barrier — the read surface, a loop
helper's entry and every call to one, an align, the bytes of a string,
and the arms of a branch or a union case.

MEASURED, wall clock, canonical shape, 100k ops (after lever A -> after C):

  read             3.561 -> 2.384 us/op     1.49x
    stats           27.6 ->  ~15  ns/stat   (rd calls 2/elem -> 1)
    entities                               (rdw calls 14/elem -> 4)
  round_trip       8.886 -> 8.126 us/op
  write            5.208 -> 5.208 us/op     (untouched, as expected)

  bench --quick: round_trip 0.12 -> 0.13 M msg/s

Wire bytes unmoved: corpus_id 6b213fbfa1a03a99, the full variant
round-trip gate green, test/elixir and test/elixir-ludicrous OK,
mix format --check-formatted clean.

* elixir: lever B — one map read per scope, not one per field

Elixir's `.` on a struct is a get_map_elements of its own with its own
raise branch. The writer spent one per field: 14 for an entity, 2 for a
stat, ~30 at the top level. A scope now reads its fields ONCE —
`%{stat_id: e_stat_id, delta: e_delta} = e` — and every reference to
those fields is a local from there on.

Scope, precisely: a scope binds the fields it reads UNCONDITIONALLY at
its own level — its field items and the conditions of its branches. A
branch arm's fields are NOT bound at the enclosing level; the arm is its
own scope and binds them when it is taken, so a value the wire never asks
for is still never demanded of the caller. Below two fields the pattern
would not pay for itself and none is emitted.

Refusals are unchanged. A struct always carries every key, so the bound
locals are exactly as unconditional as the dotted accesses they replace,
and every range, count, mask and length check is where it was. The one
delta, named: a MAP missing a key now raises MatchError where it raised
KeyError — a raise either way, on exactly the same inputs, never a wrong
answer. The raise TEXT is unchanged: g.dsp resolves a local back to the
dotted access it stands for, so the message still reads "e.delta is above
the wire maximum" and never names a local the caller never wrote.

MEASURED, wall clock, canonical shape, 100k ops (after lever C -> after B):

  write            5.208 -> 3.202 us/op     1.63x
    stats loop      23.4 ->  15.4 ns/stat
    entities       212.6 ->  96.4 ns/entity
  round_trip       8.126 -> 5.466 us/op
  read             2.384 -> 2.276 us/op     (the read side binds locals already)

  bench --quick: write 0.18 -> 0.31 M msg/s, round_trip 0.13 -> 0.18

Cumulative over A + C + B, against the round's baseline:

  write            7.344 -> 3.202 us/op     2.29x
  read             3.561 -> 2.276 us/op     1.56x
  round_trip      10.748 -> 5.466 us/op     1.97x

Wire bytes unmoved: corpus_id 6b213fbfa1a03a99, the full variant
round-trip gate green, test/elixir and test/elixir-ludicrous OK,
mix format --check-formatted clean.

* elixir: lever E — the float32 step, and a declaration constant folded

Three changes to the compressed-float helpers, all of them removing work
that was provably redundant:

fr/1 lets the REFUSAL be the test. A float segment does not match a
non-finite pattern, so the finite path is one construction and one match
and never touches the exponent field; the second clause reads the sign of
exactly the patterns the first refused. NaN still maps by sign, as it did.

cf_quantize takes miv32 — the float32 of the step count — as a
generation-time literal instead of computing fr(miv * 1.0) on every call.
It is a declaration constant; folding it removes one of the six float32
steps. cf_decode takes it directly in place of the integer count, which
was only ever used to compute the same rounding.

trunc(Float.floor(x)) becomes floor(x): one BIF returning an integer
instead of a float floor and a truncation. The argument is finite by
construction (normalized is clamped to [0, 1] and miv32 is finite), so
the paths differ only in the class of an unreachable raise.

MEASURED, wall clock, canonical shape, 100k ops, and reported honestly:

  write            3.202 -> 3.18 us/op   AT OR BELOW the noise floor
  read             2.276 -> 2.19 us/op   ~3%
  round_trip       5.466 -> 5.30 us/op

The isolated micro said more (cf_quantize 203.6 -> 135.2 ns/call), and
the micro was wrong to say it: it passed fr as a closure, so every fr
call it removed was an inflated indirect call rather than the direct
local call the generated module makes. The in-situ number is the ruling.
The lever stays because it costs nothing and removes real work; it is
recorded as small, not as what the micro promised.

Wire bytes unmoved: corpus_id 6b213fbfa1a03a99, the full variant
round-trip gate green, test/elixir and test/elixir-ludicrous OK,
mix format --check-formatted clean.

* elixir: re-pin the four source goldens the emitter moved

Deliberate emitter change, and only the SOURCE goldens move: no file under
testdata/wire is touched by this commit or by any of the four levers, and
the full `make test` chain — every language's conformance suite, the wire
goldens, the fuzzers, the format refuser — is green.

* elixir: lever D — the static offsets the generator already knew

The group model carried its widths statically and then computed with them
at runtime anyway: every merge shifted by the scratch_bits VARIABLE and
added to it, and every flush divided that variable by eight to size a
binary segment whose width was therefore dynamic.

A write function starts at a known empty scratch and stays known until the
message's own data decides a length. Through that whole region the emitter
now tracks the offset itself, so:

  - a merge is one statement, `scratch = scratch ||| v <<< 16`, with a
    literal shift and no scratch_bits arithmetic at all; the first merge of
    an empty group is a bare bind, `scratch = v`
  - a flush is `data = <<data::binary, scratch::little-size(4)-unit(8)>>`
    with a LITERAL segment width — the form the BEAM's binary construction
    is built for — plus a literal shift, in place of four statements
  - an align that lands on a boundary emits nothing, and one that does not
    appends the residual byte unconditionally instead of testing for it
  - the function's tail is `data` or `<<data::binary, scratch>>` outright,
    never `if scratch_bits != 0`
  - scratch_bits is not even bound where the surface never needs it

Staticness is a property the emitter EARNS and gives up honestly. It is
given up where a loop helper is called (how many elements rode is the
message's business) and where a branch's arms end on offsets that disagree;
in that case each arm publishes the offset it reached and the emitter goes
back to maintaining the variable. It is REGAINED at every align, which
lands the position on a byte whatever the data did. Where every arm of a
branch or a union case does agree, the offset stays static past the join
and scratch_bits does not ride the join's tuple at all.

Behaviour is unchanged: the same statements in the same order over the same
values, with the arithmetic the generator can do moved to generation time.
Every range, count, mask and length check is where it was, and every raise
carries the text it carried.

Wire bytes unmoved: no file under testdata/wire is touched, corpus_id
6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and
test/elixir-ludicrous OK, mix format --check-formatted clean, and the whole
nine-backend `make test` chain green. Only the four SOURCE goldens move,
re-pinned here with the emitter that moved them.

MEASURED: in the sitting's A/B, recorded on the PR with the round's table.

* elixir: lever J — a scalar array element sized its own read window

Lever C sizes each window decode by the fused static run it sits in, and
the run fuser is reached through emitReadItems. A scalar array element is
not: readHelper hands it straight to the scalar read, so the run was zero —
unknown — and readR fell back to its widest window. A one-byte element was
opening the 56-bit window to take eight bits.

That is not merely a wider mask. rdw needs seven bytes ahead of the
position to match its window and rd needs five, so the wide window drops
into the :binary.decode_unsigned tail fallback two bytes sooner — and a
scalar array is very often the LAST thing in a message, which is exactly
where that boundary lies.

A scalar element's own width is the run, so readHelper sets it. The loadout
loop now reads `rd(data, bits_read, 8)` where it read `rdw(data, bits_read,
49)`. Nothing else moves: a struct or union element still goes through
emitReadItems and fuses its own runs, and readR's choice of window was
already free to be any width that covers the field.

Wire bytes unmoved: no file under testdata/wire is touched, corpus_id
6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and
test/elixir-ludicrous OK, mix format --check-formatted clean, the whole
nine-backend `make test` chain green, and the two source goldens the
emitter moved re-pinned here.

MEASURED: in the sitting's A/B, recorded on the PR with the round's table.

* elixir: lever K — one append for a whole clause of elements, not one each

The write loop appended once per ELEMENT: 80 bs_append calls for the stats
array of one bench_mixed message, 4 for the loadout. Lever A closed the
group across the fields of an element; it could not close it across the
call boundary between elements, because a helper is entered at an offset
the caller's element count decides.

Measured in isolation, that boundary is where the write path's cost is.
Twelve appends of one 32-bit segment each cost 103.6 ns; the same 48 bytes
in three appends of four segments cost 79.0, and in one append of twelve,
73.3. The same twelve appends with a DYNAMIC segment width cost 103.7 —
identical. The append is the expense; the arithmetic around it is not.

So a clause takes SEVERAL elements off the list. k is chosen by the group
budget the fixnum boundary already fixed — the most whole elements whose
widths fit in 52 bits, capped at four so one array field cannot cost
unbounded generated code — and the k element bodies merge into one group
and flush once. The single-element clause behind the wide one is the
remainder, so a list length never has to divide anything, and an element
whose width the wire decides keeps one clause per element as before.

  stats  (18 bits): 2 elements per clause, 80 appends -> 40
  loadout (8 bits): 4 elements per clause,  4 appends ->  1

Nothing about an element's emission changes. The clause names its slots e1
and e2 where it used to have just e, and the raise TEXT is held: the
display map resolves every slot back to "e", so a message still reads
"e.delta is above the wire maximum" and never names a slot the caller never
wrote. Every range, count, mask and length check is where it was.

MEASURED, wall clock, the canonical shape from the committed variant
corpus, 64 rotating instances, 150k ops, 4 runs x 4 interleaved passes,
best per pass, median of passes:

  write            3.097 -> 2.65 us/op      1.17x
  round_trip       5.268 -> 4.74 us/op      1.11x
  read             2.132 -> 2.09 us/op      (untouched, as expected)

Wire bytes unmoved: no file under testdata/wire is touched, corpus_id
6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and
test/elixir-ludicrous OK, every generated module compiles without a
warning, mix format --check-formatted clean, the nine-backend `make test`
chain green, and the source goldens the emitter moved re-pinned here.

* elixir: lever L — one window decode for a whole clause of elements

The read loop's other half of lever K. Lever C reads a GROUP into one
window and cuts each field out of it with a static shift and mask, but a
loop helper is a function boundary, so a group could never span two
elements: 80 window decodes for the stats array, 4 for the loadout.

A read clause now decodes k elements under ONE window. k is chosen by the
wide window's usable width — the most whole elements whose widths fit in
49 bits, which is the same fixnum boundary the write budget rests on,
capped at four — and the clause carries a guard on the remaining count,
with the single-element clause behind it as the remainder.

Two things had to give way for a window to span elements. The element
emitters now take the variable they bind, so a clause can hold e1 and e2
without either shadowing the other. And the run fuser no longer overwrites
an OUTER run: a named element's own scope would otherwise size the window
to one element and undo the clause. Where no outer run is open the fuser
behaves exactly as it did.

  stats  (18 bits): 2 elements per rdw window of 36 bits, 80 decodes -> 40
  loadout (8 bits): 4 elements per rd window of 32 bits,   4 decodes ->  1

Reading a window wider than one element is safe on exactly the grounds
lever C established: rd and rdw never raise, the tail falls back to the
bytes that exist, and bits past the bounds-checked run are discarded and
never observed. The bounds check itself is unchanged — the call site
proved count * elem for a counted array, and an unbounded run still checks
its own span before it reads it.

MEASURED, wall clock, the canonical shape from the committed variant
corpus, 64 rotating instances, 150k ops, 4 runs x 4 interleaved passes,
best per pass, median of passes, against lever K:

  read             2.089 -> 1.838 us/op     1.14x
  round_trip       4.736 -> 4.409 us/op     1.07x
  write            2.703 -> 2.691 us/op     (untouched, as expected)

Wire bytes unmoved: no file under testdata/wire is touched, corpus_id
6b213fbfa1a03a99, the full variant round-trip gate green, test/elixir and
test/elixir-ludicrous OK, every generated module compiles without a
warning, mix format --check-formatted clean, the nine-backend `make test`
chain green, and the source goldens the emitter moved re-pinned here.

* elixir: the emitter answers modernize and gofmt

Four findings, all of them this branch's own, none of them a behaviour
change: the two scope-binding map copies lever B introduced become
maps.Copy, and the two unroll caps levers K and L introduced become min.
The generator emits identical bytes for every schema in the tree.

CI's lint job runs modernize at @latest, so a check that ships tomorrow
lands on the next branch the same way; this is the branch paying for its
own four.

* elixir: re-pin Degenerate's source golden after the rebase onto main

#198 landed `examples/Degenerate.schema` and pinned its Elixir source
golden with the pre-round emitter, so this branch merging as-is turned
main red: `TestGoldenSourceElixir` refused
`testdata/golden/elixir/Degenerate.ex`.

The re-pin is a SOURCE re-pin only. `make update-goldens` rewrote every
wire golden under `SCHEMA_UPDATE_WIRE_GOLDENS=1` and every one came back
byte-identical, `testdata/wire/` has no diff, and the nine-backend chain
byte-compares `degenerate.bin` against the C++ pin in all nine legs and
is green. The new text is the round's own shapes: one map destructure
per scope (B) and literal-width flushes with `scratch_bits` gone (D).

* corpus: Clauses.schema and Joins.schema — the arrangements Degenerate cannot reach

Degenerate.schema's standing property is that every type in it is a whole
number of bytes. That is load bearing for what it catches and it is also a
ceiling: no clause boundary inside it ever lands mid-byte, so an emitter
that groups array elements picks the same group size on the write and the
read side of every type in the file. The Elixir round's write clause has a
52-bit budget and its read clause a 49-bit window, so on Degenerate the two
always agree and the disagreement is untested.

Clauses.schema picks element widths where they do not. At 13 bits the write
clause takes four elements (52, the whole budget) and the read clause three
(39, inside the window); at 17 it is three against two; at 26, two against
one. Counts run 0, below a clause, exactly a clause, one past it, and the
bound, so the remainder path is entered from every phase. It also carries a
fixed mid-byte array, grouping across a nested struct boundary, a union of
empty arms behind a tag, and string/bytes at zero, partial and full length
behind a 5-bit lead so the align inside them is a real barrier.

Joins.schema does the same to the static-offset state machine: arms that
agree and disagree on width, a branch with no else, a branch inside a
branch, an align that regains staticness on one path only, an array that
gives it up on one path only, unions of unequal arms at mid-byte offsets,
and a long static run after an align.

Unlike Degenerate these shapes are NOT byte-aligned, so one shared stream
would not equal a concatenation of the shapes written alone — and the Elixir
emitter returns each message as its own binary from bit zero, so it cannot
write a shared stream at all. Every shape is therefore written to its own
stream and flushed, and the golden is those concatenated. Every leg can
reproduce that, and each shape's bytes stay individually attributable.

C++ pins, Go and Elixir byte-compare; the remaining six legs follow. Adding
units to the corpus moves the example unit's protocol id, as #198 did when
it added Degenerate. No existing wire golden moved.

* corpus: the Rust, C and JS legs byte-compare Clauses and Joins

* corpus: the C#, Dart and Java legs byte-compare Clauses and Joins

All nine legs now hold the two units to the C++ pin. The corpus README
gains a row for each, stating what each reaches that Degenerate cannot.

* bench: commit both halves of the sweep the round's numbers come from

The published figures were unverifiable: `git log origin/main..HEAD --
bench/results/` was empty, so no re-runner could check them. #198, #199
and #204 all committed their before/after data this week; this matches
that.

Both halves are a fresh paired A/B taken in one sitting on the M2 Air
after the rebase, three minutes apart, corpus_id 6b213fbfa1a03a99 on
every row. The BEFORE half stamps main at 52691a0 and the AFTER half
elixir-swizzle at fef86ed — the last CODE commit on the branch, so both
stamps resolve on origin and reconstruct the tree that was measured.

Elixir, bench_mixed, family gen:
  write      135,696 -> 378,044 msg/s  (2.79x; spread 2.08% -> 0.22%)
  round_trip  95,413 -> 239,723 msg/s  (2.51x; spread 0.46% -> 0.38%)

As a share of generated C++ (§2.9, max rates): 3743% -> 1457%. Both
elixir rows are far inside §2.3's 15% noise gate, so neither is a row
bench/tools/relative.go would exclude.
rowan-claude added a commit that referenced this pull request Aug 31, 2026
Main at d00cf49, all of the night's emitter work landed (#198 go, #202
elixir), box idle, nine legs, corpus_id 6b213fbfa1a03a99.

  c 98 / cpp 100 / rust 154 / java 157 / go 239 / cs 361 / dart 381 /
  js 486 / elixir 1498

This is ledger point one for #194: the first sitting where the canonical
shape, the data-driven harness, and the night's optimizations are all on
main together, measured with nothing else running.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gafferongames added a commit that referenced this pull request Aug 31, 2026
The third port of the template Rust (#183) and Go (#198) already carry:
field values into locals, OR'd into 64-bit chunk words at literal shifts,
one packer step per chunk instead of one per field.

WHY IT PAYS HERE IS NOT WHY IT PAID IN GO. #198's finding was that Go's
per-field entry points sit past the compiler's inlining budget, so every
field cost a real call and ~86% of the codec's time was inside them. C# has
no such wall: the JitDisasm audit found ZERO direct calls into serialize.cs
from any generated method — every entry point inlines. What C# pays instead
is what those inlined bodies DO. BitWriter's packer step masks, shifts, ORs
into the scratch word, tests `newScratchBits >= 64`, conditionally stores a
qword and recovers the spill: a data-dependent branch and a conditional
store PER FIELD. MixedEntity paid fourteen of them for 135 bits. It now
pays three.

THREE INVARIANTS, RE-DERIVED RATHER THAN INHERITED — #198's review record
is why they are spelled out in flat.go's header rather than assumed:

  1. 64-bit chunks are wire-identical because serialize.cs's SerializeBits64
     splits low-dword-first-then-remainder, which IS the 32-bit chunk order.
  2. Every piece is masked to its width, because BitWriter.WriteBitsUnchecked
     TRUNCATES a too-wide value rather than refusing it — the flat form must
     truncate identically or a wide value corrupts its neighbours in the
     chunk instead of losing its own high bits. This is the invariant the Go
     port lost by inheriting Rust's fallback; here it is enforced at the one
     place a piece's value expression is built.
  3. Write refusals come first, in declaration order — same values refused,
     same non-latching false. What moves is how many bits reached the stream
     before the refusal, and a refused write leaves a partial message either
     way.

A run breaks at align, strings/bytes, arrays, branches, nested struct and
union calls, and the float, fixed-point, compressed-float and 128-bit
families. A run that would not REDUCE the packer-step count is not
flattened. A run that does not pay falls back ITEM BY ITEM — the item rides
inside every piece for exactly that reason, because the item is the only
unit the per-field path can re-emit.

THE READ HALF IS NOT IN THIS COMMIT, and the blocker is named rather than
worked around: folding a ranged read replaces SerializeInt with a generated
comparison, and serialize.cs's ranged read LATCHES ValueOutOfRange where a
generated guard returns false without latching. cs publishes checks=always
and test/cs pins that latch, so the read fold waits on a public
refusal-latch on ReadStream/ReadBatch — a serialize.cs change, its own PR.
The write path has no such dependence: its folded range refusals already
return false without latching. Measured ceiling for the read half, from a
hand-written flat prototype of the two hot cores: round_trip 1.08 -> 1.55
M msg/s, a 42% cut in read time.

MEASURED, against the previous commit, same box, median of 3 sittings:

  bench_mixed write       2.82 -> 3.30 M msg/s  (+17%)
  bench_mixed round_trip  1.08 -> 1.18 M msg/s  (+9%)

and against main:

  bench_mixed write       2.10 -> 3.30 M msg/s  (+57%)
  bench_mixed round_trip  0.97 -> 1.18 M msg/s  (+22%)

flatMaxRunBits is Go's 384 carried unmeasured: swept at 128/256/512 the C#
numbers do not move, because this corpus's longest run is 135 bits and no
cap in that range binds it. Said out loud in the file rather than presented
as a C# measurement.

Wire bytes unmoved: full `make test` green across nine backends, the
degenerate corpus included, and the bench's golden + per-variant round-trip
gates pass at corpus_id 6b213fbfa1a03a99.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gafferongames added a commit that referenced this pull request Sep 1, 2026
…word codec (write 1.59x, round_trip 1.60x; 361% -> 219% of generated C++) (#208)

* C# emitter: union batch cores, and a scoped batch per composition site

The C# backend was the only one never audited for generation quality, and
the profile of the bench shape found the cost in one place: composition.

MEASURED FIRST (DOTNET_JitDisasm + DOTNET_JitDisasmSummary, FullOpts,
tiering off, on generated/bench/cs/Bench.cs). Everything the runtime offers
already inlines — zero direct calls into serialize.cs from any generated
method. What did not inline was the generated code's own composition: the
entity array (8 elements) and the stat array (80 elements) each reached
their element type through its STREAM entry, and each of those entries
opened a batch and closed it again. 88 BeginBatch captures and 88 End
restores per 438-byte message, plus 88 real calls the JIT will not inline
through, on a message where those two arrays carry 72% of the wire bits.

They reached the stream entry because BenchMixed reaches a union, and the
batch plan excluded any type that reached one — a union's arm dispatch had
no core to compose with. That exclusion was landed as land-simple with the
note "a follow-on if a profile ever convicts the plain path". It has.

TWO CHANGES, one law each.

1. Unions carry batch cores. A union's arm dispatch is a switch over calls,
   and inside a core those calls go core-to-core by ref like any other
   composition — nothing about the inline-only rule ever stood in the way,
   only the missing emitter half. densityUnion weighs the tag plus the
   heaviest arm; the core closure walks arms as well as struct fields.

2. A composition site inside a stream body opens a SCOPED batch — one
   capture and one restore for the whole site, an array's entire loop
   included. Only the composition call rides inside the scope (an array's
   count site stays on the stream ahead of it), so the scope has exactly one
   way to fail and End always runs before the refusal leaves the function.

AND ONE MEASURED REFUSAL, recorded because it is a result. The obvious
third change — let the whole message body run on one batch now that the
union no longer blocks it — is SLOWER, and materially: bench_mixed write
2.10 -> 2.04 M msg/s and round_trip 0.97 -> 0.76 M msg/s against the same
baseline the two changes above improve. The JIT did inline the 12.7 KB
core (IL size 23 at the entry), so this is not the address-exposure law
firing; it is the delegated sites. A body with any bulk, 128-bit or
fixed-point site must Sync the batch down and Recapture it around that
site, and between them it hands the register allocator a ref struct that
has to survive the whole body. batchEntry states it: density is necessary,
b == 0 is the rest.

MEASURED, this branch vs main, same box, bench --quick, median of 3
sittings (iteration instrument, not certification):

  bench_mixed write       2.10 -> 2.82 M msg/s  (+34%)
  bench_mixed round_trip  0.97 -> 1.08 M msg/s  (+11%)

Wire bytes unmoved: full `make test` green across nine backends, the
degenerate corpus included, and the bench's own golden + per-variant
round-trip gates pass at corpus_id 6b213fbfa1a03a99. Checks semantics are
untouched — the same inputs are refused with the same errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* C# emitter: the flat word codec, write half

The third port of the template Rust (#183) and Go (#198) already carry:
field values into locals, OR'd into 64-bit chunk words at literal shifts,
one packer step per chunk instead of one per field.

WHY IT PAYS HERE IS NOT WHY IT PAID IN GO. #198's finding was that Go's
per-field entry points sit past the compiler's inlining budget, so every
field cost a real call and ~86% of the codec's time was inside them. C# has
no such wall: the JitDisasm audit found ZERO direct calls into serialize.cs
from any generated method — every entry point inlines. What C# pays instead
is what those inlined bodies DO. BitWriter's packer step masks, shifts, ORs
into the scratch word, tests `newScratchBits >= 64`, conditionally stores a
qword and recovers the spill: a data-dependent branch and a conditional
store PER FIELD. MixedEntity paid fourteen of them for 135 bits. It now
pays three.

THREE INVARIANTS, RE-DERIVED RATHER THAN INHERITED — #198's review record
is why they are spelled out in flat.go's header rather than assumed:

  1. 64-bit chunks are wire-identical because serialize.cs's SerializeBits64
     splits low-dword-first-then-remainder, which IS the 32-bit chunk order.
  2. Every piece is masked to its width, because BitWriter.WriteBitsUnchecked
     TRUNCATES a too-wide value rather than refusing it — the flat form must
     truncate identically or a wide value corrupts its neighbours in the
     chunk instead of losing its own high bits. This is the invariant the Go
     port lost by inheriting Rust's fallback; here it is enforced at the one
     place a piece's value expression is built.
  3. Write refusals come first, in declaration order — same values refused,
     same non-latching false. What moves is how many bits reached the stream
     before the refusal, and a refused write leaves a partial message either
     way.

A run breaks at align, strings/bytes, arrays, branches, nested struct and
union calls, and the float, fixed-point, compressed-float and 128-bit
families. A run that would not REDUCE the packer-step count is not
flattened. A run that does not pay falls back ITEM BY ITEM — the item rides
inside every piece for exactly that reason, because the item is the only
unit the per-field path can re-emit.

THE READ HALF IS NOT IN THIS COMMIT, and the blocker is named rather than
worked around: folding a ranged read replaces SerializeInt with a generated
comparison, and serialize.cs's ranged read LATCHES ValueOutOfRange where a
generated guard returns false without latching. cs publishes checks=always
and test/cs pins that latch, so the read fold waits on a public
refusal-latch on ReadStream/ReadBatch — a serialize.cs change, its own PR.
The write path has no such dependence: its folded range refusals already
return false without latching. Measured ceiling for the read half, from a
hand-written flat prototype of the two hot cores: round_trip 1.08 -> 1.55
M msg/s, a 42% cut in read time.

MEASURED, against the previous commit, same box, median of 3 sittings:

  bench_mixed write       2.82 -> 3.30 M msg/s  (+17%)
  bench_mixed round_trip  1.08 -> 1.18 M msg/s  (+9%)

and against main:

  bench_mixed write       2.10 -> 3.30 M msg/s  (+57%)
  bench_mixed round_trip  0.97 -> 1.18 M msg/s  (+22%)

flatMaxRunBits is Go's 384 carried unmeasured: swept at 128/256/512 the C#
numbers do not move, because this corpus's longest run is 135 bits and no
cap in that range binds it. Said out loud in the file rather than presented
as a C# measurement.

Wire bytes unmoved: full `make test` green across nine backends, the
degenerate corpus included, and the bench's golden + per-variant round-trip
gates pass at corpus_id 6b213fbfa1a03a99.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* C# emitter: the flat word codec, read half

The read half of the same form: chunks read first, ONE sticky-error test for
the whole run, then every value unpacked, validated and stored with literal
shifts and literal masks. Where the per-field form paid a bounds check and an
error test per field, MixedEntity now pays one of each per chunk.

WHAT IT WILL NOT ABSORB, AND WHY. A ranged read whose refusal can actually
fire goes through serialize.cs's SerializeInt/SerializeInt64, which LATCH
SerializeError.ValueOutOfRange; a generated comparison returns false without
latching. cs publishes checks=always and test/cs pins that latch, so a
ranged piece joins a run ONLY where its check is VACUOUS — max - min is
exactly 2^bits - 1, so no value the bits can hold is out of range — or where
this emitter's own read path ALREADY refuses without latching (the
full-range unsigned bits64 path, which has carried a generated refusal all
along). Everything else closes the run and keeps the runtime call. Same
inputs refused, same errors, on every path.

AND THAT RESTRICTION COSTS ALMOST NOTHING — which retires a lever rather
than deferring it. The hand-written prototype that folded EVERY ranged read
of the two hot cores, latch semantics and all, measured round_trip 1.55
M msg/s. This commit, which leaves four of MixedEntity's fourteen fields on
the runtime call because their range checks can fire, measures 1.54. The
public refusal-latch on ReadStream/ReadBatch that the full fold would need
is worth about 1% here, and is NOT worth a serialize.cs API addition on this
evidence. Recorded as ranked-and-refused, not as a follow-on.

TWO CONSEQUENCES OF FUSION, named in the file rather than found later:

  - A stream BOTH truncated inside a run AND carrying an out-of-range value
    before the truncation now surfaces the stream's overflow error where the
    per-field form surfaced the range refusal. Both refuse the packet; the
    set of ACCEPTED streams is unchanged, which is the property that matters
    at the trust boundary. Rust, Go, Java, Dart and JS-flat already carry it.
  - A failed run leaves the destination object untouched where the per-field
    form left the fields before the failure updated. Both are partial states
    a failed read gives no contract over (SPEC §5).

NEGATIVE CONTROL, run rather than assumed: with flatMaskLit forced to emit
no mask — invariant 2 deliberately broken — `make test` produces 65 failure
lines across the backends. The gate that proves this form does see it.

MEASURED, against the previous commit, same box, median of 3 sittings:

  bench_mixed write       3.30 -> 3.30 M msg/s  (unchanged, as expected)
  bench_mixed round_trip  1.18 -> 1.54 M msg/s  (+30%)

and against main:

  bench_mixed write       2.10 -> 3.30 M msg/s  (+57%)
  bench_mixed round_trip  0.97 -> 1.54 M msg/s  (+59%)

Wire bytes unmoved: full `make test` green across nine backends, the
degenerate corpus included, test/cs and test/cs-ludicrous (which pin the
corrupt-wire refusals) among them, and the bench's golden + per-variant
round-trip gates pass at corpus_id 6b213fbfa1a03a99.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* bench/run.sh: the schema commit line carries its branch

Every runtime line in the preamble already records `<sha> (<branch>)`; the
schema commit line recorded a bare sha. Two failures that closes, both of
which this repo has already had: a sweep that silently measured the wrong
tree and a CSV that could not have said so, and a published result whose sha
stopped resolving the moment the branch was rebased. The branch name
survives the rebase.

One line in run.sh (it reuses commit_of, which the runtime lines already
use) and its law-home in BENCH-STANDARD.md §3.5, beside the provenance
clause the same class of defect earned in August.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* bench/results: the cs-word-codec before/after pair, both halves, two sittings

Both halves committed, --only cs, taken back to back in the same window,
twice. Preambles carry the schema commit AND its branch on the after half;
the before half runs main's own run.sh, which predates that line, so the
file says so in its own note.

bench_mixed, cs, family gen, checks=always (M msg/s):

  path         before max   after max    before median  after median
  write         2,104,563   3,342,814      2,103,454     3,334,630   1.59x
  round_trip      992,155   1,594,245        990,865     1,586,844   1.60x

Against the quiet-box reference sweep of the same day and the same
corpus_id (bench/results/2026-09-01-arm64-macbook.csv, cpp round_trip
median 3,467,551; cpp write median 7,287,687), on the round_trip statistic
the published ledger reads at:

  cs before   350% of generated C++   (the published ledger row: 361%)
  cs after    219%

and on write, 347% -> 219%.

The `bits` family rows in this pair are NOT attributable to the change and
the files say so where a reader will meet them: BitsBench.cs exercises
serialize.cs's raw BitWriter/BitReader and no generated code at all, yet its
rows still move 11% across the halves at sub-3% spread. That can only be
process-level — JIT method placement and cache state after a differently
shaped bench_mixed pass in the same process. An open question, not a result.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* flat.go: the bare-integer width test stops pretending to be a switch

staticcheck QF1002 on CI: a two-arm `switch { case w == 64: default: }` is an
if. Generated output is byte-identical — the regenerated tree is unchanged
and `make test` stays green across nine backends.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gafferongames added a commit that referenced this pull request Sep 1, 2026
* cs: the 1:1 piece↔item gate lands before lever F — closes #212

The flat codec re-derives #198-safety from every classifier returning
one piece per item or false; nothing asserted it and the package had no
tests. TestFlatFallbackReEmitsItemsNotPieces now pins the output
properties the invariant protects: a fixed array emits exactly one
element loop per direction, a single-scalar fallback (the one fallback
path that runs today) emits exactly one per-field serialize per
direction, and a nested struct at the run cap never names inner fields
on the outer base. Sabotage control: doubling the fallback's item
emission goes red (3 serializes detected); restored, green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* modernize: min() in the straddle builder

---------

Co-authored-by: Rowan Claude <rowan@mas-bandwidth.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gafferongames added a commit that referenced this pull request Sep 1, 2026
…3% of generated C++ (#237)

* js flat: single-word 32-bit write merge replaces the two-lane staging pair

Profile (node --prof, bench_mixed --quick): 51% of ticks inside
writeBenchMixedFlatProduction itself, 27% in V8 runtime calls it makes.
The per-field merge carried two data-dependent branches (the s<32 lane
split and the 64-bit flush) plus an s>0 guard; the single 32-bit staging
word does one shift-or, one literal add, and one flush branch storing one
word. Micro (interleaved, node 26, M2): 1.44x on a mixed-width group of
ten. On the leg (quick, production): write 1.35 -> 1.49 M msg/s (+10%),
round_trip 0.77 -> 0.82 (+6.5%). Wire bytes unchanged: an LSB-first
packer into consecutive little-endian 32-bit words emits the identical
byte stream; golden + 64-variant re-encode gates and test/js (both
modes, degenerate corpus included) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* js flat: write-side BigInt lane discipline — the DataView scratch consumes, nothing allocates

The profile's 27% V8-runtime bucket was BigInt: every BigInt '&', '>>',
asUintN-that-truncates and Number() on the write path allocates a BigInt
and calls the runtime, where a DataView setBigUint64 consumes the value
into plain memory without allocating. Micro (node 26, M2): 128-bit split
15.0 -> 84.3 M ops/s via two scratch stores + one shift (5.6x); sub-32
truncation 116.9 -> 718.9 M ops/s via scratch wrap (6.1x); the existing
<=64 scratch form measured already optimal (674 M ops/s) and keeps its
shape minus the redundant asUintN(64) wrap (ToBigUint64 IS that wrap;
for the >64 half, shift-then-wrap equals the asUintN(128) high half).

Reworked: emitWriteWideOffset takes the raw BigInt offset and routes all
four width classes through the scratch; flags <=32 wire mask through the
scratch instead of Number(asUintN); the >32-storage int32-range truncation
becomes SC.getInt32 (ToBigUint64 wrap + signed lane == asIntN(32) for
every BigInt). wideOffsetWidth deleted — nothing reduces ahead of the
store anymore. Checked-mode guards unchanged.

Leg (quick, production): write 1.49 -> 2.22 M msg/s (+49%), round_trip
0.82 -> 1.01 (+23%). Golden + 64-variant gates and test/js{,-ludicrous}
green in both modes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* js flat: wide (>64-bit) read assembly through the scratch — two halves, one shift-or

The read twin of the write-side lane discipline: the 96/128-bit
assembly built its BigInt from four BigInt(v) constructions chained
with three shift-ors (seven allocating steps); it now assembles each
64-bit half in the number domain through SC.setUint32 pairs and joins
two getBigUint64 loads with a single shift-or. Micro: 12.5 -> 40.3
M ops/s on the 128-bit assembly (3.2x, node 26, M2). Leg (quick,
production): round_trip 1.01 -> 1.06 M msg/s, derived read
1.86 -> 2.07; write untouched within spread. Gates green both modes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* js flat: chunked write merges — adjacent constant-width fields pack into one staged word

The family's grouping lever (go #198's runs, cs #208's word codec)
in the form this backend can hold: adjacent <=32-bit fields whose
value expressions are pure (field loads, hoisted element refs,
literals — never a mutable temp) pack into one 'v' with literal
RELATIVE shifts, then merge once. Relative offsets inside a chunk
are static even where the absolute cursor is dynamic, so the lever
reaches the loop bodies carrying most of the corpus wire (the
80-element stat array drops from two merges per element to one; the
entity body from ~14 to ~6). Chunks never cross a scope boundary or
a statement-form merge — wire order is emission order, enforced by
chunkFlush at every loop, branch, switch, align, and scratch-based
field; a temp-based piece (the >32-storage int32-range 'n') flushes
around itself so no piece outlives its inputs.

Leg (quick, production): write 2.22 -> 2.48 M msg/s (+12%),
round_trip 1.06 -> 1.11 (+5%). Wire identical: golden + 64-variant
gates, test/js{,-ludicrous} both modes, degenerate corpus — green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* js flat: string/bytes bodies merge four bytes per staged word

Measured first as a hand-patched ceiling on the generated leg (+3.6%
write), then landed in the emitter: the byte-copy loop for string(N)/
bytes(N) takes 4 bytes per 32-bit merge with a byte-at-a-time tail.
The counter lives in a brace scope so two byte fields can share a
nesting depth. Leg (quick, production): write 2.48 -> 2.55 M msg/s,
round_trip 1.11 -> 1.12. Gates green both modes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* js flat: re-pin the SOURCE goldens for the restructured emission

Source goldens only — testdata/wire is untouched by this round (the
stop-the-line invariant), and the full nine-language make test is
green against the unchanged wire pins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* bench/results: the js writes round's certification pair (family precedent: the pair rides with the round)

before 1dba683 (main) / after 3b3e6f6 (the round), one sitting 39s
apart, spreads 0.22-0.82%. The before leg reproduces the standing
ledger row within 0.3% — the pair's validity check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Rowan Claude <rowan@mas-bandwidth.com>
gafferongames added a commit that referenced this pull request Sep 1, 2026
…k-bytes path; 363% -> 221% of generated C++ (#238)

* dart: chunked write merges — 64-bit lanes, single-merge wide fields

Adjacent constant-width pure-expression fields pack into one staged word
with literal relative shifts (the family lever, #198/#208/#237), capped at
Dart's native 64-bit lane where the js number domain held it to 32. Wide
33..64-bit fields merge once instead of twice — their wire bits are
contiguous, so the bytes are identical by the concatenation argument.

Leg (quick, arm64 M2): write 1.894 -> 2.380 M msg/s (1.26x),
round_trip 0.974 -> 1.088 (1.12x). Golden gate + 64-variant re-encode
green; dart conformance suites green with asserts on.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* dart: bulk-bytes write path — the #165 lead

String/bytes bodies and bare [N]uint8 arrays stop paying one merge per
byte: static runs of 8 or fewer bytes queue as chunk pieces (joining
neighboring fields in one merge), longer and dynamic runs take 4 bytes
per merge with a byte tail. No alignment requirement: LSB-first merging
of the concatenation equals merging the bytes in sequence at any bit
position, so the fix reaches the unaligned loadout too.

Micro (AOT, M2): 4-byte grouping 2.33x, per-byte baseline 1.00x.
Leg (quick): write 2.380 -> 2.499 M msg/s (+5.0%), round_trip
1.088 -> 1.104 (+1.4%) — bulk is only 3.65% of bench_mixed's wire by
construction; the lever is for the real packets the issue names.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* dart: chunked read windows — one load and one tail branch per run, not per field

The reader kept reloading its 64-bit window at every field. A load now
folds the byte-interior shift in and guarantees 57 valid bits (in the
tail arm, every bounds-proven remaining bit), so consecutive
constant-width fields extract at literal relative shifts from one
window through a compile-time ledger. Wide 33..57-bit fields extract in
one masked read — their groups are contiguous on the wire. Short bare
byte arrays unroll into the same windows (#165's read twin). The ledger
invalidates at every dynamic bitsRead move and scope boundary.

Micro (AOT, M2): chunked window 1.35x over per-field reload.
Leg (quick): round_trip 1.104 -> 1.367 M msg/s (+23.8%), write flat;
derived read 2.01 -> 3.03 M msg/s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* dart: grouped counted loops — k elements share one lane per iteration

A counted array of static-width elements unrolls k = lane/elemBits
elements per iteration (64-bit chunk lane on the write side, 57-bit
window on the read side), with a remainder loop. The chunk accumulator
and window ledger span the unrolled elements, so short elements merge
together and extract from one load — bench_mixed's 80-element stat
array drops from one merge and one window load per element to one per
three. Hoisted element refs get unique names (e0, e0g1, e0g2) so the
unroll shares one scope; correctness rides the same ledger/flush
machinery as straight-line code.

Bounded first by a hand-patched generated leg (+10% round_trip on the
read side alone). Leg (quick): write 2.493 -> 2.856 M msg/s (+14.6%),
round_trip 1.367 -> 1.592 (+16.5%).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* dart: re-pin source goldens for the chunked emission

Wire goldens untouched — testdata/wire byte pins hold in every leg; the
re-pin is the SOURCE form only (SPEC §3.1's deliberate-change path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* dart: range-over-int in the emitter's unroll loops (modernize)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* bench/results: the dart round's certification pair

before 1,900,191/972,818 (reproducing the standing ledger row within
0.6%), after 2,857,586/1,591,336; spreads 0.07-2.28%; one sitting,
corpus 6b213fbfa1a03a99.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Rowan Claude <rowan@mas-bandwidth.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant