Skip to content

BitWriter: single-word 32-bit staging — family bits 1.18x, writeBytes 1.37x - #11

Merged
gafferongames merged 2 commits into
mainfrom
bitwriter-single-word
Sep 1, 2026
Merged

BitWriter: single-word 32-bit staging — family bits 1.18x, writeBytes 1.37x#11
gafferongames merged 2 commits into
mainfrom
bitwriter-single-word

Conversation

@gafferongames

@gafferongames gafferongames commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Closes #10. The runtime half of schema's js READS round; its twin is
mas-bandwidth/schema#244, and the two PRs pair (see "Pairing").

BitWriter staged into a two-lane 64-bit scratch (#scratchLo / #scratchHi,
#scratchBits in [0,63]) and flushed a pair of little-endian 32-bit words
every 64 bits. Every merge carried a data-dependent lane split (s < 32), a
nested s > 0 guard inside it, and the 64-bit flush branch. One 32-bit
staging word
with #scratchBits in [0,31] writes the same wire in one
shift-or, one add and one branch storing one word — the form schema's
generated flat tier retired the two-lane staging for in schema#237.

Headline (BENCH-STANDARD.md §1.4 family bits: the 16-width table,
227 bits/group, over a 65536-byte buffer — the estate's one bitpacker
workload; max of 7 rounds, three invocations):

path ratio A/A null
family bits 1.1743, 1.1809, 1.1816 ±1.1%
writeBytes 1.3594, 1.3605 ±0.2%

(Before the review fix below these read 1.1832 / 1.2177 / 1.1825 and 1.3694 /
1.3801 / 1.3871; the flush's now-unconditional pairing store costs one branch
per packet. The figures above are the ones that describe the branch.)

The instrument — A/A NULL before any number

A paired instrument loads two RUNTIME checkouts as arms, each arm's timing
loop built from its own source text so the two never share a
SharedFunctionInfo or a feedback vector. One path per invocation, arm order
rotated by round parity, max of 7 measured rounds after two discarded warmups
in both orders. Every pair run gates before it times anything: both arms
write the same 65,518 bytes byte-identically, each reads back exactly what it
wrote, and each arm's READER re-reads the other arm's buffer.

The A/A null — both arms origin/main, the second a copy on a different path
— is 0.9965 / 0.9973 / 1.0113 on bits and 0.9978 / 0.9979 on bytes.

Byte-equivalence, stated

An LSB-first packer into consecutive little-endian 32-bit words emits the
identical byte stream either way: the old pair of setUint32(base) and
setUint32(base+4) IS two consecutive 32-bit words. The invariant is that the
staging word's bits at and above scratchBits are zero; value << s
contributes bits [s, min(s+bits, 32)) — a JavaScript shift drops the rest —
and the flush recovers the dropped high bits as the next word's low bits.
spill === 0 is the single case where the recovery shift would be 32, a no-op
in JavaScript rather than zero, so it is taken as a literal zero. With s in
[0,31] no shift here is ever the mod-32 no-op the two-lane form needed guards
for.

Scope, exactly as the issue named it

  • The four fused copiesCheckedBitWriter.writeBits / .tryWriteBits,
    ProductionBitWriter.writeBits / .tryWriteBits. The file's own KEEP THE
    FOUR COPIES IDENTICAL rule holds: the edit was applied by one script that
    refuses unless it finds exactly four.
  • The flush and reset invariants — see "The review condition" below,
    which is where the flush's memory guarantee was actually made true.
  • writeBytes, whose "the scratch is empty at a word boundary" argument
    now holds every 32 bits rather than every 64: head and tail through the
    scratch drop from ≤7 bytes to ≤3, and the bulk-copy unit is 4 bytes. That is
    where its 1.37x comes from.

The review condition (#11): the flush now pairs its span for EVERY bit count

The pairing store sat inside if (this.#scratchBits !== 0). That guard is
false exactly when the last write ended on a 32-bit boundary
— the merge had
already stored the word and left scratchBits at 0 — so for
bitsWritten ≡ 32 (mod 64) the flush did nothing at all and the second half
of the 8-byte span kept whatever the caller's buffer held before. main
always wrote the full span.

Payload bytes were never affected. The exposure is a reused dirty buffer whose
consumer reads past bytesWritten() — 4 stale bytes on a transmitted
8-byte-aligned buffer — and, just as much, the class doc's "bytes past the end
of the written data are only ever written as zeros" being false.

The fix: the store moves OUTSIDE the guard. After the guard the cursor is
at word ceil(bitsWritten / 32) on both paths — the partial word was stored,
or there was none to store — so an odd index means the span's second half has
never been written, either way. The store does not advance the cursor, so a
second flushBits is a no-op rather than a walk off the end. Both class
copies moved.

Reproduced before it was fixed, with origin/main as the reference: a
tail oracle writes k bits into a 0xff-prefilled buffer for every k in [0, 384]
and compares the WHOLE buffer against the same write through main. Unfixed:
6 divergent counts — k = 32, 96, 160, 224, 288, 352 — every one past
bytesWritten(), never payload.
Fixed: 0 divergent across all 385.

The missing test class is now in the repo.
test/writer-tail-span.test.js pins k = 32 and k = 96 by name, sweeps every k
in [0, 384], covers the writeBytes bulk-copy cursor, and pins flush
idempotence. It asserts the full shape — packet bytes, zeros to
8*ceil(k/64), untouched 0xff beyond — so over-zeroing fails it as loudly as
under-zeroing. It joins production-tests.mjs under that file's stated
membership rule (it asserts no dev-only caller validation, and both class
copies carry the flush). Negative control: with the store back inside the
guard the file reports 9 failures and exits 1 in BOTH modes.

Class and method documentation moved with the code — present state only, no
archaeology about the shape that was there before.

Gates

  • node --test290 tests, 0 fail (281 pass, 9 skipped), including the
    golden-wire battery and the new tail-span suite.
  • node production-tests.mjs129 tests, 0 fail.
  • STANDARD.md untouched, so the vendored-copy job is unaffected.

Pairing

schema's test/js and test/js-ludicrous (both NODE_ENV modes) and its bench
leg all run green against this checkout through schema's §3.5 SERIALIZE_JS
override, at corpus_id 6b213fbfa1a03a99. The two halves do not overlap and
schema#244's certification pair shows it: the generated flat tier imports
nothing, so this lever adds nothing to the bench_mixed round_trip row
(1.2126x emitter-only, 1.2123x paired), and schema's emitter lands nothing on
family bits.

Honest note on that certification leg: its before and emitter-only after
bitpacker write rows printed at 47.7% and 49.3% spread — over BENCH-STANDARD
§2.3's 40% INVALID threshold — so no ratio is taken from them there. This PR's
numbers stand on the paired instrument above, whose spreads are 0.13–1.13%;
the paired certification leg's own bitpacker write row (7,024 at 0.08% spread)
is consistent with them.

Landing follow-on, named not done: schema CI pins SERIALIZE_JS_TAG: v1.1.0. Bumping that pin after this lands is a separate change in schema.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

All contributors have signed the CAA. Thank you.
Posted by the CLA Assistant Lite bot.

rowan-claude and others added 2 commits September 2, 2026 02:10
…ratch

Every merge carried a data-dependent lane split (s < 32), a nested s > 0
guard inside it, and a 64-bit flush branch storing two words. One 32-bit
staging word with scratchBits in [0,31] does the same wire in one shift-or,
one add and one branch storing one word.

Byte-equivalence: an LSB-first packer into consecutive little-endian 32-bit
words emits the identical byte stream — the old pair of setUint32(base) and
setUint32(base+4) IS two consecutive 32-bit words. The invariant is that the
staging word's bits at and above scratchBits are zero; `value << s`
contributes bits [s, min(s+bits, 32)), a JavaScript shift drops the rest, and
the flush recovers the dropped high bits as the next word's low bits. spill
=== 0 is the single case where the recovery shift would be 32 — a no-op in
JavaScript, not zero — so it is taken as a literal zero.

Scope: the four fused copies (the file's KEEP THE FOUR COPIES IDENTICAL rule
holds, applied by one scripted edit that refuses unless it finds exactly
four), the flush and reset invariants, and writeBytes, whose empty-scratch
word boundary is now every 32 bits rather than every 64 — head and tail
through the scratch drop from 7 bytes to 3. flushBits keeps the memory
behaviour: it stores the pending word and, when that lands on an odd 4-byte
word, zeroes the next, so the bytes past the written data are still only ever
written as zeros over the same 8-byte span and the buffer-length contract is
unchanged.

Paired instrument, arm rotation, max of 7 rounds, three invocations, the
BENCH-STANDARD §1.4 family-bits workload: 1.1832 / 1.2177 / 1.1825 against a
+/-1.1% A/A null. The writeBytes path 1.3694 / 1.3801 / 1.3871 against
+/-0.2%. Wire gate every run: 65,518 bytes byte-identical, read-back and
cross read-back exact.

node --test 285/0 fail, node production-tests.mjs 124/0 fail, both including
the golden-wire battery. schema's test/js and its bench leg run green against
this checkout through the SERIALIZE_JS override.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…artial word

The pairing store sat inside `if (this.#scratchBits !== 0)`. That guard is
false exactly when the last write ended on a 32-bit boundary — the merge had
already stored the word and left scratchBits at 0 — so for bitsWritten ≡ 32
(mod 64) the flush did nothing and the second half of the 8-byte span kept
whatever the caller's buffer held before. main always wrote the full span.

Payload bytes were never affected; the exposure is a reused dirty buffer whose
consumer reads past bytesWritten() (4 stale bytes on a transmitted 8-byte
aligned buffer), and the class doc's "bytes past the end of the written data
are only ever written as zeros" being false.

The store moves outside the guard. After the guard the cursor is at word
ceil(bitsWritten / 32) on both paths — the partial word was stored, or there
was none to store — so an odd index means the span's second half has never
been written either way. The store does not advance the cursor, so a second
flushBits is a no-op rather than a walk off the end. Both class copies moved.

Reproduced first, against origin/main as the reference: a tail oracle writes k
bits into a 0xff-prefilled buffer for every k in [0,384] and compares the whole
buffer against main. Unfixed, 6 divergent counts — k = 32, 96, 160, 224, 288,
352 — every one past bytesWritten() and never payload. Fixed, 0 of 385.

test/writer-tail-span.test.js pins k=32 and k=96 by name, sweeps every k in
[0,384], covers the writeBytes bulk-copy cursor and flush idempotence, and
asserts the full shape (packet, zeros to 8*ceil(k/64), untouched beyond) so
over-zeroing fails as loudly as under-zeroing. It joins production-tests.mjs
under that file's membership rule: it asserts no dev-only caller validation,
and both class copies carry the flush. Negative control: with the store back
inside the guard it reports 9 failures and exits 1 in both modes.

The lever's own numbers move with it and the ROUND-LOG now carries these:
family bits 1.1743 / 1.1809 / 1.1816 (was 1.1832 / 1.2177 / 1.1825), writeBytes
1.3594 / 1.3605 (was 1.3694 / 1.3801 / 1.3871) — one unconditional branch per
packet is the difference. node --test 290/0 fail, production 129/0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gafferongames
gafferongames marked this pull request as ready for review September 1, 2026 16:12
@gafferongames

Copy link
Copy Markdown
Contributor Author

Adversarially reviewed with schema#244: the byte-equivalence algebra verified, 65,536-byte wire identity at zero differing bytes, all four staging copies sabotage-covered by exactly their own suites, and the review's one condition — the odd-word tail zeroing at bitsWritten ≡ 32 (mod 64) — fixed with the previously-missing test class (bytes past bytesWritten on a dirty buffer, k-sweep 0/385 divergent, negative control 9 red). The lever's numbers were honestly republished downward post-fix (bits 1.17-1.18x, writeBytes 1.36x — far over the ±1.1%/±0.2% nulls). Verdict: MERGE.

@gafferongames
gafferongames merged commit 1b7ecb3 into main Sep 1, 2026
8 checks passed
@gafferongames
gafferongames deleted the bitwriter-single-word branch September 1, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BitWriter.writeBits carries two-lane 64-bit staging — the single-word form measured 1.44x in schema's js round

2 participants