Skip to content

fix(ceremony): read JSON fields with the whitespace beside structure removed - #37

Merged
xgreenx merged 1 commit into
mainfrom
fix/json-whitespace-normalized
Sep 10, 2026
Merged

xgreenx merged 1 commit into
mainfrom
fix/json-whitespace-normalized

Conversation

@xgreenx

@xgreenx xgreenx commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

An alternative to #36 for the same defect: GitHub pretty-prints /user for the media type the profile pins (accept: application/vnd.github+json is the trigger, checked live; without it the JSON is compact), so readers matching "login":" and "id": byte for byte refuse every real identity read.

The approach

Normalize once, then keep every reader's exact template, the way normalizeHeaderBytes already serves the request head. CeremonyFields.normalizeJsonBytes removes a run of JSON whitespace (0x20 0x09 0x0a 0x0d) only where it touches a structural byte, : , { } [ ], which is exactly where JSON puts insignificant whitespace. A run between two tokens stays: 123 4 does not read as 1234, and a space after digits is still the byte the terminator check judges.

  • tryJsonString and tryJsonInteger normalize their range first; the cross-range duplicate counts in the base normalize the concatenation.
  • Bearer framing: the revealed range ending where the commitment starts is the anchor and its normalized bytes must end with "access_token":"; the prefix may appear at most once across everything revealed; a prefix split across ranges is refused, since no single range carries it.
  • Stateless by design: a reader with a notion of "inside a string" is one a prover desynchronises by cutting a revealed range mid-value. No needle can be manufactured by removing whitespace, since one needs unescaped quotes, and nothing read carries whitespace beside a structural byte inside its value.

Compared with #36: the same acceptance and the same refusals, one normalization instead of a tolerant matcher at each reader, the templates unchanged, and 20 fewer lines in the verifier.

Tests

458 pass, up from 450. CeremonyFields.t.sol: pretty-printed members read, a duplicate in another spelling is still a duplicate, only the four JSON bytes are removed (a vertical tab is not), 123 } reads and 123 4 is refused at the space. CeremonyAttestation.t.sol: a bearer framed behind whitespace, one byte off refused, a second compact prefix ambiguous, a prefix split across ranges refused. GitHubPlatformVerifier.t.sol: the response GitHub actually serves reads as 583231 / octocat; a second login spelled with spaces is FieldNotUnique; test_rejectsAnIdWithoutAStructuralTerminator now uses 583231 4, since 583231 } is JSON.

Sizes stay well under the 24,576 limit, measured on this branch: GitHubPlatformVerifier 16,636, XPlatformVerifier 15,728, GoogleJwtRoots 14,725 runtime bytes.

The prover side reveals the whitespace inside the member at its offsets: libid-org/libid-rs follows in its own PR. Specification wording: libid-org/libID#32 covers the tolerance; if this shape is preferred, its "MUST NOT reconstruct by dropping whitespace" sentence should say "across ranges", since dropping it within one range is what this does.

🤖 Generated with Claude Code

…removed

GitHub pretty-prints /user for the media type the profile pins: a
newline and two spaces before every member, a space after every colon.
The readers matched compact delimiters byte for byte, so every real
identity read was refused.

Rather than teach each reader the places whitespace may sit, the bytes
are normalized once before any reader looks, the way
`normalizeHeaderBytes` already serves the request head: a run of JSON
whitespace that touches `:` `,` `{` `}` `[` or `]` is removed, which is
exactly where JSON puts insignificant whitespace, and a run between two
tokens stays, so `123 4` does not read as `1234` and a trailing space
is still the byte the terminator check judges. Stateless on purpose:
a reader that tracks strings is one a prover desynchronises by cutting
a range mid-value. Every reader keeps its one template, and a member
in any spelling is the same member, so a duplicate spelled with spaces
is counted as one.

The bearer framing follows: the revealed range ending where the
commitment starts is the anchor, its normalized bytes end with the
prefix, and the prefix appears at most once across everything revealed.
One range, never a join -- a prefix assembled across a seam is one the
platform never wrote.

The prover reveals the whitespace inside the member at its offsets;
libid-rs follows in its own change. An alternative to #36, which
reaches the same acceptance with a tolerant matcher per reader.

Assisted-by: Claude Opus 5
Signed-off-by: xgreenx <xgreenx9999@gmail.com>
@xgreenx xgreenx self-assigned this Sep 10, 2026
@xgreenx
xgreenx merged commit 56f0b10 into main Sep 10, 2026
11 checks passed
xgreenx added a commit that referenced this pull request Sep 10, 2026
#37 landed the JSON whitespace normalization on main and touched the
same three files. The two library files merged on their own; the
GitHub suite needed the two sides of one hunk kept in order, with the
closing brace git had moved past the conflict put back. 477 tests,
both sets.

Assisted-by: Claude Opus 5
Signed-off-by: xgreenx <xgreenx9999@gmail.com>
xgreenx added a commit to libid-org/libID that referenced this pull request Sep 11, 2026
GitHub pretty-prints `/user` for the media type the profile pins, so the
compact delimiters this specification spells matched nothing it serves.
REQ-COMMON-19F fixes what libid-org/libID-contracts#37 does: the Platform
Verifier removes the JSON whitespace touching a structural byte and matches,
counts and reads over what is left; the Implementation reveals a member as
the wire carried it, whitespace inside, and never commits that whitespace
with a bearer; the compact spellings name the member after removal.
TEST-COMMON-10A lists the vectors. REQ-PLAT-51 judges GitHub's terminator
after that removal and REQ-PLAT-60 keeps the whitespace in the reveal.

Carries the vectors of #32 in the form the merged verifier
implements.

Co-authored-by: Wondertan <hlibwondertan@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: xgreenx <xgreenx9999@gmail.com>
Wondertan added a commit to libid-org/libID that referenced this pull request Sep 13, 2026
GitHub pretty-prints `/user` for the media type the profile pins, so the
compact delimiters this specification spells matched nothing it serves.
REQ-COMMON-19F fixes what libid-org/libID-contracts#37 does: the Platform
Verifier removes the JSON whitespace touching a structural byte and matches,
counts and reads over what is left; the Implementation reveals a member as
the wire carried it, whitespace inside, and never commits that whitespace
with a bearer; the compact spellings name the member after removal.
TEST-COMMON-10A lists the vectors. REQ-PLAT-51 judges GitHub's terminator
after that removal and REQ-PLAT-60 keeps the whitespace in the reveal.

Carries the vectors of #32 in the form the merged verifier
implements.

Co-authored-by: Wondertan <hlibwondertan@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: xgreenx <xgreenx9999@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant