Skip to content

test: add portable renderer conformance checks - #188

Draft
hallerite wants to merge 3 commits into
codex/laguna-bridge-separatorfrom
codex/renderer-conformance
Draft

hallerite wants to merge 3 commits into
codex/laguna-bridge-separatorfrom
codex/renderer-conformance

Conversation

@hallerite

@hallerite hallerite commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

Exercise renderer contracts through pinned reference encoders and the public generate client. Existing single-turn checks can miss errors at the actual generation boundary, across tool cycles, or when a bridge reuses sampled tokens.

Depends on #206 (preserve string tool arguments) and #207 (Laguna bridge separators). This PR is stacked on #207 so its own diff contains the conformance tests and their CI integration.

Changes

  • Cover all 31 renderer formats and 47 catalog checkpoints using ci: split offline checks from pinned model parity tests #118's shared asset pins. Execute DeepSeek V4's model-supplied Python encoder at the pinned revision after verifying its source hash. Harmony uses the existing explicit analysis-channel profile of the official encoder.
  • Add 1,159 cases for reference coverage, exact generation prefixes, three-turn client conversations, token-budget stops, and two successive tool cycles with ordinary, empty, and whitespace/Unicode arguments.
  • Reuse parsed public-client messages in subsequent turns. Check completion IDs/log-probabilities and require bridges to preserve sampled tokens and append the independently encoded boundary suffix; declined bridges fall back to a reference-checked render.
  • Strengthen the existing transcript parity matrix with render/render_ids agreement and attribution-array checks instead of adding a duplicate transcript matrix. Bound the tokenizer cache and release assets between modules.
  • Run model-backed cases in the existing pinned text CI lane. Allow that workflow to run on stacked codex/** PR bases. No vLLM installation or GPU sampling is required.

Validation

  • Portable conformance plus the strengthened transcript parity matrix: 9,599 passed, with no skips or xfails, using cached pinned assets and model downloads disabled.
  • Complete suite with model downloads disabled: 15,740 passed, 118 existing skips.
  • Offline lane with an empty asset cache and network sockets blocked: 214 passed, 1 existing skip. Collection audit: all 14,654 previous cases retained; 15,858 total cases across exhaustive offline, text, and multimodal lanes.
  • Ruff, formatting, pre-commit, and ty check renderers passed.
  • GitHub Actions: all four offline jobs, multimodal parity, and style/type checks passed. Text parity is still running.

Base automatically changed from codex/fix-reasoning-template-parity to main October 12, 2026 20:38
@hallerite
hallerite changed the base branch from main to codex/laguna-bridge-separator October 29, 2026 02:49
@hallerite hallerite changed the title Add black-box renderer conformance and pinned vLLM audits test: add portable renderer conformance checks Oct 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant