Repository navigation
llm-conformance: named conformance gaps are zero; the rig refuses any named gap, ceiling or SKIP row [final; stacked on #969 + #958] - #979
Merged
Conversation
…s, and an orphan named gap is refused; 4 of the 5 named gaps close
Owner rule 22 (2026-10-14): named conformance gaps are zero; a gap is a defect to fix. Two of the
named gaps in testing/llm-conformance/named-gaps.json were the instrument's defects, not busbar's.
anthropic-ping-event (4 rows, llm|anthropic|{bedrock,cohere,openai,responses}|request|ok_stream#response).
Anthropic's stream contract is two documents: the OpenAPI MessageStreamEvent union (six members)
and the streaming guide, which documents `ping` ("any number of ping events") and an in-stream
`error` event carrying ErrorResponse. 1.5.5 sends the ping after message_start on every translated
stream (golden/1.5.5/cells/llm__anthropic__openai__request__ok_stream.json). validate.py now judges
each documented event against its own shape (ping: exactly {"type":"ping"}; error: the spec's
ErrorResponse) under its own SSE event name, and every other type against the union. No byte
changes (ARCHITECT ruling 2026-10-14, CONFORMANCE-GAPS B).
gemini-vertex-{traffic-type,create-time,grounding-chunk-domain} (3 entries). They named rows of
testing/llm-conformance/fixtures/gemini-vertex-recording. That fixture is not in cells.json or the
selftest, so no gate judged it, and its bodies are byte-identical copies of
crates/busbar-plane-llm/src/codec/tests/proto/golden/vendor/{resp,stream}_g2g_vertex_grounding.*,
which the codec tests keep. The bytes are Vertex AI's. Vertex's own aiplatform v1beta1 discovery
document declares GenerateContentResponse.createTime, UsageMetadata.trafficType and
GroundingChunkWeb.domain, so these were Gemini-API-schema-vs-Vertex-bytes, not a gap. Fixture and
entries removed.
The loader now refuses a `gaps` entry naming a row the production cell universe
(testing/shadow-oracle/cells.json) does not owe. Such an entry can never forgive anything and can
never go stale; that is how the three vertex entries sat unseen.
RED -> GREEN:
- selftest (v): the 1.5.5 cross-dialect Anthropic event sequence, ping included, in a new fixture
fixtures/anthropic-stream-recording. Old validate.py: FAIL `sse[1]:/ discriminator: 'type':
'ping' is not one of`. New: PASS, no SKIP, no named gap.
- (v2) a documented error event is GREEN.
- (v3) an undocumented type, (v4) a ping under another event name and (v5) a ping with an extra
member are each RED.
- (m3) an orphan entry is REFUSED by name.
- selftest GREEN 32/32 locally (python + bash only).
The remaining named gap is bedrock-service-quota-exceeded (code fix stacked on this). `expected`
is NOT raised: the run observes 17 SKIP rows against 16 declared until the recorder cells and that
fix land.
…error Converse declares; the last named conformance gap closes Owner rule 22 (2026-10-14): named conformance gaps are zero; a gap is a defect to fix in code. ARCHITECT ruling 2026-10-14 (CONFORMANCE-GAPS C). The gap: llm|bedrock|bedrock|request|over_budget_total#response failed `error.declared`. busbar answered 400 `ServiceQuotaExceededException`, the same as v1.5.5 (golden/1.5.5/cells/llm__bedrock__bedrock__request__over_budget_total.json). That exception is not in the `errors` of Converse or ConverseStream in the pinned botocore bedrock-runtime model (e5c8122617a2; re-fetched and checked); only InvokeModel declares it. The fix: - crates/busbar-plane-llm/src/codec/bedrock/mod.rs maps insufficient_quota to `ThrottlingException` (AWS: "Your request was denied due to exceeding the account quotas for Amazon Bedrock", httpStatusCode 429), and its quota_exceeded_status is 429. - crates/busbar-plane-llm/src/refusal.rs drops the bedrock OVER_BUDGET 400 row, so the class default 429 answers, as on the other five dialects. The message, Retry-After and the charge (none) are unchanged. Not money. Registered as accepted difference "C-22 bedrock budget refusal is a declared Converse error" (kind breaking: status/body/headers, one cell; premise 429 + __type), with its CHANGELOG line. The parity bindings PB-19/PB-22/PB-79 now say 1.5.5's Bedrock 400 is 429 in 1.6.0 and cite C-22: qa/parity-bindings.md, qa/design-bindings.json and design_bindings/tables.rs, the same strings in all three. Tests: - llm_refusal_statuses reads that register entry for the one cell, so it can forgive nothing the register does not. - The legacy engine tests' bedrock 400 assertions move to 429/ThrottlingException. RED: phase1_5_relocated_tests::every_bedrock_error_kind_is_a_converse_declared_exception. Every kind the writer can be handed must answer one of the nine Converse-declared exceptions. On predev, insufficient_quota -> ServiceQuotaExceededException fails it. named-gaps.json `gaps` is now empty.
…, against the in-stream error shape each vendor defines Owner rule 22 (2026-10-14): named conformance gaps are zero. ARCHITECT ruling 2026-10-14 (CONFORMANCE-GAPS D). These are runner defects that the stream_upstream_error cells (busbar-release #369) and the stream_upstream_cut cells (row 155, #893) expose the moment they record. Each would read as a FAIL on rows that are correct. - stream_upstream_error and stream_upstream_cut ask for a stream and open 2xx. validate.py treated only `ok_stream` as a stream outcome, so both read "non-stream request answered with a stream", and bedrock's request was judged as ConverseRequest. The cut outcome was not classified at all ("outcome ... is not classified"). STREAM_OUTCOMES now names all three, and the cut joins OUTCOME_STATUS_EXEMPT. - OpenAI chat and Gemini define their in-stream error only in the official SDK. Pinned citations: - openai-python src/openai/_streaming.py at 47b9e4e95e44: any `data:` object with a truthy `error` raises APIError(body=data["error"]). Shape: the spec's ErrorResponse. - python-genai google/genai/_api_client.py at f1499bb86c70: a chunk starting `{"error":` raises. Shape: google.rpc.Status, the transcribed schemas/google-rpc-status.json. The event is judged against that shape and must be last. A stream that ends on it owes no [DONE]. - Bedrock's `:exception-type` is the event-stream union's MEMBER name (internalServerException). The judge looked it up as a shape name, so every legitimate exception frame failed. It now resolves through ConverseStreamOutput's members and requires an error member. RED -> GREEN: new fixture fixtures/stream-error-recording (the mock's openai/gemini/bedrock stream-error answers and an anthropic first-frame cut, as a same-dialect relay hands them on). - On #955's validate.py: 4 of 4 response rows FAIL (outcome not classified; non-stream request answered with a stream; `error` not declared; exception type not a shape). - Here, (x): 8/8 rows PASS. - (x2) a chunk after the error, (x3) a malformed Status, (x4) a non-member exception and (x5) an openai stream ending on neither the error nor [DONE] are each RED. - Selftest GREEN 37/37 locally (python + bash only).
… named gap, any declared ceiling and any SKIP row Owner rule 22 (2026-10-14): named conformance gaps must be 0; a non-zero count means a larger issue, and a gap is a defect to fix, never a re-baseline. #619 (expected 16 -> 21) was closed for that reason. ARCHITECT ruling 2026-10-14 (CONFORMANCE-GAPS E). This lands last, after: - #955: Anthropic documented events, orphan entries; - #958: Bedrock 429 ThrottlingException; - #969: stream outcomes; - the recorder cells (busbar-release #369 and its busbar pin PR). - named-gaps.json: `gaps: []`, `expected: 0`, `accepted: []`, with a header that says why. - validate.py refuses ANY `gaps` entry at load ("owner rule 22"), so nothing is judged. - run.sh's reconciliation row `gate|llm-conformance|named-gaps` is RED on: - any `accepted` entry; - any `expected` other than 0; - every SKIP row the run produced, naming the row and its reason. The validator's SKIP paths now all read as RED: an unrecorded cell, an eventstream with no raw body, an error status with no schema. RED -> GREEN in selftest.sh; each of these was GREEN on #969's runner: - (f2) two SKIPs signed for in `accepted` with `expected: 2`; - (i) a well-formed gap over a real violation; - (i2) a gaps-only list. Now (f2) is RED, naming `accepted` and `expected`; (i) and (i2) are REFUSED at load. (j) and (k) are refused at load. (f), (f3) and (j2) are RED and now name rule 22. Selftest GREEN 37/37 locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRAFT, and it lands LAST (rule 19). It is stacked on #969 (and through it #955) and merges #958. It also waits on the recorder cells: busbar-release #369 plus the busbar pin and golden PR. Until those land, production still produces 18 recorder SKIP rows, and this PR makes them RED, which is the intent of owner rule 22.
Changes
named-gaps.json:gaps: [],expected: 0,accepted: [].validate.py: refuses anygapsentry at load.run.sh: the reconciliation row is RED on anyacceptedentry, on anyexpectedother than 0, and on every SKIP row.RED. Selftest cases (f2), (i) and (i2) were GREEN on #969's runner. Here (f2) is RED and (i) and (i2) are refused. Selftest GREEN 37/37 locally.