Repository navigation
llm-conformance: failing/cut streams judged as streams; openai/gemini in-stream errors per official SDK; bedrock exception member names [stacked on #955] - #969
Merged
Conversation
…s, and an orphan named gap is refused; 4 of the 5 named gaps close
Owner rule 22 (2026-10-14): named conformance gaps are zero; a gap is a defect to fix. Two of the
named gaps in testing/llm-conformance/named-gaps.json were the instrument's defects, not busbar's.
anthropic-ping-event (4 rows, llm|anthropic|{bedrock,cohere,openai,responses}|request|ok_stream#response).
Anthropic's stream contract is two documents: the OpenAPI MessageStreamEvent union (six members)
and the streaming guide, which documents `ping` ("any number of ping events") and an in-stream
`error` event carrying ErrorResponse. 1.5.5 sends the ping after message_start on every translated
stream (golden/1.5.5/cells/llm__anthropic__openai__request__ok_stream.json). validate.py now judges
each documented event against its own shape (ping: exactly {"type":"ping"}; error: the spec's
ErrorResponse) under its own SSE event name, and every other type against the union. No byte
changes (ARCHITECT ruling 2026-10-14, CONFORMANCE-GAPS B).
gemini-vertex-{traffic-type,create-time,grounding-chunk-domain} (3 entries). They named rows of
testing/llm-conformance/fixtures/gemini-vertex-recording. That fixture is not in cells.json or the
selftest, so no gate judged it, and its bodies are byte-identical copies of
crates/busbar-plane-llm/src/codec/tests/proto/golden/vendor/{resp,stream}_g2g_vertex_grounding.*,
which the codec tests keep. The bytes are Vertex AI's. Vertex's own aiplatform v1beta1 discovery
document declares GenerateContentResponse.createTime, UsageMetadata.trafficType and
GroundingChunkWeb.domain, so these were Gemini-API-schema-vs-Vertex-bytes, not a gap. Fixture and
entries removed.
The loader now refuses a `gaps` entry naming a row the production cell universe
(testing/shadow-oracle/cells.json) does not owe. Such an entry can never forgive anything and can
never go stale; that is how the three vertex entries sat unseen.
RED -> GREEN:
- selftest (v): the 1.5.5 cross-dialect Anthropic event sequence, ping included, in a new fixture
fixtures/anthropic-stream-recording. Old validate.py: FAIL `sse[1]:/ discriminator: 'type':
'ping' is not one of`. New: PASS, no SKIP, no named gap.
- (v2) a documented error event is GREEN.
- (v3) an undocumented type, (v4) a ping under another event name and (v5) a ping with an extra
member are each RED.
- (m3) an orphan entry is REFUSED by name.
- selftest GREEN 32/32 locally (python + bash only).
The remaining named gap is bedrock-service-quota-exceeded (code fix stacked on this). `expected`
is NOT raised: the run observes 17 SKIP rows against 16 declared until the recorder cells and that
fix land.
…, against the in-stream error shape each vendor defines Owner rule 22 (2026-10-14): named conformance gaps are zero. ARCHITECT ruling 2026-10-14 (CONFORMANCE-GAPS D). These are runner defects that the stream_upstream_error cells (busbar-release #369) and the stream_upstream_cut cells (row 155, #893) expose the moment they record. Each would read as a FAIL on rows that are correct. - stream_upstream_error and stream_upstream_cut ask for a stream and open 2xx. validate.py treated only `ok_stream` as a stream outcome, so both read "non-stream request answered with a stream", and bedrock's request was judged as ConverseRequest. The cut outcome was not classified at all ("outcome ... is not classified"). STREAM_OUTCOMES now names all three, and the cut joins OUTCOME_STATUS_EXEMPT. - OpenAI chat and Gemini define their in-stream error only in the official SDK. Pinned citations: - openai-python src/openai/_streaming.py at 47b9e4e95e44: any `data:` object with a truthy `error` raises APIError(body=data["error"]). Shape: the spec's ErrorResponse. - python-genai google/genai/_api_client.py at f1499bb86c70: a chunk starting `{"error":` raises. Shape: google.rpc.Status, the transcribed schemas/google-rpc-status.json. The event is judged against that shape and must be last. A stream that ends on it owes no [DONE]. - Bedrock's `:exception-type` is the event-stream union's MEMBER name (internalServerException). The judge looked it up as a shape name, so every legitimate exception frame failed. It now resolves through ConverseStreamOutput's members and requires an error member. RED -> GREEN: new fixture fixtures/stream-error-recording (the mock's openai/gemini/bedrock stream-error answers and an anthropic first-frame cut, as a same-dialect relay hands them on). - On #955's validate.py: 4 of 4 response rows FAIL (outcome not classified; non-stream request answered with a stream; `error` not declared; exception type not a shape). - Here, (x): 8/8 rows PASS. - (x2) a chunk after the error, (x3) a malformed Status, (x4) a non-member exception and (x5) an openai stream ending on neither the error nor [DONE] are each RED. - Selftest GREEN 37/37 locally (python + bash only).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STACKED on #955, DRAFT until #955 lands (rule 19).
Owner rule 22 (2026-10-14): named conformance gaps are zero. ARCHITECT ruling 2026-10-14 (CONFORMANCE-GAPS D). These runner defects would mis-FAIL correct rows the moment stream_upstream_error (busbar-release #369) and stream_upstream_cut (#893) record.
stream_upstream_errorandstream_upstream_cutare now stream outcomes. The cut outcome is classified (exempt from a side of 2xx).[DONE]is owed after it:_streaming.py@47b9e4e9, judged as the spec's ErrorResponse._api_client.py@f1499bb8, judged as google.rpc.Status.:exception-typeresolves through ConverseStreamOutput's member names.RED. New fixture
fixtures/stream-error-recording. On #955's validator all 4 response rows FAIL; here they PASS. Selftest cases (x2)-(x5) are each RED. Selftest GREEN 37/37 locally.