Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -559,6 +559,9 @@ event-feed-fixtures-check:
--check-metaschema conformance/event-feed/schema.json
uvx --from 'check-jsonschema==$(CHECK_JSONSCHEMA_VERSION)' check-jsonschema \
--schemafile conformance/event-feed/schema.json conformance/event-feed/fixtures/*.json
@echo "==> Verifying event-feed schema pins via derived mutants..."
python3 scripts/check-event-feed-pin-probes.py \
conformance/event-feed/schema.json conformance/event-feed/pin-probes '$(CHECK_JSONSCHEMA_VERSION)'

event-feed-digest-fixtures-check:
@echo "==> Validating event-feed srv1 digest vectors..."
Expand Down
14 changes: 14 additions & 0 deletions conformance/event-feed/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,20 @@ against the JSON Schema metaschema and every fixture against the schema, with a
pinned `check-jsonschema` run through `uvx` (part of `make conformance`, so
`make check` gates it).

The same gate then verifies the schema's load-bearing `allOf` pins with the
probes in `pin-probes/` (`scripts/check-event-feed-pin-probes.py`). Each probe
declares a `control` scenario and one `mutation`; the gate requires the control
to VALIDATE, derives the mutant from it in-process, and requires the mutant to be
REJECTED. Deriving (rather than committing an invalid file) is what makes the
isolation claim true by construction: the accepted/rejected delta is exactly the
declared mutation, a control that stops validating fails the gate rather than
masking a wrong-reason rejection, and there is no invalid artifact to go
malformed or drift extra deltas. A mutation whose path is absent from the
control, or whose value equals the control's, fails as vacuous. The current pair
pins the per-signal default-terminal rules: a phantom invocation of a signal kind
whose disposition key is absent. Harnesses must never glob `pin-probes/` — probe
files are gate inputs, not scenario fixtures (and are not scenario-shaped).

## Directory is a schema boundary

This directory contains exactly one shape: tier-2 scenario scripts. If a second
Expand Down
150 changes: 150 additions & 0 deletions conformance/event-feed/pin-probes/phantom-gap-invocation.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
{
"description": "Pin probe for the per-signal feedGap default-terminal rule, guarding BOTH directions. The control configures only a bufferOverflow disposition, ends feed_gap default-terminal, and records the legal {bufferOverflow, accept} invocation \u2014 so a regression to the over-strict whole-record-empty form (const []) rejects the control and fails the gate. The mutant retains that record and adds a phantom {feedGap, accept} \u2014 an unregistered handler cannot be invoked \u2014 so a regression that widens or removes the pin accepts the mutant and fails the gate.",
"control": {
"name": "pin-probe-gap-invocation-control",
"description": "Control half: feed_gap default-terminal with only a bufferOverflow disposition configured; the record holds exactly the legal bufferOverflow invocation. Must validate \u2014 under an over-strict whole-record-empty pin it would not.",
"config": {
"liveBufferCapacity": 2,
"signalDisposition": {
"bufferOverflow": "accept"
}
},
"steps": [
{
"expectMint": {
"respond": {
"status": 200,
"body": {
"ticket": "{{TICKET:1}}",
"expires_in": 120,
"url": "{{CABLE_URL:1}}"
}
}
}
},
{
"expectConnect": {
"url": "{{CABLE_URL:1}}"
}
},
{
"serve": {
"frame": "welcome"
}
},
{
"expectSubscribe": {
"channel": "EventsChannel"
}
},
{
"serve": {
"frame": "message",
"event": {
"id": 51,
"kind": "message",
"event_type": "message.created",
"action": "created",
"created_at": "2026-08-01T12:00:00Z",
"bucket_id": 2,
"creator_id": 3,
"recording_id": 900,
"visible_to_clients": false
}
}
},
{
"serve": {
"frame": "message",
"event": {
"id": 52,
"kind": "message",
"event_type": "message.created",
"action": "created",
"created_at": "2026-08-01T12:00:00Z",
"bucket_id": 2,
"creator_id": 3,
"recording_id": 900,
"visible_to_clients": false
}
}
},
{
"serve": {
"frame": "message",
"event": {
"id": 53,
"kind": "message",
"event_type": "message.created",
"action": "created",
"created_at": "2026-08-01T12:00:00Z",
"bucket_id": 2,
"creator_id": 3,
"recording_id": 900,
"visible_to_clients": false
}
}
},
{
"expectSignal": {
"kind": "bufferOverflow",
"droppedIds": [
51
],
"droppedCount": 1
}
},
{
"expectClientClose": {}
},
{
"expectError": {
"reason": "feed_gap"
}
}
],
"finally": {
"state": "terminal",
"mintCount": 1,
"connectCount": 1,
"delivered": {
"exact": []
},
"checkpoints": {
"exact": []
},
"timers": {
"exact": {}
},
"socket": "closed",
"error": {
"reason": "feed_gap"
},
"handlerInvocations": {
"exact": [
{
"kind": "bufferOverflow",
"disposition": "accept"
}
]
}
}
},
"mutation": {
"path": [
"finally",
"handlerInvocations",
"exact"
],
"value": [
{
"kind": "bufferOverflow",
"disposition": "accept"
},
{
"kind": "feedGap",
"disposition": "accept"
}
]
}
}
150 changes: 150 additions & 0 deletions conformance/event-feed/pin-probes/phantom-overflow-invocation.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
{
"description": "Pin probe for the per-signal bufferOverflow default-terminal rule, guarding BOTH directions. The control configures only a feedGap disposition, ends buffer_overflow default-terminal, and records the legal {feedGap, accept} invocation \u2014 so a regression to the over-strict whole-record-empty form (const []) rejects the control and fails the gate. The mutant retains that record and adds a phantom {bufferOverflow, accept} \u2014 an unregistered handler cannot be invoked \u2014 so a regression that widens or removes the pin accepts the mutant and fails the gate.",
"control": {
"name": "pin-probe-overflow-invocation-control",
"description": "Control half: buffer_overflow default-terminal with only a feedGap disposition configured; the record holds exactly the legal feedGap invocation. Must validate \u2014 under an over-strict whole-record-empty pin it would not.",
"config": {
"liveBufferCapacity": 2,
"signalDisposition": {
"feedGap": "accept"
}
},
"steps": [
{
"expectMint": {
"respond": {
"status": 200,
"body": {
"ticket": "{{TICKET:1}}",
"expires_in": 120,
"url": "{{CABLE_URL:1}}"
}
}
}
},
{
"expectConnect": {
"url": "{{CABLE_URL:1}}"
}
},
{
"serve": {
"frame": "welcome"
}
},
{
"expectSubscribe": {
"channel": "EventsChannel"
}
},
{
"serve": {
"frame": "message",
"event": {
"id": 51,
"kind": "message",
"event_type": "message.created",
"action": "created",
"created_at": "2026-08-01T12:00:00Z",
"bucket_id": 2,
"creator_id": 3,
"recording_id": 900,
"visible_to_clients": false
}
}
},
{
"serve": {
"frame": "message",
"event": {
"id": 52,
"kind": "message",
"event_type": "message.created",
"action": "created",
"created_at": "2026-08-01T12:00:00Z",
"bucket_id": 2,
"creator_id": 3,
"recording_id": 900,
"visible_to_clients": false
}
}
},
{
"serve": {
"frame": "message",
"event": {
"id": 53,
"kind": "message",
"event_type": "message.created",
"action": "created",
"created_at": "2026-08-01T12:00:00Z",
"bucket_id": 2,
"creator_id": 3,
"recording_id": 900,
"visible_to_clients": false
}
}
},
{
"expectSignal": {
"kind": "bufferOverflow",
"droppedIds": [
51
],
"droppedCount": 1
}
},
{
"expectClientClose": {}
},
{
"expectError": {
"reason": "buffer_overflow"
}
}
],
"finally": {
"state": "terminal",
"mintCount": 1,
"connectCount": 1,
"delivered": {
"exact": []
},
"checkpoints": {
"exact": []
},
"timers": {
"exact": {}
},
"socket": "closed",
"error": {
"reason": "buffer_overflow"
},
"handlerInvocations": {
"exact": [
{
"kind": "feedGap",
"disposition": "accept"
}
]
}
}
},
"mutation": {
"path": [
"finally",
"handlerInvocations",
"exact"
],
"value": [
{
"kind": "feedGap",
"disposition": "accept"
},
{
"kind": "bufferOverflow",
"disposition": "accept"
}
]
}
}
Loading