Repository navigation
An agent's answer is held to what the agent says it gives - #33
Merged
Merged
Conversation
dittops
force-pushed
the
rulings/answers-held-to-shape
branch
from
November 14, 2026 20:05
85ce08c to
7488e6b
Compare
The owner's ruling: "if there is a promised shape, it must be honored." - `answers_fit`: a reply that would be the answer is held to `answers-with:` and to `checked-by:`, from one count of `checks-at-most:` (2 unwritten). Asked again, the model is told why at its next request and at no other. Still not fitting, the run ends `checks-failed` and hands back no answer. The closing answer of a run that reached a limit is held the same way and keeps the ceiling's word (`Reached.unanswered`). - `RunResult.answer`: the declared fields the reply carried, and no others. PACT's agent facade hands those back. - A field that streams is the reply itself. - `checks-at-most:` is read beside `answers-with:`, `checked-by:` or both (`needs-any-of:`, a new schema attribute; `schema/missing-any-of`). - `turn.message.failed` is emitted for each reply that did not fit. - 136 existing tests scripted answers that are not what their agent gives; each now scripts one that is (`tests/fitting.py`). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…r held; the ending said to a person - TypeScript port: `checked-by:` and `checks-at-most:` on `unenforced` (§7.28 list B, fifteen keys). It puts the same shape to the model and holds nothing. - `tests/conformance/answers.json`: thirteen cases on `trees/decided-refunds`. Every agent case's `expect` gains `asked-again`, `said`, `answer`. - A run that ends `checks-failed` says what was wrong with the last reply in the third person, not in the words the model was told. - Three live tests: a desk ends with its fields; told why, it answers again; its closing answer has them. - `one_that_fits` is a test helper, not library API. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
dittops
force-pushed
the
rulings/answers-held-to-shape
branch
from
November 14, 2026 20:29
7488e6b to
0e136e3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The owner's ruling: "if there is a promised shape, it must be honored."
What changes
answers-with:is a promise to whoever calls the agent. The reference harness put the shape to the model and handed back whatever came: "Sure, I think we should refund them." from an agent that gives a decision and an amount was the run's answer,halted: final.checked-by:, the author's own rules for an answer, was read by nothing in the loop.answers_fit): to the shape the agent was asked for and, when it fits, to itschecked-by:rules.checks-at-most:times in all, 2 unwritten: one count for the shape and the rules, and the closing answer of a run that reached a limit spends from it. An empty reply is a try. The count crosses a pause.checks-failed, no answer handed back, and the ending says how many times it was asked and what was wrong with the last reply. A closing answer that never fits keeps the ceiling's own word.RunResult.answer: the declared fields the answer carried, and no undeclared field. PACT's agent facade hands those back.checks-at-most:is read besideanswers-with:,checked-by:or both (needs-any-of:, a new schema attribute;needs-one-of:means exactly one and refused an agent with both).turn.message.failedfor each reply that was not the answer.What it cost
136 existing tests in 37 files scripted "done" or "Approved." for agents that promise fields. Each now scripts an answer that fits (
tests/fitting.py). One test documented the opposite rule (the facade read a reply "and not validated, deliberately") and is rewritten, with the reason.The second port
The TypeScript port puts the same shape to the model and holds nothing. It now says so on
unenforced(checked-by:,checks-at-most:), in §7.28 list B (fifteen keys), held by the Rust and Python tests that hold that list.What a runtime must match
tests/conformance/answers.json: thirteen cases on a new treetrees/decided-refunds.expectgainsasked-again,said,answer. The 24 existing agent cases are otherwise unchanged (verified key by key).Verified
scripts/test-all.sh: 5,127 tests (1,692 Rust + 3,435 adapter), clippy, every shipped tree under--deny-warnings, TypeScript, seven adapter targets.mkdocs build --strict.Still owed
checked-by:retries separately.checked-by:is held by nothing in the reference workflow runtime.🤖 Generated with Claude Code