Skip to content

An agent's answer is held to what the agent says it gives - #33

Merged
dittops merged 3 commits into
masterfrom
rulings/answers-held-to-shape
Nov 14, 2026
Merged

dittops merged 3 commits into
masterfrom
rulings/answers-held-to-shape

Conversation

@dittops

@dittops dittops commented Nov 14, 2026

Copy link
Copy Markdown
Member

The owner's ruling: "if there is a promised shape, it must be honored."

What changes

answers-with: is a promise to whoever calls the agent. The reference harness put the shape to the model and handed back whatever came: "Sure, I think we should refund them." from an agent that gives a decision and an amount was the run's answer, halted: final. checked-by:, the author's own rules for an answer, was read by nothing in the loop.

  • Held before it is the answer (answers_fit): to the shape the agent was asked for and, when it fits, to its checked-by: rules.
  • Asked again and told why, in the author's own words for each field, at the next request and no other, last in the system text.
  • checks-at-most: times in all, 2 unwritten: one count for the shape and the rules, and the closing answer of a run that reached a limit spends from it. An empty reply is a try. The count crosses a pause.
  • Still not the answer: checks-failed, no answer handed back, and the ending says how many times it was asked and what was wrong with the last reply. A closing answer that never fits keeps the ceiling's own word.
  • RunResult.answer: the declared fields the answer carried, and no undeclared field. PACT's agent facade hands those back.
  • A field that streams is the reply itself.
  • checks-at-most: is read beside answers-with:, checked-by: or both (needs-any-of:, a new schema attribute; needs-one-of: means exactly one and refused an agent with both).
  • turn.message.failed for each reply that was not the answer.

What it cost

136 existing tests in 37 files scripted "done" or "Approved." for agents that promise fields. Each now scripts an answer that fits (tests/fitting.py). One test documented the opposite rule (the facade read a reply "and not validated, deliberately") and is rewritten, with the reason.

The second port

The TypeScript port puts the same shape to the model and holds nothing. It now says so on unenforced (checked-by:, checks-at-most:), in §7.28 list B (fifteen keys), held by the Rust and Python tests that hold that list.

What a runtime must match

  • tests/conformance/answers.json: thirteen cases on a new tree trees/decided-refunds.
  • Every agent case's expect gains asked-again, said, answer. The 24 existing agent cases are otherwise unchanged (verified key by key).

Verified

  • scripts/test-all.sh: 5,127 tests (1,692 Rust + 3,435 adapter), clippy, every shipped tree under --deny-warnings, TypeScript, seven adapter targets.
  • mkdocs build --strict.
  • Live on gpt-4o-mini: 16 of 16, three of them new. With its first reply replaced by one naming no field, a real model told why answers again with every field (also on gpt-5-nano).

Still owed

  • The Pydantic AI path counts shape retries and checked-by: retries separately.
  • A workflow stage's checked-by: is held by nothing in the reference workflow runtime.
  • The TypeScript port has no shape reader.

🤖 Generated with Claude Code

@dittops
dittops force-pushed the rulings/answers-held-to-shape branch from 85ce08c to 7488e6b Compare November 14, 2026 20:05
jithinAB and others added 3 commits November 15, 2026 01:53
The owner's ruling: "if there is a promised shape, it must be honored."

- `answers_fit`: a reply that would be the answer is held to `answers-with:`
  and to `checked-by:`, from one count of `checks-at-most:` (2 unwritten).
  Asked again, the model is told why at its next request and at no other.
  Still not fitting, the run ends `checks-failed` and hands back no answer.
  The closing answer of a run that reached a limit is held the same way and
  keeps the ceiling's word (`Reached.unanswered`).
- `RunResult.answer`: the declared fields the reply carried, and no others.
  PACT's agent facade hands those back.
- A field that streams is the reply itself.
- `checks-at-most:` is read beside `answers-with:`, `checked-by:` or both
  (`needs-any-of:`, a new schema attribute; `schema/missing-any-of`).
- `turn.message.failed` is emitted for each reply that did not fit.
- 136 existing tests scripted answers that are not what their agent gives;
  each now scripts one that is (`tests/fitting.py`).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…r held; the ending said to a person

- TypeScript port: `checked-by:` and `checks-at-most:` on `unenforced` (§7.28
  list B, fifteen keys). It puts the same shape to the model and holds nothing.
- `tests/conformance/answers.json`: thirteen cases on `trees/decided-refunds`.
  Every agent case's `expect` gains `asked-again`, `said`, `answer`.
- A run that ends `checks-failed` says what was wrong with the last reply in
  the third person, not in the words the model was told.
- Three live tests: a desk ends with its fields; told why, it answers again;
  its closing answer has them.
- `one_that_fits` is a test helper, not library API.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@dittops
dittops force-pushed the rulings/answers-held-to-shape branch from 7488e6b to 0e136e3 Compare November 14, 2026 20:29
@dittops
dittops merged commit 0e593bb into master Nov 14, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants