An agent's answer is held to what the agent says it gives

CHANGELOG.md

@@ -3,6 +3,77 @@
 What changed in PACT, newest first. Each entry says what an author can now write, or what now
 holds that did not; the commits are listed under it for whoever wants the detail.
 
+## 2026-11 — an agent's answer is held to what the agent says it gives (15 November)
+
+`answers-with:` is a promise to whoever calls the agent. The reference harness put the shape to
+the model and handed back whatever came: *"Sure, I think we should refund them."* from an agent that
+gives a decision and an amount was the run's answer, `halted: final`. `checked-by:`, the author's
+own rules for an answer, was read by nothing in the loop; the run said so and returned the answer
+unchecked. The owner's ruling: *"if there is a promised shape, it must be honored."*
+
+### What holds now
+
+- **A reply that would be the answer is held before it is one** (`answers_fit`): to the shape the
+  agent was asked for, and, when it fits, to its `checked-by:` rules.
+- **Asked again, and told why.** A field left out (unless `, optional`) or a value its line refuses
+  is named in the author's own words; a broken rule is named with its `because:`. The model is told
+  at its next request and at no other, last in its system text, never in the person's voice.
+- **`checks-at-most:` times in all**, 2 unwritten: one count for the shape and the rules, and the
+  closing answer of a run that reached a limit spends from it too. An empty reply that arrived is a
+  try. The count crosses a pause with the rest of what a run has used.
+- **Still not the answer, the run ends `checks-failed`** and hands back none. Its `output` says how
+  many times it was asked and what was wrong with the last reply. A closing answer that never fits
+  keeps the ceiling's own word (`Reached.unanswered`).
+- **`RunResult.answer`**: each declared field the answer carried, as written, and no field the agent
+  does not declare. `output` stays the reply as the model wrote it. PACT's own agent facade
+  (`pact_agent`) hands back `answer`; a run that ended `checks-failed` arrives there as `Halted`.
+- **A field that streams** (`reply: stream of text`) is the reply itself, not a line in it.
+- **`turn.message.failed`** is emitted for each reply that was not the answer, with `attempt` and
+  `reason`; a watch may name it.
+
+### What `pact check` now says
+
+- **`checks-at-most:` is read beside `answers-with:`, `checked-by:` or both.** It needed
+  `checked-by:`. Beside neither it is refused as a line nothing reads (`schema/missing-any-of`).
+  `needs-any-of:` is a new attribute of the schema: `needs-one-of:` means exactly one, and refused an
+  agent that writes a shape and rules both.
+
+### What changed for whoever scripts a model
+
+A scripted answer has to be what the agent gives. 136 tests in 37 files scripted *"done"* or
+*"Approved."* for agents that promise fields; each now scripts an answer that fits
+(`tests/fitting.py`). One test documented the opposite rule and is rewritten: PACT's agent facade
+read a reply "and not validated, deliberately", because the harness checked nothing and a check at
+the door would have been a second one. The harness is now the one check, and the facade hands back
+what it read.
+
+### The second port
+
+The TypeScript port puts the same shape to the model and holds nothing: it asks once and returns
+the reply. It says so on every run where an answer is asked for in a shape or held to rules
+(`checks-at-most:`, `checked-by:` on `unenforced`; §7.28 list B, two rows, fifteen keys in all).
+
+### What a runtime must match
+
+- **`tests/conformance/answers.json`**, thirteen cases on a new tree `trees/decided-refunds`.
+- Every agent case's `expect` (reactions, limits, answers) has three new keys: `asked-again`,
+  `said`, `answer`. The 24 existing cases are otherwise unchanged.
+- `digests.json`: `decided-refunds` is new; no listed tree moved.
+
+### Checked live
+
+On gpt-4o-mini and gpt-5-nano: a desk ends its reply with the fields it gives; with its first
+reply replaced by one naming no field, told why, it answers again with every field; out of
+lookups, its closing answer has them too.
+
+### Still owed
+
+- The Pydantic AI path (`pydantic_ai_interop`) holds the shape with the SDK's own output retries
+  and `checked-by:` with `checks-at-most:`: two counts where the reference has one.
+- A workflow's stage that writes `checked-by:` is still held by nothing in the reference workflow
+  runtime.
+- The TypeScript port does not hold an answer; it has no shape reader.
+
 ## 2026-11 — the kernel's operations are a file, and each field says which it is (13 November)
 
 A runtime that runs PACT one operation at a time needs to know which fields reach each of the 76

README.md

@@ -763,7 +763,7 @@ produces *byte-identical traces, tool sequences and model-call counts*. The clai
 is bounded and the bound is written down: it covers `instructions:`, `tools:`,
 `skills:`, `knowledge:`, `team:`, `answers-with:`, the `loop:`, the `limits:`
 ceilings and whether a `plugins:` line stops the agent before it starts — and
-**not** the thirteen governance keys the Node port reports on
+**not** the fifteen governance keys the Node port reports on
 `unenforced`, nor the documents under `knowledge/`, which nothing on that port
 can look anything up in and which it names on `unretrieved` for every set an
 answer did not come from. See
@@ -1000,11 +1000,11 @@ framework is an extra.
 |---|---|
 | **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete |
 | **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers |
-| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5149 tests (1691 Rust + 3458 adapter), clippy clean, TypeScript type-checked** |
+| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5195 tests (1696 Rust + 3499 adapter), clippy clean, TypeScript type-checked** |
 | **Adapters** | **All 7 named targets**, proven against one shared conformance suite |
 
 ```bash
-./scripts/test-all.sh          # 5149 tests, Rust + 7 adapters, fully offline
+./scripts/test-all.sh          # 5195 tests, Rust + 7 adapters, fully offline
 ```
 
 **What is still owed** is not hidden — it lives in

adapters/python/src/pact_adapters/answers_fit.py

@@ -0,0 +1,193 @@
+"""Whether an agent's answer is what the agent says it gives.
+
+`answers-with:` is a promise to whoever calls the agent. The harness put the
+shape to the model and then took whatever came back: `decision: maybe` from an
+agent that gives `one of approved, declined` was handed on as its answer, and
+the caller found out when it read the field. `checked-by:`, the author's own
+rules for an answer, was read by nothing in this loop at all. This is the
+reader that holds an answer to both, and the one place its refusal is worded,
+so a model that is asked again is told what was wrong in the author's own
+words and two runtimes tell it the same thing.
+
+What is fixed (ruled with Bud Agent Flow, whose engine checked the answer and
+asked again without limit; the owner's ruling, 14 November: "if there is a
+promised shape, it must be honored"):
+
+* **A field left out** does not fit, unless its shape says `, optional`.
+* **A value its shape refuses** does not fit: outside a `one of`, not a number
+  where a number is written.
+* **A field that streams** (`reply: stream of text`) is the reply itself, as it
+  arrives: it is not looked for on a line of its own, and its value is the
+  reply's words. Only an empty reply leaves it out.
+* **A field the shape does not have** is not a reason to ask again. It breaks
+  nothing that was promised; it is not handed back (`fields_of`).
+* **Asked for at most `checks-at-most:` times in all**, 2 when unwritten. An
+  attempt that does not fit and an attempt that breaks a `checked-by:` rule
+  spend from the same count, and so does the closing answer of a run that
+  reached a limit.
+* **An empty reply that arrived is an attempt.** A call that failed is not: it
+  never produced an answer to hold.
+* **Still not fitting, the run ends `checks-failed` and hands back no answer.**
+  What it says is why. The answer that did not fit is in the steps.
+* **An agent that promises no shape and has no rules is held to nothing**, an
+  empty reply included.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass, field
+from typing import Any, Callable, Mapping
+
+from .questions import Rejected, Shape
+
+#: How a run that could not get a fitting answer ended (`RunResult.halted`).
+CHECKS_FAILED = "checks-failed"
+
+#: How a failed `checked-by:` check opens what the model is told. The words
+#: `pydantic_ai_interop` already sent for the same thing.
+CHECK_FAILED = "Your answer breaks a rule it must keep: "
+#: How a reply that does not fit `answers-with:` opens what the model is told.
+DID_NOT_FIT = "Your last answer did not fit what your answer must contain:"
+
+#: `name: value` on a line of its own, as the model is asked to end its reply:
+#: `- decision: approved`. A bullet or bold in front is the model's habit.
+_LINE = re.compile(r"^\s*(?:[-*]\s*)?(?:\*\*)?([A-Za-z0-9][A-Za-z0-9_-]*)(?:\*\*)?\s*:\s*(.*?)\s*$")
+
+
+def _streams(written: Any) -> bool:
+    """Whether a field is written as a stream: the reply itself, arriving."""
+    return str(written).lstrip().startswith("stream of ")
+
+
+def fields_of(said: str, shape: Mapping[str, str]) -> dict[str, Any]:
+    """The declared fields a reply carries, and no others. A reply that is one
+    JSON record is read as that; otherwise each declared name is read off its
+    own line, the last such line winning, because the model is told to END
+    with them. A value written as JSON (a list, a record) is read as JSON. A
+    field that streams is the reply's own words, unless a line of its own
+    says otherwise."""
+    try:
+        read = json.loads(said)
+    except (TypeError, ValueError):
+        read = None
+    if isinstance(read, Mapping):
+        return {k: v for k, v in read.items() if k in shape}
+    found: dict[str, Any] = {
+        name: str(said) for name, written in shape.items() if _streams(written) and str(said).strip()
+    }
+    for line in str(said).splitlines():
+        m = _LINE.match(line)
+        if not m or m.group(1) not in shape:
+            continue
+        value: Any = m.group(2)
+        if value[:1] in "[{":
+            try:
+                value = json.loads(value)
+            except ValueError:
+                pass
+        found[m.group(1)] = value
+    return found
+
+
+def does_not_fit(
+    said: str, shape: Mapping[str, str], shapes: Mapping[str, Mapping[str, str]] | None = None
+) -> tuple[str, ...]:
+    """Why `said` is not what `shape` promises: one line for each field, in the
+    order the shape writes them. Empty when it fits."""
+    fields = fields_of(said, shape)
+    problems: list[str] = []
+    for name, written in shape.items():
+        if name not in fields or (isinstance(fields[name], str) and not fields[name].strip()):
+            if not str(written).rstrip().endswith(", optional"):
+                problems.append(f"`{name}` is missing. It should be: {written}.")
+            continue
+        try:
+            Shape.parse(written, dict(shapes or {})).read(fields[name])
+        except Rejected as no:
+            # The shape reader's own words: what it should be, and what it is.
+            problems.append(f"`{name}` {no}.")
+    return tuple(problems)
+
+
+def ask_again(problems: tuple[str, ...]) -> str:
+    """What a model is told when its answer did not fit and it is asked again."""
+    return (
+        DID_NOT_FIT
+        + "\n"
+        + "\n".join(f"- {p}" for p in problems)
+        + "\nAnswer again, with every field as it is asked for."
+    )
+
+
+def rule_broken(broken: str) -> str:
+    """What a model is told when its answer broke a `checked-by:` rule."""
+    return f"{CHECK_FAILED}{broken}. Answer again so that it keeps every rule."
+
+
+@dataclass
+class Held:
+    """One run's hold on its answer: what the answer is held to, and how many
+    times it has been asked for.
+
+    `wrong` is the one question, asked of every reply that would be the
+    answer. It counts the attempt. The shape is read first: a reply with a
+    field missing is told that, and its `checked-by:` rules are read only of a
+    reply that fits, because a rule about what an answer says is not usefully
+    decided on one that is not an answer yet."""
+
+    shape: Mapping[str, str] = field(default_factory=dict)
+    shapes: Mapping[str, Mapping[str, str]] = field(default_factory=dict)
+    #: `evals.first_broken`, already bound to the agent's rules: given the
+    #: reply, why it breaks one, or `""`.
+    rules: "Callable[[str], str] | None" = None
+    at_most: int = 2
+    #: How many replies have been held so far.
+    asked: int = 0
+    #: Why the last reply held did not fit, as the model was (or would be) told.
+    last: str = ""
+    #: The same, as its parts: the fields that did not fit, or the rule broken.
+    problems: tuple[str, ...] = ()
+    broken: str = ""
+
+    @property
+    def holds_anything(self) -> bool:
+        return bool(self.shape) or self.rules is not None
+
+    @property
+    def may_ask_again(self) -> bool:
+        return self.asked < self.at_most
+
+    def wrong(self, said: str) -> str:
+        """Why `said` is not the answer, in the words a model is told, or `""`
+        when it is. Counts one attempt either way."""
+        if not self.holds_anything:
+            return ""
+        self.asked += 1
+        self.last, self.broken = "", ""
+        self.problems = does_not_fit(said, self.shape, self.shapes) if self.shape else ()
+        if self.problems:
+            self.last = ask_again(self.problems)
+        elif self.rules is not None and (broken := self.rules(said)):
+            self.broken = broken
+            self.last = rule_broken(broken)
+        return self.last
+
+    def gave_up(self) -> str:
+        """What a run says when it ends `checks-failed`: how many times it was
+        asked, and what was wrong with the last reply. Said to a person, so in
+        the third person: the words the MODEL was told (`last`) begin "Your
+        last answer", and whoever reads a run's ending is not the model."""
+        times = "once" if self.asked == 1 else f"{self.asked} times"
+        if self.broken:
+            kept = "it did not keep" if self.asked == 1 else "none kept"
+            return (
+                f"Stopped without an answer: it was asked for its answer {times} and {kept} "
+                f"its `checked-by:` rules. The last one {self.broken}."
+            )
+        fit = "it did not fit" if self.asked == 1 else "none fit"
+        return (
+            f"Stopped without an answer: it was asked for its answer {times} and {fit} "
+            f"`answers-with:`. In the last one:\n" + "\n".join(f"- {p}" for p in self.problems)
+        )