From 4eaa8c9a612b310e50ec2cc79ed47c47250a9f27 Mon Sep 17 00:00:00 2001 From: Jithin V G Date: Sun, 15 Nov 2026 00:59:33 +0530 Subject: [PATCH 1/3] An agent's answer is held to what the agent says it gives The owner's ruling: "if there is a promised shape, it must be honored." - `answers_fit`: a reply that would be the answer is held to `answers-with:` and to `checked-by:`, from one count of `checks-at-most:` (2 unwritten). Asked again, the model is told why at its next request and at no other. Still not fitting, the run ends `checks-failed` and hands back no answer. The closing answer of a run that reached a limit is held the same way and keeps the ceiling's word (`Reached.unanswered`). - `RunResult.answer`: the declared fields the reply carried, and no others. PACT's agent facade hands those back. - A field that streams is the reply itself. - `checks-at-most:` is read beside `answers-with:`, `checked-by:` or both (`needs-any-of:`, a new schema attribute; `schema/missing-any-of`). - `turn.message.failed` is emitted for each reply that did not fit. - 136 existing tests scripted answers that are not what their agent gives; each now scripts one that is (`tests/fitting.py`). Co-Authored-By: Claude Fable 5.2 --- .../python/src/pact_adapters/answers_fit.py | 183 +++++++++ adapters/python/src/pact_adapters/harness.py | 135 ++++++- adapters/python/src/pact_adapters/limits.py | 29 +- .../python/src/pact_adapters/pact_agent.py | 28 +- adapters/python/src/pact_adapters/watches.py | 1 + adapters/python/tests/fitting.py | 74 ++++ ...ne_reaches_a_server_without_a_handshake.py | 3 +- ...tool_is_data_and_a_cancelled_run_is_not.py | 9 +- ...ed_run_does_not_pretend_to_be_an_answer.py | 75 ++-- .../python/tests/test_a_person_can_say_no.py | 5 +- .../test_a_read_only_workspace_still_runs.py | 3 +- ...hes_a_ceiling_does_what_the_author_said.py | 7 +- .../tests/test_action_scoped_approvals.py | 3 +- ...swer_is_checked_before_it_is_the_answer.py | 25 +- ...is_held_to_what_the_agent_says_it_gives.py | 367 ++++++++++++++++++ .../python/tests/test_connection_consent.py | 5 +- adapters/python/tests/test_context_policy.py | 19 +- adapters/python/tests/test_join_policy.py | 27 +- adapters/python/tests/test_judged_rules.py | 8 +- .../tests/test_metrics_an_expert_brings.py | 5 +- adapters/python/tests/test_model_pin.py | 15 +- .../python/tests/test_model_portability.py | 3 +- ...test_no_adapter_reads_the_authors_files.py | 4 +- .../tests/test_one_vocabulary_for_hiding.py | 5 +- adapters/python/tests/test_portability.py | 7 +- .../test_priced_rows_beyond_one_vendor.py | 11 +- adapters/python/tests/test_scoring_a_suite.py | 4 +- adapters/python/tests/test_suspension.py | 13 +- adapters/python/tests/test_termination.py | 29 +- .../test_the_golden_set_runs_everywhere.py | 18 +- ...del_choosing_door_survives_being_opened.py | 8 +- ...ment_is_the_same_run_through_the_facade.py | 22 +- .../tests/test_typed_questions_in_the_loop.py | 25 +- adapters/python/tests/test_watching_a_run.py | 3 +- ...t_what_the_author_wrote_reaches_the_run.py | 23 +- .../python/tests/test_worked_example_loop.py | 7 +- ..._gets_is_read_beside_what_it_is_held_to.rs | 111 ++++++ crates/pact-schema/src/from_doc.rs | 3 + crates/pact-schema/src/lib.rs | 53 +++ docs/20-ARCHITECTURE-DRAFT.md | 2 +- spec/schema.yaml | 24 +- 41 files changed, 1226 insertions(+), 175 deletions(-) create mode 100644 adapters/python/src/pact_adapters/answers_fit.py create mode 100644 adapters/python/tests/fitting.py create mode 100644 adapters/python/tests/test_an_answer_is_held_to_what_the_agent_says_it_gives.py create mode 100644 crates/pact-cli/tests/how_many_tries_an_answer_gets_is_read_beside_what_it_is_held_to.rs diff --git a/adapters/python/src/pact_adapters/answers_fit.py b/adapters/python/src/pact_adapters/answers_fit.py new file mode 100644 index 0000000..77f1cb9 --- /dev/null +++ b/adapters/python/src/pact_adapters/answers_fit.py @@ -0,0 +1,183 @@ +"""Whether an agent's answer is what the agent says it gives. + +`answers-with:` is a promise to whoever calls the agent. The harness put the +shape to the model and then took whatever came back: `decision: maybe` from an +agent that gives `one of approved, declined` was handed on as its answer, and +the caller found out when it read the field. `checked-by:`, the author's own +rules for an answer, was read by nothing in this loop at all. This is the +reader that holds an answer to both, and the one place its refusal is worded, +so a model that is asked again is told what was wrong in the author's own +words and two runtimes tell it the same thing. + +What is fixed (ruled with Bud Agent Flow, whose engine checked the answer and +asked again without limit; the owner's ruling, 14 November: "if there is a +promised shape, it must be honored"): + +* **A field left out** does not fit, unless its shape says `, optional`. +* **A value its shape refuses** does not fit: outside a `one of`, not a number + where a number is written. +* **A field that streams** (`reply: stream of text`) is the reply itself, as it + arrives: it is not looked for on a line of its own, and its value is the + reply's words. Only an empty reply leaves it out. +* **A field the shape does not have** is not a reason to ask again. It breaks + nothing that was promised; it is not handed back (`fields_of`). +* **Asked for at most `checks-at-most:` times in all**, 2 when unwritten. An + attempt that does not fit and an attempt that breaks a `checked-by:` rule + spend from the same count, and so does the closing answer of a run that + reached a limit. +* **An empty reply that arrived is an attempt.** A call that failed is not: it + never produced an answer to hold. +* **Still not fitting, the run ends `checks-failed` and hands back no answer.** + What it says is why. The answer that did not fit is in the steps. +* **An agent that promises no shape and has no rules is held to nothing**, an + empty reply included. +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field +from typing import Any, Callable, Mapping + +from .questions import Rejected, Shape + +#: How a run that could not get a fitting answer ended (`RunResult.halted`). +CHECKS_FAILED = "checks-failed" + +#: How a failed `checked-by:` check opens what the model is told. The words +#: `pydantic_ai_interop` already sent for the same thing. +CHECK_FAILED = "Your answer breaks a rule it must keep: " +#: How a reply that does not fit `answers-with:` opens what the model is told. +DID_NOT_FIT = "Your last answer did not fit what your answer must contain:" + +#: `name: value` on a line of its own, as the model is asked to end its reply: +#: `- decision: approved`. A bullet or bold in front is the model's habit. +_LINE = re.compile(r"^\s*(?:[-*]\s*)?(?:\*\*)?([A-Za-z0-9][A-Za-z0-9_-]*)(?:\*\*)?\s*:\s*(.*?)\s*$") + + +def _streams(written: Any) -> bool: + """Whether a field is written as a stream: the reply itself, arriving.""" + return str(written).lstrip().startswith("stream of ") + + +def fields_of(said: str, shape: Mapping[str, str]) -> dict[str, Any]: + """The declared fields a reply carries, and no others. A reply that is one + JSON record is read as that; otherwise each declared name is read off its + own line, the last such line winning, because the model is told to END + with them. A value written as JSON (a list, a record) is read as JSON. A + field that streams is the reply's own words, unless a line of its own + says otherwise.""" + try: + read = json.loads(said) + except (TypeError, ValueError): + read = None + if isinstance(read, Mapping): + return {k: v for k, v in read.items() if k in shape} + found: dict[str, Any] = { + name: str(said) for name, written in shape.items() if _streams(written) and str(said).strip() + } + for line in str(said).splitlines(): + m = _LINE.match(line) + if not m or m.group(1) not in shape: + continue + value: Any = m.group(2) + if value[:1] in "[{": + try: + value = json.loads(value) + except ValueError: + pass + found[m.group(1)] = value + return found + + +def does_not_fit( + said: str, shape: Mapping[str, str], shapes: Mapping[str, Mapping[str, str]] | None = None +) -> tuple[str, ...]: + """Why `said` is not what `shape` promises: one line for each field, in the + order the shape writes them. Empty when it fits.""" + fields = fields_of(said, shape) + problems: list[str] = [] + for name, written in shape.items(): + if name not in fields or (isinstance(fields[name], str) and not fields[name].strip()): + if not str(written).rstrip().endswith(", optional"): + problems.append(f"`{name}` is missing. It should be: {written}.") + continue + try: + Shape.parse(written, dict(shapes or {})).read(fields[name]) + except Rejected as no: + # The shape reader's own words: what it should be, and what it is. + problems.append(f"`{name}` {no}.") + return tuple(problems) + + +def ask_again(problems: tuple[str, ...]) -> str: + """What a model is told when its answer did not fit and it is asked again.""" + return ( + DID_NOT_FIT + + "\n" + + "\n".join(f"- {p}" for p in problems) + + "\nAnswer again, with every field as it is asked for." + ) + + +def rule_broken(broken: str) -> str: + """What a model is told when its answer broke a `checked-by:` rule.""" + return f"{CHECK_FAILED}{broken}. Answer again so that it keeps every rule." + + +@dataclass +class Held: + """One run's hold on its answer: what the answer is held to, and how many + times it has been asked for. + + `wrong` is the one question, asked of every reply that would be the + answer. It counts the attempt. The shape is read first: a reply with a + field missing is told that, and its `checked-by:` rules are read only of a + reply that fits, because a rule about what an answer says is not usefully + decided on one that is not an answer yet.""" + + shape: Mapping[str, str] = field(default_factory=dict) + shapes: Mapping[str, Mapping[str, str]] = field(default_factory=dict) + #: `evals.first_broken`, already bound to the agent's rules: given the + #: reply, why it breaks one, or `""`. + rules: "Callable[[str], str] | None" = None + at_most: int = 2 + #: How many replies have been held so far. + asked: int = 0 + #: Why the last reply held did not fit, as the model was (or would be) told. + last: str = "" + + @property + def holds_anything(self) -> bool: + return bool(self.shape) or self.rules is not None + + @property + def may_ask_again(self) -> bool: + return self.asked < self.at_most + + def wrong(self, said: str) -> str: + """Why `said` is not the answer, in the words a model is told, or `""` + when it is. Counts one attempt either way.""" + if not self.holds_anything: + return "" + self.asked += 1 + self.last = "" + problems = does_not_fit(said, self.shape, self.shapes) if self.shape else () + if problems: + self.last = ask_again(problems) + elif self.rules is not None and (broken := self.rules(said)): + self.last = rule_broken(broken) + return self.last + + def gave_up(self) -> str: + """What a run says when it ends `checks-failed`: how many times it was + asked, and what was wrong with the last answer.""" + times = "once" if self.asked == 1 else f"{self.asked} times" + none = "it did not fit" if self.asked == 1 else "none fit" + what = "`answers-with:`" if self.last.startswith(DID_NOT_FIT) else "its `checked-by:` rules" + why = self.last.rsplit("\nAnswer again", 1)[0].rsplit(". Answer again so that", 1)[0] + return ( + f"Stopped without an answer: it was asked for its answer {times} and {none} " + f"{what}. {why}" + ) diff --git a/adapters/python/src/pact_adapters/harness.py b/adapters/python/src/pact_adapters/harness.py index 30f38c5..fd06d2b 100644 --- a/adapters/python/src/pact_adapters/harness.py +++ b/adapters/python/src/pact_adapters/harness.py @@ -63,6 +63,7 @@ from .holes import fill from .interceptors import Chain from .ir import RUN_WITHOUT_IT, AgentSpec, PluginSpec, SkillSpec +from .answers_fit import CHECKS_FAILED, Held, fields_of from .limits import RAN_OUT, Action, Ceiling, Limits, Meter, Reached, money, step_ceiling from .limits import _CEILING_FIELDS as CEILING_FIELDS from .loops import DONE, Does, Loop, LoopError, Phase @@ -144,6 +145,12 @@ class RunResult: output: str steps: list[Step] = field(default_factory=list) halted: str = "final" + #: The fields of the answer, for an agent that says what it gives + #: (`answers-with:`): each declared field the answer carried, as it was + #: written, and no other. A field the agent does not declare is not here; + #: `output` is the reply as the model wrote it. Empty for an agent with no + #: `answers-with:`, and for a run that did not end in an answer. + answer: dict[str, Any] = field(default_factory=dict) suspension: "Suspension | None" = None #: One entry per step in which the team was asked. Kept because "two of the #: three agreed and the third was never waited for" is a fact about the run @@ -1450,6 +1457,55 @@ def closing_words(reached: "Reached", not_made: "tuple[ToolCall, ...]" = ()) -> def after_a_yes() -> str: return "\n\n" + told_once.pop() if told_once else "" + + # What this run's answer is held to (`answers_fit`): the shape it was asked + # for, and the author's own `checked-by:` rules, `checks-at-most:` attempts + # in all. A `judged:` rule needs a grader and this loop is handed none, so + # it is said (`unenforced`) and not passed in silence, once. + unjudged: list[str] = [] + + def _breaks_a_rule(said: str) -> str: + from .evals import first_broken + + noted: list[str] = [] + broken = first_broken( + spec.checked_by, RunResult(output=said, steps=list(result.steps)), + unenforced=noted, run_program=run_program, + ) + for line in noted: + if line not in unjudged: + unjudged.append(line) + result.unenforced = result.unenforced + (line,) + return broken + + held = Held( + shape=dict(shape_to_ask_for), + shapes=spec.shapes, + rules=_breaks_a_rule if spec.checked_by else None, + at_most=max(1, spec.checks_at_most), + asked=int((resume.used or {}).get("answers-asked-for", 0)) if resume is not None else 0, + ) + + def hold(said: str, at: int, *, can_ask_again: bool = True) -> str: + """Hold a reply that would be the run's answer. `""` when it is the + answer; `"again"` when it is not and the model is asked once more (it + is told why at its next request, and at no other); `"failed"` when it + is not and the run has ended `checks-failed`, handing back nothing.""" + why = held.wrong(said) + if not why: + result.answer = fields_of(said, held.shape) if held.shape else {} + return "" + # Across a pause with everything else the run has used. + result.used.answers = held.asked + bus.emit("turn.message.failed", at=(at,), attempt=held.asked, reason=why) + if can_ask_again and held.may_ask_again: + told_once.append(why) + return "again" + result.halted = CHECKS_FAILED + result.output = held.gave_up() + result.answer = {} + bus.emit("turn.run.failed", at=(at,), reason=CHECKS_FAILED) + return "failed" result.reaction_notes = reactor.notes result.reactions = reactor.log reports_usage = callable(getattr(transport, "usage", None)) @@ -2012,7 +2068,7 @@ def at_ceiling() -> "Reached | None": out, spec=spec, transport=transport, result=result, history=history, already=already, bus=bus, meter=meter, now=now, clock=clock, at=i, phase=phase_name, visits=visits, asking=asking, once=once, chain=chain, given=given, - closing_words=closing_words, + closing_words=closing_words, held=held, ) # Which stage runs now. A stage that has used up its `at-most` is routed # past here rather than after the fact, so it never spends a model call @@ -2023,6 +2079,10 @@ def at_ceiling() -> "Reached | None": if gave_up: result.halted = "stage-limit" bus.emit("turn.run.failed", at=(i,), reason=gave_up) + # No stage is left to ask again in: these words are the answer, or + # there is none. + elif hold(result.output, i, can_ask_again=False): + pass else: result.halted = "final" bus.emit("turn.run.completed", at=(i,)) @@ -2334,6 +2394,13 @@ def at_ceiling() -> "Reached | None": if nxt is None: return result if nxt == DONE: + verdict = hold(ran, i) + if verdict == "again": + # The same stage, once more: it is not another visit. + visits[phase_name] -= 1 + continue + if verdict == "failed": + return _turn_ended(result, spec.facts) return _finish(result, chain, bus, i, ran, spec.facts) phase_name = nxt continue @@ -2369,7 +2436,7 @@ def at_ceiling() -> "Reached | None": out, spec=spec, transport=transport, result=result, history=history, already=already, bus=bus, meter=meter, now=now, clock=clock, at=i, phase=phase_name, visits=visits, asking=asking, once=once, chain=chain, given=given, - closing_words=closing_words, + closing_words=closing_words, held=held, ) # The reactions on what the agent just wrote (docs/99 §5) — AFTER # the step's own call has been metered. A judge may be the same @@ -2420,10 +2487,29 @@ def at_ceiling() -> "Reached | None": if nxt is None: return result if nxt == DONE: + # THE ANSWER IS HELD BEFORE IT IS ONE (`answers_fit`). What the + # model wrote is what is held, before any rule rewrites it: a + # hiding rule that removed a field's value must not be what + # makes an answer stop fitting. + verdict = hold(text, i) + if verdict: + # The reply is a step either way: it was written, and a + # model asked again has to be able to see what it said. + kept, decision = chain.run("step.message.after", {"content": text}) + shown = decision.stop if decision.stop is not None else kept["content"] + result.steps.append(Step(index=i, text=shown)) + if verdict == "failed": + return _turn_ended(result, spec.facts) + history.append({"role": "assistant", "content": shown}) + bus.emit("step.message.completed", at=(i,), text=shown) + # The same stage, once more: it is not another visit. + visits[phase_name] -= 1 + continue answered, decision = chain.run("turn.message.after", {"content": text}) if decision.stop is not None: result.halted = "stopped-by-rule" result.output = decision.stop + result.answer = {} bus.emit("turn.run.cancelled", at=(i,), reason=decision.stop) return result text = answered["content"] @@ -3271,7 +3357,7 @@ def at_ceiling() -> "Reached | None": out_of_calls, spec=spec, transport=transport, result=result, history=history, already=already, bus=bus, meter=meter, now=now, clock=clock, at=i, phase=phase_name, visits=visits, asking=asking, once=once, chain=chain, given=given, - closing_words=closing_words, not_made=not_made, + closing_words=closing_words, held=held, not_made=not_made, ) visits[phase_name] = visits.get(phase_name, 0) + 1 @@ -3344,7 +3430,7 @@ def at_ceiling() -> "Reached | None": spec=spec, transport=transport, result=result, history=history, already=already, bus=bus, meter=meter, now=now, clock=clock, at=limit, phase=phase_name, visits=visits, asking=asking, once=once, chain=chain, given=given, - closing_words=closing_words, + closing_words=closing_words, held=held, ) @@ -3459,6 +3545,10 @@ async def _ran_out( #: it has been telling the model. Required for the reason `chain` is: a #: caller that forgets it sends the closing request less than every other. closing_words: "Callable[[Reached, tuple[ToolCall, ...]], str]", + #: What the run's answer is held to. Required for the reason `chain` is: + #: the closing answer is the one nobody planned for, and a caller that + #: forgets this hands it back unheld. + held: "Held", #: The calls of this reply that were not made because the ceiling was #: reached part-way through it. The closing request tells the model. not_made: "tuple[ToolCall, ...]" = (), @@ -3574,6 +3664,25 @@ async def _ran_out( # answer must have, and without anything a reaction had reminded it # of. text, _ = await transport.model_call(closing_words(reached, not_made), history, []) + # THE CLOSING ANSWER IS HELD LIKE ANY OTHER, from the same count: the + # closing request is one more time the run asks for its answer. Asked + # again, it is told why, after everything the closing request is told. + # What never fits is not handed back: the run keeps the ceiling's word, + # because it did run out, and says it was left with no answer. + while why := held.wrong(text): + bus.emit("turn.message.failed", at=(at,), attempt=held.asked, reason=why) + kept, _ = chain.run("step.message.after", {"content": text}) + result.steps.append(Step(index=at, text=kept["content"])) + if not held.may_ask_again: + result.output = reached.unanswered(held.asked) + result.answer = {} + bus.emit("turn.run.failed", at=(at,), reason=CHECKS_FAILED) + return result + history.append({"role": "assistant", "content": kept["content"]}) + text, _ = await transport.model_call( + closing_words(reached, not_made) + "\n\n" + why, history, [] + ) + result.answer = fields_of(text, held.shape) if held.shape else {} # THE SAME RULES AS EVERY OTHER REPLY. `_finish`'s docstring states the # guarantee this used to break, word for word: "a redaction rule cannot # be escaped by finishing from a stage rather than by running out of @@ -3591,6 +3700,7 @@ async def _ran_out( # survive in the trace: a redaction visible nowhere but the # transcript is a redaction in name only. result.output = decision.stop + result.answer = {} result.steps.append(Step(index=at, text=decision.stop)) bus.emit("turn.run.cancelled", at=(at,), reason=decision.stop) return result @@ -3809,22 +3919,14 @@ def _unchecked_answers(spec: AgentSpec) -> tuple[str, ...]: def _unheld(spec: AgentSpec) -> tuple[str, ...]: """The lines this loop reads nothing of, said rather than skipped. - `checked-by:`, `teamwork.may-start:` and a list under `model:` are checked - by `pact check` and carried by `ir`, and nothing in `run` holds any of them: - an answer is returned without being held to its rules, nobody can be + `teamwork.may-start:` and a list under `model:` are checked by `pact check` + and carried by `ir`, and nothing in `run` holds either: nobody can be started who is not on `team:`, and the one transport this run was handed is - the only model it will ever ask. Silence here let `pact-eval` score an agent + the only model it will ever ask. (`checked-by:` was a third, and is held: + `answers_fit`.) Silence here let `pact-eval` score an agent whose checks never ran as though they had. """ out: list[str] = [] - if spec.checked_by: - out.append( - f"checked-by: {spec.key or 'this agent'} holds an answer to " - f"{len(spec.checked_by)} rule(s) before it is the answer, and this " - "runtime returns the answer unchecked. fix: run it on a runtime that " - "holds `checked-by:`, or hold the answer yourself with " - "`evals.first_broken(spec.checked_by, result)`." - ) if spec.teamwork.may_start: allowed = ", ".join(sorted(str(getattr(w, "value", w)) for w in spec.teamwork.may_start)) out.append( @@ -4601,6 +4703,7 @@ def _finish( if decision.stop is not None: result.halted = "stopped-by-rule" result.output = decision.stop + result.answer = {} bus.emit("turn.run.cancelled", at=(at,), reason=decision.stop) return result said = answered["content"] diff --git a/adapters/python/src/pact_adapters/limits.py b/adapters/python/src/pact_adapters/limits.py index 6d7858a..c216aba 100644 --- a/adapters/python/src/pact_adapters/limits.py +++ b/adapters/python/src/pact_adapters/limits.py @@ -153,6 +153,21 @@ def declined(self, words: str = "", *, nobody_answered: bool = False) -> str: words = words.strip() return f"{said} {words}" if words else said + def unanswered(self, attempts: int) -> str: + """What a person is told when the run reached a ceiling whose action + is `answer-with-what-it-has`, and what it had was never an answer: the + closing reply did not fit what the agent says it gives + (`answers-with:`, `checked-by:`), however many times it was asked. + + The run keeps the ceiling's own word (`halted`), because it did run + out; it hands back this and not the reply, because a reply that does + not fit is not what the agent gives.""" + times = "once and it did not fit" if attempts == 1 else f"{attempts} times and none fit" + return ( + f"Stopped before finishing: this run reached {self._whose()}. It was asked for " + f"its answer {times} what it says it gives." + ) + def closing(self, not_made: "tuple[str, ...]" = ()) -> str: """What the MODEL is told at the closing request of a run that answers with what it has. @@ -241,6 +256,12 @@ class Meter: tool_calls: int = 0 tokens: int = 0 money: float = 0.0 + #: How many times this run has asked for its answer and held what came + #: back (`answers_fit.Held`). Not a ceiling of `limits:`; it is here + #: because it has to cross a pause with the rest of what a run has used, + #: or a run that parks between two attempts is given `checks-at-most:` + #: more. + answers: int = 0 def seconds(self, now: float) -> float: """Time spent WORKING. Time parked for a person is excluded on resume, @@ -252,13 +273,18 @@ def reading(self, reads: str, now: float) -> float: def as_dict(self, now: float) -> dict[str, float]: """The shape that crosses a process boundary inside a pause.""" - return { + used = { "seconds": self.seconds(now), "steps": float(self.steps), "tool_calls": float(self.tool_calls), "tokens": float(self.tokens), "money": self.money, } + # Written only once an answer has been held and found wanting, so a + # pause of a run that never was reads as it always did. + if self.answers: + used["answers-asked-for"] = float(self.answers) + return used @staticmethod def restored(used: dict[str, float], now: float, steps: int) -> "Meter": @@ -272,6 +298,7 @@ def restored(used: dict[str, float], now: float, steps: int) -> "Meter": tool_calls=int(used.get("tool_calls", 0)), tokens=int(used.get("tokens", 0)), money=float(used.get("money", 0.0)), + answers=int(used.get("answers-asked-for", 0)), ) diff --git a/adapters/python/src/pact_adapters/pact_agent.py b/adapters/python/src/pact_adapters/pact_agent.py index 358722a..52f2aeb 100644 --- a/adapters/python/src/pact_adapters/pact_agent.py +++ b/adapters/python/src/pact_adapters/pact_agent.py @@ -1529,7 +1529,7 @@ def _answer_of(self, ran: RunResult) -> Any: rather than being a second, quieter description of the same park. """ if ran.halted == "final": - return self._declared(ran.output) + return self._declared(ran.output, ran.answer) parked = ran.suspension # `_pending` and not `awaiting`, because `awaiting` is the WHOLE batch of # the parked step and the calls the gate cleared have already run. A park @@ -1555,7 +1555,7 @@ def _answer_of(self, ran: RunResult) -> Any: words=(parked.in_words if parked is not None else "") or ran.output, ) - def _declared(self, said: str) -> Any: + def _declared(self, said: str, fields: "Mapping[str, Any] | None" = None) -> Any: """The answer in the shape the author declared, when they declared one. `answers-with:` makes `_output_type_for` a `StructuredDict`, so handing a @@ -1563,12 +1563,22 @@ def _declared(self, said: str) -> Any: how they report a failure is exactly the divergence between two runtimes that PACT exists to remove. - Read and not VALIDATED, deliberately. The author's shape is put to the - model by the harness (`shape_to_ask_for`) and a second check here would be - the `same-setting-twice` mistake: two places deciding whether one answer - keeps one contract. Text that is not the declared shape comes back as the - text the run wrote, because inventing a parse failure at the door would - turn a bad answer into no answer. + HELD ONCE, by the harness, and read from there. The harness holds a + reply to `answers-with:` before it is the answer (`answers_fit`), asks + again when it does not fit, and ends the run `checks-failed` when it + never does — which arrives here as a `Halted`, not as an answer. So a + run that ended `final` has an answer that fits, and `fields` is what + the harness read off it: each declared field, and no other. A second + check here would be the `same-setting-twice` mistake, two places + deciding whether one answer keeps one contract; handing back what the + one check read is not a second check. + + This used to read the reply itself, and "deliberately" handed back + text that was not the declared shape as the text the run wrote: the + harness asked for the shape and checked nothing, so there was no + verdict to pass on. The reading below is what is left of that, for a + run nothing held: a shape asked for in a mode this harness does not + put to the model. """ declared = _output_type_for(self._spec) # `str` is what `_output_type_for` returns for `answers-with-mode: text` @@ -1576,6 +1586,8 @@ def _declared(self, said: str) -> Any: # either would contradict the type this same object publishes. if isinstance(declared, type): return said + if fields: + return dict(fields) try: read = json.loads(said) except (TypeError, ValueError): diff --git a/adapters/python/src/pact_adapters/watches.py b/adapters/python/src/pact_adapters/watches.py index 8a9c685..292a7d9 100644 --- a/adapters/python/src/pact_adapters/watches.py +++ b/adapters/python/src/pact_adapters/watches.py @@ -234,6 +234,7 @@ def write_down(base: "Path", watch: "Watch", record: Mapping[str, Any]) -> bool: "turn.delegate.failed", "turn.limit.completed", "turn.message.completed", + "turn.message.failed", "turn.run.cancelled", "turn.run.completed", "turn.run.failed", diff --git a/adapters/python/tests/fitting.py b/adapters/python/tests/fitting.py new file mode 100644 index 0000000..1a1247e --- /dev/null +++ b/adapters/python/tests/fitting.py @@ -0,0 +1,74 @@ +"""Scripted answers that are what the agent says it gives. + +An agent's answer is held to its `answers-with:` (`pact_adapters.answers_fit`). +A test that scripts a model for the worked example — the Refund Desk gives a +`decision`, a `reason` and an `amount` — and is not ABOUT the answer has to +script one that fits, or what it measures is the harness asking again. +""" + +from __future__ import annotations + +import re +from typing import Mapping + +from pact_adapters.questions import Shape + +#: The three lines the Refund Desk's answer ends with, for a refund it grants. +REFUND = "- decision: approved\n- reason: as explained above\n- amount: 45 USD" + + +def fits(said: str = "", decision: str = "", amount: str = "") -> str: + """`said`, ended the way the Refund Desk is asked to end its reply. + + The three fields say what `said` says, so a test that grades the words is + grading the same answer: `declined` when the words decline, the amount the + words name (none for a decline). The reason names no rule, so it cannot + satisfy a check the words were meant to.""" + decision = decision or ("declined" if "declin" in said.lower() else "approved") + named = re.search(r"\d+(?:\.\d+)? USD", said) + amount = amount or (named.group(0) if named else "0 USD" if decision == "declined" else "45 USD") + lines = f"- decision: {decision}\n- reason: as explained above\n- amount: {amount}" + return f"{said}\n{lines}" if said else lines + + +#: What the worked example's two checkers end their replies with. +FRAUD = "- concern: none\n- why: nothing suspicious" +POLICY = "- allowed: yes\n- clause: rule 2, faulty on arrival" + + +def fraud_says(said: str = "") -> str: + return f"{said}\n{FRAUD}" if said else FRAUD + + +def policy_says(said: str = "") -> str: + return f"{said}\n{POLICY}" if said else POLICY + + +def fits_whoever_asks(said: str = "") -> str: + """`said`, ended with what each of the worked example's three agents gives. + + For a stand-in that is handed to the desk AND to whichever checker the + desk consults, and cannot tell which is asking. A field an agent does not + declare is not a reason to ask again and is dropped, so the one reply is + an answer for all three.""" + return f"{fits(said)}\n{POLICY}\n{FRAUD}" + + +def one_that_fits( + shape: Mapping[str, str], + shapes: Mapping[str, Mapping[str, str]] | None = None, + said: str = "", +) -> str: + """A reply that fits `shape`: `said`, and then each field the shape + requires on a line of its own, with the example its shape reader gives + (`Shape.example`), the way the model is asked to end its reply. + + For a stand-in that answers for ANY agent (the golden set runs every agent + in `examples/`). An optional field is left out, which fits. With no shape + it is `said`.""" + lines = [] + for name, written in shape.items(): + if str(written).rstrip().endswith(", optional"): + continue + lines.append(f"- {name}: {Shape.parse(written, dict(shapes or {})).example()}") + return "\n".join(part for part in (said, "\n".join(lines)) if part) diff --git a/adapters/python/tests/test_a_connect_line_reaches_a_server_without_a_handshake.py b/adapters/python/tests/test_a_connect_line_reaches_a_server_without_a_handshake.py index 96b8ba7..384e7a0 100644 --- a/adapters/python/tests/test_a_connect_line_reaches_a_server_without_a_handshake.py +++ b/adapters/python/tests/test_a_connect_line_reaches_a_server_without_a_handshake.py @@ -58,6 +58,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) from consenting import allowing_the_connection # noqa: E402 +from fitting import fits # noqa: E402 from pact_adapters.harness import ToolCall # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.mcp.calling import tool_impls_for # noqa: E402 @@ -781,7 +782,7 @@ def script() -> Script: ), ), ), - Turn("Done."), + Turn(fits("Done.")), ] ) diff --git a/adapters/python/tests/test_a_failing_tool_is_data_and_a_cancelled_run_is_not.py b/adapters/python/tests/test_a_failing_tool_is_data_and_a_cancelled_run_is_not.py index 37469a8..7bb6b19 100644 --- a/adapters/python/tests/test_a_failing_tool_is_data_and_a_cancelled_run_is_not.py +++ b/adapters/python/tests/test_a_failing_tool_is_data_and_a_cancelled_run_is_not.py @@ -51,6 +51,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters import harness # noqa: E402 from pact_adapters.harness import RunResult, ToolCall # noqa: E402 from pact_adapters.ir import AgentSpec, ToolSpec # noqa: E402 @@ -129,7 +130,7 @@ def three_turns() -> Script: return Script([ Turn("Checking the ticket.", (ToolCall(LOOKS_UP, {"ticket": "T-1"}),)), Turn("Issuing the refund.", (ToolCall(SPENDS, {"amount": "40"}),)), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) @@ -485,9 +486,9 @@ def careful_turns() -> Script: """One lookup, then one plain answer per remaining stage of `loop: careful`.""" return Script([ Turn("Checking the ticket.", (ToolCall(LOOKS_UP, {"ticket": "T-1"}),)), - Turn("The ticket says the lamp arrived broken 6 days ago."), - Turn("Rule 2 applies: faulty on arrival, and 6 days is inside the window."), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("The ticket says the lamp arrived broken 6 days ago.")), + Turn(fits("Rule 2 applies: faulty on arrival, and 6 days is inside the window.")), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) diff --git a/adapters/python/tests/test_a_halted_run_does_not_pretend_to_be_an_answer.py b/adapters/python/tests/test_a_halted_run_does_not_pretend_to_be_an_answer.py index aef0c16..1d95be0 100644 --- a/adapters/python/tests/test_a_halted_run_does_not_pretend_to_be_an_answer.py +++ b/adapters/python/tests/test_a_halted_run_does_not_pretend_to_be_an_answer.py @@ -995,31 +995,32 @@ def test_a_run_id_and_a_conversation_id_the_caller_gave_reach_the_result( @pytest.mark.parametrize( - "wrote,comes_back", + "wrote", [ - ("just the words, no JSON at all", "just the words, no JSON at all"), - ("[1, 2]", "[1, 2]"), - ('{"decision": "approved"}', {"decision": "approved"}), + "just the words, no JSON at all", + "[1, 2]", + # A mapping, and still not what the agent gives: two fields are missing. + '{"decision": "approved"}', + '{"decision": "maybe", "reason": "r", "amount": "40 USD"}', ], ) -def test_an_answer_that_is_not_the_declared_shape_comes_back_as_what_the_run_wrote( - pact_agent: Any, spec: AgentSpec, wrote: str, comes_back: Any +def test_an_answer_that_is_not_the_declared_shape_is_not_an_answer( + pact_agent: Any, halted_type: Any, spec: AgentSpec, wrote: str ) -> None: - """Read and not VALIDATED, deliberately — and a JSON list is not a mapping. - - `answers-with:` makes `_output_type_for` a `StructuredDict`, so a caller - handed the JSON TEXT still has to parse it, and where they parse it and how - they report a failure is exactly the divergence between two runtimes PACT - exists to remove. But the shape is put to the model by the harness, and a - SECOND check here would be the `same-setting-twice` mistake: two places - deciding whether one answer keeps one contract. So text that is not the - declared shape comes back as the text the run wrote — inventing a parse - failure at the door turns a bad answer into no answer. - - The list is the case a bare `json.loads` gets wrong: it parses, so a - crossing that only catches `ValueError` hands a caller expecting a mapping - something with no keys at all, and the failure surfaces wherever they first - subscript it. + """Held ONCE, by the harness, and what it decided is what arrives here. + + This test was `..._comes_back_as_what_the_run_wrote`, and held the opposite: + text that was not the declared shape came back as text, "deliberately", + because the harness put the shape to the model and checked nothing, and a + check at this door would have been a second place deciding one contract. + The reasoning stands and the conclusion moved with the harness: it now + holds a reply to `answers-with:` (`answers_fit`), so a run whose reply + never fits ends `checks-failed`, and that is a `Halted` here — a TYPE, the + way every other run that did not finish arrives — and never the text, and + never a mapping with fields missing. + + The JSON list is still the case a bare `json.loads` gets wrong, and it is + refused for the same reason the prose is: no field the agent gives is in it. """ agent = pact_agent.for_spec( replace(spec, loop=Loop.from_library(STANDARD)), @@ -1027,8 +1028,36 @@ def test_an_answer_that_is_not_the_declared_shape_comes_back_as_what_the_run_wro tool_impls={}, ) said = agent.run_sync(ASKED).output - assert said == comes_back - assert type(said) is type(comes_back) + assert isinstance(said, halted_type), said + assert said.halted == "checks-failed" + assert said.words.startswith( + "Stopped without an answer: it was asked for its answer 2 times and none fit " + "`answers-with:`." + ) + + +@pytest.mark.parametrize( + "wrote", + [ + '{"decision": "approved", "reason": "damaged on arrival", "amount": "40 USD", "mood": "x"}', + "Approved.\n- decision: approved\n- reason: damaged on arrival\n- amount: 40 USD\n- mood: x", + ], +) +def test_an_answer_that_fits_comes_back_as_the_fields_the_agent_gives_and_no_others( + pact_agent: Any, spec: AgentSpec, wrote: str +) -> None: + """One JSON record, or the fields one to a line the way the model is asked + to end its reply: either is the answer, and what a caller is handed is the + mapping of declared fields. `mood` is not something this agent gives, so it + is not a reason to ask again and it is not handed back.""" + agent = pact_agent.for_spec( + replace(spec, loop=Loop.from_library(STANDARD)), + transport=PydanticAITransport(Script([Turn(wrote)])), + tool_impls={}, + ) + assert agent.run_sync(ASKED).output == { + "decision": "approved", "reason": "damaged on arrival", "amount": "40 USD", + } def test_an_agent_that_declares_no_shape_is_handed_its_answer_as_the_text_it_wrote( diff --git a/adapters/python/tests/test_a_person_can_say_no.py b/adapters/python/tests/test_a_person_can_say_no.py index 5e5fb37..4a1fc4a 100644 --- a/adapters/python/tests/test_a_person_can_say_no.py +++ b/adapters/python/tests/test_a_person_can_say_no.py @@ -45,6 +45,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) from consenting import allowing_the_connection # noqa: E402 +from fitting import fits # noqa: E402 from pact_adapters.events import Bus # noqa: E402 from pact_adapters.harness import ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec, ToolSpec # noqa: E402 @@ -102,7 +103,7 @@ def batch() -> Script: ToolCall("payments", {"amount": 40}), ToolCall("email", {"to": "customer"}), )), - Turn("All done."), + Turn(fits("All done.")), ]) @@ -421,7 +422,7 @@ def a_refund_over_the_written_limit() -> Script: return Script([ Turn("Refunding.", (ToolCall("payments", {"order-number": "A-1182", "amount": "300.00 USD"}),)), - Turn("Refunded 300.00 USD."), + Turn(fits("Refunded 300.00 USD.")), ]) diff --git a/adapters/python/tests/test_a_read_only_workspace_still_runs.py b/adapters/python/tests/test_a_read_only_workspace_still_runs.py index 36efb50..caef650 100644 --- a/adapters/python/tests/test_a_read_only_workspace_still_runs.py +++ b/adapters/python/tests/test_a_read_only_workspace_still_runs.py @@ -37,6 +37,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.harness import ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.script import Script, Turn # noqa: E402 @@ -73,7 +74,7 @@ def _go(spec: AgentSpec) -> Any: """ script = Script([ Turn("Looking it up.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("done"), + Turn(fits("done")), ]) return asyncio.run( run(spec, ReferenceTransport(script), "hello", {"zendesk": lambda a: "a ticket"}) diff --git a/adapters/python/tests/test_a_run_that_reaches_a_ceiling_does_what_the_author_said.py b/adapters/python/tests/test_a_run_that_reaches_a_ceiling_does_what_the_author_said.py index a70e5be..fe81582 100644 --- a/adapters/python/tests/test_a_run_that_reaches_a_ceiling_does_what_the_author_said.py +++ b/adapters/python/tests/test_a_run_that_reaches_a_ceiling_does_what_the_author_said.py @@ -369,10 +369,13 @@ def test_a_budget_once_granted_is_carried_by_the_park_and_across_a_process(tmp_p def test_the_closing_request_is_told_what_every_other_request_is_told(tmp_path: Path) -> None: - turns = [look("a"), look("b"), Turn("a and b are there.")] + # The closing answer is held to what the agent gives (`found: list of + # text`) like any other, so it ends with it. + closing_answer = 'a and b are there.\n- found: ["a", "b"]' + turns = [look("a"), look("b"), Turn(closing_answer)] out, _again, looked, told = go(tmp_path, "closes", turns) assert out.halted == "tool-call-limit" and looked == ["a", "b"] - assert out.output == "a and b are there." + assert out.output == closing_answer and out.answer == {"found": ["a", "b"]} closing = told[-1] assert closing.startswith("0 tools\n"), "the closing request offers no tool" # The agent's own instructions, the shape its answer must have, and what diff --git a/adapters/python/tests/test_action_scoped_approvals.py b/adapters/python/tests/test_action_scoped_approvals.py index 57a89e4..5ab841b 100644 --- a/adapters/python/tests/test_action_scoped_approvals.py +++ b/adapters/python/tests/test_action_scoped_approvals.py @@ -42,6 +42,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) from consenting import allowing_the_connection # noqa: E402 +from fitting import fits # noqa: E402 from pact_adapters.harness import ToolCall # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.questions import questions_for # noqa: E402 @@ -99,7 +100,7 @@ def desk(document: dict) -> AgentSpec: def one_call(name: str, args: dict): """A factory, not a script: a `Script` is a cursor and `ran` resumes.""" return lambda: Script( - [Turn("working on it", (ToolCall(name, args),)), Turn("Approved.")] + [Turn("working on it", (ToolCall(name, args),)), Turn(fits("Approved."))] ) diff --git a/adapters/python/tests/test_an_answer_is_checked_before_it_is_the_answer.py b/adapters/python/tests/test_an_answer_is_checked_before_it_is_the_answer.py index ce4bee1..bf04aba 100644 --- a/adapters/python/tests/test_an_answer_is_checked_before_it_is_the_answer.py +++ b/adapters/python/tests/test_an_answer_is_checked_before_it_is_the_answer.py @@ -86,10 +86,14 @@ async def respond(messages: list, info: object) -> ModelResponse: assert retries and str(retries[0].content).startswith(CHECK_FAILED) -def test_pacts_own_harness_says_it_holds_none_of_the_three_rather_than_staying_silent() -> None: - """`harness.run` has no reader for `checked-by:`, `may-start:` or a model - list. It returned a failing answer with nothing on `unenforced`, so - `pact-eval` scored an agent whose checks never ran as though they had.""" +def test_pacts_own_harness_holds_the_rules_and_says_the_two_lines_it_does_not_hold() -> None: + """`harness.run` had no reader for `checked-by:`, `may-start:` or a model + list. First it returned a failing answer with nothing on `unenforced`, so + `pact-eval` scored an agent whose checks never ran as though they had; then + it said all three were unheld. It holds the first now (`answers_fit`): an + answer that breaks a rule is asked for again, `checks-at-most:` times in + all, and is never returned. The other two it still does not hold, and + still says so.""" import asyncio from pact_adapters.harness import run @@ -103,12 +107,19 @@ def test_pacts_own_harness_says_it_holds_none_of_the_three_rather_than_staying_s "team": {"helper": "helps"}, "teamwork": {"may-start": ["narrowed-new"]}, }) - out = asyncio.run(run(spec, ReferenceTransport(Script([Turn("Sorry about that.")])), "Refund?", {})) - assert out.output == "Sorry about that.", "the answer breaks the rule and is still returned" + sorry = Script([Turn("Sorry about that.")]) + out = asyncio.run(run(spec, ReferenceTransport(sorry), "Refund?", {})) + assert out.halted == "checks-failed" and sorry.calls == 2, (out.halted, sorry.calls) + assert out.output != "Sorry about that.", "the answer breaks the rule and is not returned" + assert "does not say ['refund']" in out.output said = {u.split(":", 1)[0]: u for u in out.unenforced} - assert "returns the answer unchecked" in said["checked-by"], out.unenforced + assert "checked-by" not in said, "held, so there is nothing to report" assert "narrowed-new" in said["may-start"], out.unenforced assert "big-one" in said["model"] and "small-one" in said["model"], out.unenforced + # An answer that keeps the rule is the answer, at the first asking. + kept = Script([Turn("A refund is on its way.")]) + out = asyncio.run(run(spec, ReferenceTransport(kept), "Refund?", {})) + assert out.halted == "final" and kept.calls == 1 # And an agent that wrote none of them is told nothing about them. plain = asyncio.run(run(_spec({"name": "Desk"}), ReferenceTransport(Script([Turn("ok")])), "x", {})) assert not [u for u in plain.unenforced if u.startswith(("checked-by", "may-start", "model"))] diff --git a/adapters/python/tests/test_an_answer_is_held_to_what_the_agent_says_it_gives.py b/adapters/python/tests/test_an_answer_is_held_to_what_the_agent_says_it_gives.py new file mode 100644 index 0000000..cd6bdc9 --- /dev/null +++ b/adapters/python/tests/test_an_answer_is_held_to_what_the_agent_says_it_gives.py @@ -0,0 +1,367 @@ +"""An agent's answer is held to what the agent says it gives. + +`answers-with:` is a promise to whoever calls the agent: a workflow stage that +binds `steps.decide.amount`, a program that reads the field, a person shown it. +The harness put the shape to the model and then handed back whatever came: +"Sure, I think we should refund them." from an agent that gives a decision and +an amount was the run's answer, `halted: final`, and the caller found out when +it read a field that was not there. `checked-by:`, the author's own rules for +an answer, was read by nothing in this loop: the run SAID so (`unenforced`) +and returned the answer unchecked. + +The owner's ruling (14 November): "if there is a promised shape, it must be +honored." What is held, and how (`answers_fit`, ruled with Bud Agent Flow): + +* a field left out, or a value its shape refuses, does not fit; +* the model is asked again and told what was wrong, in the author's own words + for the shape, at its next request and at no other; +* at most `checks-at-most:` times in all, 2 unwritten, one count for the shape + and for `checked-by:`, and the closing answer of a run that reached a limit + spends from it too; +* still not fitting, the run ends `checks-failed` and hands back no answer; +* a field the agent does not declare is not a reason to ask again and is not + handed back; an empty reply that arrived is an attempt. +""" + +from __future__ import annotations + +import asyncio +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from pact_adapters.answers_fit import ( # noqa: E402 + CHECK_FAILED, + CHECKS_FAILED, + DID_NOT_FIT, + Held, + does_not_fit, + fields_of, +) +from pact_adapters.harness import ToolCall, run # noqa: E402 +from pact_adapters.ir import AgentSpec # noqa: E402 +from pact_adapters.limits import RAN_OUT # noqa: E402 +from pact_adapters.script import Script, Turn # noqa: E402 +from pact_adapters.suspension import Suspension # noqa: E402 +from pact_adapters.transports.mock import ReferenceTransport # noqa: E402 +from trees import shown # noqa: E402 + +WORKSPACE = """\ +workspace "Refunds" + description: A desk that decides refunds, and says what it decided. + +resource office + resource-kind: mcp-server + endpoint: host/office + description: Where the orders are reached. + +tool orders + description: The order book. + connect: office + actions: + look: + description: Read one order. + reads-only: yes + takes: + order: text + refund: + description: Refund one order. + needs-a-person: yes + takes: + order: text + +agent desk + description: Decides a refund request. + uses orders + gives decision: one of approved, declined, amount: money, note: text, optional + instructions: Decide the refund. + +agent strict + description: Decides a refund request, and gets one try at saying so. + gives decision: one of approved, declined + checks-at-most: 1 + instructions: Decide the refund. + +agent ruled + description: Decides a refund request and must say what happens to the money. + gives decision: one of approved, declined + checked-by: + - must-contain: [refund] + because: say what happens to the money + instructions: Decide the refund. + +agent only-ruled + description: Says what happens to the money, in its own words. + checked-by: + - must-contain: [refund] + checks-at-most: 3 + instructions: Say what happens to the money. + +agent free + description: Promises nothing about its answer. + instructions: Help. + +agent closes + description: Looks orders up, and answers with what it has when it runs out. + uses orders + gives decision: one of approved, declined + at most 1 tool calls + when it runs out: answer with what it has + instructions: Decide the refund. +""" + +GOOD = "The item arrived broken.\\n- decision: approved\\n- amount: 45 USD".replace("\\n", "\n") +VAGUE = "Sure, I think we should refund them." + + +class Counted(ReferenceTransport): + """A scripted model that answers by how many times it has been ASKED, and + keeps the system text of each request. By count and not by the + conversation (`Script`'s way), because a closing request and the request + before it are sent the same conversation: only the count tells them apart.""" + + def __init__(self, turns: list[Turn], told: list[str]) -> None: + super().__init__(Script(turns)) + self.turns, self.told = turns, told + + async def model_call(self, system, history, tools): # type: ignore[override] + turn = self.turns[min(len(self.told), len(self.turns) - 1)] + self.told.append(system) + return turn.text, list(turn.tool_calls) + + +class Asked: + def __init__(self, told: list[str]) -> None: + self.told = told + + @property + def calls(self) -> int: + return len(self.told) + + +def go(tmp_path: Path, agent: str, turns: list[str | Turn], **kw: Any): + (tmp_path / "refunds.pact").write_text(WORKSPACE, encoding="utf-8") + spec = AgentSpec.from_document(shown(tmp_path), agent) + told: list[str] = [] + scripted = [t if isinstance(t, Turn) else Turn(t) for t in turns] + tools = {"orders": lambda args: "order 7: 45 USD, delivered broken"} + + def again(**more: Any): + return asyncio.run( + run(spec, Counted(scripted, told), "Order 7 arrived broken.", tools, **more) + ) + + return again(**kw), told, Asked(told), again + + +# ── an answer that fits ────────────────────────────────────────────────────── + + +def test_an_answer_that_fits_is_the_answer_and_its_fields_are_handed_back(tmp_path: Path) -> None: + out, told, script, _ = go(tmp_path, "desk", [GOOD]) + assert out.halted == "final" and out.output == GOOD + assert out.answer == {"decision": "approved", "amount": "45 USD"} + assert script.calls == 1, "nothing was asked twice" + assert DID_NOT_FIT not in told[0] + + +def test_an_optional_field_left_out_fits_and_one_the_agent_does_not_declare_is_dropped( + tmp_path: Path, +) -> None: + said = GOOD + "\n- mood: cheerful" + out, _, script, _ = go(tmp_path, "desk", [said]) + assert out.halted == "final" and script.calls == 1, "an extra field is no reason to ask again" + assert out.output == said, "the reply is the model's, as written" + assert out.answer == {"decision": "approved", "amount": "45 USD"}, "and it is not handed back" + + +def test_an_answer_written_as_one_json_record_fits_too(tmp_path: Path) -> None: + out, _, _, _ = go(tmp_path, "desk", ['{"decision": "declined", "amount": "0 USD", "x": 1}']) + assert out.halted == "final" + assert out.answer == {"decision": "declined", "amount": "0 USD"} + + +# ── one that does not ──────────────────────────────────────────────────────── + + +def test_an_answer_that_does_not_fit_is_asked_for_again_and_told_why(tmp_path: Path) -> None: + out, told, script, _ = go(tmp_path, "desk", [VAGUE, GOOD]) + assert out.halted == "final" and out.output == GOOD + assert out.answer == {"decision": "approved", "amount": "45 USD"} + assert script.calls == 2 + # Told at the second request, last, in the author's words for each field. + assert DID_NOT_FIT not in told[0] + assert told[1].endswith( + "Your last answer did not fit what your answer must contain:\n" + "- `decision` is missing. It should be: one of approved, declined.\n" + "- `amount` is missing. It should be: money.\n" + "Answer again, with every field as it is asked for." + ), told[1][-400:] + # `note` is optional, so it is not named. And both replies are on the record. + assert "`note`" not in told[1] + assert [s.text for s in out.steps] == [VAGUE, GOOD] + + +def test_a_value_the_shape_refuses_is_named_with_what_it_should_be(tmp_path: Path) -> None: + out, told, _, _ = go(tmp_path, "desk", ["- decision: maybe\n- amount: 45 USD", GOOD]) + assert out.halted == "final" + assert "- `decision` " in told[1] and "approved, declined" in told[1] and "maybe" in told[1] + assert "`amount`" not in told[1], "the field that fit is not complained about" + + +def test_it_is_told_once_and_no_request_after_that(tmp_path: Path) -> None: + # Misfit, then a tool call, then the answer: the notice is in the request + # that follows the misfit and in no other. + look = Turn("", (ToolCall("orders", {"action": "look", "order": "7"}),)) + out, told, _, _ = go(tmp_path, "desk", [VAGUE, look, GOOD]) + assert out.halted == "final", out.output + assert [DID_NOT_FIT in t for t in told] == [False, True, False] + + +def test_still_not_fitting_the_run_ends_checks_failed_and_hands_back_nothing(tmp_path: Path) -> None: + out, told, script, _ = go(tmp_path, "desk", [VAGUE, VAGUE, GOOD]) + assert out.halted == CHECKS_FAILED + assert script.calls == 2, "`checks-at-most:` is 2 unwritten: two attempts in all, no third" + assert out.answer == {} + assert out.output == ( + "Stopped without an answer: it was asked for its answer 2 times and none fit " + "`answers-with:`. Your last answer did not fit what your answer must contain:\n" + "- `decision` is missing. It should be: one of approved, declined.\n" + "- `amount` is missing. It should be: money." + ) + # What it wrote is in the steps, for whoever has to find out why. + assert [s.text for s in out.steps] == [VAGUE, VAGUE] + + +def test_checks_at_most_one_fails_at_the_first_miss(tmp_path: Path) -> None: + out, _, script, _ = go(tmp_path, "strict", [VAGUE, "- decision: approved"]) + assert out.halted == CHECKS_FAILED and script.calls == 1 + assert out.output.startswith( + "Stopped without an answer: it was asked for its answer once and it did not fit " + "`answers-with:`." + ) + + +def test_an_empty_reply_that_arrived_is_an_attempt(tmp_path: Path) -> None: + out, _, script, _ = go(tmp_path, "strict", ["", "- decision: approved"]) + assert out.halted == CHECKS_FAILED and script.calls == 1 + + +def test_an_agent_that_promises_nothing_is_held_to_nothing(tmp_path: Path) -> None: + out, _, script, _ = go(tmp_path, "free", [""]) + assert out.halted == "final" and out.output == "" and out.answer == {} + assert script.calls == 1 + + +# ── the author's own rules, from the same count ────────────────────────────── + + +def test_a_rule_an_answer_breaks_is_told_and_the_answer_asked_for_again(tmp_path: Path) -> None: + out, told, script, _ = go( + tmp_path, "ruled", ["- decision: approved", "A refund is on its way.\n- decision: approved"] + ) + assert out.halted == "final" and script.calls == 2 + assert told[1].endswith( + f"{CHECK_FAILED}does not say ['refund'] (say what happens to the money). " + "Answer again so that it keeps every rule." + ), told[1][-300:] + assert all("checked-by" not in u for u in out.unenforced), "it is held now, not reported" + + +def test_the_shape_and_the_rules_spend_from_one_count(tmp_path: Path) -> None: + # First it does not fit; then it fits and breaks the rule. Two attempts in + # all, so that is the end: it does not get two for each. + out, _, script, _ = go(tmp_path, "ruled", [VAGUE, "- decision: approved", GOOD]) + assert out.halted == CHECKS_FAILED and script.calls == 2 + assert out.output.startswith( + "Stopped without an answer: it was asked for its answer 2 times and none fit " + "its `checked-by:` rules." + ) + assert "does not say ['refund']" in out.output + + +def test_rules_alone_hold_an_answer_that_promises_no_shape(tmp_path: Path) -> None: + out, _, script, _ = go(tmp_path, "only-ruled", ["Sorry.", "Sorry again.", "We will refund it."]) + assert out.halted == "final" and script.calls == 3 and out.answer == {} + + +# ── a run that reached a limit ─────────────────────────────────────────────── + + +LOOK = Turn("", (ToolCall("orders", {"action": "look", "order": "7"}),)) + + +def test_a_closing_answer_that_fits_is_the_answer(tmp_path: Path) -> None: + out, told, asked, _ = go(tmp_path, "closes", [LOOK, "- decision: approved"]) + assert out.halted in RAN_OUT and out.stopped_by is not None + assert out.output == "- decision: approved" and out.answer == {"decision": "approved"} + assert asked.calls == 2, "one step, which used the one call, and one closing request" + + +def test_a_closing_answer_that_does_not_fit_is_asked_for_again_with_no_tools(tmp_path: Path) -> None: + out, told, _, _ = go(tmp_path, "closes", [LOOK, VAGUE, "- decision: declined"]) + assert out.halted in RAN_OUT, "it did run out, and still says so" + assert out.answer == {"decision": "declined"} + # The second closing request is the first, and then why. + assert told[-1].startswith(told[-2]) and told[-1].endswith( + "Answer again, with every field as it is asked for." + ) + + +def test_a_closing_answer_that_never_fits_is_not_handed_back(tmp_path: Path) -> None: + out, _, _, _ = go(tmp_path, "closes", [LOOK, VAGUE, VAGUE, "- decision: declined"]) + # The ceiling's own word, because it did run out; and no answer. + assert out.halted in RAN_OUT and out.halted != CHECKS_FAILED + assert out.answer == {} + assert out.output == ( + "Stopped before finishing: this run reached the limit you set with " + "`tool-calls-at-most` (1 of 1 tool calls). It was asked for its answer 2 times and " + "none fit what it says it gives." + ) + + +# ── across a pause ─────────────────────────────────────────────────────────── + + +def test_a_run_that_parks_between_two_attempts_is_not_given_the_count_again(tmp_path: Path) -> None: + # A miss, then a call a person must approve, then two more misses. With the + # count lost at the pause, the run would be asked a third time. + refund = Turn("", (ToolCall("orders", {"action": "refund", "order": "7"}),)) + out, _, script, again = go(tmp_path, "desk", [VAGUE, refund, VAGUE, GOOD]) + assert out.halted == "suspended", (out.halted, out.output) + parked = Suspension.from_json(out.suspension.to_json()) + assert parked.used["answers-asked-for"] == 1 + resumed = again(resume=parked, answer=parked.answer(**{n.name: "yes" for n in parked.asks})) + assert resumed.halted == CHECKS_FAILED, (resumed.halted, resumed.output) + assert "2 times" in resumed.output + + +# ── the reader itself ──────────────────────────────────────────────────────── + + +def test_fields_are_read_off_their_own_lines_the_last_one_winning() -> None: + shape = {"decision": "one of approved, declined", "items": "list of text"} + said = "decision: declined\nOn reflection:\n**decision**: approved\n* items: [\"a\", \"b\"]\nother: x" + assert fields_of(said, shape) == {"decision": "approved", "items": ["a", "b"]} + assert does_not_fit(said, shape) == () + assert fields_of("nothing here", shape) == {} + + +def test_a_hold_with_nothing_to_hold_counts_nothing() -> None: + held = Held() + assert held.wrong("") == "" and held.asked == 0 and not held.holds_anything + + +def test_a_field_that_streams_is_the_reply_itself() -> None: + # `reply: stream of text` is what the reply IS as it arrives. Nobody ends a + # stream with `- reply: ...`; only saying nothing leaves it out. + shape = {"reply": "stream of text"} + assert does_not_fit("The total is 42.", shape) == () + assert fields_of("The total is 42.", shape) == {"reply": "The total is 42."} + assert does_not_fit("", shape) == ("`reply` is missing. It should be: stream of text.",) + both = {"reply": "stream of text", "total": "number"} + assert does_not_fit("The total is 42.", both) == ("`total` is missing. It should be: number.",) + assert fields_of("The total is 42.\n- total: 42", both)["total"] == "42" diff --git a/adapters/python/tests/test_connection_consent.py b/adapters/python/tests/test_connection_consent.py index eb665ef..0ec7b25 100644 --- a/adapters/python/tests/test_connection_consent.py +++ b/adapters/python/tests/test_connection_consent.py @@ -43,6 +43,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters import suspension # noqa: E402 from pact_adapters.harness import ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 @@ -96,7 +97,7 @@ def a_refund_of(amount: str) -> Script: return Script([ Turn("Issuing the refund.", (ToolCall("payments", {"order-number": "A-1182", "amount": amount}),)), - Turn(f"Approved: refunded {amount}."), + Turn(fits(f"Approved: refunded {amount}.")), ]) @@ -394,7 +395,7 @@ def two_refunds_and_a_lookup() -> Script: ToolCall("payments", {"order-number": "A-300", "amount": "300.00 USD"}), ToolCall("payments", {"order-number": "A-040", "amount": "40.00 USD"}), )), - Turn("Done."), + Turn(fits("Done.")), ]) diff --git a/adapters/python/tests/test_context_policy.py b/adapters/python/tests/test_context_policy.py index cee2be1..2828e84 100644 --- a/adapters/python/tests/test_context_policy.py +++ b/adapters/python/tests/test_context_policy.py @@ -23,6 +23,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits, fraud_says, policy_says # noqa: E402 from pact_adapters.context_policy import ( # noqa: E402 CHARS_PER_TOKEN, ContextPolicy, @@ -1170,7 +1171,7 @@ def a_refund_worth_checking() -> "Script": # noqa: F821 Turn("Checking the ticket.", (ToolCall("zendesk", {"ticket": "T-1"}),)), Turn("The ticket says the lamp arrived broken 6 days ago."), Turn("Rule 2 applies: faulty on arrival, and 6 days is inside the window."), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) @@ -1203,9 +1204,13 @@ def a_refund_with_more_prose_than_shortening_can_fix(window: int) -> "Script": said = _RAMBLE * ((window * CHARS_PER_TOKEN) // len(_RAMBLE) + 1) return Script([ Turn("Checking the ticket.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("The ticket says the lamp arrived broken 6 days ago. " + said), - Turn("Rule 2 applies: faulty on arrival, and 6 days is inside the window."), - Turn("Approved: the item arrived damaged within 30 days."), + # Every reply ends with what the agent gives, as the shape asks of + # every stage: shortening the conversation moves which of these a + # script keyed on the conversation says next, and whichever one lands + # in the last stage has to be an answer. + Turn(fits("The ticket says the lamp arrived broken 6 days ago. " + said)), + Turn(fits("Rule 2 applies: faulty on arrival, and 6 days is inside the window.")), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) @@ -1882,19 +1887,19 @@ async def model_call(self, system, history, tools): Turn("Checking the ticket.", (ToolCall("zendesk", {"ticket": "T-1"}),)), Turn("The lamp arrived broken 6 days ago. " + said, (ToolCall("zendesk", {"ticket": "T-2"}),)), - Turn("Rule 2 applies, and 6 days is inside the window."), + Turn(policy_says("Rule 2 applies, and 6 days is inside the window.")), ]) def transport_for(member: AgentSpec): if member.name == "Policy Checker": return Blocking(rambles) - return Unhurried(Script([Turn("nothing suspicious")])) + return Unhurried(Script([Turn(fraud_says("nothing suspicious"))])) asking_both = Script([ Turn("Consulting the team.", (ToolCall("policy-checker", {"question": "is this covered?"}), ToolCall("fraud-checker", {"question": "does this look real?"}))), - Turn("Approved."), + Turn(fits("Approved.")), ]) desk = AgentSpec.from_document(doc, "refund-desk", EXAMPLE) diff --git a/adapters/python/tests/test_join_policy.py b/adapters/python/tests/test_join_policy.py index d51a1a2..9cec17a 100644 --- a/adapters/python/tests/test_join_policy.py +++ b/adapters/python/tests/test_join_policy.py @@ -24,6 +24,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits, fraud_says, policy_says # noqa: E402 from pact_adapters.delegation import ( # noqa: E402 ANSWERED, Answer, @@ -523,7 +524,7 @@ def asking_both() -> Script: Turn("Consulting the team.", (ToolCall("policy-checker", {"question": "is this covered?"}), ToolCall("fraud-checker", {"question": "does this look real?"}))), - Turn("Approved."), + Turn(fits("Approved.")), ] ) @@ -582,7 +583,7 @@ async def ask(grant: Grant) -> str: [ Turn("First pass.", (ToolCall("policy-checker", {"question": "a"}),)), Turn("Second pass.", (ToolCall("policy-checker", {"question": "b"}),)), - Turn("Approved."), + Turn(fits("Approved.")), ] ) result = asyncio.run( @@ -779,11 +780,11 @@ def test_every_transport_reaches_the_same_place_when_the_team_is_asked() -> None def test_a_run_that_never_asks_the_team_is_unchanged_by_the_mechanism() -> None: """It must cost nothing when unused — the same bar the interceptor chain is held to.""" - plain = Script([Turn("checking", (ToolCall("zendesk", {}),)), Turn("Approved.")]) + plain = Script([Turn("checking", (ToolCall("zendesk", {}),)), Turn(fits("Approved."))]) without = asyncio.run(run(SPEC, ReferenceTransport(plain), "refund?", TOOLS)) with_team = asyncio.run( run(SPEC, ReferenceTransport( - Script([Turn("checking", (ToolCall("zendesk", {}),)), Turn("Approved.")])), + Script([Turn("checking", (ToolCall("zendesk", {}),)), Turn(fits("Approved."))])), "refund?", TOOLS, ask_member=answering(), teamwork=Teamwork()) ) assert without.trace() == with_team.trace() @@ -797,8 +798,8 @@ def test_a_teammate_is_run_by_this_same_harness_one_level_down(document: dict) - """Delegation is not a second execution model, so it does not get a second loop. The child is an agent in the same folder, run by the same `run`.""" child_scripts = { - "policy-checker": Script([Turn("Allowed under clause 3.")]), - "fraud-checker": Script([Turn("Nothing suspicious.")]), + "policy-checker": Script([Turn(policy_says("Allowed under clause 3."))]), + "fraud-checker": Script([Turn(fraud_says("Nothing suspicious."))]), } def transport_for(member_spec: AgentSpec): @@ -812,7 +813,9 @@ def transport_for(member_spec: AgentSpec): teamwork=Teamwork()) ) assert result.halted == "final" - assert result.steps[0].tool_results == ("Allowed under clause 3.", "Nothing suspicious.") + assert result.steps[0].tool_results == ( + policy_says("Allowed under clause 3."), fraud_says("Nothing suspicious."), + ) def test_the_parents_question_reaches_the_teammate_it_was_meant_for(document: dict) -> None: @@ -831,7 +834,7 @@ def transport_for(member_spec: AgentSpec): key = {"Policy Checker": "policy-checker", "Fraud Checker": "fraud-checker"}[ member_spec.name ] - return Listening(Script([Turn("ok")]), key) + return Listening(Script([Turn(fits("ok"))]), key) desk = AgentSpec.from_document(document, "refund-desk") asyncio.run( @@ -1073,7 +1076,7 @@ def usage(self) -> tuple[int, float]: run(AgentSpec.from_document(document, "refund-desk"), ReferenceTransport(asking_both()), "Can I get a refund?", TOOLS, ask_member=delegate_by_running( - document, lambda member: Costly(Script([Turn("checked")])))) + document, lambda member: Costly(Script([Turn(fits("checked"))])))) ) handoff = result.handoffs[0] @@ -1102,7 +1105,7 @@ def test_what_the_team_spent_counts_against_the_ceiling_the_author_wrote( twice = Script([ Turn("asking", (ToolCall("policy-checker", {"message": "a"}),)), Turn("asking again", (ToolCall("policy-checker", {"message": "b"}),)), - Turn("Approved."), + Turn(fits("Approved.")), ]) seen: list[float] = [] @@ -1142,7 +1145,7 @@ class Watching(ReferenceTransport): def transport_for(member: AgentSpec) -> ReferenceTransport: ceilings[member.name] = member.limits.cost_per_request_under - return Watching(Script([Turn("checked")])) + return Watching(Script([Turn(fits("checked"))])) asyncio.run( run(AgentSpec.from_document(document, "refund-desk"), @@ -1177,7 +1180,7 @@ def script() -> Script: (ToolCall("policy-checker", {"question": "is this covered?"}), ToolCall("policy-checker", {"question": "and the fee?"}), ToolCall("fraud-checker", {"question": "does this look real?"}))), - Turn("Approved."), + Turn(fits("Approved.")), ] ) diff --git a/adapters/python/tests/test_judged_rules.py b/adapters/python/tests/test_judged_rules.py index cfe65db..e40b05f 100644 --- a/adapters/python/tests/test_judged_rules.py +++ b/adapters/python/tests/test_judged_rules.py @@ -36,6 +36,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.evals import ( # noqa: E402 JUDGED, Case, @@ -98,6 +99,9 @@ def one_sentence_reader(system: str, user: str) -> str: asked for in `judge.RUBRIC`. """ answer = user.split("ANSWER:", 1)[-1] + # The words written to the customer, and not the three fields the answer + # ends with (`answers-with:`): a field is not a sentence of the reason. + answer = answer.split("\n- decision:", 1)[0] sentences = [s for s in answer.replace("\n", " ").split(".") if s.strip()] if len(sentences) <= 2: return "PASS\nthe decision and one reason, which is all a customer needs." @@ -168,8 +172,8 @@ async def model_call(self, system, history, tools): asked = " ".join(str(m.get("content") or "") for m in history).lower() for marker, said in ANSWERS: if marker in asked: - return said, [] - return "DECISION: declined. Nothing here matched a rule we hold.", [] + return fits(said), [] + return fits("DECISION: declined. Nothing here matched a rule we hold."), [] def _agent(doc: dict) -> AgentSpec: diff --git a/adapters/python/tests/test_metrics_an_expert_brings.py b/adapters/python/tests/test_metrics_an_expert_brings.py index 6d8ee73..996839d 100644 --- a/adapters/python/tests/test_metrics_an_expert_brings.py +++ b/adapters/python/tests/test_metrics_an_expert_brings.py @@ -41,6 +41,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters import providers # noqa: E402 from pact_adapters.evals import ( # noqa: E402 Case, @@ -145,8 +146,8 @@ async def model_call(self, system, history, tools): asked = " ".join(str(m.get("content") or "") for m in history).lower() for marker, said in ANSWERS: if marker in asked: - return said, [] - return "DECISION: declined. Nothing here matched a rule we hold.", [] + return fits(said), [] + return fits("DECISION: declined. Nothing here matched a rule we hold."), [] def _answers(doc: dict) -> list[tuple[Case, RunResult]]: diff --git a/adapters/python/tests/test_model_pin.py b/adapters/python/tests/test_model_pin.py index 6e08568..3ce9d69 100644 --- a/adapters/python/tests/test_model_pin.py +++ b/adapters/python/tests/test_model_pin.py @@ -33,6 +33,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits, policy_says # noqa: E402 from pact_adapters.events import Bus # noqa: E402 from pact_adapters.harness import ToolCall, delegate_by_running, run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 @@ -139,7 +140,7 @@ def go(agent: AgentSpec): agent, ReferenceTransport(Script([ Turn("Checking.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ])), "my lamp arrived broken, please refund", {"zendesk": lambda a: ticket, "payments": lambda a: "refunded"}, @@ -205,7 +206,7 @@ def test_a_workspace_may_add_a_model_the_distribution_has_never_heard_of( spec, ReferenceTransport(Script([ Turn("Checking.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ])), "my lamp arrived broken, please refund", { @@ -237,7 +238,7 @@ def test_a_pin_the_run_did_not_honour_is_reported_rather_than_measured_around( out = asyncio.run( run( spec, - PydanticAITransport(Script([Turn("Approved.")])), # binds the default + PydanticAITransport(Script([Turn(fits("Approved."))])), # binds the default "my lamp arrived broken, please refund", {"zendesk": lambda a: "broken on arrival", "payments": lambda a: "refunded"}, bus=bus, @@ -279,7 +280,7 @@ def test_a_team_members_pin_reaching_the_running_harness_is_honoured_or_reported def reads_the_pin(member: AgentSpec): honoured.append(member.model) return PydanticAITransport( - Script([Turn("Rule 2 applies.")]), model=member.model or None + Script([Turn(policy_says("Rule 2 applies."))]), model=member.model or None ) asyncio.run( @@ -292,7 +293,7 @@ def reads_the_pin(member: AgentSpec): bus = Bus() def ignores_the_pin(member: AgentSpec): - return PydanticAITransport(Script([Turn("Rule 2 applies.")])) + return PydanticAITransport(Script([Turn(policy_says("Rule 2 applies."))])) asyncio.run( delegate_by_running(doc, ignores_the_pin, bus=bus)( @@ -359,7 +360,7 @@ def test_a_model_this_workspace_added_itself_is_priced_like_any_other( # And through a transport, which is the hop that was missing: the row is # read by the thing that meters, not only by the thing that checks. made = PydanticAITransport( - Script([Turn("Approved.")]), model="our-own-model", workspace=ws + Script([Turn(fits("Approved."))]), model="our-own-model", workspace=ws ) assert made.prices_money is True assert made.context_window() == 32768 @@ -399,7 +400,7 @@ def test_a_ceiling_measured_against_a_free_model_says_it_can_never_fire( script = lambda: Script([ # noqa: E731 Turn("Checking.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) tools = {"zendesk": lambda a: "lamp, broken", "payments": lambda a: "refunded"} spec = AgentSpec.from_document(document, "refund-desk", EXAMPLE) diff --git a/adapters/python/tests/test_model_portability.py b/adapters/python/tests/test_model_portability.py index c526f67..df815c3 100644 --- a/adapters/python/tests/test_model_portability.py +++ b/adapters/python/tests/test_model_portability.py @@ -31,6 +31,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.evals import Case, bar_of # noqa: E402 from pact_adapters.harness import ToolCall # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 @@ -143,7 +144,7 @@ def transport_for(model: str, strategy: str): """Build the scripted executor for one (model, strategy) pair.""" def make(prompt: str) -> Script: - good = Turn(_answer_for(prompt)) + good = Turn(fits(_answer_for(prompt))) if model == FRONTIER: return Script([good]) if model == MID: diff --git a/adapters/python/tests/test_no_adapter_reads_the_authors_files.py b/adapters/python/tests/test_no_adapter_reads_the_authors_files.py index 2f16a8f..56ce57f 100644 --- a/adapters/python/tests/test_no_adapter_reads_the_authors_files.py +++ b/adapters/python/tests/test_no_adapter_reads_the_authors_files.py @@ -220,9 +220,9 @@ def test_a_spec_built_from_a_document_alone_runs(tmp_path: Path) -> None: })) spec = AgentSpec.from_document(document, "desk") out = asyncio.run( - run(spec, ReferenceTransport(Script([Turn("yes")])), "is this ok?") + run(spec, ReferenceTransport(Script([Turn("- decision: yes")])), "is this ok?") ) assert out.halted == "final", out.halted - assert out.output == "yes" + assert out.output == "- decision: yes" and out.answer == {"decision": "yes"} # And the authored answer shape reached the model without a tree in sight. assert spec.answers_with == {"decision": "one of yes, no"} diff --git a/adapters/python/tests/test_one_vocabulary_for_hiding.py b/adapters/python/tests/test_one_vocabulary_for_hiding.py index 4888d83..df69b0a 100644 --- a/adapters/python/tests/test_one_vocabulary_for_hiding.py +++ b/adapters/python/tests/test_one_vocabulary_for_hiding.py @@ -51,6 +51,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) from consenting import allowing_the_connection # noqa: E402 +from fitting import fits # noqa: E402 from pact_adapters.harness import ToolCall # noqa: E402 from pact_adapters.interceptors import ( # noqa: E402 ON_ITS_OWN, @@ -414,7 +415,7 @@ def script() -> Script: Turn("paying", (ToolCall("payments", { "order-number": "A-1182", "contact": {"email": EMAIL}, }),)), - Turn(f"Refunded. We have written to {EMAIL}."), + Turn(fits(f"Refunded. We have written to {EMAIL}.")), ]) result = allowing_the_connection( @@ -458,7 +459,7 @@ def script() -> Script: Turn("paying", (ToolCall("payments", { "order-number": "A-1182", "contact": {"email": EMAIL}, }),)), - Turn(f"Refunded. We have written to {EMAIL}."), + Turn(fits(f"Refunded. We have written to {EMAIL}.")), ]) result = allowing_the_connection( diff --git a/adapters/python/tests/test_portability.py b/adapters/python/tests/test_portability.py index 99028ff..b9ef62f 100644 --- a/adapters/python/tests/test_portability.py +++ b/adapters/python/tests/test_portability.py @@ -25,6 +25,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.harness import ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.loops import STANDARD, Loop # noqa: E402 @@ -145,7 +146,7 @@ def script() -> Script: return Script( [ Turn("Checking the ticket.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ] ) @@ -243,7 +244,7 @@ def test_the_step_limit_is_reported_not_disguised(spec: AgentSpec) -> None: def test_an_unknown_tool_is_reported_to_the_model_not_swallowed(spec: AgentSpec) -> None: - s = Script([Turn("trying", (ToolCall("nonexistent", {}),)), Turn("done")]) + s = Script([Turn("trying", (ToolCall("nonexistent", {}),)), Turn(fits("done"))]) result = asyncio.run(run(spec, ReferenceTransport(s), "hello", TOOLS)) assert "no tool named" in result.steps[0].tool_results[0] @@ -288,7 +289,7 @@ def _ts_trace(spec: AgentSpec) -> dict: "turns": [ {"text": "Checking the ticket.", "toolCalls": [{"name": "zendesk", "args": {"ticket": "T-1"}}]}, - {"text": "Approved: the item arrived damaged within 30 days."}, + {"text": fits("Approved: the item arrived damaged within 30 days.")}, ] }) out = subprocess.run( diff --git a/adapters/python/tests/test_priced_rows_beyond_one_vendor.py b/adapters/python/tests/test_priced_rows_beyond_one_vendor.py index 708e38f..5639875 100644 --- a/adapters/python/tests/test_priced_rows_beyond_one_vendor.py +++ b/adapters/python/tests/test_priced_rows_beyond_one_vendor.py @@ -43,6 +43,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.harness import run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.resolve import load_catalogue, price_of, window_of # noqa: E402 @@ -193,7 +194,7 @@ def test_a_spend_cap_stops_a_run_bound_to_a_vendor_no_anthropic_key_can_reach( after the author's figure was crossed. """ spec = pinned_to(document, OPENAI_ROW) - script = Script([Turn(TALKS_PAST_THE_CAP)]) + script = Script([Turn(fits(TALKS_PAST_THE_CAP))]) transport = OpenAIAgentsTransport(script, model=spec.model or None) assert transport.model == OPENAI_ROW, "the pin is what the transport bound" assert transport.prices_money is True, "the catalogue prices this row" @@ -226,7 +227,7 @@ def test_the_same_run_on_the_row_nobody_can_price_refuses_nothing( list was empty. """ spec = pinned_to(document, UNPRICED_ROW) - script = Script([Turn(TALKS_PAST_THE_CAP)]) + script = Script([Turn(fits(TALKS_PAST_THE_CAP))]) transport = OpenAIAgentsTransport(script, model=spec.model or None) assert transport.prices_money is False @@ -253,7 +254,7 @@ def test_a_second_vendor_stops_at_the_same_ceiling_on_a_different_framework( happened to run (D12). """ spec = pinned_to(document, XAI_ROW) - script = Script([Turn(TALKS_PAST_THE_CAP)]) + script = Script([Turn(fits(TALKS_PAST_THE_CAP))]) transport = AutoGenTransport(script, model=spec.model or None) assert transport.model == XAI_ROW assert transport.prices_money is True @@ -284,7 +285,7 @@ def test_a_hosted_row_outside_anthropic_can_now_say_what_it_holds( along. The control is the same run pinned to an id nothing has a row for. """ spec = pinned_to(document, UNPRICED_ROW) - transport = OpenAIAgentsTransport(Script([Turn("Approved.")]), model=spec.model) + transport = OpenAIAgentsTransport(Script([Turn(fits("Approved."))]), model=spec.model) assert transport.context_window() == 1_050_000, ( "the window comes off the row, through the transport the run uses" ) @@ -297,7 +298,7 @@ def test_a_hosted_row_outside_anthropic_can_now_say_what_it_holds( assert r.tidyings == [] nowhere = pinned_to(document, NOT_IN_THE_CATALOGUE) - blind = OpenAIAgentsTransport(Script([Turn("Approved.")]), model=nowhere.model) + blind = OpenAIAgentsTransport(Script([Turn(fits("Approved."))]), model=nowhere.model) assert blind.context_window() is None out = asyncio.run( run(nowhere, blind, "my lamp arrived broken, please refund", TOOLS) diff --git a/adapters/python/tests/test_scoring_a_suite.py b/adapters/python/tests/test_scoring_a_suite.py index 883def8..a6775de 100644 --- a/adapters/python/tests/test_scoring_a_suite.py +++ b/adapters/python/tests/test_scoring_a_suite.py @@ -42,6 +42,8 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 + REPO = Path(__file__).resolve().parents[3] EXAMPLE = REPO / "examples" / "refund-desk" PACT_BIN = REPO / "target" / "debug" / "pact" @@ -138,7 +140,7 @@ def reply(self, model: str, messages: list[dict]) -> dict: {"action": "issue-refund", "order-number": "A-1", "amount": "40 USD"}, ) return _tool_call("payments", {"action": "look-up-order", "order-number": "A-1"}) - return _text(_decides_from(said, self.wrong)) + return _text(fits(_decides_from(said, self.wrong))) def _decides_from(ticket: str, wrong: str) -> str: diff --git a/adapters/python/tests/test_suspension.py b/adapters/python/tests/test_suspension.py index 58e64a4..ca801ef 100644 --- a/adapters/python/tests/test_suspension.py +++ b/adapters/python/tests/test_suspension.py @@ -26,6 +26,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) from consenting import allowing_the_connection # noqa: E402 +from fitting import fits # noqa: E402 from pact_adapters.harness import Step, ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec, ToolSpec # noqa: E402 from pact_adapters.questions import ANYTHING, Shape # noqa: E402 @@ -65,7 +66,7 @@ def desk(**kw) -> AgentSpec: def pay_script() -> Script: return Script([ Turn("Issuing the refund.", (ToolCall("payments", {"amount": 40}),)), - Turn("Approved: refunded 40 USD."), + Turn(fits("Approved: refunded 40 USD.")), ]) @@ -413,7 +414,7 @@ def test_approval_and_delegation_park_the_same_way() -> None: delegating = AgentSpec(name="a", description="d", instructions="i", team={"policy-checker": "checks the policy"}) - asks_teammate = Script([Turn("asking", (ToolCall("policy-checker", {}),)), Turn("done")]) + asks_teammate = Script([Turn("asking", (ToolCall("policy-checker", {}),)), Turn(fits("done"))]) parked_for[WAITING_FOR_ANOTHER_AGENT] = asyncio.run( run(delegating, ReferenceTransport(asks_teammate), "refund me", {}) ) @@ -431,7 +432,7 @@ def test_a_teammate_who_is_not_here_is_waited_for_rather_than_reported_missing() spec = AgentSpec(name="a", description="d", instructions="i", team={"policy-checker": "checks the policy"}) script = Script([Turn("asking", (ToolCall("policy-checker", {}),)), - Turn("Approved, the policy allows it.")]) + Turn(fits("Approved, the policy allows it."))]) first = asyncio.run(run(spec, ReferenceTransport(script), "refund me", {})) assert first.halted == "suspended" @@ -542,7 +543,7 @@ def a_refund_that_reaches_payments() -> Script: return Script([ Turn("Issuing the refund.", (ToolCall("payments", {"order-number": "A-1182", "amount": "40.00 USD"}),)), - Turn("Approved: refunded 40.00 USD."), + Turn(fits("Approved: refunded 40.00 USD.")), ]) @@ -770,7 +771,7 @@ def test_a_wait_for_another_agent_borrows_nobodys_audience_or_deadline() -> None spec = AgentSpec.from_document(doc, "refund-desk") asks_teammate = Script([ Turn("Asking the fraud team.", (ToolCall("fraud-checker", {}),)), - Turn("Done."), + Turn(fits("Done.")), ]) waiting = asyncio.run( run(spec, ReferenceTransport(asks_teammate), "refund A-1182", TOOLS, now=0.0) @@ -830,7 +831,7 @@ def script() -> Script: ToolCall("payments", {"order-number": "A-300", "amount": "300.00 USD"}), ToolCall("payments", {"order-number": "A-250", "amount": "250.00 USD"}), ]), - Turn("done"), + Turn(fits("done")), ]) # Two calls over one connection are ONE consent — the question's own file diff --git a/adapters/python/tests/test_termination.py b/adapters/python/tests/test_termination.py index c4fbe67..56c6db2 100644 --- a/adapters/python/tests/test_termination.py +++ b/adapters/python/tests/test_termination.py @@ -24,6 +24,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.context_policy import CHARS_PER_TOKEN # noqa: E402 from pact_adapters.harness import ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec, ToolSpec # noqa: E402 @@ -182,10 +183,10 @@ def test_a_run_with_no_ceilings_declared_behaves_exactly_as_before() -> None: """The algebra must cost nothing when unused.""" plain = asyncio.run( run(SPEC, ReferenceTransport(Script([Turn("checking", (ToolCall("zendesk", {}),)), - Turn("Approved.")])), "refund?", TOOLS) + Turn(fits("Approved."))])), "refund?", TOOLS) ) assert plain.halted == "final" - assert plain.output == "Approved." + assert plain.output == fits("Approved.") assert plain.stopped_by is None @@ -244,7 +245,7 @@ def test_the_declared_action_decides_what_happens_and_nothing_else_does() -> Non def test_a_run_that_finished_and_a_run_that_ran_out_are_never_reported_alike() -> None: finished = asyncio.run( - run(SPEC, ReferenceTransport(Script([Turn("Approved.")])), "refund?", TOOLS) + run(SPEC, ReferenceTransport(Script([Turn(fits("Approved."))])), "refund?", TOOLS) ) ran_out = asyncio.run( run(replace(SPEC, max_steps=2), ReferenceTransport(never_finishes()), "refund?", TOOLS) @@ -270,10 +271,10 @@ def test_a_ceiling_the_transport_cannot_measure_is_reported_not_silently_ignored """T7. An author who wrote a spend cap, got no enforcement, and was told nothing has been told something untrue — which is worse than having no cap.""" spec = with_limits(cost_per_request_under=0.05, tokens_at_most=100) - r = asyncio.run(run(spec, ReferenceTransport(Script([Turn("done")])), "hello", TOOLS)) + r = asyncio.run(run(spec, ReferenceTransport(Script([Turn(fits("done"))])), "hello", TOOLS)) assert set(r.unmetered) == {"cost-per-request-under", "tokens-at-most"} - metered = asyncio.run(run(spec, Costing(Script([Turn("done")])), "hello", TOOLS)) + metered = asyncio.run(run(spec, Costing(Script([Turn(fits("done"))])), "hello", TOOLS)) assert metered.unmetered == (), "a transport that can measure is not warned about" @@ -342,7 +343,7 @@ def test_an_unenforceable_ceiling_is_announced_before_any_work_is_done() -> None bus = Bus() spec = with_limits(cost_per_request_under=0.05) - asyncio.run(run(spec, ReferenceTransport(Script([Turn("done")])), "hello", TOOLS, bus=bus)) + asyncio.run(run(spec, ReferenceTransport(Script([Turn(fits("done"))])), "hello", TOOLS, bus=bus)) addresses = [str(e.address) for e in bus.log] assert "session.limit.failed" in addresses assert addresses.index("session.limit.failed") < addresses.index("step.model.requested") @@ -407,9 +408,9 @@ def test_a_promise_with_no_separate_ceiling_is_enforced_as_the_ceiling(document: def a_refund_worth_four_turns() -> Script: return Script([ Turn("Checking the ticket.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("The ticket says the lamp arrived broken 6 days ago."), - Turn("Rule 2 applies: faulty on arrival, and 6 days is inside the window."), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("The ticket says the lamp arrived broken 6 days ago.")), + Turn(fits("Rule 2 applies: faulty on arrival, and 6 days is inside the window.")), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) @@ -437,9 +438,9 @@ def talks_past_what_the_model_holds() -> Script: said = _SAYS_TOO_MUCH * ((window * CHARS_PER_TOKEN) // len(_SAYS_TOO_MUCH) + 1) return Script([ Turn("Checking the ticket.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("The ticket says the lamp arrived broken 6 days ago. " + said), - Turn("Rule 2 applies: faulty on arrival, and 6 days is inside the window."), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("The ticket says the lamp arrived broken 6 days ago. " + said)), + Turn(fits("Rule 2 applies: faulty on arrival, and 6 days is inside the window.")), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) @@ -897,7 +898,7 @@ def test_both_ports_name_the_currency_the_author_wrote() -> None: limits=Limits.from_mapping(written), ) mine = asyncio.run( - run(spec, ReferenceTransport(Script([Turn("done")])), "hello", {}) + run(spec, ReferenceTransport(Script([Turn(fits("done"))])), "hello", {}) ) assert mine.halted == "cost-limit", mine.halted @@ -941,7 +942,7 @@ def test_a_cap_with_no_currency_leaves_no_gap_where_one_would_go() -> None: limits=Limits.from_mapping(written), ) mine = asyncio.run( - run(spec, ReferenceTransport(Script([Turn("done")])), "hello", {}) + run(spec, ReferenceTransport(Script([Turn(fits("done"))])), "hello", {}) ) assert mine.stopped_by is not None said = mine.stopped_by.sentence() diff --git a/adapters/python/tests/test_the_golden_set_runs_everywhere.py b/adapters/python/tests/test_the_golden_set_runs_everywhere.py index 13c8fbb..f140eee 100644 --- a/adapters/python/tests/test_the_golden_set_runs_everywhere.py +++ b/adapters/python/tests/test_the_golden_set_runs_everywhere.py @@ -46,6 +46,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import one_that_fits # noqa: E402 from pact_adapters.reactions import JUDGE_SYSTEM # noqa: E402 from pact_adapters.harness import run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 @@ -84,6 +85,19 @@ ANSWER = "Here is what I make of it." +def _answer(spec: AgentSpec) -> str: + """What the model says back to THIS agent: `ANSWER`, and then each field + the agent says it gives, as the model is asked to end its reply. + + One sentence was the answer for every agent, and an agent that gives a + decision and an amount was "answered" by it: the run ended `final` with an + answer that had none of what it promises. An answer is now held to + `answers-with:` (`answers_fit`), so a scripted one has to fit. The second + port does not hold it and says so (`notDoneHere`); handed the same fitting + answer, the two make the same single call, which is what is compared.""" + return one_that_fits(spec.answers_with, spec.shapes, ANSWER) + + def _workspaces() -> list[Path]: """Every directory under `examples/` holding a `workspace.yaml`.""" return sorted(p.parent for p in EXAMPLES.rglob("workspace.yaml")) @@ -142,7 +156,7 @@ async def model_call(self, system, history, tools): # type: ignore[no-untyped-d def _here(spec: AgentSpec, transport_cls: Any) -> dict[str, Any]: - s = Script([Turn(ANSWER)]) + s = Script([Turn(_answer(spec))]) transport = transport_cls(s) result = asyncio.run(run(spec, transport, ASKED, {})) # A judged reaction (docs/99) puts a yes-or-no question to a model, on this @@ -259,7 +273,7 @@ def _there(spec: AgentSpec) -> dict[str, Any] | None: }) out = subprocess.run( ["node", "--experimental-strip-types", "src/run-trace.ts", - payload, json.dumps({"turns": [{"text": ANSWER}]}), ASKED], + payload, json.dumps({"turns": [{"text": _answer(spec)}]}), ASKED], cwd=TS_DIR, capture_output=True, text=True, ) if out.returncode != 0: diff --git a/adapters/python/tests/test_the_model_choosing_door_survives_being_opened.py b/adapters/python/tests/test_the_model_choosing_door_survives_being_opened.py index 22a5745..5c86ff8 100644 --- a/adapters/python/tests/test_the_model_choosing_door_survives_being_opened.py +++ b/adapters/python/tests/test_the_model_choosing_door_survives_being_opened.py @@ -186,6 +186,8 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits, fits_whoever_asks # noqa: E402 + REPO = Path(__file__).resolve().parents[3] PACT_BIN = REPO / "target" / "debug" / "pact" EXAMPLE = "examples/refund-desk" @@ -910,7 +912,7 @@ def lattice(self) -> dict[str, str]: async def model_call(self, system, history, tools): self.calls += 1 - return "Approved. The lamp arrived damaged within 30 days, refund 40 USD.", [] + return fits_whoever_asks("Approved. The lamp arrived damaged within 30 days, refund 40 USD."), [] def test_a_teammate_the_agent_asks_is_run_rather_than_parked() -> None: @@ -1438,6 +1440,10 @@ def do_POST(self) -> None: # noqa: N802 ) else: answer = "Approved. Sale items follow the same rules and this one was faulty." + if sent.get("model") == self.right_for: + # The model that gets it right ends its reply with what the agent + # says it gives; the other one never does, and never has an answer. + answer = fits(answer) self._send({ "choices": [{"message": {"content": answer}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 12, "completion_tokens": 8}, diff --git a/adapters/python/tests/test_the_same_document_is_the_same_run_through_the_facade.py b/adapters/python/tests/test_the_same_document_is_the_same_run_through_the_facade.py index b156f7c..a2fd410 100644 --- a/adapters/python/tests/test_the_same_document_is_the_same_run_through_the_facade.py +++ b/adapters/python/tests/test_the_same_document_is_the_same_run_through_the_facade.py @@ -52,6 +52,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.harness import RunResult, ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.limits import Meter # noqa: E402 @@ -125,9 +126,9 @@ def careful_turns() -> Script: """ return Script([ Turn("Checking the ticket.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("The ticket says the lamp arrived broken 6 days ago."), - Turn("Rule 2 applies: faulty on arrival, and 6 days is inside the window."), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("The ticket says the lamp arrived broken 6 days ago.")), + Turn(fits("Rule 2 applies: faulty on arrival, and 6 days is inside the window.")), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) @@ -500,9 +501,14 @@ def test_what_comes_back_is_the_sdks_own_result_carrying_the_pact_run( f"AbstractAgent was promised an AgentRunResult" ) assert isinstance(ours.pact, RunResult) - # The SDK-facing answer and the PACT answer are the same words. Two spellings - # of one run is how a caller comes to log one and act on the other. - assert ours.output == ours.pact.output + # The SDK-facing answer and the PACT answer are the same answer. This agent + # says what it gives (`answers-with:`), so what a caller is handed is those + # fields, as the harness read them off the reply when it held it: one + # reading, in one place. Two readings of one run is how a caller comes to + # log one and act on the other. + assert ours.output == ours.pact.answer + assert set(ours.output) == set(spec.answers_with) + assert all(str(value) in ours.pact.output for value in ours.output.values()) def test_the_facade_is_an_agent_the_sdk_itself_would_accept(spec: AgentSpec) -> None: @@ -586,7 +592,7 @@ def test_the_one_loop_that_forks_needs_no_driver_of_its_own() -> None: ) agent = pact_agent_class().for_spec( spec, - script=Script([Turn("option A"), Turn("A is best"), Turn("final answer")]), + script=Script([Turn(fits("option A")), Turn(fits("A is best")), Turn(fits("final answer"))]), ) answered = agent.run_sync("pick one") @@ -594,4 +600,4 @@ def test_the_one_loop_that_forks_needs_no_driver_of_its_own() -> None: # not merely "it did not crash". assert answered.pact.phases == ["propose", "judge", "follow"] assert answered.pact.halted == "final" - assert answered.output == "final answer" + assert answered.output == fits("final answer") diff --git a/adapters/python/tests/test_typed_questions_in_the_loop.py b/adapters/python/tests/test_typed_questions_in_the_loop.py index 786e547..86aee3d 100644 --- a/adapters/python/tests/test_typed_questions_in_the_loop.py +++ b/adapters/python/tests/test_typed_questions_in_the_loop.py @@ -28,6 +28,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) from consenting import allowing_the_connection # noqa: E402 +from fitting import fits # noqa: E402 from pact_adapters.harness import ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec, ToolSpec # noqa: E402 from pact_adapters.loops import Loop # noqa: E402 @@ -90,7 +91,7 @@ def script() -> Script: Turn("Reading the ticket and issuing the refund.", (ToolCall("zendesk", {"id": "T-1"}), ToolCall("payments", {"amount": "40.00 USD", "order-number": "A-1182"}))), - Turn("Done."), + Turn(fits("Done.")), ]) @@ -376,7 +377,7 @@ def test_a_refund_over_the_written_limit_waits_for_a_person_with_nothing_passed_ def over() -> Script: return Script([ Turn("refunding", (ToolCall("payments", {"amount": "300.00 USD"}),)), - Turn("Refunded."), + Turn(fits("Refunded.")), ]) result = allowing_the_connection( @@ -408,7 +409,7 @@ def test_a_refund_under_the_written_limit_is_not_stopped_by_the_same_rule( def under() -> Script: return Script([ Turn("refunding", (ToolCall("payments", {"amount": "40.00 USD"}),)), - Turn("Refunded."), + Turn(fits("Refunded.")), ]) result = allowing_the_connection( @@ -439,7 +440,7 @@ def test_a_call_that_does_not_say_which_action_it_is_is_not_gated_on_a_guess( seen = Watching() looking = Script([ Turn("looking", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("Approved."), + Turn(fits("Approved.")), ]) result = asyncio.run(run(spec, ReferenceTransport(looking), "refund me", seen.tools())) @@ -459,7 +460,7 @@ def test_an_eval_run_says_out_loud_that_it_turns_the_gate_off(document: dict) -> seen = Watching() over = Script([ Turn("refunding", (ToolCall("payments", {"amount": "300.00 USD"}),)), - Turn("Refunded."), + Turn(fits("Refunded.")), ]) result = asyncio.run( run(spec, ReferenceTransport(over), "refund me", seen.tools(), @@ -489,7 +490,7 @@ def test_the_policys_own_gate_survives_a_park_and_lets_the_call_through_once( def script() -> Script: return Script([ Turn("refunding", (ToolCall("payments", {"amount": "300.00 USD"}),)), - Turn("Refunded."), + Turn(fits("Refunded.")), ]) parked = allowing_the_connection( @@ -508,7 +509,7 @@ def script() -> Script: ) assert done.halted == "final" - assert done.output == "Refunded." + assert done.output == fits("Refunded.") assert [n for n, _ in seen.calls] == ["payments"], seen.calls @@ -574,7 +575,7 @@ async def ask(grant): Turn("Consulting the team.", (ToolCall("policy-checker", {"question": "covered?"}), ToolCall("fraud-checker", {"question": "genuine?"}))), - Turn("Approved."), + Turn(fits("Approved.")), ]) return asyncio.run( run(spec, ReferenceTransport(both), "refund me", @@ -589,7 +590,7 @@ def _a_stage_that_stops_to_ask(document: dict): doc = _asking_workspace(asks="is-this-within-policy") spec = AgentSpec.from_document(doc, "desk") return asyncio.run( - run(spec, ReferenceTransport(Script([Turn("Approved.")])), "refund me", + run(spec, ReferenceTransport(Script([Turn(fits("Approved."))])), "refund me", Watching().tools()) ) @@ -727,7 +728,7 @@ async def ask(grant): calls_them = Script([ Turn("Consulting.", (ToolCall("fraud-checker", {"question": "genuine?"}),)), - Turn("Approved."), + Turn(fits("Approved.")), ]) parked = asyncio.run( run(spec, ReferenceTransport(calls_them), "refund me", {}, ask_member=ask) @@ -763,7 +764,7 @@ async def ask(grant): def calls_them() -> Script: return Script([ Turn("Consulting.", (ToolCall("fraud-checker", {"question": "worth?"}),)), - Turn("Approved at 400 USD."), + Turn(fits("Approved at 400 USD.")), ]) parked = asyncio.run( @@ -818,7 +819,7 @@ def test_a_loop_handed_in_by_a_host_still_shows_the_words_that_stage_carries() - """ handed_in = Loop.resolve(_asking_workspace(asks=None), "checked") parked = asyncio.run( - run(SPEC, ReferenceTransport(Script([Turn("Approved.")])), "refund me", + run(SPEC, ReferenceTransport(Script([Turn(fits("Approved."))])), "refund me", Watching().tools(), loop=handed_in) ).suspension diff --git a/adapters/python/tests/test_watching_a_run.py b/adapters/python/tests/test_watching_a_run.py index 72ea068..cc44a3b 100644 --- a/adapters/python/tests/test_watching_a_run.py +++ b/adapters/python/tests/test_watching_a_run.py @@ -36,6 +36,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.events import Address, Bus # noqa: E402 from pact_adapters.harness import ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 @@ -125,7 +126,7 @@ def desk(workspace: Path) -> AgentSpec: def _run(desk: AgentSpec, said: str = "please refund order A-1182"): script = Script([ Turn("checking the ticket", (ToolCall("zendesk", {"id": "A-1182"}),)), - Turn("approved"), + Turn(fits("approved")), ]) return asyncio.run(run(desk, ReferenceTransport(script), said, TOOLS)) diff --git a/adapters/python/tests/test_what_the_author_wrote_reaches_the_run.py b/adapters/python/tests/test_what_the_author_wrote_reaches_the_run.py index 1a74280..f8caaf1 100644 --- a/adapters/python/tests/test_what_the_author_wrote_reaches_the_run.py +++ b/adapters/python/tests/test_what_the_author_wrote_reaches_the_run.py @@ -37,6 +37,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) from consenting import allowing_the_connection # noqa: E402 +from fitting import fits # noqa: E402 from pact_adapters.harness import ToolCall, _system_for, run # noqa: E402 from pact_adapters.ir import AgentSpec, SkillSpec, ToolSpec # noqa: E402 from pact_adapters.learning import ( # noqa: E402 @@ -139,7 +140,7 @@ def test_a_procedure_with_no_procedure_in_it_is_named_on_the_result() -> None: name="a", description="d", instructions="i", skills=(SkillSpec(name="refund-policy", description="The policy."),), ) - out = asyncio.run(run(spec, ReferenceTransport(Script([Turn("done")])), "hello")) + out = asyncio.run(run(spec, ReferenceTransport(Script([Turn(fits("done"))])), "hello")) assert any("refund-policy" in u and "no body" in u for u in out.unenforced), out.unenforced @@ -164,7 +165,7 @@ def fresh(): "Looking it up.", [ToolCall("payments", {"action": "look-up-order", "order-number": "O-9"})], ), - Turn("Done."), + Turn(fits("Done.")), ])) out = allowing_the_connection( @@ -185,7 +186,7 @@ def test_a_bind_nothing_supplies_is_named_rather_than_filled_with_nothing( """An empty string in place of a customer id is worse than a run that says out loud that the identity never arrived.""" spec = AgentSpec.from_document(document, "refund-desk") - out = asyncio.run(run(spec, ReferenceTransport(Script([Turn("hi")])), "hello")) + out = asyncio.run(run(spec, ReferenceTransport(Script([Turn(fits("hi"))])), "hello")) said = [u for u in out.unenforced if u.startswith("bind:")] assert said, out.unenforced assert all("customer-id" in u and "run-inputs.customer-id" in u for u in said), said @@ -211,7 +212,7 @@ def test_a_later_step_calling_the_same_tool_really_calls_it_again() -> None: script = Script([ Turn("one", [ToolCall("zendesk", {"ticket-id": "T-1"})]), Turn("two", [ToolCall("zendesk", {"ticket-id": "T-999"})]), - Turn("done"), + Turn(fits("done")), ]) out = asyncio.run( run( @@ -606,7 +607,7 @@ def test_the_authors_propose_only_line_holds_a_wording_change_for_a_person( outcome = learner.cycle( Proposal("instructions", spec.instructions, spec.instructions + "\n\nBe brief."), lambda _spec: ReferenceTransport( - Script([Turn("approved — 40 USD, the item arrived damaged")] * 40) + Script([Turn(fits("approved — 40 USD, the item arrived damaged"))] * 40) ), ) assert not outcome.applied, "propose-only must not apply anything" @@ -722,7 +723,7 @@ def test_a_mode_no_transport_can_deliver_is_reported_not_silently_downgraded( AgentSpec.from_document(document, "refund-desk", source=EXAMPLE), answers_with_mode="native-json-schema", ) - out = asyncio.run(run(spec, ReferenceTransport(Script([Turn("ok")] * 8)), "hello")) + out = asyncio.run(run(spec, ReferenceTransport(Script([Turn(fits("ok"))] * 8)), "hello")) said = [u for u in out.unenforced if u.startswith("answers-with-mode:")] assert said, f"nothing reported the unhonoured mode: {out.unenforced}" assert "prompted" in said[0], "and it must name the mode that does work" @@ -750,8 +751,8 @@ def test_a_better_model_for_checking_is_used_for_the_checking_stages( "this test needs the example's checking stage" ) - doing = ReferenceTransport(Script([Turn("ok")] * 8)) - checking = ReferenceTransport(Script([Turn("looks right")] * 8)) + doing = ReferenceTransport(Script([Turn(fits("ok"))] * 8)) + checking = ReferenceTransport(Script([Turn(fits("looks right"))] * 8)) asked: dict[str, int] = {"doing": 0, "checking": 0} for name, t in (("doing", doing), ("checking", checking)): original = t.model_call @@ -783,7 +784,7 @@ def test_asking_for_a_better_checking_model_nobody_serves_is_reported( AgentSpec.from_document(document, "refund-desk", source=EXAMPLE), model_for_checking="a-better-model", ) - out = asyncio.run(run(spec, ReferenceTransport(Script([Turn("ok")] * 8)), "hello")) + out = asyncio.run(run(spec, ReferenceTransport(Script([Turn(fits("ok"))] * 8)), "hello")) said = [u for u in out.unenforced if u.startswith("model-for-checking:")] assert said, f"nothing reported the missing second model: {out.unenforced}" assert "a-better-model" in said[0] @@ -898,7 +899,7 @@ def test_a_promoted_case_carries_what_the_rules_allowed_not_what_was_typed( spec = AgentSpec.from_document(document, "refund-desk", source=EXAMPLE) card = "refund card 4111 1111 1111 1111 please" out = asyncio.run( - run(spec, ReferenceTransport(Script([Turn("Approved.")] * 8)), card) + run(spec, ReferenceTransport(Script([Turn(fits("Approved."))] * 8)), card) ) assert "4111" not in out.asked, out.asked @@ -919,7 +920,7 @@ def test_a_promoted_case_leaves_the_right_answer_for_a_person(document: dict) -> spec = AgentSpec.from_document(document, "refund-desk", source=EXAMPLE) out = asyncio.run( - run(spec, ReferenceTransport(Script([Turn("Approved.")] * 8)), "a question") + run(spec, ReferenceTransport(Script([Turn(fits("Approved."))] * 8)), "a question") ) case = case_from_a_run(out.as_record(), "went-wrong") assert case["expect"] == {}, case diff --git a/adapters/python/tests/test_worked_example_loop.py b/adapters/python/tests/test_worked_example_loop.py index 98db763..542d675 100644 --- a/adapters/python/tests/test_worked_example_loop.py +++ b/adapters/python/tests/test_worked_example_loop.py @@ -44,6 +44,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from fitting import fits # noqa: E402 from pact_adapters.harness import RunResult, ToolCall, run # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.loops import DONE, Does, Loop # noqa: E402 @@ -127,9 +128,9 @@ def a_refund_worth_checking() -> Script: """ return Script([ Turn("Checking the ticket.", (ToolCall("zendesk", {"ticket": "T-1"}),)), - Turn("The ticket says the lamp arrived broken 6 days ago."), - Turn("Rule 2 applies: faulty on arrival, and 6 days is inside the window."), - Turn("Approved: the item arrived damaged within 30 days."), + Turn(fits("The ticket says the lamp arrived broken 6 days ago.")), + Turn(fits("Rule 2 applies: faulty on arrival, and 6 days is inside the window.")), + Turn(fits("Approved: the item arrived damaged within 30 days.")), ]) diff --git a/crates/pact-cli/tests/how_many_tries_an_answer_gets_is_read_beside_what_it_is_held_to.rs b/crates/pact-cli/tests/how_many_tries_an_answer_gets_is_read_beside_what_it_is_held_to.rs new file mode 100644 index 0000000..4d1cbd3 --- /dev/null +++ b/crates/pact-cli/tests/how_many_tries_an_answer_gets_is_read_beside_what_it_is_held_to.rs @@ -0,0 +1,111 @@ +//! `checks-at-most:` is read beside `answers-with:`, or `checked-by:`, or both. +//! +//! It says how many times an agent is asked for its answer. An answer is held +//! to the rules the author wrote (`checked-by:`) and, since the owner's ruling +//! of 14 November ("if there is a promised shape, it must be honored"), to the +//! shape the agent says it gives (`answers-with:`), from the same count. +//! +//! The line was `needs-also: [checked-by]`, so an agent with a shape and no +//! rules of its own could not say how many tries it gets. Written as +//! `needs-one-of:`, which means EXACTLY one, the most careful agent in the tree +//! — a shape and rules both — was refused for having both +//! (`examples/kernel/fallback`). It is `needs-any-of:`, a new attribute of the +//! schema: at least one of these, or nothing reads the line. +//! +//! Each case runs the real `pact check` over files on disk. + +use std::process::Command; + +/// A one-agent workspace whose agent writes these lines besides its own. +fn desk(name: &str, lines: &str) -> std::path::PathBuf { + let dst = std::env::temp_dir().join(format!("pact-tries-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dst); + std::fs::create_dir_all(dst.join("agents/desk")).unwrap(); + std::fs::write( + dst.join("workspace.yaml"), + "name: Probe\ndescription: A probe workspace.\n", + ) + .unwrap(); + std::fs::write( + dst.join("agents/desk/agent.yaml"), + format!("name: Desk\ndescription: Answers.\ninstructions: Answer the question.\n{lines}"), + ) + .unwrap(); + dst +} + +fn check(root: &std::path::Path) -> (bool, String) { + let out = Command::new(env!("CARGO_BIN_EXE_pact")) + .args(["check", root.to_str().unwrap(), "--deny-warnings"]) + .output() + .expect("runs"); + let _ = std::fs::remove_dir_all(root); + ( + out.status.success(), + format!( + "{}{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ), + ) +} + +const SHAPE: &str = "answers-with:\n decision: one of approved, declined\n"; +const RULES: &str = "checked-by:\n - must-contain: [refund]\n"; + +#[test] +fn beside_a_shape_alone_it_loads() { + let (ok, said) = check(&desk("shape", &format!("{SHAPE}checks-at-most: 3\n"))); + assert!(ok, "{said}"); +} + +#[test] +fn beside_rules_alone_it_loads_as_it_always_did() { + let (ok, said) = check(&desk("rules", &format!("{RULES}checks-at-most: 3\n"))); + assert!(ok, "{said}"); +} + +#[test] +fn beside_both_it_loads() { + // Refused while the line was `needs-one-of:`, which means exactly one. + let (ok, said) = check(&desk("both", &format!("{SHAPE}{RULES}checks-at-most: 1\n"))); + assert!(ok, "{said}"); +} + +#[test] +fn beside_neither_nothing_reads_it_and_it_is_refused() { + let (ok, said) = check(&desk("neither", "checks-at-most: 3\n")); + assert!(!ok, "{said}"); + assert!(said.contains("rule: schema/missing-any-of"), "{said}"); + assert!( + said.contains( + "'checks-at-most' is set, and it is read only beside `checked-by` or \ + `answers-with` — none of which is written, so nothing reads it." + ), + "{said}" + ); + assert!( + said.contains( + "fix: Add one of them beside it: `checked-by: …`, `answers-with: …`. Or take \ + 'checks-at-most' out." + ), + "{said}" + ); +} + +#[test] +fn a_workflow_stage_still_needs_its_rules() { + // A stage has no shape of its own (it is the workflow's, or the agent's it + // calls), so there the line is read beside `checked-by:` and nothing else. + let dst = desk("stage", ""); + std::fs::create_dir_all(dst.join("workflows")).unwrap(); + std::fs::write( + dst.join("workflows/ask.yaml"), + "description: Ask the desk.\naccepts:\n message: text\nsteps:\n ask:\n does: call\n \ + call: desk\n bind: { message: input.message }\n checks-at-most: 2\n", + ) + .unwrap(); + let (ok, said) = check(&dst); + assert!(!ok, "{said}"); + assert!(said.contains("'checks-at-most' is set, but 'checked-by' is not"), "{said}"); +} diff --git a/crates/pact-schema/src/from_doc.rs b/crates/pact-schema/src/from_doc.rs index 96a68e7..a3d9ed5 100644 --- a/crates/pact-schema/src/from_doc.rs +++ b/crates/pact-schema/src/from_doc.rs @@ -138,6 +138,8 @@ impl Schema { // `Field::needs_one_of` for the rule `needs-also:` could not // state and the author it refused. let needs_one_of = names(f, "needs-one-of"); + // At least one of these. See `Field::needs_any_of`. + let needs_any_of = names(f, "needs-any-of"); // Carries an amount of money when what it compares is one. See // `Field::may_be_money`. let may_be_money = f.get("may-be-money").and_then(as_bool).unwrap_or(false); @@ -251,6 +253,7 @@ impl Schema { field.may_be_a_folder = may_be_a_folder; field.reaches_outside = reaches_outside; field.needs_one_of = needs_one_of; + field.needs_any_of = needs_any_of; field.may_be_money = may_be_money; field.names = names_in; field.key_names = key_names; diff --git a/crates/pact-schema/src/lib.rs b/crates/pact-schema/src/lib.rs index 1ffe02e..09d7fa7 100644 --- a/crates/pact-schema/src/lib.rs +++ b/crates/pact-schema/src/lib.rs @@ -320,6 +320,15 @@ pub struct Field { /// siblings"* — which is the second use, already visible, that makes this an /// attribute rather than one more module. pub needs_one_of: Vec, + /// AT LEAST one of these fields must be set whenever this one is. + /// + /// `needs_also` names one companion and `needs_one_of` means exactly one, + /// and neither says "this line is read beside either of those, or both". + /// `checks-at-most:` is that: how many times an agent is asked for its + /// answer, which is held to `answers-with:` and to `checked-by:`. Written + /// as `needs-one-of`, an agent with a shape AND rules of its own — the + /// most careful one in the tree — was refused for having both. + pub needs_any_of: Vec, /// Whether this field's value is an amount of money WHEN the thing it /// compares is one. /// @@ -446,6 +455,7 @@ impl Field { may_be_a_folder: false, reaches_outside: false, needs_one_of: Vec::new(), + needs_any_of: Vec::new(), may_be_money: false, names: Vec::new(), key_names: Vec::new(), @@ -1530,6 +1540,28 @@ impl Schema { )); continue; } + // The at-least-one-of set: either companion will do, and so will both. + if let Some(listed) = want.strip_prefix("__any_of__") { + let choices: Vec<&str> = listed.split("`, `").collect(); + diags.push(Diagnostic::error( + "schema/missing-any-of", + first_span.clone(), + format!( + "'{first_name}' is set, and it is read only beside `{}` — none of \ + which is written, so nothing reads it.", + choices.join("` or `") + ), + format!( + "Add one of them beside it: {}. Or take '{first_name}' out.", + choices + .iter() + .map(|c| format!("`{c}: …`")) + .collect::>() + .join(", ") + ), + )); + continue; + } let Some(other) = g.fields.iter().find(|x| x.name == want) else { diags.push(Diagnostic::error( "schema/unknown-companion", @@ -3239,6 +3271,27 @@ fn note_companions( .push((f.name.clone(), at.clone())); } } + // At least one of a set: the same reading of "is it written", and owed + // only when none of them is. + if !f.needs_any_of.is_empty() { + let any = f.needs_any_of.iter().any(|want| { + g.fields + .iter() + .find(|x| &x.name == want) + .map(|other| { + std::iter::once(&other.name) + .chain(other.aliases.iter()) + .any(|n| map.contains_key(n)) + }) + .unwrap_or(false) + }); + if !any { + let listed = f.needs_any_of.join("`, `"); + owed.entry(format!("__any_of__{listed}")) + .or_default() + .push((f.name.clone(), at.clone())); + } + } for want in &f.needs_also { let written = g .fields diff --git a/docs/20-ARCHITECTURE-DRAFT.md b/docs/20-ARCHITECTURE-DRAFT.md index d9d51b4..7701b24 100644 --- a/docs/20-ARCHITECTURE-DRAFT.md +++ b/docs/20-ARCHITECTURE-DRAFT.md @@ -7065,7 +7065,7 @@ choose from.** All three, verbatim from `Address.parse`: | `tool` | `step.tool.started`, `step.tool.completed`, `step.tool.cancelled` | | `delegate` | `step.delegate.requested`, `step.delegate.started`, `step.delegate.completed`, `step.delegate.failed`, `step.delegate.cancelled`, `turn.delegate.completed`, `turn.delegate.failed` | | `approval` | `step.approval.requested`, `step.approval.completed`, `turn.approval.failed` | -| `message` | `step.message.completed`, `turn.message.completed` | +| `message` | `step.message.completed`, `turn.message.completed`, `turn.message.failed` (a reply that was to be the answer did not fit `answers-with:` or broke a `checked-by:` rule; `attempt` and `reason` say which try and why) | | `compaction` | `step.compaction.started`, `step.compaction.completed` | | `input` | `step.input.requested` | | `limit` | `session.limit.failed`, `turn.limit.completed` | diff --git a/spec/schema.yaml b/spec/schema.yaml index b2d3e14..b482464 100644 --- a/spec/schema.yaml +++ b/spec/schema.yaml @@ -60,6 +60,10 @@ # own (`answer-within` no shorter than 30s, say), that is a new # attribute, not this one, because 30 here would have to mean # milliseconds and nobody would guess that. +# `needs-any-of:` at least one of these is written beside this field, or +# nothing reads it. `needs-also:` names one companion and +# `needs-one-of:` means exactly one; `checks-at-most:` is read +# beside `answers-with:` or `checked-by:` or both. # `needed-when:` the same as `needs-also:`, conditioned on a VALUE. # `needs-also:` fires on presence, which is right for a ceiling # (any ceiling needs an action) and wrong for the three pairings @@ -836,7 +840,10 @@ groups: help: > what it hands back when it is done, one line per thing — for example `decision: one of approved, declined`. The same shapes `accepts:` - takes. + takes. The answer is held to it: one with a field missing, or a value + the line refuses, is asked for again with the reason, `checks-at-most:` + times in all, and is never handed back as it is. A field you did not + write here is dropped, not refused. answers-with-mode: type: one-of choices: [text, native-json-schema, tool, prompted] @@ -862,15 +869,23 @@ groups: asks again with the reason; still failing after `checks-at-most:` attempts, the run stops and says which rule. checks-at-most: + # Was `needs-also: [checked-by]`. An answer is held to `answers-with:` + # as well (the owner's ruling, 14 November: "if there is a promised + # shape, it must be honored"), from the same count, so an agent with a + # shape and no rules of its own could not say how many tries it gets. + # Beside neither, it is still a line nothing reads, and still refused. type: integer at-least: 1 - needs-also: [checked-by] + needs-any-of: [checked-by, answers-with] surface: S-GOV tier: core kernel-op: allowance.quantity help: > - how many attempts in all, asking again with the reason after each - failed check. Leave it out for 2; write `1` to fail at the first miss. + how many times in all this agent is asked for its answer. An answer + that does not fit `answers-with:`, or that breaks a `checked-by:` + rule, is asked for again with the reason; both count here. Leave it + out for 2; write `1` to fail at the first miss. Still failing, the + run stops (`checks-failed`) and hands back no answer. run-inputs: type: map of answer-shape shapes: *the-answer-shapes @@ -7261,6 +7276,7 @@ groups: - turn.delegate.failed - turn.limit.completed - turn.message.completed + - turn.message.failed - turn.run.cancelled - turn.run.completed - turn.run.failed From 592d753b2178bef2cf2e0daef42f0c9b3f77035e Mon Sep 17 00:00:00 2001 From: Jithin V G Date: Sun, 15 Nov 2026 01:08:42 +0530 Subject: [PATCH 2/3] The second port says it asks once; thirteen shared cases for an answer held; the ending said to a person MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - TypeScript port: `checked-by:` and `checks-at-most:` on `unenforced` (§7.28 list B, fifteen keys). It puts the same shape to the model and holds nothing. - `tests/conformance/answers.json`: thirteen cases on `trees/decided-refunds`. Every agent case's `expect` gains `asked-again`, `said`, `answer`. - A run that ends `checks-failed` says what was wrong with the last reply in the third person, not in the words the model was told. - Three live tests: a desk ends with its fields; told why, it answers again; its closing answer has them. - `one_that_fits` is a test helper, not library API. Co-Authored-By: Claude Fable 5.2 --- CHANGELOG.md | 71 ++ README.md | 2 +- .../python/src/pact_adapters/answers_fit.py | 30 +- adapters/python/tests/conformance_kit.py | 29 +- ...is_held_to_what_the_agent_says_it_gives.py | 10 +- adapters/python/tests/test_portability.py | 4 + .../test_the_golden_set_runs_everywhere.py | 4 + .../test_the_language_runs_on_a_live_model.py | 57 ++ ...shared_cases_are_what_this_runtime_does.py | 53 +- .../test_the_subset_the_second_port_runs.py | 47 +- adapters/typescript/src/harness.ts | 35 +- .../tests/the_subset_the_second_port_runs.rs | 11 +- docs/20-ARCHITECTURE-DRAFT.md | 10 +- site-docs/concepts/harness.md | 48 + tests/conformance/answers.json | 818 ++++++++++++++++++ tests/conformance/digests.json | 4 + tests/conformance/limits.json | 74 ++ tests/conformance/reactions.json | 92 ++ .../trees/decided-refunds/refunds.pact | 61 ++ 19 files changed, 1435 insertions(+), 25 deletions(-) create mode 100644 tests/conformance/answers.json create mode 100644 tests/conformance/trees/decided-refunds/refunds.pact diff --git a/CHANGELOG.md b/CHANGELOG.md index dc0be03..f77c3cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,77 @@ What changed in PACT, newest first. Each entry says what an author can now write, or what now holds that did not; the commits are listed under it for whoever wants the detail. +## 2026-11 — an agent's answer is held to what the agent says it gives (15 November) + +`answers-with:` is a promise to whoever calls the agent. The reference harness put the shape to +the model and handed back whatever came: *"Sure, I think we should refund them."* from an agent that +gives a decision and an amount was the run's answer, `halted: final`. `checked-by:`, the author's +own rules for an answer, was read by nothing in the loop; the run said so and returned the answer +unchecked. The owner's ruling: *"if there is a promised shape, it must be honored."* + +### What holds now + +- **A reply that would be the answer is held before it is one** (`answers_fit`): to the shape the + agent was asked for, and, when it fits, to its `checked-by:` rules. +- **Asked again, and told why.** A field left out (unless `, optional`) or a value its line refuses + is named in the author's own words; a broken rule is named with its `because:`. The model is told + at its next request and at no other, last in its system text, never in the person's voice. +- **`checks-at-most:` times in all**, 2 unwritten: one count for the shape and the rules, and the + closing answer of a run that reached a limit spends from it too. An empty reply that arrived is a + try. The count crosses a pause with the rest of what a run has used. +- **Still not the answer, the run ends `checks-failed`** and hands back none. Its `output` says how + many times it was asked and what was wrong with the last reply. A closing answer that never fits + keeps the ceiling's own word (`Reached.unanswered`). +- **`RunResult.answer`**: each declared field the answer carried, as written, and no field the agent + does not declare. `output` stays the reply as the model wrote it. PACT's own agent facade + (`pact_agent`) hands back `answer`; a run that ended `checks-failed` arrives there as `Halted`. +- **A field that streams** (`reply: stream of text`) is the reply itself, not a line in it. +- **`turn.message.failed`** is emitted for each reply that was not the answer, with `attempt` and + `reason`; a watch may name it. + +### What `pact check` now says + +- **`checks-at-most:` is read beside `answers-with:`, `checked-by:` or both.** It needed + `checked-by:`. Beside neither it is refused as a line nothing reads (`schema/missing-any-of`). + `needs-any-of:` is a new attribute of the schema: `needs-one-of:` means exactly one, and refused an + agent that writes a shape and rules both. + +### What changed for whoever scripts a model + +A scripted answer has to be what the agent gives. 136 tests in 37 files scripted *"done"* or +*"Approved."* for agents that promise fields; each now scripts an answer that fits +(`tests/fitting.py`). One test documented the opposite rule and is rewritten: PACT's agent facade +read a reply "and not validated, deliberately", because the harness checked nothing and a check at +the door would have been a second one. The harness is now the one check, and the facade hands back +what it read. + +### The second port + +The TypeScript port puts the same shape to the model and holds nothing: it asks once and returns +the reply. It says so on every run where an answer is asked for in a shape or held to rules +(`checks-at-most:`, `checked-by:` on `unenforced`; §7.28 list B, two rows, fifteen keys in all). + +### What a runtime must match + +- **`tests/conformance/answers.json`**, thirteen cases on a new tree `trees/decided-refunds`. +- Every agent case's `expect` (reactions, limits, answers) has three new keys: `asked-again`, + `said`, `answer`. The 24 existing cases are otherwise unchanged. +- `digests.json`: `decided-refunds` is new; no listed tree moved. + +### Checked live + +On gpt-4o-mini and gpt-5-nano: a desk ends its reply with the fields it gives; with its first +reply replaced by one naming no field, told why, it answers again with every field; out of +lookups, its closing answer has them too. + +### Still owed + +- The Pydantic AI path (`pydantic_ai_interop`) holds the shape with the SDK's own output retries + and `checked-by:` with `checks-at-most:`: two counts where the reference has one. +- A workflow's stage that writes `checked-by:` is still held by nothing in the reference workflow + runtime. +- The TypeScript port does not hold an answer; it has no shape reader. + ## 2026-11 — the kernel's operations are a file, and each field says which it is (13 November) A runtime that runs PACT one operation at a time needs to know which fields reach each of the 76 diff --git a/README.md b/README.md index b29fd69..3cce2d1 100644 --- a/README.md +++ b/README.md @@ -763,7 +763,7 @@ produces *byte-identical traces, tool sequences and model-call counts*. The clai is bounded and the bound is written down: it covers `instructions:`, `tools:`, `skills:`, `knowledge:`, `team:`, `answers-with:`, the `loop:`, the `limits:` ceilings and whether a `plugins:` line stops the agent before it starts — and -**not** the thirteen governance keys the Node port reports on +**not** the fifteen governance keys the Node port reports on `unenforced`, nor the documents under `knowledge/`, which nothing on that port can look anything up in and which it names on `unretrieved` for every set an answer did not come from. See diff --git a/adapters/python/src/pact_adapters/answers_fit.py b/adapters/python/src/pact_adapters/answers_fit.py index 77f1cb9..427d915 100644 --- a/adapters/python/src/pact_adapters/answers_fit.py +++ b/adapters/python/src/pact_adapters/answers_fit.py @@ -147,6 +147,9 @@ class Held: asked: int = 0 #: Why the last reply held did not fit, as the model was (or would be) told. last: str = "" + #: The same, as its parts: the fields that did not fit, or the rule broken. + problems: tuple[str, ...] = () + broken: str = "" @property def holds_anything(self) -> bool: @@ -162,22 +165,29 @@ def wrong(self, said: str) -> str: if not self.holds_anything: return "" self.asked += 1 - self.last = "" - problems = does_not_fit(said, self.shape, self.shapes) if self.shape else () - if problems: - self.last = ask_again(problems) + self.last, self.broken = "", "" + self.problems = does_not_fit(said, self.shape, self.shapes) if self.shape else () + if self.problems: + self.last = ask_again(self.problems) elif self.rules is not None and (broken := self.rules(said)): + self.broken = broken self.last = rule_broken(broken) return self.last def gave_up(self) -> str: """What a run says when it ends `checks-failed`: how many times it was - asked, and what was wrong with the last answer.""" + asked, and what was wrong with the last reply. Said to a person, so in + the third person: the words the MODEL was told (`last`) begin "Your + last answer", and whoever reads a run's ending is not the model.""" times = "once" if self.asked == 1 else f"{self.asked} times" - none = "it did not fit" if self.asked == 1 else "none fit" - what = "`answers-with:`" if self.last.startswith(DID_NOT_FIT) else "its `checked-by:` rules" - why = self.last.rsplit("\nAnswer again", 1)[0].rsplit(". Answer again so that", 1)[0] + if self.broken: + kept = "it did not keep" if self.asked == 1 else "none kept" + return ( + f"Stopped without an answer: it was asked for its answer {times} and {kept} " + f"its `checked-by:` rules. The last one {self.broken}." + ) + fit = "it did not fit" if self.asked == 1 else "none fit" return ( - f"Stopped without an answer: it was asked for its answer {times} and {none} " - f"{what}. {why}" + f"Stopped without an answer: it was asked for its answer {times} and {fit} " + f"`answers-with:`. In the last one:\n" + "\n".join(f"- {p}" for p in self.problems) ) diff --git a/adapters/python/tests/conformance_kit.py b/adapters/python/tests/conformance_kit.py index c5e8817..7cb0751 100644 --- a/adapters/python/tests/conformance_kit.py +++ b/adapters/python/tests/conformance_kit.py @@ -42,6 +42,19 @@ given and the time the question allows (`answer-within:`) passes: the run is resumed with no answer, at a moment after the wait has run out. +**`asked-again`**, in an agent's `expect`, is what the model was told each time +a reply that would have been the answer was not one (`answers_fit`): it did +not fit what the agent says it gives, or broke one of its `checked-by:` rules. +One text for each such reply that was asked for again, told at the request +that follows it and at no other; a closing request asked again is told it +after everything the closing request is told. **`said`** is what the model +wrote at each step, in order, a reply that was not the answer included. +**`answer`** is the fields of the answer for an agent that says what it gives: +each declared field the reply carried, as it was written, and no field the +agent does not declare; `{}` for an agent that declares none and for a run +that did not end in an answer. A run that never got a fitting answer ends +`checks-failed`, or keeps its ceiling's word when it had already run out. + A workflow's case has three more things it may say: * **`people`**, by question: an answer, or a list of answers in the order they @@ -73,6 +86,7 @@ from pathlib import Path from typing import Any +from pact_adapters.answers_fit import CHECK_FAILED, DID_NOT_FIT from pact_adapters.harness import ToolCall, delegate_by_running, run from pact_adapters.ir import AgentSpec, WorkflowSpec from pact_adapters.script import Script, Turn @@ -108,6 +122,7 @@ def replay_agent(case: dict[str, Any]) -> dict[str, Any]: offered: list[list[str]] = [] closing: list[str] = [] after_a_yes: list[str] = [] + asked_again: list[str] = [] class Lead(ReferenceTransport): async def model_call(self, system, history, tools): # type: ignore[override] @@ -121,6 +136,15 @@ async def model_call(self, system, history, tools): # type: ignore[override] at = system.rfind(RAISED_BEGINS) if at >= 0: after_a_yes.append(system[at:]) + # What a request was told because the reply before it was not the + # answer: it did not fit `answers-with:`, or broke a rule. + at = max(system.rfind(DID_NOT_FIT), system.rfind(CHECK_FAILED)) + if at >= 0: + asked_again.append(system[at:]) + # A closing request asked again is the closing request, and + # then why: `closing` is what the limit told it, without that. + if closing and closing[-1].endswith(system[at:]): + closing[-1] = closing[-1][: -len(system[at:])].rstrip("\n") return await super().model_call(system, history, tools) def tool(name: str): @@ -193,8 +217,11 @@ async def model_call(self, system, history, tools): # type: ignore[override] for s in out.steps ], "offered": offered, - "closing": closing[-1] if closing else "", + "closing": closing[0] if closing else "", "after-a-yes": after_a_yes, + "asked-again": asked_again, + "said": [s.text for s in out.steps], + "answer": json.loads(json.dumps(out.answer, default=str)), "executed": executed, "woken": woken, "parks": parks, diff --git a/adapters/python/tests/test_an_answer_is_held_to_what_the_agent_says_it_gives.py b/adapters/python/tests/test_an_answer_is_held_to_what_the_agent_says_it_gives.py index cd6bdc9..0e5f2fe 100644 --- a/adapters/python/tests/test_an_answer_is_held_to_what_the_agent_says_it_gives.py +++ b/adapters/python/tests/test_an_answer_is_held_to_what_the_agent_says_it_gives.py @@ -228,7 +228,7 @@ def test_still_not_fitting_the_run_ends_checks_failed_and_hands_back_nothing(tmp assert out.answer == {} assert out.output == ( "Stopped without an answer: it was asked for its answer 2 times and none fit " - "`answers-with:`. Your last answer did not fit what your answer must contain:\n" + "`answers-with:`. In the last one:\n" "- `decision` is missing. It should be: one of approved, declined.\n" "- `amount` is missing. It should be: money." ) @@ -276,11 +276,11 @@ def test_the_shape_and_the_rules_spend_from_one_count(tmp_path: Path) -> None: # all, so that is the end: it does not get two for each. out, _, script, _ = go(tmp_path, "ruled", [VAGUE, "- decision: approved", GOOD]) assert out.halted == CHECKS_FAILED and script.calls == 2 - assert out.output.startswith( - "Stopped without an answer: it was asked for its answer 2 times and none fit " - "its `checked-by:` rules." + assert out.output == ( + "Stopped without an answer: it was asked for its answer 2 times and none kept its " + "`checked-by:` rules. The last one does not say ['refund'] (say what happens to the " + "money)." ) - assert "does not say ['refund']" in out.output def test_rules_alone_hold_an_answer_that_promises_no_shape(tmp_path: Path) -> None: diff --git a/adapters/python/tests/test_portability.py b/adapters/python/tests/test_portability.py index b9ef62f..ba4072e 100644 --- a/adapters/python/tests/test_portability.py +++ b/adapters/python/tests/test_portability.py @@ -118,6 +118,10 @@ def _payload_for(spec: AgentSpec) -> str: # the model for the shape the author declared and nothing noticed. "answersWith": dict(spec.answers_with), "answersWithMode": spec.answers_with_mode, + # What an answer is held to and how many tries it gets. This port holds + # it to neither and says so; sent so that it can. + "checkedBy": [r.kind for r in spec.checked_by], + "checksAtMost": spec.checks_at_most, }) diff --git a/adapters/python/tests/test_the_golden_set_runs_everywhere.py b/adapters/python/tests/test_the_golden_set_runs_everywhere.py index f140eee..124e45d 100644 --- a/adapters/python/tests/test_the_golden_set_runs_everywhere.py +++ b/adapters/python/tests/test_the_golden_set_runs_everywhere.py @@ -248,6 +248,10 @@ def _there(spec: AgentSpec) -> dict[str, Any] | None: "loops": _loops_of(spec), "answersWith": dict(spec.answers_with), "answersWithMode": spec.answers_with_mode, + # What an answer is held to and how many tries it gets. This port holds + # it to neither and says so; sent so that it can. + "checkedBy": [r.kind for r in spec.checked_by], + "checksAtMost": spec.checks_at_most, # docs/98 WP-8: each memory shared with the team, which Python offers as # a tool named after it. Sent so the port offers the same names and says # it keeps no board, rather than offering fewer tools in silence. diff --git a/adapters/python/tests/test_the_language_runs_on_a_live_model.py b/adapters/python/tests/test_the_language_runs_on_a_live_model.py index 99d3458..fecc2aa 100644 --- a/adapters/python/tests/test_the_language_runs_on_a_live_model.py +++ b/adapters/python/tests/test_the_language_runs_on_a_live_model.py @@ -151,6 +151,63 @@ def ledger(args: dict[str, Any]) -> str: assert not made_up, f"the model reported lookups that were never made: {made_up}" +# ── an answer held to what the agent says it gives ─────────────────────────── + +REFUNDS = REPO / "tests/conformance/trees/decided-refunds" +ORDER = "Order 7 was a lamp for 45 USD and it arrived broken. Refund it?" + + +def _orders(args: dict[str, Any]) -> str: + return "order 7: one lamp, 45 USD, delivered broken, bought 6 days ago" + + +def test_a_live_model_ends_its_reply_with_what_the_agent_gives() -> None: + """The shape is put to the model in the author's own words, and the answer + is held to it. A live model has to actually write the fields.""" + spec = AgentSpec.from_document(shown(REFUNDS), "desk") + live = Live("desk") + out = asyncio.run(run(spec, live, ORDER, {"orders": _orders})) + assert out.halted == "final", said(out) + assert out.answer.get("decision") in ("approved", "declined"), said(out) + assert "amount" in out.answer, said(out) + + +class Forgetful(Live): + """A live model whose FIRST reply is thrown away and replaced by one that is + not the answer. What follows is the live model's own doing: it is sent the + conversation with that reply in it as its own, told why it was not the + answer, and asked again.""" + + async def model_call(self, system, history, tools): # type: ignore[override] + first = not self.told + text, calls = await super().model_call(system, history, tools) + return ("Sure, I think we should refund them.", []) if first else (text, calls) + + +def test_a_live_model_told_why_its_answer_did_not_fit_answers_again_and_fits() -> None: + """The words a model is told when its reply was not the answer + (`answers_fit.ask_again`) are only any good if a model that reads them + then writes the fields. Scripted models prove the harness asks again; + this proves the asking works.""" + spec = AgentSpec.from_document(shown(REFUNDS), "desk") + live = Forgetful("desk") + out = asyncio.run(run(spec, live, ORDER, {"orders": _orders})) + assert out.halted == "final", said(out) + assert out.steps[0].text == "Sure, I think we should refund them." + # Told at the request after the reply that did not fit, last in its system text. + assert "Your last answer did not fit what your answer must contain:" in live.told[1], live.told[1][-500:] + assert "`decision` is missing" in live.told[1] and "`amount` is missing" in live.told[1] + assert out.answer.get("decision") in ("approved", "declined"), said(out) + assert "amount" in out.answer, said(out) + + +def test_a_live_model_out_of_lookups_still_ends_with_what_the_agent_gives() -> None: + """The closing answer of a run that reached a limit is held like any other.""" + spec = AgentSpec.from_document(shown(REFUNDS), "closes") + out = asyncio.run(run(spec, Live("closes"), ORDER, {"orders": _orders})) + assert out.answer.get("decision") in ("approved", "declined"), said(out) + + def _ledger(looked: list[str]): def ledger(args: dict[str, Any]) -> str: looked.append(str(args.get("entry"))) diff --git a/adapters/python/tests/test_the_shared_cases_are_what_this_runtime_does.py b/adapters/python/tests/test_the_shared_cases_are_what_this_runtime_does.py index 1e9ddda..864c4b0 100644 --- a/adapters/python/tests/test_the_shared_cases_are_what_this_runtime_does.py +++ b/adapters/python/tests/test_the_shared_cases_are_what_this_runtime_does.py @@ -29,7 +29,7 @@ from conformance_kit import CASES, REPO, replay # noqa: E402 -FILES = ("reactions.json", "workflows.json", "limits.json") +FILES = ("reactions.json", "workflows.json", "limits.json", "answers.json") def _cases() -> list[tuple[str, dict[str, Any]]]: @@ -321,3 +321,54 @@ def test_the_cases_say_the_things_they_are_there_to_say() -> None: assert finished["answer"]["ended"] == "finished" and finished["shown"] == [] label = by_name["a-label-from-an-agent-that-ran-out-counts-and-the-record-says-it-ran-out"] assert label["answer"] == {"team": "powerplant", "ended": "ran-out"} + + +def test_the_answer_cases_say_what_an_answer_is_held_to() -> None: + by_name = {case["name"]: case["expect"] for name, case in _cases() if name.startswith("answers:")} + assert len(by_name) == 13 + + fits = by_name["an-answer-that-fits-is-the-answer-and-only-the-fields-it-gives-are-handed-back"] + # Asked once. `mood` is in the reply and not in the answer. + assert fits["halted"] == "final" and fits["asked-again"] == [] and len(fits["offered"]) == 1 + assert "mood" in fits["output"] and fits["answer"] == {"decision": "approved", "amount": "45 USD"} + + again = by_name["an-answer-that-does-not-fit-is-asked-for-again-and-told-why"] + assert again["halted"] == "final" and len(again["said"]) == 2 + assert again["asked-again"] == [ + "Your last answer did not fit what your answer must contain:\n" + "- `decision` is missing. It should be: one of approved, declined.\n" + "- `amount` is missing. It should be: money.\n" + "Answer again, with every field as it is asked for." + ], "the optional `note` is not named" + + once = by_name["it-is-told-why-at-the-next-request-and-at-no-other"] + assert len(once["offered"]) == 3 and len(once["asked-again"]) == 1 + + never = by_name["still-not-fitting-the-run-ends-checks-failed-and-hands-back-nothing"] + # Two tries when none is written, and the scripted third is never asked for. + assert never["halted"] == "checks-failed" and never["answer"] == {} and len(never["offered"]) == 2 + assert never["output"].startswith( + "Stopped without an answer: it was asked for its answer 2 times and none fit `answers-with:`." + ) + empty = by_name["with-one-try-an-empty-reply-is-that-try"] + assert empty["halted"] == "checks-failed" and len(empty["offered"]) == 1 + + shared = by_name["the-shape-and-the-rules-spend-from-one-count"] + # A miss on the shape, then a miss on the rule: two in all, not two each. + assert shared["halted"] == "checks-failed" and len(shared["offered"]) == 2 + assert "none kept its `checked-by:` rules" in shared["output"] + + free = by_name["an-agent-that-promises-nothing-is-held-to-nothing"] + assert (free["halted"], free["output"], free["answer"]) == ("final", "", {}) + + closing = by_name["a-closing-answer-that-does-not-fit-is-asked-for-again"] + # It ran out, and still says so; the second closing request offers no tool + # either, and what the limit told it is what it told the first. + assert closing["halted"] == "tool-call-limit" and closing["answer"] == {"decision": "declined"} + assert closing["offered"][1:] == [[], []] and len(closing["asked-again"]) == 1 + assert "did not fit" not in closing["closing"] + left = by_name["a-closing-answer-that-never-fits-is-not-handed-back-and-the-run-still-ran-out"] + assert left["halted"] == "tool-call-limit" and left["answer"] == {} + assert left["output"].endswith( + "It was asked for its answer 2 times and none fit what it says it gives." + ) diff --git a/adapters/python/tests/test_the_subset_the_second_port_runs.py b/adapters/python/tests/test_the_subset_the_second_port_runs.py index 705f467..bb75629 100644 --- a/adapters/python/tests/test_the_subset_the_second_port_runs.py +++ b/adapters/python/tests/test_the_subset_the_second_port_runs.py @@ -75,6 +75,9 @@ ("slo", "`slo.*`", {"first-reply-within": "2s"}, "first-reply-within:"), ("model", "`model:`", "claude-haiku-4-5", "model:"), ("watches", "`watch/`", ["everything"], "watch:"), + # The author's own rules for an answer. The Python harness holds a reply to + # them before it is the answer; this port names them and holds nothing. + ("checkedBy", "`checked-by:`", ["must-contain"], "checked-by:"), ) #: A teammate, which is the one key on BOTH sides of the bound: the name is @@ -105,6 +108,13 @@ #: back. Both ports stop the run in that stage; only Python puts the question. STAGE_QUESTION = "`asks:`" +#: How §7.28 list B spells the row for how many times an agent is asked for +#: its answer. The Python harness holds a reply to the shape the agent says it +#: gives and to its `checked-by:` rules, and asks again; this port asks once. +#: Named by `run()` whenever a shape is asked for or rules are written, so it +#: is in the report of a spec that writes `checked-by:` too. +TRIES = "`checks-at-most:`" + #: A loop whose first stage stops to ask a person, and names the question. #: `approve-refund` is an entry under `questions/` — which is §7.28 list C, a #: category this port refuses outright — so there is nothing here for the port to @@ -303,8 +313,13 @@ def test_the_architecture_lists_exactly_the_keys_the_run_says_it_ignores() -> No # fifth: the sentence about a set of documents nobody looked in belongs on # `unretrieved`, and arriving here instead would be one fact in two channels # across the two ports and a ticket sent to an author with nothing to edit. + assert _named_by_a_row(section, TRIES), ( + f"§7.28 list B has no row for {TRIES}.\n" + " fix: add a row — the Python harness asks for an answer again when it does not " + "fit, that many times in all, and this port asks once and returns the reply." + ) said = _run(_everything_written())["unenforced"] - accounted = tuple(starts for _, _, _, starts in KEYS) + ("team:",) + accounted = tuple(starts for _, _, _, starts in KEYS) + ("team:", "checks-at-most:") stray = [line for line in said if not line.startswith(accounted)] assert not stray, ( "this port reports a line §7.28 list B does not account for:\n " @@ -480,3 +495,33 @@ def test_a_document_carrying_none_of_them_is_told_nothing() -> None: "a spec carrying no unenforced key was told about one anyway: " f"{quiet['unenforced']}" ) + + +def test_an_answer_this_port_does_not_hold_is_said_to_be_asked_for_once() -> None: + """The Python harness holds a reply to what the agent says it gives, and + this port does not: a reply with no field in it is `checks-failed` there + and the run's answer here. The shape is put to the model in the same words + on both (list A), so nothing in the instructions tells the two apart; the + run has to say it, on every run where an answer was asked for in a shape. + """ + plain = {"name": "Refund Desk", "instructions": "Decide.", "tools": [], "maxSteps": 2} + shaped = _run({**plain, "answersWith": {"decision": "one of approved, declined"}}) + tries = [u for u in shaped["unenforced"] if u.startswith("checks-at-most:")] + assert len(tries) == 1, shaped["unenforced"] + assert tries[0].startswith("checks-at-most: 2 — "), "2 when the author wrote none" + assert "asked once" in tries[0] and "`checks-failed`" in tries[0] + # The reply had no `decision` in it, and it is this port's answer all the same. + assert shaped["halted"] == "final" and shaped["output"] == "done" + + written = _run({**plain, "answersWith": {"decision": "text"}, "checksAtMost": 3}) + assert any(u.startswith("checks-at-most: 3 — ") for u in written["unenforced"]) + + # Rules alone: the rules are named, and so is the one asking. + ruled = _run({**plain, "checkedBy": ["must-contain"]}) + starts = sorted(u.split(":", 1)[0] for u in ruled["unenforced"]) + assert starts == ["checked-by", "checks-at-most"], ruled["unenforced"] + + # A shape asked for in a mode this port cannot deliver is not asked for at + # all, which `answers-with-mode:` says; there is then no answer to hold. + unasked = _run({**plain, "answersWith": {"decision": "text"}, "answersWithMode": "tool"}) + assert not [u for u in unasked["unenforced"] if u.startswith("checks-at-most:")] diff --git a/adapters/typescript/src/harness.ts b/adapters/typescript/src/harness.ts index e6cecb0..7fab4d5 100644 --- a/adapters/typescript/src/harness.ts +++ b/adapters/typescript/src/harness.ts @@ -174,6 +174,13 @@ export type AgentSpec = { // and PACT picks `prompted` — the one mode every target can honour and the // only one that needs nothing off this machine. answersWithMode?: string; + // The author's own rules for an answer (`checked-by:`), each by the kind of + // rule it is, and how many times in all the agent is asked for its answer + // (`checks-at-most:`, 2 when unwritten). Declared so the run can NAME them: + // the Python harness holds a reply to its shape and to these rules before it + // is the answer (`answers_fit.py`), and this port holds it to neither. + checkedBy?: string[]; + checksAtMost?: number; // Teammates, by name, with what each is for. team?: Record; // Absent means `ir.DEFAULT_STEPS`. A bound is never absent — but this port @@ -232,7 +239,7 @@ export const AGENT_SPEC_FIELDS = [ "name", "instructions", "tools", "skills", "team", "maxSteps", "limits", "loop", "loops", "interceptors", "contextPolicy", "policy", "teamwork", "settings", "slo", "model", "watches", "answersWith", "answersWithMode", - "knowledge", "boards", "reactions", "plugins", + "knowledge", "boards", "reactions", "plugins", "checkedBy", "checksAtMost", ] as const; //: The tool an agent says its own topic with. The same name `reactions.py` uses. @@ -474,6 +481,14 @@ export function notDoneHere(spec: AgentSpec): string[] { `nothing on this runtime records what the run did.`, ); } + if ((spec.checkedBy ?? []).length > 0) { + out.push( + `checked-by: ${(spec.checkedBy ?? []).length} rule(s) an answer must keep before ` + + `it is the answer (${(spec.checkedBy ?? []).join(", ")}) — read and not held on ` + + `this runtime, so a reply that breaks one is returned as it is. The Python ` + + `harness asks again with the reason and ends the run \`checks-failed\`.`, + ); + } return out; } @@ -894,6 +909,24 @@ export async function run( ); } + // HOW MANY TIMES THE AGENT IS ASKED FOR ITS ANSWER: once, here. The Python + // harness holds a reply that would be the answer to the shape it was asked + // for and to the author's `checked-by:` rules, asks again with the reason, + // `checks-at-most:` times in all, and ends the run `checks-failed` when none + // fits. This port puts the same shape to the model (list A: the instructions + // are byte-identical) and returns whatever comes back. Said beside the call + // and not in `notDoneHere`, for the reason `answers-with-mode:` is: it turns + // on whether a shape was ASKED FOR this turn, which is a fact about the run. + if (Object.keys(shapeToAskFor).length > 0 || (spec.checkedBy ?? []).length > 0) { + unenforced.push( + `checks-at-most: ${spec.checksAtMost ?? 2} — how many times in all this agent is ` + + `asked for its answer, and on this runtime it is asked once: the reply is ` + + `returned as written, whether or not it is what \`answers-with:\` says it ` + + `gives. The Python harness asks again with the reason and ends the run ` + + `\`checks-failed\` when none fits.`, + ); + } + // An outcome the author routed nowhere. The run ends carrying the message // `route` composed — which names the stage, the outcome and the line to add — // and the steps already taken stay on the result, because a run that got eight diff --git a/crates/pact-cli/tests/the_subset_the_second_port_runs.rs b/crates/pact-cli/tests/the_subset_the_second_port_runs.rs index be63636..72e4358 100644 --- a/crates/pact-cli/tests/the_subset_the_second_port_runs.rs +++ b/crates/pact-cli/tests/the_subset_the_second_port_runs.rs @@ -179,6 +179,15 @@ const REPORTED: &[(&str, &str, &str)] = &[ // docs/99: the reactions that cover an agent — the workspace's and every // other agent's `listens-for:` — named and not carried out. ("reactions", "reactions:", "unenforced"), + // An answer is held to what the agent says it gives, and to the author's + // own rules for it, by the Python harness (`answers_fit.py`): asked again + // with the reason, `checks-at-most:` times in all, then `checks-failed`. + // This port asks once and returns the reply. `answers-with:` itself stays + // in list A — the shape is put to the model in the same words — so the + // rows here are the rules and the count, which is the line that governs + // the asking again. + ("checkedBy", "checked-by:", "unenforced"), + ("checksAtMost", "checks-at-most:", "unenforced"), ]; /// Rows of list B whose key is NOT a top-level `AgentSpec` field, because the @@ -214,7 +223,7 @@ const REPORTED_INSIDE: &[(&str, &str)] = &[("asks:", "unenforced"), ("forks:", " fn word_for(n: usize) -> String { const WORDS: &[&str] = &[ "zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten", - "eleven", "twelve", "thirteen", "fourteen", + "eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", "seventeen", "eighteen", ]; WORDS .get(n) diff --git a/docs/20-ARCHITECTURE-DRAFT.md b/docs/20-ARCHITECTURE-DRAFT.md index 7701b24..686edfb 100644 --- a/docs/20-ARCHITECTURE-DRAFT.md +++ b/docs/20-ARCHITECTURE-DRAFT.md @@ -8969,11 +8969,11 @@ declared on `AgentSpec` for one purpose: so the run can name it, one line at a t what does not happen. A run over this target carrying any of them is still a run whose trace matches — it is a run that *did less*, and said which less. -Thirteen of the fourteen are named on `RunResult.unenforced`: **ten** by `notDoneHere`, which reads -the `AgentSpec`'s own top-level keys, and **three** — a loop stage's `asks:` and `forks:`, and -`answers-with-mode:` — by `run()` itself, for the two different reasons given at the end of +Fifteen of the sixteen are named on `RunResult.unenforced`: **eleven** by `notDoneHere`, which reads +the `AgentSpec`'s own top-level keys, and **four** — a loop stage's `asks:` and `forks:`, +`answers-with-mode:` and `checks-at-most:` — by `run()` itself, for the two different reasons given at the end of this section. (`notDoneHere` returns one line more, for `team:`, but `team:` is a list A key and -not a row here.) The fourteenth — the +not a row here.) The sixteenth — the documents a corpus declares — is named on `RunResult.unretrieved`, which is the same fifth channel `harness.py` has carried since A7 and is a fifth for the same reason `never_reached` is not `unmetered`: *the rule could not be evaluated* sends the reader to the rule, *the @@ -9004,6 +9004,8 @@ every shipped tree agreed anyway. | `reactions:` on an agent, and every other agent's `listens-for:` | docs/99 §5 | The Python harness stops at six moments of its own loop and carries out what a reaction says there — a reminder added to what the model is told, another agent asked first or told after, the run stopped. This port has no reactor: each reaction that covers the agent is named on `unenforced`, and nothing is reminded, asked, told or stopped. The `working-on` tool an agent says its own topic with IS offered, under the same name and description, so both ports show the model the same tools; a call to it is noted and nobody is asked. Carried on the payload as `reactions`. | | `forks:` on a loop stage | docs/98 WP-10 | Python starts each attempt of a forking stage from a copy of the conversation as the stage began; this port runs every attempt on the one conversation, so a later attempt reads the earlier ones and the samples are not independent. Like `asks:`, written inside a loop and named by `run()` once the loop is resolved, for every stage the loop declares. | | `answers-with-mode:` | A5 | Only `prompted` is deliverable anywhere — `native-json-schema` and `tool` constrain the answer at the provider, and no transport in either port does that. So the shape is asked for in the prompt (list A above) and the two stricter modes are named on `unenforced` rather than quietly served as prose, which would be the silent degradation T7 forbids. Both ports pick `prompted` when the line is absent, and both report the same sentence when it is not. | +| `checked-by:` | 02P O7 | The Python harness holds a reply that would be the answer to the author's own rules before it is one (`answers_fit.py`, `evals.first_broken`): a reply that breaks a rule is asked for again with the reason, and is never returned. This port has no rule reader, so the rules are named and a reply that breaks one is returned as it is. | +| `checks-at-most:` | 02P O7 | How many times in all an agent is asked for its answer — 2 when unwritten — held to `checked-by:` and to the shape it says it gives. Python asks again with the reason and ends the run `checks-failed` when none fits; this port asks once and returns the reply as written. The shape itself is list A: both ports put it to the model in the same words, and a reply that fits is the answer on both. Named by `run()` and not by `notDoneHere`, because it turns on whether a shape was asked for this turn. | **What list B counts, and what a run actually prints.** The thirteen is a count of list B's ROWS on `unenforced`, not of the lines a run comes back with. Two other things reach the same channel diff --git a/site-docs/concepts/harness.md b/site-docs/concepts/harness.md index eaa0578..8f73ed1 100644 --- a/site-docs/concepts/harness.md +++ b/site-docs/concepts/harness.md @@ -65,6 +65,54 @@ doubts."* entire package**, at `harness/tool-loop.ts:907`. The continue-decision is not authorable there, and the file holding it is 2,547 lines. +**An answer is held to what the agent says it gives.** An agent that writes +`answers-with:` has promised a shape to whoever calls it, and may write rules +of its own for an answer under `checked-by:`. The harness holds a reply to both +before it is the answer. + +```yaml +answers-with: + decision: one of approved, declined + amount: money + note: text, optional +checked-by: + - must-contain: [refund] + because: say what happens to the money +checks-at-most: 2 +``` + +| The reply | What happens | +|---|---| +| has every field, each as its line asks | it is the answer; the run's `answer` is those fields | +| leaves a field out, or writes a value its line refuses (`decision: maybe`) | the model is asked again and told what was wrong, in the author's own words for each field | +| has a field the agent did not declare | nothing: it is not a reason to ask again, and it is not handed back | +| fits, and breaks a `checked-by:` rule | the model is asked again and told which rule and why | +| still is not the answer after `checks-at-most:` tries (2 when unwritten) | the run ends `checks-failed`, says how many times it was asked and what was wrong with the last reply, and hands back no answer | + +The shape and the rules spend from one count. An empty reply that arrived is +one of the tries; a call to the model that failed is not. A field written +`stream of text` is the reply itself as it arrives, so it is not looked for on +a line of its own. An agent that writes neither line is held to nothing. + +The closing answer of a run that reached a ceiling is held the same way, from +the same count. One that never fits is not handed back: the run keeps the +ceiling's own word, because it did run out, and says it was left with no +answer. + +What the model is told when it is asked again is added to its instructions for +that one request, never put in the customer's voice: + +```text +Your last answer did not fit what your answer must contain: +- `decision` is missing. It should be: one of approved, declined. +- `amount` is missing. It should be: money. +Answer again, with every field as it is asked for. +``` + +Checked live, by replacing a real model's first reply with one that names no +field: told this, gpt-4o-mini and gpt-5-nano each answered again with every +field. + **Ceilings that actually stop things.** `steps-at-most`, `runs-for-at-most`, `tool-calls-at-most`, `tokens-at-most`, `cost-per-request-under` — each with the author's own `when-it-runs-out:` action, and each reported distinctly so diff --git a/tests/conformance/answers.json b/tests/conformance/answers.json new file mode 100644 index 0000000..d31f875 --- /dev/null +++ b/tests/conformance/answers.json @@ -0,0 +1,818 @@ +{ + "about": "What an agent's answer is held to (`answers-with:`, `checked-by:`, `checks-at-most:`). Replayed as conformance_kit.replay_agent does: `model` is the lead agent's turns, turn N being what the model writes when the conversation it is sent holds N replies of its own, so a reply that was not the answer and was asked for again is followed by the next turn; `tools` is what each tool hands back. The rules these hold: a reply that would be the answer is held to the shape the agent says it gives (a field left out does not fit unless its line says `, optional`; a value its line refuses does not fit; a field the agent does not declare is not a reason to ask again and is not handed back) and, when it fits, to the agent's own `checked-by:` rules. One that is not the answer is asked for again, the model told why at its next request and at no other, last in its system text (`asked-again`). It is asked for at most `checks-at-most:` times in all, 2 when unwritten, the shape and the rules spending from one count; an empty reply that arrived is one of them. Still not the answer, the run ends `checks-failed`, its `output` says how many times it was asked and what was wrong with the last reply, and it hands back no `answer`. An agent that declares no shape and no rules is held to nothing. The closing answer of a run that reached a limit (`when-it-runs-out: answer-with-what-it-has`) is held the same way, from the same count: asked again, the closing request is the same request and then why; never fitting, the run keeps its ceiling's own word in `halted`, because it did run out, and its `output` says it was left with no answer. In `expect`: `said` is what the model wrote at each step, a reply that was not the answer included; `answer` is each declared field the answer carried, as written; `output` is the reply as the model wrote it, or what the run says when it has no answer.", + "cases": [ + { + "name": "an-answer-that-fits-is-the-answer-and-only-the-fields-it-gives-are-handed-back", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "desk", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "The item arrived broken.\n- decision: approved\n- amount: 45 USD\n- mood: cheerful" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": { + "amount": "45 USD", + "decision": "approved" + }, + "asked-again": [], + "closing": "", + "executed": [], + "halted": "final", + "offered": [ + [ + "orders" + ] + ], + "output": "The item arrived broken.\n- decision: approved\n- amount: 45 USD\n- mood: cheerful", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "The item arrived broken.\n- decision: approved\n- amount: 45 USD\n- mood: cheerful" + ], + "steps": [ + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "an-answer-written-as-one-json-record-fits-too", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "desk", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "{\"decision\": \"declined\", \"amount\": \"0 USD\", \"mood\": \"sorry\"}" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": { + "amount": "0 USD", + "decision": "declined" + }, + "asked-again": [], + "closing": "", + "executed": [], + "halted": "final", + "offered": [ + [ + "orders" + ] + ], + "output": "{\"decision\": \"declined\", \"amount\": \"0 USD\", \"mood\": \"sorry\"}", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "{\"decision\": \"declined\", \"amount\": \"0 USD\", \"mood\": \"sorry\"}" + ], + "steps": [ + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "an-answer-that-does-not-fit-is-asked-for-again-and-told-why", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "desk", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "Sure, I think we should refund them." + }, + { + "text": "The item arrived broken.\n- decision: approved\n- amount: 45 USD" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": { + "amount": "45 USD", + "decision": "approved" + }, + "asked-again": [ + "Your last answer did not fit what your answer must contain:\n- `decision` is missing. It should be: one of approved, declined.\n- `amount` is missing. It should be: money.\nAnswer again, with every field as it is asked for." + ], + "closing": "", + "executed": [], + "halted": "final", + "offered": [ + [ + "orders" + ], + [ + "orders" + ] + ], + "output": "The item arrived broken.\n- decision: approved\n- amount: 45 USD", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "Sure, I think we should refund them.", + "The item arrived broken.\n- decision: approved\n- amount: 45 USD" + ], + "steps": [ + { + "calls": [], + "results": [] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "a-value-the-shape-refuses-is-named-and-the-field-that-fit-is-not", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "desk", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "- decision: maybe\n- amount: 45 USD" + }, + { + "text": "The item arrived broken.\n- decision: approved\n- amount: 45 USD" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": { + "amount": "45 USD", + "decision": "approved" + }, + "asked-again": [ + "Your last answer did not fit what your answer must contain:\n- `decision` should be one of: approved, declined, but it is 'maybe'.\nAnswer again, with every field as it is asked for." + ], + "closing": "", + "executed": [], + "halted": "final", + "offered": [ + [ + "orders" + ], + [ + "orders" + ] + ], + "output": "The item arrived broken.\n- decision: approved\n- amount: 45 USD", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "- decision: maybe\n- amount: 45 USD", + "The item arrived broken.\n- decision: approved\n- amount: 45 USD" + ], + "steps": [ + { + "calls": [], + "results": [] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "it-is-told-why-at-the-next-request-and-at-no-other", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "desk", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "Sure, I think we should refund them." + }, + { + "text": "", + "calls": [ + { + "name": "orders", + "args": { + "action": "look", + "order": "7" + } + } + ] + }, + { + "text": "The item arrived broken.\n- decision: approved\n- amount: 45 USD" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": { + "amount": "45 USD", + "decision": "approved" + }, + "asked-again": [ + "Your last answer did not fit what your answer must contain:\n- `decision` is missing. It should be: one of approved, declined.\n- `amount` is missing. It should be: money.\nAnswer again, with every field as it is asked for." + ], + "closing": "", + "executed": [ + [ + "orders", + { + "action": "look", + "order": "7" + } + ] + ], + "halted": "final", + "offered": [ + [ + "orders" + ], + [ + "orders" + ], + [ + "orders" + ] + ], + "output": "The item arrived broken.\n- decision: approved\n- amount: 45 USD", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "Sure, I think we should refund them.", + "", + "The item arrived broken.\n- decision: approved\n- amount: 45 USD" + ], + "steps": [ + { + "calls": [], + "results": [] + }, + { + "calls": [ + { + "args": { + "action": "look", + "order": "7" + }, + "name": "orders" + } + ], + "results": [ + "order 7: 45 USD, delivered broken" + ] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "still-not-fitting-the-run-ends-checks-failed-and-hands-back-nothing", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "desk", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "Sure, I think we should refund them." + }, + { + "text": "Sure, I think we should refund them." + }, + { + "text": "The item arrived broken.\n- decision: approved\n- amount: 45 USD" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": {}, + "asked-again": [ + "Your last answer did not fit what your answer must contain:\n- `decision` is missing. It should be: one of approved, declined.\n- `amount` is missing. It should be: money.\nAnswer again, with every field as it is asked for." + ], + "closing": "", + "executed": [], + "halted": "checks-failed", + "offered": [ + [ + "orders" + ], + [ + "orders" + ] + ], + "output": "Stopped without an answer: it was asked for its answer 2 times and none fit `answers-with:`. In the last one:\n- `decision` is missing. It should be: one of approved, declined.\n- `amount` is missing. It should be: money.", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "Sure, I think we should refund them.", + "Sure, I think we should refund them." + ], + "steps": [ + { + "calls": [], + "results": [] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "with-one-try-an-empty-reply-is-that-try", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "strict", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "" + }, + { + "text": "- decision: approved" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": {}, + "asked-again": [], + "closing": "", + "executed": [], + "halted": "checks-failed", + "offered": [ + [] + ], + "output": "Stopped without an answer: it was asked for its answer once and it did not fit `answers-with:`. In the last one:\n- `decision` is missing. It should be: one of approved, declined.", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "" + ], + "steps": [ + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "a-rule-the-answer-breaks-is-told-and-the-answer-asked-for-again", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "ruled", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "- decision: approved" + }, + { + "text": "A refund is on its way.\n- decision: approved" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": { + "decision": "approved" + }, + "asked-again": [ + "Your answer breaks a rule it must keep: does not say ['refund'] (say what happens to the money). Answer again so that it keeps every rule." + ], + "closing": "", + "executed": [], + "halted": "final", + "offered": [ + [], + [] + ], + "output": "A refund is on its way.\n- decision: approved", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "- decision: approved", + "A refund is on its way.\n- decision: approved" + ], + "steps": [ + { + "calls": [], + "results": [] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "the-shape-and-the-rules-spend-from-one-count", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "ruled", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "Sure, I think we should refund them." + }, + { + "text": "- decision: approved" + }, + { + "text": "A refund is on its way.\n- decision: approved" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": {}, + "asked-again": [ + "Your last answer did not fit what your answer must contain:\n- `decision` is missing. It should be: one of approved, declined.\nAnswer again, with every field as it is asked for." + ], + "closing": "", + "executed": [], + "halted": "checks-failed", + "offered": [ + [], + [] + ], + "output": "Stopped without an answer: it was asked for its answer 2 times and none kept its `checked-by:` rules. The last one does not say ['refund'] (say what happens to the money).", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "Sure, I think we should refund them.", + "- decision: approved" + ], + "steps": [ + { + "calls": [], + "results": [] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "an-agent-that-promises-nothing-is-held-to-nothing", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "free", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": {}, + "asked-again": [], + "closing": "", + "executed": [], + "halted": "final", + "offered": [ + [] + ], + "output": "", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "" + ], + "steps": [ + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "a-closing-answer-that-fits-is-the-answer-of-a-run-that-ran-out", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "closes", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "", + "calls": [ + { + "name": "orders", + "args": { + "action": "look", + "order": "7" + } + } + ] + }, + { + "text": "- decision: approved" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": { + "decision": "approved" + }, + "asked-again": [], + "closing": "This run has reached its limit, `tool-calls-at-most` (1 of 1 tool calls). No tool can be called any more.\nAnswer now, from the results you already have. Say plainly what you did not get to. Never write a tool's result yourself.", + "executed": [ + [ + "orders", + { + "action": "look", + "order": "7" + } + ] + ], + "halted": "tool-call-limit", + "offered": [ + [ + "orders" + ], + [] + ], + "output": "- decision: approved", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "", + "- decision: approved" + ], + "steps": [ + { + "calls": [ + { + "args": { + "action": "look", + "order": "7" + }, + "name": "orders" + } + ], + "results": [ + "order 7: 45 USD, delivered broken" + ] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "a-closing-answer-that-does-not-fit-is-asked-for-again", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "closes", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "", + "calls": [ + { + "name": "orders", + "args": { + "action": "look", + "order": "7" + } + } + ] + }, + { + "text": "Sure, I think we should refund them." + }, + { + "text": "- decision: declined" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": { + "decision": "declined" + }, + "asked-again": [ + "Your last answer did not fit what your answer must contain:\n- `decision` is missing. It should be: one of approved, declined.\nAnswer again, with every field as it is asked for." + ], + "closing": "This run has reached its limit, `tool-calls-at-most` (1 of 1 tool calls). No tool can be called any more.\nAnswer now, from the results you already have. Say plainly what you did not get to. Never write a tool's result yourself.", + "executed": [ + [ + "orders", + { + "action": "look", + "order": "7" + } + ] + ], + "halted": "tool-call-limit", + "offered": [ + [ + "orders" + ], + [], + [] + ], + "output": "- decision: declined", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "", + "Sure, I think we should refund them.", + "- decision: declined" + ], + "steps": [ + { + "calls": [ + { + "args": { + "action": "look", + "order": "7" + }, + "name": "orders" + } + ], + "results": [ + "order 7: 45 USD, delivered broken" + ] + }, + { + "calls": [], + "results": [] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + }, + { + "name": "a-closing-answer-that-never-fits-is-not-handed-back-and-the-run-still-ran-out", + "tree": "tests/conformance/trees/decided-refunds", + "agent": "closes", + "says": "Order 7 arrived broken. Refund it?", + "model": [ + { + "text": "", + "calls": [ + { + "name": "orders", + "args": { + "action": "look", + "order": "7" + } + } + ] + }, + { + "text": "Sure, I think we should refund them." + }, + { + "text": "Sure, I think we should refund them." + }, + { + "text": "- decision: declined" + } + ], + "tools": { + "orders": "order 7: 45 USD, delivered broken" + }, + "expect": { + "after-a-yes": [], + "answer": {}, + "asked-again": [ + "Your last answer did not fit what your answer must contain:\n- `decision` is missing. It should be: one of approved, declined.\nAnswer again, with every field as it is asked for." + ], + "closing": "This run has reached its limit, `tool-calls-at-most` (1 of 1 tool calls). No tool can be called any more.\nAnswer now, from the results you already have. Say plainly what you did not get to. Never write a tool's result yourself.", + "executed": [ + [ + "orders", + { + "action": "look", + "order": "7" + } + ] + ], + "halted": "tool-call-limit", + "offered": [ + [ + "orders" + ], + [], + [] + ], + "output": "Stopped before finishing: this run reached the limit you set with `tool-calls-at-most` (1 of 1 tool calls). It was asked for its answer 2 times and none fit what it says it gives.", + "parks": [], + "reactions": [], + "reminders": [], + "said": [ + "", + "Sure, I think we should refund them.", + "Sure, I think we should refund them." + ], + "steps": [ + { + "calls": [ + { + "args": { + "action": "look", + "order": "7" + }, + "name": "orders" + } + ], + "results": [ + "order 7: 45 USD, delivered broken" + ] + }, + { + "calls": [], + "results": [] + }, + { + "calls": [], + "results": [] + } + ], + "unenforced": [], + "woken": [] + } + } + ] +} diff --git a/tests/conformance/digests.json b/tests/conformance/digests.json index 1a38f76..263ad24 100644 --- a/tests/conformance/digests.json +++ b/tests/conformance/digests.json @@ -14,6 +14,10 @@ } } }, + "tests/conformance/trees/decided-refunds": { + "digest": "sha256:abbbc6314cbbe93380130b92ce43e798ecf4fa49464d56d7e8c5b6e1dff4aebc", + "workflows": {} + }, "tests/conformance/trees/guarded-desk": { "digest": "sha256:96f2462be2e6bcef216efffa0cb93edd5b79beb681a22a66f94605ee3b6ba9bf", "workflows": {} diff --git a/tests/conformance/limits.json b/tests/conformance/limits.json index bff2462..8e598cb 100644 --- a/tests/conformance/limits.json +++ b/tests/conformance/limits.json @@ -44,6 +44,8 @@ "people": null, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "This run has reached its limit, `tool-calls-at-most` (2 of 2 tool calls). No tool can be called any more.\nAnswer now, from the results you already have. Say plainly what you did not get to. Never write a tool's result yourself.", "executed": [ [ @@ -75,6 +77,11 @@ "parks": [], "reactions": [], "reminders": [], + "said": [ + "", + "", + "Rent and gas are there; I did not get to water." + ], "steps": [ { "calls": [ @@ -156,6 +163,8 @@ "people": null, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -186,6 +195,10 @@ "parks": [], "reactions": [], "reminders": [], + "said": [ + "", + "" + ], "steps": [ { "calls": [ @@ -278,6 +291,8 @@ "after-a-yes": [ "A person was asked and said to keep going. The limit this run had reached, `tool-calls-at-most` (2 of 2 tool calls), is now 4 tool calls.\nDo not repeat a call that already has a result." ], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -330,6 +345,12 @@ ], "reactions": [], "reminders": [], + "said": [ + "", + "", + "", + "All three are there." + ], "steps": [ { "calls": [ @@ -447,6 +468,8 @@ "after-a-yes": [ "A person was asked and said to keep going. The limit this run had reached, `tool-calls-at-most` (2 of 2 tool calls), is now 4 tool calls.\nThese calls you asked for were not made and can be made now: ledger {\"action\": \"look\", \"entry\": \"water\"}.\nDo not repeat a call that already has a result." ], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -499,6 +522,12 @@ ], "reactions": [], "reminders": [], + "said": [ + "", + "", + "", + "All three are there." + ], "steps": [ { "calls": [ @@ -607,6 +636,8 @@ ], "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -646,6 +677,10 @@ ], "reactions": [], "reminders": [], + "said": [ + "", + "" + ], "steps": [ { "calls": [ @@ -736,6 +771,8 @@ ], "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -775,6 +812,10 @@ ], "reactions": [], "reminders": [], + "said": [ + "", + "" + ], "steps": [ { "calls": [ @@ -861,6 +902,8 @@ ], "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -900,6 +943,10 @@ ], "reactions": [], "reminders": [], + "said": [ + "", + "" + ], "steps": [ { "calls": [ @@ -973,6 +1020,8 @@ "people": null, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "This run has reached its limit, `tool-calls-at-most` (2 of 2 tool calls). No tool can be called any more.\nAnswer now, from the results you already have. Say plainly what you did not get to. Never write a tool's result yourself.", "executed": [ [ @@ -1001,6 +1050,10 @@ "parks": [], "reactions": [], "reminders": [], + "said": [ + "", + "Rent and gas are there." + ], "steps": [ { "calls": [ @@ -1083,6 +1136,8 @@ "people": null, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "This run has reached its limit, `tool-calls-at-most` (2 of 2 tool calls). No tool can be called any more.\nThese calls you asked for were NOT made and have no result: ledger {\"action\": \"look\", \"entry\": \"water\"}.\nAnswer now, from the results you already have. Say plainly what you did not get to. Never write a tool's result yourself.", "executed": [ [ @@ -1114,6 +1169,11 @@ "parks": [], "reactions": [], "reminders": [], + "said": [ + "", + "", + "Rent and gas are there." + ], "steps": [ { "calls": [ @@ -1206,6 +1266,8 @@ "people": null, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "This run has reached its limit, `steps-at-most` (3 of 3 steps). No tool can be called any more.\nAnswer now, from the results you already have. Say plainly what you did not get to. Never write a tool's result yourself.", "executed": [ [ @@ -1247,6 +1309,12 @@ "parks": [], "reactions": [], "reminders": [], + "said": [ + "", + "", + "", + "All three are there." + ], "steps": [ { "calls": [ @@ -1331,6 +1399,8 @@ "people": null, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -1354,6 +1424,10 @@ "parks": [], "reactions": [], "reminders": [], + "said": [ + "", + "Rent is there." + ], "steps": [ { "calls": [ diff --git a/tests/conformance/reactions.json b/tests/conformance/reactions.json index 3b78dbd..85409d6 100644 --- a/tests/conformance/reactions.json +++ b/tests/conformance/reactions.json @@ -51,6 +51,8 @@ ], "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -125,6 +127,11 @@ } ], "reminders": [], + "said": [ + "", + "", + "Refunded." + ], "steps": [ { "calls": [ @@ -224,6 +231,8 @@ ], "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -298,6 +307,11 @@ } ], "reminders": [], + "said": [ + "", + "", + "Refunded." + ], "steps": [ { "calls": [ @@ -397,6 +411,8 @@ ], "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [], "halted": "final", @@ -447,6 +463,11 @@ } ], "reminders": [], + "said": [ + "", + "", + "I could not refund it." + ], "steps": [ { "calls": [ @@ -552,6 +573,8 @@ ], "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -643,6 +666,12 @@ } ], "reminders": [], + "said": [ + "", + "", + "", + "Refunded." + ], "steps": [ { "calls": [ @@ -762,6 +791,8 @@ "judge": "no", "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [], "halted": "final", @@ -805,6 +836,12 @@ } ], "reminders": [], + "said": [ + "", + "", + "", + "VAT on shipping is not refunded." + ], "steps": [ { "calls": [ @@ -882,6 +919,8 @@ "judge": "yes", "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [], "halted": "final", @@ -907,6 +946,9 @@ "reminders": [ "The customer is upset. Apologise once, plainly, before anything else." ], + "said": [ + "I am sorry. Your order is on its way." + ], "steps": [ { "calls": [], @@ -940,6 +982,8 @@ "judge": "no", "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [], "halted": "final", @@ -954,6 +998,9 @@ "parks": [], "reactions": [], "reminders": [], + "said": [ + "Yes, it was delivered." + ], "steps": [ { "calls": [], @@ -1064,6 +1111,8 @@ "judge": "no", "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -1174,6 +1223,16 @@ "reminders": [ "You have taken more than six steps on this request. Answer with what you have, and say plainly what you could not find out." ], + "said": [ + "", + "", + "", + "", + "", + "", + "", + "It was delivered." + ], "steps": [ { "calls": [ @@ -1316,6 +1375,8 @@ }, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [], "halted": "stopped-by-rule", @@ -1338,6 +1399,7 @@ } ], "reminders": [], + "said": [], "steps": [], "unenforced": [], "woken": [] @@ -1390,6 +1452,8 @@ }, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -1459,6 +1523,11 @@ } ], "reminders": [], + "said": [ + "", + "", + "Both paid." + ], "steps": [ { "calls": [ @@ -1579,6 +1648,8 @@ }, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -1656,6 +1727,13 @@ } ], "reminders": [], + "said": [ + "", + "", + "", + "", + "Both told." + ], "steps": [ { "calls": [ @@ -1787,6 +1865,8 @@ }, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -1842,6 +1922,12 @@ "reminders": [ "You have looked at the balance three times. Use what you have." ], + "said": [ + "", + "", + "", + "1200 USD." + ], "steps": [ { "calls": [ @@ -1923,6 +2009,8 @@ "members": null, "expect": { "after-a-yes": [], + "answer": {}, + "asked-again": [], "closing": "", "executed": [ [ @@ -1958,6 +2046,10 @@ } ], "reminders": [], + "said": [ + "", + "Told." + ], "steps": [ { "calls": [ diff --git a/tests/conformance/trees/decided-refunds/refunds.pact b/tests/conformance/trees/decided-refunds/refunds.pact new file mode 100644 index 0000000..366a538 --- /dev/null +++ b/tests/conformance/trees/decided-refunds/refunds.pact @@ -0,0 +1,61 @@ +# DECIDED REFUNDS — a workspace for tests/conformance/answers.json. +# +# What an agent's answer is held to. An agent says what it gives +# (`answers-with:`, written `gives` here), and may write rules of its own for +# an answer (`checked-by:`). A reply that would be the answer is held to both +# before it is one: asked for again with the reason, `checks-at-most:` times in +# all (2 when unwritten), and never handed back as it is. +workspace "Decided refunds" + description: A desk that decides refunds, and says what it decided. + +resource office + resource-kind: mcp-server + endpoint: host/office + description: Where the orders are reached. + +tool orders + description: The order book. + connect: office + actions: + look: + description: Read one order. + reads-only: yes + takes: + order: text + +# A shape, and nothing else: two tries. +agent desk + description: Decides a refund request. + uses orders + gives decision: one of approved, declined, amount: money, note: text, optional + instructions: Decide the refund. + +# One try. +agent strict + description: Decides a refund request, and gets one try at saying so. + gives decision: one of approved, declined + checks-at-most: 1 + instructions: Decide the refund. + +# A shape and a rule of the author's own, from one count. +agent ruled + description: Decides a refund request and must say what happens to the money. + gives decision: one of approved, declined + checked-by: + - must-contain: [refund] + because: say what happens to the money + instructions: Decide the refund. + +# Nothing promised, so nothing held. +agent free + description: Promises nothing about its answer. + instructions: Help. + +# The closing answer of a run that reached a limit is held like any other. +agent closes + description: Looks orders up, and answers with what it has when it runs out. + uses orders + gives decision: one of approved, declined + at most 1 tool calls + when it runs out: answer with what it has + instructions: Decide the refund. From 0e136e341723b3560aa6c99c16e0767892b3380c Mon Sep 17 00:00:00 2001 From: Jithin V G Date: Sun, 15 Nov 2026 01:59:17 +0530 Subject: [PATCH 3/3] Counts Co-Authored-By: Claude Fable 5.2 --- README.md | 4 ++-- docs/90-REVIEW.md | 4 ++-- site-docs/index.md | 2 +- site-docs/status/verified.md | 6 +++--- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 3cce2d1..82a4cb4 100644 --- a/README.md +++ b/README.md @@ -1000,11 +1000,11 @@ framework is an extra. |---|---| | **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete | | **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers | -| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5149 tests (1691 Rust + 3458 adapter), clippy clean, TypeScript type-checked** | +| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5195 tests (1696 Rust + 3499 adapter), clippy clean, TypeScript type-checked** | | **Adapters** | **All 7 named targets**, proven against one shared conformance suite | ```bash -./scripts/test-all.sh # 5149 tests, Rust + 7 adapters, fully offline +./scripts/test-all.sh # 5195 tests, Rust + 7 adapters, fully offline ``` **What is still owed** is not hidden — it lives in diff --git a/docs/90-REVIEW.md b/docs/90-REVIEW.md index 985a771..5b963f8 100644 --- a/docs/90-REVIEW.md +++ b/docs/90-REVIEW.md @@ -80,8 +80,8 @@ them and changes nothing. | | | |---|---| | Schema | **43 groups, 262 field entries, 207 distinct field names** (core 123, expert 69, both 15), 40 closed enums, 27 cross-reference constraints, **0 fields missing `tier:` or `surface:`** | -| Rust | 5 crates, **1691 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | -| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3458 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | +| Rust | 5 crates, **1696 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | +| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3499 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | | TypeScript | 5 modules, 2,305 lines, a second port of **the stepping, the ceilings and the stage path** (`harness.ts:15-21`: *"AC-5.3's bar is the stage path"*) — **not** of the loop entire: no interceptor chain, no durable suspension record, no ledger (`notDoneHere`; a park returns `halted: "suspended"` and writes nothing down). This row is load-bearing for the Phase 3 pricing below (E4) | | CLI | **6 verbs**: `check`, `show`, `waits`, `discover`, `card`, `help` | | Worked example | 42 files, 1128 lines, **132 distinct authored keys**, `pact check` → OK (498 settings) | diff --git a/site-docs/index.md b/site-docs/index.md index 5289fb7..cd9d507 100644 --- a/site-docs/index.md +++ b/site-docs/index.md @@ -47,7 +47,7 @@ so plainly rather than leaving it to be discovered. | | | |---|---| -| Tests | **5,149** — 1691 Rust, 3458 adapter | +| Tests | **5,195** — 1696 Rust, 3499 adapter | | Schema kinds | **69** | | Framework targets | **7**, over 2 runtimes | | Eve capabilities accounted for | **101 of 101** | diff --git a/site-docs/status/verified.md b/site-docs/status/verified.md index d1677f3..c8f22b8 100644 --- a/site-docs/status/verified.md +++ b/site-docs/status/verified.md @@ -12,9 +12,9 @@ from an earlier document. | Claim | Value | How to check | |---|---|---| -| Rust tests | **1691** | `cargo test --workspace` | -| Adapter tests | **3458** | `cd adapters/python && uv run pytest tests/ -q` | -| Total | **5,149** | both of the above | +| Rust tests | **1696** | `cargo test --workspace` | +| Adapter tests | **3499** | `cd adapters/python && uv run pytest tests/ -q` | +| Total | **5,195** | both of the above | | TypeScript type-checks | clean | `cd adapters/typescript && npx tsc --noEmit` | | Lints | clean | `cargo clippy --all-targets` | | Worked example loads | 498 settings | `cargo run -p pact-cli -- check examples/refund-desk` |