Skip to content

Design: the review drive's results can be rerun by anyone, because uphold ships a reference reviewer and names the one it scored #463

Open

Description

@HackingGate

Why

uphold_check.py --review --drive -- REVIEWER... scores a reviewer against the review controls, but uphold ships no reviewer. REFERENCE.md says so on purpose: "No model client lives here; the reviewer command carries the model, the key and the budget." That keeps the core free of a model client. It also means every person who wants to run the drive writes their own reviewer, and every result depends on a command nobody else has:

So the drive is general, but the thing it measures is private to whoever ran it.

Position

uphold ships one reference reviewer as a separate, optional command, not inside the core. uphold_check.py stays free of a model client, as REFERENCE.md rules. The reference reviewer lives beside it (for example scripts/reviewers/), speaks the documented stdin/stdout contract, and carries the model, the key and the budget the way any reviewer does. It is one example of the contract, not the only reviewer: a person at a prompt, or another model behind another wrapper, still answers the same way.

The drive names the reviewer it scored. Each result line, or a header above them, carries the reviewer command as run, so a pasted result says what produced it.

Done when

  • A reference reviewer that passes the contract's own tests (one id or title per line, nothing on no violation, non-zero exit on its own failure). Check: a test that runs it against a fixture model or a stub.
  • --drive output names the reviewer command. Check: a test on the rendered output.
  • REFERENCE.md's --drive section points at the reference reviewer and keeps the "any reviewer answers the same way" contract.

Questions

  1. Where does the reference reviewer live?

    • (a) Inside uphold_check.py, as a built-in reviewer the drive calls when no command is given.
    • (b) A separate optional command beside uphold_check.py that speaks the documented stdin/stdout reviewer contract, carrying its own model, key and budget.

    Answered: (b). economy-of-mechanism L11 "Features, flags, or configuration keys are proposed for a component that other components trust."; economy-of-mechanism L12 "moving work out of it into untrusted callers is the intended outcome"; economy-of-mechanism L18 "Does a new option belong inside the protected mechanism, or can it live in untrusted code that calls it?"; mechanism-policy-separation L11 "The same capability serves multiple policies or environments.". Precedent: none. Ruled from the records policy/principles.toml adopts under [decide] and its orders, at v1.40.0; three rulers given the options in different orders agreed.

  2. Which model client does the reference reviewer use?

    • (a) claude -p, Claude Code's non-interactive mode.
    • (b) A direct call to the model provider's HTTP API with a key.

    Answered: escalated -- no record this repository adopts covers choosing a client. current-stack and reuse-maintained-components would, but policy/principles.toml does not adopt them under [decide]. Three rulers agreed on escalating.

  3. Does the drive's output name the reviewer it scored?

    • (a) Yes: the output names the reviewer command as run.
    • (b) No: the output stays as it is, and whoever pastes a result says which reviewer produced it.

    Answered: (a). single-authoritative-source L11 "The same fact is represented in multiple services, stores, reports, or workflows."; single-authoritative-source L18 "Can every copy explain its source and refresh policy?"; explicit-unknown L11 "Results are aggregated into a status that someone will treat as coverage.". A pasted result is a copy of a drive run, and the reviewer is its source. Precedent: none. At v1.40.0; three rulers given the options in different orders agreed.

Unknowns

  • Whether the reference reviewer's prompt counts as part of what is measured. A pass^k from one prompt is a statement about that prompt as much as about the document.

Activity

  1. added
    designA Design record: a ruling on what will be built, filed before a pull request
    ruling:pendingDesign record awaiting a ruling
    on Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    designA Design record: a ruling on what will be built, filed before a pull requestruling:pendingDesign record awaiting a ruling

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions