Skip to content

bug(unic-archon-dlc): the summary's § "Checks this review did not run" is false when a sub-agent ran a declared command #487

Closed

Description

@orioltf

Observed

Run 3, unic-dlc-pr-review iteration 2, summary thread on PR 5881: § "Checks this review did not run" states "this review executed none of them" over the seven declared commands. The same summary's praise section says "pnpm check-types passes across all three packages". The events table shows the review sub-agent ran pnpm check-types 2>&1 | tail -25 and wrote "pnpm check-types (clean)"; a second sub-agent recorded the same.

Run 3 is not a one-off. Measured on 2026-10-27 in the maintainer's local Archon 0.10.1 database: in at least 18 of the 50 unic-dlc-pr-review runs recorded there, a review sub-agent ran a lint, build, test or type-check command. Run 3 is not among those 50. Archon records every sub-agent tool call under the review step of its events table.

Why it matters

That block is the one sentence a reader uses to know what the run measured. It is written from the Box's instruction ("runs no build, lint, type-check or test command"), not from what the sub-agents did, so it is a statement about the process presented as a statement about the run. On a Consumer with no CI, the pull request is the only record, so a reader can take a review with few findings as evidence that the code builds and its tests pass.

Evidence

Decisions

Grilled with the maintainer on 2026-10-27, under map #498.

  1. The Box keeps its instruction, and the block says only what that instruction makes true. Of the four routes (derive the block from what ran, forbid the sub-agents from running checks, both, or rewrite the block), the block is rewritten. Deriving it from what ran needs either a self-report from review or a node reading Archon's events table, which is undocumented internal storage on one machine, reached with sqlite3, and unmeasured in the Consumer and on Archon 0.12.1. A ban cannot be enforced through allowed_tools, because review needs Bash for git, and a ban in prose already failed. ADR-0037 §6 already expects a sub-agent to run something on its own.
  2. The block stays. Its value is the warning that the review verified no check, in front of every finding, in every iteration. Deleting it would put that warning back in the plugin's own tracker, which ADR-0037 §8 exists to prevent.
  3. A check result a review sub-agent reports appears in the summary only in the block, attributed to that sub-agent, never as a fact and never in "What's good".
  4. review records what its sub-agents report having run, in a structured field of SESSION/findings/review.json. That is a self-report, and the block says so. The comparison with what actually ran happens outside the Box, when run 4 is scored.
  5. A reported check is named by the need it serves, judged by review, never by matching the command text. A need's value can take very different shapes from one Consumer to the next (see #637), so a text match fails in one direction: it reports a declared need as undeclared. The block never names a command.
  6. A finding a sub-agent raised from a failed check stays a finding. Which findings exist is the Method's procedure (ADR-0030). The block only says where the check result came from.
  7. install is not a reported check, whoever ran it. bootstrap's install report already covers it.
  8. ADR-0037 §8 is amended in place. ADR-0026 and ADR-0030 do not change. The decision to warn the reader in the summary stands; only what the warning claims changes. ADR-0030 is cited for a statement that stays true. ADR-0026 says nothing about running checks; bootstrap cites it for this by mistake.

Acceptance criteria

This ticket edits unic-dlc-pr-review.yaml after #482 lands. #482 rewrites reconcile's summary frame and post. Every criterion below that touches the summary frame is written against #482's criteria, not against develop as of 2026-10-27.

The block

  • The block's heading no longer says the review did not run the checks. It says the review was not asked to run them, for example "Checks this review was not asked to run".
  • The block states that this Box instructed none of the project's declared checks, so nothing in the review verifies a claim that a build, lint, type-check or test passes.
  • The block names the keys of $bootstrap.output.sdlc_needs whose value is set, install excluded, by key only, never by value.
  • The block states that a review sub-agent may still have run one of them on its own.
  • The block carries $bootstrap.output.install_report verbatim, as today.
  • No sentence in the block, and no other sentence in the summary, states that the review executed none of the declared checks, or that no check ran.
  • The block stays where it is today, in every iteration: its own section, above every finding.

What a sub-agent reports

  • Both sub-agent prompts in review ask the sub-agent to end its report with every command it ran other than reading files and git, each with the outcome it observed.
  • review writes a reported_checks array into SESSION/findings/review.json: one entry per check a sub-agent reported, carrying the axis (standards or spec), the need it serves, and the outcome the sub-agent reported. The array is empty when no sub-agent reported one. Its writer is the review node, and its source is the sub-agents' reports.
  • Every path in review that writes SESSION/findings/review.json writes reported_checks, including the path taken when the Agent tool is unavailable. On that path the array is empty, and the block does not say that a sub-agent may have run a check, because no sub-agent ran.
  • review assigns the need by judging what the command does, from the nine sdlc_needs key names. It never assigns it by comparing the command's text with the declared value. A check that serves no declared need is recorded as such. install never enters the array.
  • reconcile reads reported_checks from SESSION/findings/review.json and renders each entry in the block as a claim of the sub-agent, naming the need and the outcome, never the command. For example: "A review sub-agent reported that it ran typecheck and that it passed. This Box did not run or verify it." A check that serves no declared need reads as "a check that matches none of the project's declared needs".
  • synthesize puts no check result in "What's good" or in any other section as a fact.
  • A finding a sub-agent raised from a check it ran stays in its severity section like any other finding.

Documents

  • bootstrap's paragraph on checks no longer says the block tells the reader which checks this review did not run. It cites ADR-0037 §8, not ADR-0026.
  • apps/claude-code/unic-archon-dlc/docs/adr/0037-config-declares-sdlc-needs.md carries a dated amendment to § Decision item 8. The amendment says that the block states what the Box instructed, not what ran, and that a sub-agent's reported check appears attributed to it. ADR-0026 and ADR-0030 are not edited.
  • apps/claude-code/unic-archon-dlc/docs/boxes/unic-dlc-pr-review.md describes the block as the criteria above define it.
  • apps/claude-code/unic-archon-dlc/CHANGELOG.md carries an entry under ## [Unreleased].
  • The change adds no node and no edge, so the pr-review pipeline diagram under apps/claude-code/unic-archon-dlc/docs/architecture/ is not touched.

What a run here does not prove

  • The pull request states that the change is unproven in the Consumer until a Consumer run reads it. A run in this repository does not prove it.

Checked in run 4 (#502), not at merge

The implementation pull request does not tick these two. They are checked when run 4 is scored.

  • In run 4, every posted summary is read for the block. In no iteration does the summary state that the review executed no declared check.
  • Run 4's scorer, who did not run it, compares each iteration's attributed lines (or their absence) with the tool_called rows of the review step in Archon's events table, and records in the findings register whether the self-report matched what ran. The Box never reads that table.

Not decided here, on purpose

Activity

  1. added
    bugSomething isn't working
    needs-specsEnough info from reporter; ready to write a spec
    p2Medium priority
    on Sep 16, 2026
  2. orioltf commented on Oct 27, 2026

    @orioltf
    MemberAuthor

    Amended 2026-10-27: the paragraph "Do not close this ticket" and § On landing are superseded by this comment. The rest stands.

    Opener — How to start this session (written 2026-10-27 by unic-sdlc-orchestrator-8, at dispatch)

    What this session produces

    Approved acceptance criteria in this ticket's body, then the label ready-for-agent. Nothing else. The fix runs later under /afk-loop 498. This ticket blocks #502, run 4.

    Start: the maintainer types /wayfinder 498 #487. This ticket's parent is #498. When the grilling starts, the maintainer types /grill-with-docs. Both carry disable-model-invocation, so the session cannot invoke them itself.

    Do not close this ticket. This overrides /wayfinder step 4. This stream grills a ticket into its own criteria and implements it under the same number. Do not invoke /implement, /tdd or /to-tickets. Keep needs-specs until the maintainer approves.

    Read in this order

    1. This ticket's body. Its one "Not decided" point is the starting question.
    2. #482's criteria, in full. bug(unic-archon-dlc): reconcile marks a finding fixed when the next review omits it, and post resolves the thread #482 is ready-for-agent and runs under /afk-loop 498 before this ticket. It rewrites reconcile and post in the same Box, and adds unverified to the summary's delta line.
    3. apps/claude-code/unic-archon-dlc/.archon/workflows/unic-dlc-pr-review.yaml on origin/develop: bootstrap (the paragraph that starts "This Box runs no build, lint, type-check or test command by instruction"), review (its allowed_tools), and reconcile steps 5 and 5b.
    4. ADR-0026, ADR-0030 and ADR-0037 under apps/claude-code/unic-archon-dlc/docs/adr/. Step 5b cites ADR-0026 and ADR-0030 for the sentence this ticket calls false. bootstrap cites ADR-0037 §6 for the case where a sub-agent decides to run something.
    5. The findings register on develop, § "G. What run 3 taught about the instrument", row G7. The run 3 evidence directory the body names, ~/Desktop/run3-parked/, is not on this machine (measured 2026-10-27 by this seat; no run3-parked directory within four levels of the home directory). Row G7 and this body are the evidence you have. Do not write a criterion that needs the run 3 transcripts. If you need them, ask the maintainer where they are.

    The trap that costs an hour

    Treating "derive the block from the sub-agents' actual tool calls" as a free choice. reconcile writes step 5b. It has never seen the review node's transcript, and nothing in the Box hands it a record of which commands the sub-agents ran. The run 3 measurement came from Archon's events table, outside the Box. So "derive it from the tool calls" means one of two things. Either the review node reports which declared commands its sub-agents ran, which is a self-report, the same shape this ticket complains about. Or a node reads a record Archon keeps, and nobody has shown that a node can read one. Before the maintainer picks this option, find out which of the two it is. If it is the second, measure it with a small probe workflow on Archon 0.10.1, never with a run of this Box. #439's criterion 0 is the probe shape to copy. Write the result into the body.

    What this ticket does not decide yet

    Each of these is a question for the maintainer. None is yours to settle with a plausible default.

    • Which of the three routes. The body lists two: derive the block from what ran, or forbid the sub-agents from running declared commands. Both is the third. A fourth costs least: keep the Box as it is and rewrite the sentence so that it describes the instruction, not the run. For example, say that the Box did not instruct any check, and that a sub-agent may still have run one. The body does not list it. Put it to the maintainer beside the others.
    • How a ban would be enforced, if the maintainer picks one. review allows Bash and Agent. It needs Bash for git, so a tool list cannot remove pnpm. A ban written in prose is honoured or not, and run 3 shows it was not. Decide whether a prose ban is enough, or whether it needs a mechanism, and name the mechanism. Check that it works on Archon 0.10.1 and on 0.12.1, which run 4 targets.
    • The praise section. Run 3's summary also said "pnpm check-types passes across all three packages", from a sub-agent's run. Decide whether a check result a sub-agent produced may appear in the summary at all, and if it may, where and with what source.
    • Whether ADR-0026 or ADR-0030 changes. If the fix changes what the Box runs, or what it says it runs, decide whether that amends an ADR or needs a new one, and say which in the criteria.

    Other traps

    The bar that proves it

    Run the four reads from docs/agents/dispatching-and-learning.md § Before you post one on the criteria as a set. Then check:

    • Run 3's case: a review in which a sub-agent ran a declared check does not publish a summary that says the review executed none of the declared checks.
    • A review in which no sub-agent ran anything publishes a block that is still true.
    • Every criterion that says what ran names the source it reads, and that source exists, with a named writer, after the change.

    A false pass is the same sentence with softer words, still written from the Box's instruction. It reads as fixed and is still a statement about the process, presented as a statement about the run. Another false pass is a review node told to "list what its sub-agents ran", with nothing that checks the list.

    On landing

    1. Rewrite this ticket's body with the approved criteria, and post a comment saying what changed.
    2. Before you apply any label, report to unic-sdlc-orchestrator-8 with the link. The orchestrator reads the criteria as one set and replies.
    3. Then replace needs-specs with ready-for-agent, remove wayfinder:grilling (the decision is made, and this stream does not close the ticket), keep p2 unless the maintainer moves it, and leave it unassigned for /afk-loop 498.
    4. Change no file in the repository.

    Report back

    BEFORE YOU FINISH, AND WHENEVER YOU LEARN SOMETHING THAT WOULD HAVE CHANGED THIS PROMPT,
    report to unic-sdlc-orchestrator-8 with four things and nothing else:
      1. what you expected
      2. what happened
      3. the command or file that proved it
      4. whether it changes only this ticket, or every session in this repo
    Send it when you learn it, not only at the end — a session that dies unreported
    takes its learnings with it.
    Talk only to unic-sdlc-orchestrator-8. Do not message another orchestrator or its
    sessions; tell unic-sdlc-orchestrator-8, who relays it.
    

    Review and handover

    There is no pull request. The maintainer approves the criteria in the session. The orchestrator reads them as one set before the label goes on. Implementation runs under /afk-loop 498, which this seat records as unproven in the Consumer until a Consumer run reads it.

  3. self-assigned this
    on Oct 27, 2026
  4. orioltf commented on Oct 27, 2026

    @orioltf
    MemberAuthor

    Opener amendment — this ticket closes when its criteria are approved (2026-10-27, unic-sdlc-orchestrator-8)

    This supersedes two parts of the opener: the paragraph "Do not close this ticket" and § On landing. Everything else in it stands.

    Why. On 2026-09-04 the maintainer decided that a grilling on a wayfinder map closes its decision ticket at /wayfinder's own step 4, and that the orchestrator files a separate implementation ticket. That ticket links to the criteria and does not copy them, and it gets its own Azure DevOps mirror. The openers since then drifted back to "do not close" without a decision. The maintainer confirmed the two-ticket shape again today. So /wayfinder step 4 is no longer overridden here.

    Still do not invoke /implement, /tdd or /to-tickets.

    On landing, replacing the opener's section

    1. Write the approved criteria into this ticket's body, so that the implementation ticket can link to them.
    2. Report to unic-sdlc-orchestrator-8 with the link, before you close or relabel anything. The orchestrator reads the criteria as one set and replies.
    3. Wait for the orchestrator to say the implementation ticket exists. The orchestrator files it under #498, with a native blocked_by edge onto #482, and moves #502's blocked_by edge from this ticket to the new one. Closing first would drop a blocker from Dispatch run 4, end it in a merge, and score the seal #502, and run 4 would read as one step closer than it is.
    4. Then follow /wayfinder step 4: post the resolution comment, close this ticket, and append the context pointer to wayfinder: run 4 — are the chain's claims about its own work derived from the work? #498's Decisions so far. Remove needs-specs and add no state label. In this repository closed means a merged pull request, and the closed grilling tickets on the other maps (for example #540 and #631) carry no state label. Keep wayfinder:grilling. On a closed decision ticket it is true, so the opener's rule to remove it no longer applies. Do not add ready-for-agent here. It goes on the implementation ticket.
    5. Change no file in the repository.
  5. orioltf commented on Oct 27, 2026

    @orioltf
    MemberAuthor

    Resolved 2026-10-27 with the maintainer. The body now carries the approved decisions and criteria. The orchestrator read them as one set, and two amendments followed from that read.

    Decision: the Box keeps its instruction, and the block says only what that instruction makes true. It is renamed so that it no longer claims the checks did not run. A check a review sub-agent reports appears only in that block, attributed to the sub-agent and named by the need it serves, never by its command, and never in "What's good". ADR-0037 §8 is amended in place. ADR-0026 and ADR-0030 do not change. Run 4's scorer compares the attributed lines with Archon's events table, outside the Box.

    Implementation: bug(unic-archon-dlc): the pr-review summary states what the Box instructed, and attributes a check a sub-agent reports, which links to this body and is blocked by #482.

  6. removed
    needs-specsEnough info from reporter; ready to write a spec
    on Oct 27, 2026
  7. removed their assignment
    on Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions