Repository navigation
spike(unic-archon-dlc): the review Box is not deterministic across iterations on identical code #485
Description
Observed
Run 3, two unic-dlc-pr-review iterations on PR 5881 at the same commit 08e2523. Iteration 1: 14 findings (Standards 9, Spec 5), deduped to 13. Iteration 2: 13 findings (Standards 3, Spec 10). 0 of 13 prior hashes reappeared. By hand: 10 are the same defects reworded, re-anchored or axis-flipped, 3 are new (all Spec, all acceptance-criteria reads iteration 1 did not do), 6 vanished (all Standards). Praise: 13 bullets each; about 7 survive reworded, 3 file-and-line bullets vanished, one new bullet praises what iteration 1 filed as a defect.
The hash includes the normalised title (unic-dlc-pr-review.yaml:354), so rewording alone explains the 0/13; reconcile's semantic fallback (:448) matched 7. Run 2 reproduced six hashes, which now reads as less rewording, not determinism.
Why it matters
The register's run-2 comparables (dedup, reconcile, praise) assume a stable reviewer. Run 3 measured the variance once; run 4's seal needs a baseline before any fixed is read as signal (#{G4}).
Evidence
- Findings register, § "G. What run 3 taught about the instrument", row G5: https://github.com/unic/unic-agents-plugins/blob/bcf688a/docs/research/2026-08-31-dlc-run-2-findings.md
- Run record and harvested artefacts:
~/Desktop/run3-parked/on the maintainer's machine (artefacts-26f0f16c…/findings/review.json,artefacts-908e3735…/findings/review.json, bothprep.json) - Dispatch record: #457 from comment 5698289748 down
Not decided
- How many identical-code iterations make a variance baseline, and what the Box should report when the axis split inverts.
- Whether the Standards axis should be seeded with the prior iteration's findings as things to confirm or retract, rather than re-derived blind.
Filed 2026-09-17 by wayfinder-orchestrator-10 from run 3's measurements. needs-specs: grill before dispatch.
Activity
orioltf commentedon Sep 17, 2026
Scoped 2026-09-17 while charting run 4's map, #498
This ticket held two things and they have been split, because they run at opposite ends of run 4.
The measurement moved out, to #501, and runs
now. This ticket's first open question — how many identical-code iterations make a variance baseline —
goes with it, because it is the measurement's own first decision. It runs before any fix lands: the moment
#486 changes
synthesize, this distributionstops being observable on 0.28.0. Not strictly perishable, since 0.28.0 is tagged and reinstallable — but
the model behind the reviewer is not frozen, so a replay in three weeks is not the same measurement.
#486 is
blocked_by#501 so the order cannot be lost.What stays here is the treatment, and it lands after run 4. The second open question — whether the
Standards axis is seeded with the prior iteration's findings as things to confirm or retract, rather than
re-derived blind — is a change, not a spike. It reduces variance at the source instead of stopping
variance becoming false state.
It lands after the run for one reason: run 4's subject is whether a claim derives from its artefact.
Reducing the reviewer's variance is a different subject, and moving both at once leaves neither
separately observable — the class rule from
#456 § Notes. This ticket is
blocked_by#502.
One thing this split makes explicit. The reviewer's variance is probably not a defect to fix — it is a
property of an LLM reviewer. What run 4 fixes is #482: that the variance must not become a resolved thread
on the client's pull request. Seeding the axis is an attempt at the variance itself, and it is worth
trying only because #501 will have produced a distribution to judge it against.
orioltf commentedon Oct 27, 2026
A sharper question for this spike (maintainer, 2026-10-27, written by
unic-sdlc-orchestrator-7)."Is the reviewer deterministic?" already has its answer, and the answer is no. Measuring that again buys nothing.
#482's grilling makes a narrower question useful. A prior finding that the new review omits, and whose file did not change, becomes
unverified.postreplies once and does not resolve the thread. So the useful number is how often a review omits a finding whose file did not change. That rate is how oftenunverifiedwill be noise and not signal.Run 3 already gives n=1. This ticket's body says 6 of 13 findings vanished on a byte-identical commit, all of them Standards. If that rate holds, about half the prior findings get an
unverifiedreply on every iteration.This ticket's second "Not decided" point is then the treatment, not a side question: seed the Standards axis with the prior iteration's findings, to confirm or retract each one. A finding the reviewer must confirm or retract cannot be silently omitted.
Open for this ticket's grilling:
orioltf commentedon Oct 27, 2026
Decided: the seeding stays after run 4 (maintainer, 2026-10-27, written by
unic-sdlc-orchestrator-7). This answers the first open question in the comment above.unverifiedthread gets one reply and no further reply while it staysunverified. So the cost is one reply per omitted finding over the life of the pull request, not one per iteration.If run 4 shows a high omission rate, this ticket is next, and that number is its starting point.
orioltf commentedon Oct 30, 2026
Closed as superseded by #678's resolution, approved by the maintainer on 2026-10-30: run 4 is dropped and
unic-archon-dlcis frozen. Reason for this ticket: it carries one linked line to #687: the successor's reviewer is an LLM too, and 0 of 13 hashes reproduced is a design input.—
unic-sdlc-orchestrator-9