What replaces RED in /build when an e2e test for an existing feature should pass first time? #540

Activity

1. orioltf

   added

   app:unic-archon-dlc

   wayfinder:grillingWayfinder ticket: decide it by grillingWayfinder ticket: decide it by grilling

   on Sep 23, 2026
2. orioltf

   added a parent issue on Sep 23, 2026

   • wayfinder: e2e specs from acceptance criteria, at a second Consumer #538
3. orioltf

   marked this as blocked on Sep 23, 2026

   • Can the DLC's own line, adapted by context, produce a reviewed e2e suite? #539
4. orioltf

   marked this as blocking on Sep 23, 2026

   • Which Box changes does v1 need, and in what order do they land? #542
5. orioltf

   marked this as blocked by 2 issues on Sep 23, 2026

   • bug(unic-archon-dlc): open-pr precedes its gate, and a config-value gate has no safe middle #439
   • No Box can target a base branch that is not the trunk, so run 4 cannot merge to integration/dlc #505
6. orioltf commented on Sep 24, 2026

   

   orioltf

   on Sep 24, 2026

   MemberAuthor

   More actions

   Constraint from Is an untestable acceptance criterion refused by context, or by a change to /specs and /tickets?, 2026-09-24

   • Every criterion in the PRD carries one of three classes: e2e-testable, not e2e-testable, or unverifiable. /specs assigns the class, and the developer confirms it at Halt 1. The terms are defined in apps/claude-code/unic-archon-dlc/CONTEXT.md, in a draft PR.
   • Only an e2e-testable criterion gets an e2e slice from /tickets. The other two classes are accepted gaps, by a Consumer ADR, so /build never receives one as an e2e test to write.
   • The criteria sit as Acceptance criteria: bullets under each PRD user story. Each bullet keeps its ID from the source page.
7. orioltf commented on Sep 28, 2026

   

   orioltf

   on Sep 28, 2026

   MemberAuthor

   More actions

   Constraint from Does an e2e test become an outer loop around /build's unit TDD loop for a new feature?, 2026-09-28

   That ticket also changes unic-dlc-build.yaml. Two of its decisions apply to an existing feature too:

   • A validation step runs before the loop. bootstrap checks that the e2e environment is ready whenever sdlc_needs.e2e is declared, and guard-not-ready cancels the build if it is not. So a failure caused by the environment never reaches RED, in either mode.
   • A failing e2e test gets one of three verdicts. A fresh context, separate from the agent that wrote the code, gives it: environment issue, broken or stale test or genuine bug. A genuine bug gets a severity on the scale Blocker, Critical, Major, Minor, Trivial. In characterisation work, a failure against a feature that already ships is the finding. Under this taxonomy it is a genuine bug. If this ticket gives characterisation its own mode, reuse the taxonomy rather than define another.

   Both decisions edit the same Box file, so the implementation tickets from the two decisions are ordered against each other, as well as against open-pr precedes its gate and No Box can target a base branch that is not the trunk.
8. orioltf commented on Sep 30, 2026

   

   orioltf

   on Sep 30, 2026

   Last edited by orioltf

   MemberAuthor

   More actions

   Unblocked for grilling: constraints from #439 and #505, 2026-09-30

   This ticket no longer waits for #439 and #505. It is a decision and edits no Box file. The seat that owns both tickets agreed that neither one makes a decision taken here void. Each one fixes a surface that the answer here has to fit. The blocked_by edges onto #439 and #505 move to the implementation tickets that come out of this ticket. #542 keeps its edges.

   The decision taken here must hold whatever #439 and #505 decide, and must state these two points.

   The verdict of a characterisation run. With #439, evidence writes a verdict on every run: pass, fail or unresolved, the values its verdictState takes. #439's body calls fail "red". A fail or unresolved verdict stops the run before the pull request opens. If this ticket picks a declared mode in /build, the decision says which verdict a characterisation run produces when its test fails. In Case A a failing test is the finding, not a broken build. If that failure maps to fail, a characterisation run never opens a pull request. Name the mapping by the verdict, not by #439's criterion numbers. #439's re-grill is paused, so its numbering can still change.

   The base branch. #505 records that three Boxes each write the rule "develop when gitflow, else main", and leaves open whether the fix is a config key or a single source. If this ticket picks another Box, that Box does not write the rule a fourth time. It takes the base branch the way #505 decides.

   Corrected 2026-09-30: the verdict values are pass, fail and unresolved, not green, red and unresolved.
9. orioltf

   unmarked this as blocked by 2 issues on Sep 30, 2026

   • bug(unic-archon-dlc): open-pr precedes its gate, and a config-value gate has no safe middle #439
   • No Box can target a base branch that is not the trunk, so run 4 cannot merge to integration/dlc #505
10. orioltf

    self-assigned this

    on Sep 30, 2026
11. orioltf

    marked this as blocked on Sep 30, 2026

    • What do Archon releases after 0.7.0 offer for running one loop phase with its own tools and rules? #632
12. orioltf commented on Sep 30, 2026

    

    orioltf

    on Sep 30, 2026

    Last edited by orioltf

    MemberAuthor

    More actions

    Resolution, 2026-09-30

    A test for a feature that already ships goes into a Characterisation Slice, a slice kind of its own in issues.json. It has its own pair of phases and its own proof that the test can fail. run-build changes from a loop: node to a loop_group, so the phase that writes the test runs in a node with its own tool rules. There is no new Box and no run-level mode. red_unexpected_pass stays as it is for every other slice.

    The slice and where it comes from

    1. A Characterisation Slice holds a test for a criterion that describes behaviour that already exists. The test must pass on its first run. A separate context then proves that it can fail.
    2. /specs marks each criterion as existing or new, beside the class from #551. The developer confirms the mark at Halt 1. /tickets makes a Characterisation Slice for each criterion that is both e2e-testable and existing. Both commands change. Where do acceptance criteria come from for an existing feature with no specification? decides where the criteria come from. This mark fits any answer it gives.
    3. Characterisation Slices run before every other slice of the feature, and so before an Acceptance Slice's RED. They pin the current behaviour before new code touches it, so they are a regression net for the rest of the build.

    The loop

    4. run-build becomes a loop_group. In each iteration, one body node picks the next slice and phase, as Step 2 does today. Then the body node for that phase runs. A loop_group body node keeps its own allowed_tools, denied_tools and sandbox on Archon 0.7.0, the plugin's floor (#632). A loop: node does not.
    5. The agent that writes a characterisation test cannot read the application source. Its node denies Read, Grep and Glob on the source paths and uses sandbox.filesystem.denyRead on the same paths for Bash. The Consumer declares which paths are application source. The plugin names the need only. The reason: a test written from the code records today's behaviour as correct, bugs included. A test written from the criterion finds them.
    6. If a real run shows that Archon does not enforce those rules, the writer may read the source. In that case the decision records the gap: a Characterisation Slice then finds fewer bugs, and the report says so.

    The proof that the test can fail

    7. A fresh context runs the slice's test_command three times. First without sabotage, and it expects exit 0. Then with the behaviour sabotaged, and it expects a non-zero exit. Then without sabotage again, and it expects exit 0. The third run proves the restore worked. The sabotage is never committed. The plugin checks exit codes only.
    8. The sabotage technique is context, not plugin code. A Practice Pack supplies it for its e2e framework, and the Consumer can override it in config. The plugin does not know which e2e framework a Consumer uses.
    9. The second run must fail for the right reason. The failure classifier from #541 reads its output. A failure on the criterion's assertion proves the test. environment issue cancels the build. A test that still passes is broken or stale test: the writer rewrites it, within Does an e2e test become an outer loop around /build's unit TDD loop for a new feature? #541's attempt cap. At the cap the test is not committed, and the criterion becomes a deferred gap with a tracker item, as in item 10.
    10. When issues.json holds a Characterisation Slice and no sabotage technique is declared, bootstrap reports it and guard-not-ready cancels the build before the loop. This is the same shape as Does an e2e test become an outer loop around /build's unit TDD loop for a new feature? #541's readiness check.

    A failing characterisation test

    11. A failure on the first run goes to the Does an e2e test become an outer loop around /build's unit TDD loop for a new feature? #541 classifier. environment issue cancels. broken or stale test means rewrite the test. genuine bug is the finding.
    12. A genuine bug is always deferred, whatever its severity. The test is marked with the framework's skip-as-known-failure marker and links a new item on the Consumer's tracker. The severity sets that item's priority only. The build never changes production code for a Characterisation Slice. The feature already ships, so stopping the build protects nothing and throws away the tests that pass.
    13. A deferred test skips the sabotage proof. It already fails against the real feature, and the classifier confirmed that it fails on the criterion.

    What the evidence verdict is

    The evidence verdict takes the values pass, fail and unresolved. #439 calls fail "red".

    What happens	Result
    Tests pass and are proven	pass
    A genuine bug is found and deferred	pass. The finding reaches review in the pull request and on the tracker
    A test is still vacuous at the attempt cap	pass, with the criterion as a deferred gap
    The e2e environment is down, or no sabotage technique is declared	The build is cancelled before the loop, and no verdict is written

    For pass to hold after a deferral, three nodes change. On develop today each of them can turn a deferral into fail or into a doubtful pass:

    • goals-check step 3 marks a criterion covered only when a test and an implementation both exist. It must count a deferred criterion, one with a skipped test and a linked tracker item, as covered with a gap.
    • verification step 2 flags TODO and FIXME comments in changed source as stubs, and step 5 makes any stub a blocker. A skip marker that links a tracker item must not count as a stub. A // FIXME comment beside it would count as one for certain.
    • verification step 1b calls an e2e run that reports zero tests "suspicious". The report must give the count of deferred tests instead.

    The same three edits are needed for #541's item 8, which defers a minor failure with a skipped test and a tracker item. Nobody had found this gap before.

    The ADRs

    • build-generic-red-green-refactor-loop is amended. The loop becomes a loop_group, it gains the Characterisation Slice's phase pair, and slices are ordered as in item 3.
    • fresh-context-red-green-separation is amended, not excepted. It gains a second way to prove that a test can fail: the three-run sabotage proof, made by a context separate from the one that wrote the test. The anti-cheat guarantee stays. Only the mechanism differs. An exception would say that a Characterisation Slice has no anti-cheat, which is false.

    The base branch

    No new Box is made, so no Box writes the rule "develop when gitflow, else main" a fourth time. The changes stay in unic-dlc-build.yaml and take the base branch the way No Box can target a base branch that is not the trunk decides.

    Left open, on purpose

    • Added 2026-09-30: items 7 and 9 do not conflict. The exit code decides pass or fail for each of the three runs. The Does an e2e test become an outer loop around /build's unit TDD loop for a new feature? #541 classifier judges only why the second run failed.
    • The names of the two phases and their states in build-state.json go to /to-spec.
    • The config for the source paths and the sabotage technique goes into the single block that Does an e2e test become an outer loop around /build's unit TDD loop for a new feature? #541 already sends to /to-spec.
    • The attempt cap in item 9 is Does an e2e test become an outer loop around /build's unit TDD loop for a new feature? #541's cap. Its value goes to the map's "The numbers".
    • The first criteria of the implementation ticket are three measurements, each needing one real run on the target Archon version:
      • that Archon enforces scoped denied_tools entries and denyRead in a loop_group body node;
      • that a loop_group resumed after a failure continues from build-state.json, since it restarts from iteration 1;
      • that a body node accepts when:.
    • The term Characterisation Slice enters apps/claude-code/unic-archon-dlc/CONTEXT.md in the implementation pull request. Avoid: existing-feature slice, Case A slice.

    Implementation

    This is Box and command work for /to-tickets. The tickets edit unic-dlc-build.yaml, commands/specs.md and commands/tickets.md, so they carry blocked_by edges to open-pr precedes its gate and #505. They are ordered against the tickets from #541, which edit the same Box. The seat that owns the Box files hears of them first. The build diagram under apps/claude-code/unic-archon-dlc/docs/architecture/ needs a redraw.
13. orioltf

    closed this as completedon Sep 30, 2026