Han Feedback: automated-test-planning-followup-tp001 (2026-07-24)

[mxriverlynn]

Han Feedback — 2026-07-24 (follow-up)

Skills used: han-coding:automated-test-planning Agents used: han-core:test-engineer, han-core:edge-case-explorer Context: Follow-up to today's earlier feedback file: the plan's single HIGH-priority finding (TP-001, "untested term of a summing method, reachable in production") was implemented afterward and turned out not to be viable — the input it depends on cannot exist in production. Outcome: The test was withdrawn, the plan was corrected, and the attempt surfaced a real (smaller) finding the plan had missed: that term of the sum is structurally always zero for the account type under test, and a public field description overpromises it.

---

What worked well

• The failure was cheap to discover because the finding was falsifiable: it named the exact seam and setup, so a single test-writing attempt disproved it within a few minutes. A vaguer finding would have survived unchallenged.
• The attempt still produced value: it uncovered that the supposedly-missing behavior is unreachable by design (a source-side validation forbids one direction; the destination-side lookup resolves through database views whose inner joins exclude this account type), which corrected both the plan and a misleading field description.

What didn't work

• Reachability was argued from absence of restriction, not presence of a path. edge-case-explorer justified the HIGH priority with "nothing in the creation class restricts this account type as a source or destination" plus the field's own documentation promising the behavior. Both checks were real but shallow: the restriction lived one layer down, in how the destination association resolves through a database view whose joins exclude the account type, and in a source-side validation the agent never read. "Not forbidden at the model surface" was treated as equivalent to "creatable through a real flow," and that equivalence is exactly what failed.
• Documentation was used as evidence of behavior. The field description promising the behavior ("including deposits, transfers, etc.") was cited as support for reachability. Descriptions state intent; when intent and reachability disagree, the disagreement is itself a finding — here it turned out the description overpromises, which the plan should have flagged instead of leaning on.
• The claim was presented as settled fact with no evidence-strength label. TP-001's justification read as verified ("reachable in production, not a hypothetical"), and both agents independently agreeing hardened the confidence. Independent agreement between agents doing the same shallow trace is correlated error, not corroboration.
• The earlier feedback file for today praised this exact check. It credited edge-case-explorer's "reachability check" as what elevated the gap from symmetry to realistic failure mode. This follow-up corrects that: the check added confidence, not correctness. If the earlier file has not yet been posted, read it alongside this one.

Suggested improvements

1. Require positive-path evidence for reachability claims that carry priority. When a test-planning finding's priority hinges on "a real caller can produce this input," edge-case-explorer and test-engineer should trace the write path end to end — who creates the record, and how every association or foreign key in the validation and read paths resolves — rather than confirming no restriction exists at the layers they happened to read. Associations that resolve through database views or cross-table joins are a named trap worth calling out in both agents' instructions: a model-level validation can pass in a spec via in-memory assignment while being unsatisfiable through any real flow.
2. Treat documentation as intent, never as evidence. If a description, comment, or schema doc promises a behavior, the agent should verify the behavior independently and raise a mismatch finding when verification fails — not cite the promise as support.
3. Label the evidence strength of reachability claims. A one-word tag on the finding ("traced end-to-end" vs. "no restriction found at the layers read") would have told the implementer TP-001 rested on the weaker form and needed a spike before being called HIGH.
4. Recommend a setup spike for coverage-gap findings. The cheapest validator of "this input is reachable" is attempting the test setup through real factories and flows. automated-test-planning could note, for each HIGH item resting on a reachability claim, that a failing-setup attempt is the verification step — and that needing to bypass validations or hand-assign associations that production resolves by lookup is the signal the state is unreachable and the finding is wrong.

---

Overall

The pipeline's one actionable HIGH finding did not survive contact with implementation, and the failure mode is systematic rather than bad luck: both analysis agents verify reachability by looking for restrictions at the layers they read, so anything enforced by lookup resolution, database view joins, or a validation in an unread file passes their check. The saving grace was that the finding was specific enough to falsify quickly, and the falsification itself produced the session's most useful discovery. The fix is not more analysis effort across the board — it is requiring positive-path evidence (or an explicit weaker-evidence label plus a setup spike) whenever a finding's priority depends on an input being producible in production.

---

Rating

Dimension	Score
Output accuracy	2/5
Evidence discipline	3/5
Finding signal-to-noise	3/5
Output length vs. decision count	3/5
Turn efficiency	4/5

Ratings here score only the TP-001 finding chain (the subject of this follow-up), not the full run rated in today's earlier file. Accuracy 2/5 because the plan's headline recommendation was wrong in its central factual claim; evidence discipline 3/5 because the agents did gather real evidence — they stopped tracing one layer too early; turn efficiency 4/5 because the error cost little to expose once implementation began.