Skip to content

workflows: acceptance criteria without requiredEvidence can pass with zero evidence (by-design tension) #119

Description

@tt-a1i

Description

Verified by read-only audit (seal/gpt-5.6-sol subagent, verdict: PARTIAL — behavior confirmed, but an explicit by-design contract rather than an implementation oversight).

Behavior

  • requiredEvidence is explicitly optional (acceptance.ts:4-7); omitted/empty requirements are normalized away (acceptance.ts:60-82).
  • evidence must be an array but has no minItems (acceptance.ts:106-113); empty/invalid entries normalize to [] (acceptance.ts:227-233).
  • Only declared labels are enforced (acceptance.ts:253-260); all-accepted statuses make the ledger accepted (acceptance.ts:268-274) → ok:true (acceptance.ts:287-291).
  • acceptance.test.ts:17, 61-74 — the scope criterion has no requirements, returns evidence: [], and explicitly asserts accepted.ok === true. This is deliberate, not accidental.

Documentation contract

skills/workflows/REFERENCE.md:21 documents requiredEvidence? as optional; README.md:303-305 promises an "evidence ledger", not non-empty evidence per criterion. Runtime matches the documented structural contract.

Why still worth raising

Semantic tension: acceptance.ts:190-191 states acceptance is "explicit evidence, not a self-awarded success claim" — yet a description-only criterion is validated solely by the agent's own status assertion, with no structured field independently demonstrating it. The design is defensible as configurable acceptance, but callers must know that criteria wanting proof must declare requiredEvidence, or the gate silently degrades to attestation.

Options (maintainer decision)

  1. Keep the contract; strengthen docs to state explicitly that criteria without requiredEvidence are attestations, not evidence-backed gates.
  2. Tighten: require at least one non-empty evidence entry per accepted criterion (schema minItems: 1 + evaluator check), possibly behind an opt-in strict mode.

Found by automated workflow audit (run wf_56028a5b018b).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions