Skip to content

test(generate): generation-correctness census forcing suite - #637

Merged
joshua-temple merged 1 commit into
mainfrom
feat/generation-correctness-census
Jul 18, 2026
Merged

test(generate): generation-correctness census forcing suite#637
joshua-temple merged 1 commit into
mainfrom
feat/generation-correctness-census

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

Problem

PR #631's emitted-field census forces every emitted-affecting manifest field into actionlint coverage: the emitted output is VALID (real GitHub accepts it at parse). Validity is not correctness. A documented feature can emit valid YAML that does the WRONG thing (a matrix deploy that drops sha, a dry_run guard blind to a boolean, a dependent deploy gated on the immutable base result). Those defects are invisible to actionlint, act, and reviews because no test asserts the emitted output's semantic shape. Ad-hoc assertions exist for a handful of past defects, but there is no systematic, census-forced correctness coverage, so the next feature nobody asserts escapes silently.

Fix

A correctness census that reuses #631's emitted-affecting surface (the union of emittedFieldRegistry and emittedAffectingAllowlistMutators) and forces every field into exactly one classification:

  • a named correctness assertion pinning its emitted semantic shape, or
  • a reviewed marker: validity-only (contract is validity + round-trip, already covered), structural (pinned by a sibling's assertion), not-emitted, or followup (emitted-affecting, warrants an assertion not yet written).

TestCorrectnessCensus_EveryEmittedFieldClassified reds when a new emitted-affecting field appears unclassified (proven by TestCorrectnessCensus_ForcesNewField against a synthetic field), mirroring how TestActionlint_AllowlistFieldsClassified already forces validity classification. TestCorrectnessCensus_AssertionsExist grounds every named assertion in a real test function, so a typo cannot leave an entry pointing at nothing.

Assertions added (each proven red-first)

Each was verified by breaking its emitter, watching the test red, and restoring:

  • least-privilege per-job permissions (a job with no configured perms emits no block; scopes never leak across jobs)
  • secrets.map propagation direction (callee input maps to the source secret expression, not reversed)
  • concurrency cancel-in-progress honored (not defaulted)
  • action_pins ref splicing into uses:
  • dispatch input typing (type/options/default)
  • extra_triggers schedule/repository_dispatch/workflow_run emission
  • custom git identity splicing
  • environments[].environment_url per-environment case threading

Honest limit

A generation-correctness assertion pins the shape the author believes correct. It catches a missing key, a wrong-versus-baseline expression, and a regression. It does NOT prove the shape is correct against real GitHub Actions runtime semantics (whether != 'true' rescues a boolean dry_run, whether a retry shim rescues a failed ladder). That proof is the fleet's job. The runtime-semantic assertions say so in-test and defer to the fleet.

Remaining-fields followup

The followup-marked fields are the enumerated backlog for the next tranche: run_policy if-gates (builds/deploys/validate), build-side and optional depends_on gating, role ladder ordering, GPG signing wiring, pin_mode, release_trigger, and reconcile source/commit output. The census red state tracks them.

Verification

Test-only; not fleet-relevant (no emitter changed).

Extends the emitted-field census from validity (actionlint) to semantic
correctness. Every emitted-affecting manifest field now maps to either a
named correctness assertion pinning its emitted shape, or a reviewed
marker (validity-only, structural, not-emitted, followup). A new
emitted-affecting field reds TestCorrectnessCensus_EveryEmittedFieldClassified
until it is classified, so a feature whose output nobody asserts cannot
ship silently.

Adds eight red-first correctness assertions covering least-privilege
per-job permissions, secrets.map propagation direction, concurrency
cancel-in-progress, action_pins ref splicing, dispatch input typing,
extra_triggers emission, custom git identity, and environment_url
threading. Classifies the full emitted-affecting surface; the followup
markers enumerate the fields a later tranche should pin.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple
joshua-temple merged commit cb69b79 into main Jul 18, 2026
21 checks passed
@joshua-temple
joshua-temple deleted the feat/generation-correctness-census branch July 18, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant