test(generate): generation-correctness census forcing suite - #637
Merged
Conversation
Extends the emitted-field census from validity (actionlint) to semantic correctness. Every emitted-affecting manifest field now maps to either a named correctness assertion pinning its emitted shape, or a reviewed marker (validity-only, structural, not-emitted, followup). A new emitted-affecting field reds TestCorrectnessCensus_EveryEmittedFieldClassified until it is classified, so a feature whose output nobody asserts cannot ship silently. Adds eight red-first correctness assertions covering least-privilege per-job permissions, secrets.map propagation direction, concurrency cancel-in-progress, action_pins ref splicing, dispatch input typing, extra_triggers emission, custom git identity, and environment_url threading. Classifies the full emitted-affecting surface; the followup markers enumerate the fields a later tranche should pin. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PR #631's emitted-field census forces every emitted-affecting manifest field into actionlint coverage: the emitted output is VALID (real GitHub accepts it at parse). Validity is not correctness. A documented feature can emit valid YAML that does the WRONG thing (a matrix deploy that drops sha, a dry_run guard blind to a boolean, a dependent deploy gated on the immutable base result). Those defects are invisible to actionlint, act, and reviews because no test asserts the emitted output's semantic shape. Ad-hoc assertions exist for a handful of past defects, but there is no systematic, census-forced correctness coverage, so the next feature nobody asserts escapes silently.
Fix
A correctness census that reuses #631's emitted-affecting surface (the union of
emittedFieldRegistryandemittedAffectingAllowlistMutators) and forces every field into exactly one classification:validity-only(contract is validity + round-trip, already covered),structural(pinned by a sibling's assertion),not-emitted, orfollowup(emitted-affecting, warrants an assertion not yet written).TestCorrectnessCensus_EveryEmittedFieldClassifiedreds when a new emitted-affecting field appears unclassified (proven byTestCorrectnessCensus_ForcesNewFieldagainst a synthetic field), mirroring howTestActionlint_AllowlistFieldsClassifiedalready forces validity classification.TestCorrectnessCensus_AssertionsExistgrounds every named assertion in a real test function, so a typo cannot leave an entry pointing at nothing.Assertions added (each proven red-first)
Each was verified by breaking its emitter, watching the test red, and restoring:
secrets.mappropagation direction (callee input maps to the source secret expression, not reversed)cancel-in-progresshonored (not defaulted)action_pinsref splicing intouses:extra_triggersschedule/repository_dispatch/workflow_run emissionenvironments[].environment_urlper-environment case threadingHonest limit
A generation-correctness assertion pins the shape the author believes correct. It catches a missing key, a wrong-versus-baseline expression, and a regression. It does NOT prove the shape is correct against real GitHub Actions runtime semantics (whether
!= 'true'rescues a boolean dry_run, whether a retry shim rescues a failed ladder). That proof is the fleet's job. The runtime-semantic assertions say so in-test and defer to the fleet.Remaining-fields followup
The
followup-marked fields are the enumerated backlog for the next tranche:run_policyif-gates (builds/deploys/validate), build-side and optionaldepends_ongating,roleladder ordering, GPG signing wiring,pin_mode,release_trigger, andreconcilesource/commit output. The census red state tracks them.Verification
go build ./...,go test ./...(3614 pass, actionlint on PATH so fix(generate): enforce emitted-workflow validity; reject on_failure continue, gate hotfix inputs #631's guard runs),go test ./internal/generate -race,golangci-lint run ./...all clean.cascade verify --own-repo: no drift.Test-only; not fleet-relevant (no emitter changed).