Skip to content

fix(quality-gate): stop baselines from loosening - #615

Merged
Tr00d merged 2 commits into
masterfrom
ci/baseline-only-tightens
Oct 30, 2026
Merged

Tr00d merged 2 commits into
masterfrom
ci/baseline-only-tightens

Conversation

@Tr00d

@Tr00d Tr00d commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Why

The gate compares each measurement with the committed .gate-baseline.json, but it reads whatever baseline the PR provides. A PR that raises a warning count or lowers a coverage value therefore passes the gate it has just weakened.

What changes

New check: Baseline only tightens

A new build-and-test.yml job runs scripts/quality-gate/baseline-tightened.sh on every PR, once for each .gate-baseline.json the PR changes. If the PR changes none, the job passes in seconds.

The gate itself only ever moves a baseline toward stricter, so a looser value can only come from a hand edit. The job fails when:

  • warnings.total rises, or disappears
  • any warnings.byCode entry rises (a code that wasn't there before counts as 0)
  • coverage.hermeticLine falls, or disappears
  • schema, package, project or testProject changes
  • the file is deleted

A new package's first baseline and changes to updated are allowed.

Fix: the gate loosened coverage itself

I replayed every baseline change on master through the new check. Two automated refreshes lowered coverage:

Commit Package hermeticLine
4f1009c8 Gotrue 49.44 → 49.38
87d451e0 Gotrue 41.58 → 41.30

#369 also lowers coverage, but that one was a deliberate change to how coverage is measured.

The cause, in two steps:

  1. When warnings fell, save_baseline rewrote the whole file from a heredoc, which dropped coverage.hermeticLine.
  2. save_coverage_baseline then found no coverage key and recorded the current run's value, which can be up to COVERAGE_EPSILON (1 point) lower.

The fix: save_baseline now merges .warnings and .updated into an existing file and keeps every other key. Without it, the new check would block the next refresh PR that lowers warnings.

The gate fixture baseline-overwrite-keeps-coverage pins this. It fails on the old warnings.sh (expected 99.5, got null) and passes on the new one. run-fixtures.sh gains a baselineKeeps assertion so the fixture can check that a key survives a write.

Verification

  • scripts/quality-gate/fixtures-baseline-tightened/run.sh: 15/15. Every rule has a case. A loosened verdict counts only when the script prints a LOOSENED: line, so a jq error can't pass as a detected loosening.
  • scripts/quality-gate/fixtures/run-fixtures.sh: 20/20, including the new fixture.
  • History replay: every baseline change on master since the fix era passes. The only failures are the two sliding refreshes and chore(quality-gate): scope coverage collection to each assembly #369 above.

This PR is independent of #614.

The gate reads the baseline it is given, so a PR that loosened a
.gate-baseline.json passed the gate it had just weakened. A new
"Baseline only tightens" job runs baseline-tightened.sh on every PR
against each changed baseline: warnings up, coverage down, an identity
field changed or the file deleted fails it.

Replaying master's history through it found the gate itself loosening
coverage: save_baseline rewrote the whole file when warnings fell,
dropping coverage.hermeticLine, which save_coverage_baseline then
re-established from the current run up to COVERAGE_EPSILON lower
(4f1009c: 49.44 -> 49.38, 87d451e: 41.58 -> 41.30). save_baseline now
merges into an existing file; baseline-overwrite-keeps-coverage pins it.
@Tr00d
Tr00d marked this pull request as ready for review October 30, 2026 13:20
@Tr00d
Tr00d merged commit d333a89 into master Oct 30, 2026
6 checks passed
@Tr00d
Tr00d deleted the ci/baseline-only-tightens branch October 30, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant