Enforce a 15-minute CI feedback budget across the portfolio #654

🤖 Generated by the Agentic Engineer

User Story

As a maintainer and contributor,
I want every pull request's complete CI feedback in less than 15 minutes,
So that CI does not dominate task lead time and every regression forces workflow optimization or redesign before merge.

Maintainer direction, 2026-10-19: implement a portfolio-wide baseline budget check after demonstrating sound GitHub Actions optimization practice, then optimize the outliers. This issue is the implementation and rollout tracker, not evidence that the guard is installed.

Evidence and measurement contract

A fresh inventory found 22 active public mapped repositories. The last three successful first-attempt repository-owned CI runs in each produced 66 complete, paginated job observations. Archived repositories were excluded; private-product rollout and evidence stay private. This success-only sample does not establish coverage of failed/cancelled attempts, all PRs or every organization-required workflow.

Measure from native workflow creation to actual final relevant check/job completion, including runner wait, setup, dependency serialization, execution and teardown. Never use workflow updated_at as completion. Counterexample: monorepo run 39277933824 has a 39m42 update span but its complete 96 jobs finish after 10m34. World run 39277625663 has a 55m13 update span but its 32 jobs finish after 29m53. These are regression fixtures for the future collector, not optimization claims.

Confirmed repository-owned CI outliers in the 66-run sample:

The isolated agent-skills and platform-tenant-template outliers have much faster subsequent samples; diagnose the specific admission/dependency/runner cause rather than presuming persistent slow tests.

The workflow name CI is not the full portfolio feedback path. At the exact KSail revision of run39189008611, independent required Go validation took over 22 minutes as well. Conversely, go-template's own CI takes seconds while the independent Go/scaffold validations take minutes. Native rulesets also require code-quality, scanning and other workflows. Inventory those paths explicitly before claiming a repository is under budget.

Proposed direction

• A hard per-revision budget of less than 900 seconds, not an average or median target. Track PR and merge-group execution separately without disguising total delivery lead time; releases/CD remain separately visible and may not absorb mandatory pre-merge checks merely to game this budget.
• Use immutable subject repository/head/event/run-attempt identities and native timestamps. Keep the original admission clock across retries; a re-run or a late update must not manufacture a fresh 15-minute allowance. Superseded heads cannot approve the current head.
• Fail closed on incomplete pagination, duplicate/mismatched job IDs, missing expected workflows/checks, malformed or reversed timestamps, cancelled/failed current attempts and API errors. A fast failed run is not successful CI.
• Preserve all existing quality, security, test-scope, provenance, cleanup and production-serialization outcomes. A budget failure names its real critical path and links the run; the PR must optimize or redesign that path, not relax the ceiling.
• Prefer extending an existing end-of-CI aggregation surface or an event-driven immutable-source observer over a runner sleeping/polling for 15 minutes. Prove the observer cannot pass early, race a late required workflow, depend on itself or leave a stale green result after a new head/retry.
• Shared implementation belongs to this catalogue. Organization enforcement and adoption belong to declarative GitHub configuration, never imperative ruleset writes. Reuse existing identities and least-privilege metadata access; do not expand credentials merely to collect timing.
• New behavior starts default-off with both states tested; publish, pilot against genuine positive/negative PRs, then activate required enforcement across every active mapped repository. Track and remove the temporary rollout switch. Optional flags and passing YAML are not completed adoption.

Optimization order and protected floors

1. Profile exact job/step critical paths and separate admission/runner waits from execution. Balance matrices and parallelize genuinely independent work rather than raising concurrency blindly.
2. Eliminate duplicate setup/build work with correctly scoped immutable artifacts and complete cache keys. Exercise both cold and warm states; never reuse a stale verdict, cross trust boundaries or remove build inputs from a key without correctness proof.
3. Use dependency caching for dependencies and artifacts for produced outputs; keep untrusted restores and trusted cache writes separate.
4. Cancel superseded read-only PR work where safe, but never interrupt protected publication/deployment or required cleanup to meet the budget.
5. Use affected-test selection only with an independently verified input/dependency closure and full-coverage fallback. Test-directory paths can be production inputs (platform#5077).
6. Redesign genuinely serial long-running scenarios without dropping their outcomes. Moving an unchanged slow mandatory suite after merge is not a speedup.

Acceptance Criteria

• Reviewed design records exact admission/end clocks, expected native check/workflow inventory, retry/supersession semantics, PR/merge-group coverage, failure behavior, privacy, rollback and publication/adoption sequence.
• TDD covers 899/900/901-second boundaries, missing/partial/duplicate API data, API errors after plausible output, mutable update timestamps, clock skew, current-head cancellation, late/missing required workflows, retries and stale-head races.
• The actual immutable-source wrapper is tested in omitted/off/on states, with least-privilege positive and refusal cases; no candidate code or untrusted artifact executes with reporting authority.
• Shared guard is merged, released and exercised from genuine consumers; its own overhead is included and measured.
• Required declarative enforcement is merged, published, reconciled and independently read back for every active mapped repository; none is silently skipped because its workflow is named differently or its repository is private.
• Every confirmed outlier is optimized through reviewed merged PRs, retaining its intended outcomes and showing comparable cold/warm before/after evidence.
• All in-scope current-revision CI completes in less than 900 seconds in the declared verification cohort; failures/unknowns remain failures/unknowns, not exclusions or waivers.
• A deliberately over-budget/refused candidate is blocked, an under-budget complete candidate passes, and the temporary rollout controls are retired after verified adoption.

Alternatives Considered

A blanket job-level timeout-minutes: 15 does not constrain a multi-job critical path and can terminate validation or cleanup. A mutable workflow-update timestamp creates false failures. Checking only ci.yaml, only successful fast jobs or only a median misses real feedback delays. A permanently optional timing report is not enforcement.

Sources and related work

Primary guidance reviewed: workflow diagnostics, concurrency, matrix parallelism, cache scope and trust, artifacts, workflow-run security and native run API.

Related existing work, not replacements: #475 (organization contention), #617 (accurate affected-test consumers), #565 (cache-write correctness), platform#4569 (parallel validation), platform#5077 (production-input closure), ksail#7857 (binary cache correctness), ksail#7869 (separate release-performance experiment).

Rough size: large, cross-repository delivery. Ship shared measurement/enforcement, then each critical-path redesign as independently reviewed increments; do not claim the whole target from one fast run.

Implementation resumed after a fresh passing preflight on 2026-10-19. First delivery slice: #656 (exact single-run measurement); temporary rollout-switch retirement: #657. Whole-PR required enforcement and outlier optimization remain open acceptance gates.