Repository navigation
Go helper suite: refresh-flux-ghcr-auth runs close to the default 10-minute test timeout and evicts unrelated PRs #4893
Description
🤖 Generated by the Agentic Engineer
Evidence
- Merge-group CI run 38597082826 (2026-10-14 16:19Z, for fix(deploy): name unready Data Product rollout checks #4883) failed in
🧪 Validate the complete Go helper regression suitewithpanic: test timed out after 10m0sinscripts/tests/refresh-flux-ghcr-auth(601 s). Every test running at the timeout had been running for only 1–8 s, so nothing was hung: the package as a whole ran past the limit. fix(deploy): name unready Data Product rollout checks #4883 was evicted from the merge queue for a change it does not touch. - On the seven most recent passing CI runs the same package took 287, 289, 319, 348, 357, 377 and 389 s. That is 48–65 % of Go's default per-package timeout, which the step (
go test ./..., no-timeout) relies on.
Problem
The package sits within about 1.5× of the default timeout, so a slower runner evicts unrelated pull requests from the merge queue. Each eviction costs a full re-queue and a full merge-group run.
Hypothesis
Making the package's runtime comfortably smaller than its timeout removes this eviction cause without weakening any check.
Smallest useful change
Prefer making the package faster (find which tests dominate with go test -json timings; many already call t.Parallel(), and the stack trace shows tests waiting in waitParallel, so parallelism is capped by -parallel/GOMAXPROCS on the 4-core runner). If it cannot be made meaningfully faster, set an explicit -timeout on the step sized from measured runtime plus headroom, so the bound is a deliberate choice rather than Go's default.
Acceptance criteria
refresh-flux-ghcr-authruns in under half of the timeout that applies to it on hosted runners, measured over at least five CI runs.- No test in the package is skipped, removed or weakened.
Success signal
No merge-group eviction caused by a Go test timeout in this step over the following two weeks.
Activity
devantler commentedon Oct 14, 2026
A bounded local profile at the signed helper-lifetime repair head (
ffac3f3dbeed77d47d6e697d1a9d4fd288bf53be) passed with four Go processors, four parallel tests and the unchanged 10-minute package timeout. The bridge package took 419.242 seconds with no failed cases. The longest case took 33.12 seconds; the next took 21.94 seconds, with several around 13–16 seconds.This is a macOS capacity sample, not hosted evidence or a before/after improvement claim. It confirms that the lifetime fix does not settle this issue's runtime-headroom acceptance: 419 seconds is still over half of the current bound. No check was skipped and no CI timeout was raised. The existing five-run hosted measurement and two-week eviction outcome remain open.
The useful next step is to profile time inside the slow cases, separating external command launches from required convergence/recovery waits before choosing an optimization. The lifetime repair remains separately scoped to #4862.
devantler commentedon Oct 15, 2026
A new hosted sample after #4902’s lifecycle repair is available: source CI 38691261878, at b9da86a, passed the complete Go suite. The bridge package took 365.590 seconds with the unchanged default 600-second bound: 60.9% of its timeout. The corresponding merge-group sample for #4902 took 340.174 seconds.
Both are above this issue’s under-half acceptance condition. The local 283.802-second pass is not substituted for hosted evidence. #4893 remains open: the process-lifetime repair is delivered, but hosted runtime headroom and the subsequent two-week eviction outcome are not established. No timeout or assertion was weakened.
devantler commentedon Oct 16, 2026
A local parallelism sample, 16 October. It suggests a lead but does not settle it.
I ran the package's test binary twice at main
6c586407on a 10-core macOS host, withGOMAXPROCS=4and the default per-package bound replaced by 20 minutes:-parallelCaveat:
GOMAXPROCScaps only the Go scheduler. The bash processes and fake commands each case spawns can still use all ten host cores, so this sample cannot tell how much of the gain comes from overlapping waits and how much from spare cores a hosted 4-core runner does not have. It does show the cases tolerate 16-way parallelism with no failures.Next measurement, if this issue is picked up: one hosted CI run of this package with
-parallel 16, to compare against the 287–389 s samples above. If the hosted gain is small, the explicit per-step-timeoutthis issue already names is the honest fix.The two hosted Go-suite failures seen since 14 October in another test package (
TestNativeMetadataProxyPreservesAcceptAndRejectsWrites) were a separate per-test 10-second bound, fixed by #4952.devantler commentedon Oct 18, 2026
Fresh isolated baseline at signed source
9c1bdc7c7c44caeef288c12f1bf9440482462ae4, with repository-pinned Go 1.26.9 and the unchanged package timeout: 454.918s, 541 passed test/subtest records, no failed cases.I will evaluate one bounded hypothesis: reduce scheduler startup overhead in the short-lived Go command processes used by fixtures, without changing the test runner's scheduling, fixture concurrency, commands, assertions, race/coverage flags or timeouts. A small executable-startup probe using the actual fake binary showed noisy results, so it is not sufficient evidence to ship.
Before changing the fixture environment, the local decision threshold is at least 10% lower full-package elapsed time (below 409.426s against this baseline), with the same complete test identities and all cases passing. An inconclusive or negative result will not be presented as an optimization. Hosted behavior and timing remain mandatory before merge; the original five-hosted-run headroom and two-week eviction criteria stay open here.
This issue is now linked as a real child of #4979. The maintainer's WIP exception applies, not any test or merge waiver.
chore: agent-claim #4893 nonce=b5ba46e3c8c0931693347bab4b255aa3
devantler commentedon Oct 18, 2026
Child-runtime experiment readback
The predeclared local trial (comment 6253491559) used source
9c1bdc7c7c44caeef288c12f1bf9440482462ae4and Go 1.26.9 throughout. The prototype setGOMAXPROCS=1only in the fixture-created child environment; the parent Go test process and parallel test-case scheduling were unchanged. Production helpers, test assertions, command deadlines and package timeout were unchanged.Both prototype runs passed the frozen <409.426s threshold (at least 10% below baseline), with exactly identical sorted test/subtest identities, zero failures and zero skips. This is local evidence, not a hosted-run comparison.
An additional full root-module
go test -count=1 ./...run with the prototype completed successfully: 55 packages passed, two packages had no tests, and 6,128 test/subtest records passed. Six existing native/private-prerequisite tests skipped; this is not a claim that their runtime behavior was exercised. The GHCR-auth package took 396.576s in that broader run, but concurrent package execution makes it a functional observation, not a controlled performance repetition.Limitation discovered afterward: an earlier externally timed-out local run had left fixture descendants alive. They were present throughout the three comparison runs, so host load was not controlled. Only this session's verified orphaned test processes were stopped; no production operation or credential was used. A clean-runner hosted trial is still needed before adoption.
Decision: eligible for a separate hosted trial, not adopted or delivered. The prototype is preserved privately and removed from the working source; no fourth CI-performance PR is opened while the three already-started delivery PRs are being finished. This issue remains open: the original five hosted runs below half the applicable timeout and two-week no-eviction criteria have not been met. No timeout or coverage reduction is proposed.
devantler commentedon Oct 18, 2026
Completed local profile, 18 October
The unmodified, signed
mainsourcededc6033bb3b39eaed40a99c0f993d4581e4bf39passed the complete fixture package with repository-pinned Go 1.26.9,GOMAXPROCS=4,-count=1and the unchanged 10-minute package timeout: 548.981 seconds, 541 passed test/subtest records, zero failures and zero skips. This is a shared-host macOS observation, not a hosted baseline or improvement claim.Native test events show 54.223 seconds spent in top-level tests that are not marked parallel. The largest barriers are the completed-transaction cleanup group (16.401s), removed-target inventory group (11.422s), empty historical-phase case (10.073s), and three historical-phase cleanup cases (12.759s combined). Some parents contain parallel subtests but still hold the top-level scheduling barrier. The expensive fixture groups use isolated temporary state; tests that change the process environment must remain serial.
Decision: do not open a fourth scheduling PR from this profile alone. Even eliminating all observed serial time would leave roughly 495 seconds in this local sample, well above this issue's under-half-timeout goal. It warrants a bounded hosted scheduling/child-process study, not an unsupported claim that a few added parallel markers solve the problem. The previously recorded child-runtime prototype remains unadopted. The existing five-hosted-run and two-week no-eviction acceptance criteria remain open, with no timeout, assertion or coverage reduction.
devantler commentedon Oct 19, 2026
The full local run did not pass; current-head hosted evidence is separate
After the isolated 548.981-second package pass, the complete local root suite at signed head
314b3646ae99a6dc8ee9f67a288e9e9f9cf35863failed in this same fixture package at 590.622 seconds. It recorded 90 failing test/subtest records across 16 top-level families as fixture-command contexts ran out of their remaining package budget. The first failures includeTestFluxChildDependencyRetryRequiresCurrentFailedHealthEvidence/current_failed_health_followed_by_dependency_retryandTestFluxControllerRolloutAllowsObservedConvergenceWithinFiniteBudget/observed_healthy_convergence, both reportingfixture command did not finish within its test budget: context deadline exceeded.The other 55 packages passed; six existing private/native-prerequisite tests skipped. This is a failed complete local run, not an all-green suite. The fixture package, helper and Go module inputs are byte-identical to the earlier
dedc6033bb3b39eaed40a99c0f993d4581e4bf39baseline; the workflow merge repair did not change them. Shared-host load was not controlled, so this does not identify a code regression or quantify an optimization.At that exact signed head, hosted CI 39178380006 / job 117825706697 subsequently passed the complete root Go suite, with this package reporting 386.257 seconds, and passed the separate native-retirement race suite. The complete job took 9m32s; the root-suite step took 7m18s.
Decision: this issue remains open. The hosted package is still above half of its unchanged 600-second bound, and one hosted pass does not establish the five-run target or the two-week eviction outcome. No timeout, command, test selection or assertion was changed to obtain the hosted result.