Repository navigation
Recover native cache observer failure on catalogue main #655
Description
🤖 Generated by the Agentic Engineer
Delivered on: 2026-10-19
This date records delivery of the diagnostic increment in #659, not resolution of this Bug.
Blocker: a new qualifying native cache-observer rejection with the released diagnostics | outcome | last-verified 2026-10-19: Main CI has recovered, but the original rejected snapshot was not retained. Resume diagnosis when a native rejection supplies the bounded count receipt from #659; do not manufacture a failure or relax the input predicate.
Description
Catalogue CI previously failed in the native Go cache observer, leaving the required aggregate red. Main CI has recovered, and the diagnostic increment is merged, but the specific original rejected invariant remains unproven.
Steps to Reproduce
Inspect a qualifying default-branch CI rejection in the Native Go Cache Policy job and its bounded count diagnostics. The original rejected snapshot was not retained, and a later complete read passes the unchanged predicate; there is no confirmed deterministic reproduction yet. Detailed evidence remains private.
Expected Behavior
The observer validates complete current-attempt evidence before checking real cache restore and save behavior. Missing, inconsistent or unsuccessful inputs continue to fail closed.
Actual Behavior
The observer rejected its input evidence. A later complete read passes the unchanged input predicate, but the original rejected snapshot was not retained. The root cause remains unproven; do not classify this as a cache-policy or quota regression without further evidence.
Acceptance Criteria
- Identify the rejected invariant using appropriately sanitized diagnostics and add a regression for the confirmed cause.
- Restore successful current-main CI with the native observer and required aggregate executed.
- Preserve complete pagination, identity binding, successful fixture requirements, useful restores and the existing cache-writer restrictions.
- Keep CI recovery separate from release publication and consumer cache-budget outcomes.
Environment
Hosted default-branch catalogue CI. The diagnostic increment adds only bounded counts and regressions; it does not change credentials, required checks, cache-writer restrictions or the strict input predicate.
Priority and Effort
High: intermittent default-branch reliability risk. Small-to-medium remaining diagnostic and repair scope. Part of #287.
Metadata
Metadata
Assignees
Type
Fields
Priority
Effort
Projects
- Status
Activity
chore: agent-claim #655 nonce=b5ad683614d8d033a3d407e260e098c1
chore: agent-claim #655 nonce=0c5f686d531151547150cfe823410349
devantler commentedon Oct 19, 2026
One complete rerun of unchanged current-main CI has been accepted and is queued. Every selected test and the required aggregate remain enabled. No source, credential, retry default or cache policy was changed.
Collect the complete attempt after it finishes and validate the native observer together with the required aggregate. Recovery is pending; the rejected-input root cause and this issue's acceptance criteria remain open. Detailed evidence is retained privately.
devantler commentedon Oct 19, 2026
Recovery confirmed, and a likely cause to test
Recovered. The complete rerun of
mainCI at576414b(attempt 2) finished green at 16:53Z, with the native observer and the required aggregate both executed. That meets the second acceptance criterion. The first criterion, naming the rejected invariant, is still open.What failed in attempt 1. The observer started at 15:55Z and exited in
validate_jobs("incomplete, stale or unsuccessful current-attempt jobs"). All nine fixture jobs it requires had already completed successfully between 15:42Z and 15:47Z, so the fixture-job conditions could not have rejected it. That leaves the conditions on the job list itself: onetotal_countacross pages, the job count equal tototal_count, and no duplicate job IDs.Likely cause (not yet proven). In attempt 1 the observer paged through about 240 jobs, 50 per page, while many other jobs in the same attempt were still running. The attempt kept changing after the observer read it: the aggregate job was created at 15:58Z and two more jobs at 16:01Z and 16:05Z. If the jobs endpoint's order or count moves while a run is in flight, a job can appear on two pages or on none. Either fails the completeness checks even though nothing is wrong with the cache. In attempt 2 the job list was effectively static by the time it was read, which fits the rerun passing with no input changes.
Smallest next slice. When
validate_jobsfails, print sanitized counts only: the number of pages, the distincttotal_countvalues, the number of jobs fetched, the number of duplicate IDs, and the fixture-job count. The next failure then names its invariant. If the hypothesis holds, the fix is to read the job list only after every job the observer depends on has finished, or to re-read until two consecutive complete reads agree. The completeness checks themselves stay strict.Not starting implementation now. Board WIP is above its ceiling, and with
maingreen again this no longer meets the Expedite bar of a confirmed live incident.chore: agent-claim #655 nonce=9a0e68c7073b5d278fc23f20acf6a512
devantler commentedon Oct 19, 2026
Current-main CI has recovered on the single full rerun, with no source, credential or cache-policy change. The complete current-attempt inventory is accounted for; the native observer actually ran and passed all eight Go restores and the expected save policy, and the required aggregate succeeded.
This clears the immediate CI failure, not the root-cause acceptance criteria. The original rejected snapshot was not retained, so the specific rejected invariant remains unproven. Keep this issue open for appropriately sanitized failure diagnostics and a regression for the confirmed cause. Release publication and consumer cache-budget outcomes remain separate. Detailed evidence is retained privately.
chore: agent-claim #655 nonce=f7450c2c6099eadf7c2eb7776d66d4a8
devantler commentedon Oct 19, 2026
The bounded diagnostic increment and its regressions are merged in #659. The latest main CI at
68927b3801d388baf8243583def438d87d5cbeceis successful; this does not identify the original rejected invariant or close this Bug.No delivery PR is still in review for this issue. Returning the continuation to Ready with an explicit outcome blocker: resume when a qualifying native observer rejection supplies the new bounded count receipt. Preserve the original acceptance criteria and work history; do not manufacture a failure, broaden cache writers or label the historical cause as proven. Publication and portfolio-budget adoption remain separate work.