Repository navigation
Recover catalogue CI from primary API quota exhaustion #596
Description
🤖 Generated by the Agentic Engineer
Description
Recorded required catalogue CI on main and an adapted dependency pull request failed when the repository workflow API budget is exhausted. The existing secondary-limit work in #288 explicitly retains this as a separate failure class. Those checks correctly failed; this issue must improve reliability without converting incomplete work into success.
Steps to Reproduce
- Read main CI run 38603765042 at
ed3d9c4767b304923f06c74bc5564f8da8c306c9. - The Ubuntu skill-installation job exhausts all three default attempts while GitHub reports primary installation API quota exhaustion.
- The read-only cleanup fixture refuses inventory retrieval because the primary reset exceeds its bounded retry envelope.
- PR #592 CI run 38607731850 at
5ad5bf4ea9e3e66a5e2990eef047c01e5fd75831shows the same primary-quota class in its auto-merge fixture.
Expected Behavior
Normal repository validation remains reliable within its API allowance. Inventory, installation and security checks examine their complete inputs and fail closed when evidence is incomplete.
Actual Behavior
At the recorded failing heads, installation could not resolve source repository metadata and cleanup could not retrieve its complete inventory. Default-branch CI has since recovered: run 38771224501 succeeded at main 226e6e7 on October 15. This does not establish request attribution or pressure reduction, so the acceptance criteria remain open. A bounded fast retry does not replenish an hourly primary budget.
The main TODO fixture separately reports a 502 through its local read-only proxy; its upstream cause is not established by this sample. PR #592 dependency review says the repository is unsupported, while a later authenticated dependency comparison at the same base/head succeeds. That later read proves support for the caller used here, not availability or permissions for the failing workflow token. Neither diagnostic is relabelled as proven primary quota exhaustion.
Environment
Hosted catalogue CI at the two immutable heads above. No credential, repository setting, retry default or security control was changed during diagnosis.
Acceptance Criteria
- Attribute the confirmed primary-quota failure to measured request sources and overlapping workflow activity; retain unknowns where the evidence does not support a join.
- Deliver the smallest source or scheduling repair that reduces or safely budgets that pressure, with a before/after request measurement and a real hosted consumer run.
- Demonstrate successful installation and complete inventory reads at the repaired current head, including required aggregate CI.
- Keep operational errors distinct from unsupported configuration and security findings; diagnose the TODO and dependency-review failures independently if they remain.
- Preserve all existing happy-path, refusal, mutation and completeness checks, credential boundaries, and the bounded retry exhaustion result. Do not raise quotas, broaden credentials, skip checks or change retry defaults to manufacture recovery.
Priority and Effort
High: confirmed main breakage prevents dependable release and dependency delivery across catalogue consumers. Medium effort: request attribution, a focused repair and hosted verification; this is not a measured estimate of the cause.
Part of #287. Related: #288 preserves its secondary-limit trial; #546 delivered bounded inventory retry behavior and must not be implemented again.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
chore: agent-claim #596 nonce=de430c4ff5701a60448f67869b1e63ee
chore: agent-claim #596 nonce=83bbc2f46f97b02c0b2b718f10927901
chore: agent-claim #596 nonce=ed9632c2bb12691b1af9cecd644ba8a7
chore: agent-claim #596 nonce=9984bddb84e89b6d4920ef2698c3854c
devantler commentedon Oct 14, 2026
The carried recovery run has completed. Main CI 38603765042, attempt 2 at unchanged
ed3d9c4767b304923f06c74bc5564f8da8c306c9now has 200 successful jobs, 47 skipped jobs and one failed required gate. A complete nine-page native job read reconciles all 248 advertised jobs, including three successful queue slots at this exact run, attempt and head.The remaining required-gate failure is different from the original installation quota failure: its job-result and selection checks passed, then all three complete queue-inventory attempts returned HTTP 502. A later bounded read of the same attempt endpoint returned HTTP 200. The response-size hypothesis is unproven; this does not establish a source defect or justify changing retry defaults, permissions or completeness checks.
The underlying installer, cleanup and TODO consumer jobs recovered without a source change. That is recovery evidence for this attempt, not the request attribution, pressure reduction or recurrence acceptance required by this issue. Keep #596 open. The earlier primary-quota evidence remains valid; the later server error must not be classified as that same failure.
PR #593 has independently begun a fresh recovery run. I am preserving that work and adding no competing full-catalogue rerun during it. The next invocation should collect that run and main's required-gate state once, then rebind current heads and ownership before deciding the next recovery action.
chore: agent-claim #596 nonce=fb9ad22ce30062b481cf5311a727307e
devantler commentedon Oct 14, 2026
A catalogue main CI run has recovered following the independently merged PR #593. Its merge commit
d4ce0b1d00e624574fea1e74f69111060f8be8d1was the default head when the check set was collected. Main CI 38627520915, attempt 1 completed with a successful required aggregate.A complete three-page GraphQL check read reconciles all 253 advertised unique contexts at this exact commit: 205 successful and 48 skipped, with no pending or failed context. Of these, 248 check runs belong to that CI run and attempt: 201 successful and 47 skipped. The Ubuntu skill-installation consumer, read-only cleanup fixture, TODO consumers and all three queue slots succeeded. This is complete check evidence; a native Actions job/log read remains unavailable in this run, so it is not presented as an independent complete job-inventory receipt.
The earlier primary-quota failures and later HTTP 502 remain separate historical failures. This successful main run establishes recovery at that commit, but does not attribute the primary request pressure, prove that #593 reduced it, or meet recurrence acceptance. Keep #596 open with its acceptance criteria unchanged. Main has since advanced to
ffb11aec8160e060a89ddfdc54ee882d007d8d28; its checks are pending, so the earlier success is not transferred to that successor. No competing rerun, retry-default change, credential change or check exception was introduced.chore: agent-claim #596 nonce=81bc352a148f4e2c832950edfc62bc6e
devantler commentedon Oct 15, 2026
A native installer experiment rejected the proposed exact-path optimization. Using GitHub CLI 2.102.0, the repository's actual installer wrapper, a fixed public source commit (
github/awesome-copilot@8fbf6c4a798df51d1d1d8fd37a1aa7e94203109c) and disposable project installations, the short skill namegit-commitmade 8 HTTP requests. Bothskills/git-commitandskills/git-commit/SKILL.mdmade 10. All installed skill bytes matched. The observer retained only request methods and sanitized public paths, not headers or response bodies. No CI input change follows from a candidate that increased requests.This measures client requests for one pinned-source installation using the interactive account. It does not attribute the original workflow installation's hourly quota, establish a fleet-wide saving, or measure latency. The original Ubuntu failure used the same CLI version but failed during an unpinned default-branch lookup; its primary-quota failure remains distinct from the later HTTP 502 inventory failure. There was no catalogue rerun for this experiment.
Separate cleanup follow-up work is tracked in child #608. It is not a quota-reduction claim. Request attribution and recurrence acceptance remain outstanding, so #596 stays open.
devantler commentedon Oct 19, 2026
The same failure happened again on main. Main CI 39225588155, attempt 1 at
8f02981c80c8f96644d26907b17d81e1db5c2a56(08:50Z on 2026-10-19) failed 11 jobs, all for the same reason: the repository's API allowance ran out.test-setup-agent-skills-*,test-update-agent-skills-dry-run,test-ensure-gh-skill-script) could not look upgithub/awesome-copilot@8fbf6c4a…. GitHub answeredHTTP 403: API rate limit exceeded for installationon all three attempts.[Test] Delete Workflow Runsfixtures stopped withprimary rate-limit reset exceeds bound.complete list read failed.These checks were right to fail: none of them reported success on input it never read. The last green main was
8d2203af13(2026-10-18). Pull requests #648 and #649 fail on the same set of jobs, so the cause is the shared allowance, not those changes.Recovery: I started attempt 2 for the failed jobs only, about an hour after the failure, when the hourly allowance should have reset. This is the second recorded time main has gone red this way, after the 10-14 one above. This issue is still where the lasting fix belongs: keep normal validation within the allowance. Re-running failed jobs only recovers main each time; it does not stop the next failure.
devantler commentedon Oct 19, 2026
Independent readback of all 245 jobs in main run
39225588155, immutable attempt 1 at8f02981c80c8f96644d26907b17d81e1db5c2a56, confirms the primary-quota class in the cleanup fixture, pinned skill installation, reviewed CLI verification and updater dry run. The logs explicitly report installation API rate-limit exhaustion or a primary reset beyond the bounded envelope.One causal distinction remains: the TODO fixture reports HTTP
502from its read-only proxy for the comparison and fallback reads, then correctly refuses an incomplete scan. These logs do not expose the upstream response or establish that quota caused that proxy failure. Preserve this as a separate unconfirmed cause, as the issue's acceptance criteria require; temporal overlap alone does not prove the join.Another writer has already started attempt 2; it was in progress at collection. I did not dispatch another retry or change checks, credentials, retry defaults or quota controls. The durable request-budget repair and hosted recovery proof remain open on this issue.
devantler commentedon Oct 19, 2026
Main recovery is still unverified. Run 39225588155, attempt 2, at
8f02981c80c8f96644d26907b17d81e1db5c2a56reports completed / failure, with its latest run update at 2026-10-19 09:41:33 UTC.The initial inventory reads failed (cancelled stream / HTTP 502). A bounded read using 50 jobs per page then recovered all 245 unique jobs, matching the advertised count and binding every job to this run, attempt and head. Only CI - Required Checks failed; all other jobs succeeded or were skipped. The three named queue-slot jobs each succeeded at attempt 2.
The aggregate log identifies the remaining failure: three complete queue-evidence reads returned HTTP 502, ending with
Could not read complete queue evidence. The attempt-1 installation-quota failures recovered; they are not the cause shown by this remaining attempt-2 job. A fresh control using the workflow's original 100-job page size also failed with a cancelled stream. The successful smaller-page read establishes evidence availability, not a proven page-size root cause.No further retry was dispatched. Retrying only the aggregate would not reproduce the required same-attempt three-slot burst; the reviewed gate explicitly requires a full repeat. Keep main red until the workflow can read complete evidence and the full required gate passes. This issue's lasting request-budget acceptance also remains open.
chore: agent-claim #596 nonce=a1ec663c793924987eb5cdf67e7f8c02
chore: agent-claim #596 nonce=070a158473271d8d08ddded051098191
devantler commentedon Oct 19, 2026
Started one full recovery run for current main: CI 39225588155, attempt 3 at
8f02981c80c8f96644d26907b17d81e1db5c2a56. GitHub accepted it as queued at 2026-10-19 10:17:54 UTC, with the registered maintainer identity as the triggering actor. Completion is pending; main is not yet cleared.Before dispatch, the original 100-job page read again returned a partial inventory followed by a server-error response. A complete 50-job-page read recovered all 245 unique jobs from immutable attempt 2, bound to its run, attempt and head. Its sole failed job is the required aggregate; all three queue-slot jobs succeeded. This independently confirms evidence availability at the smaller page size, without establishing a page-size root cause or a quota-pressure reduction.
The reviewed aggregate requires the three-slot burst to execute at the same run attempt. Accordingly, this recovery repeats all jobs once; it does not retry only the aggregate, weaken completeness checks, or change retry defaults. No further retry or watcher is armed. The next invocation should collect attempt 3 once, rebind current main and its complete required gate, and retain this issue's request-attribution and durable-budget acceptance until separately proved.
chore: agent-claim #596 nonce=2da84144ec8ce276009e451cf65c5283
devantler commentedon Oct 19, 2026
The pending recovery succeeded. CI run 39225588155, attempt 3 completed successfully at 10:25:44 UTC on 2026-10-19, at unchanged commit
8f02981c80c8f96644d26907b17d81e1db5c2a56.I independently retrieved all 245 unique jobs, matched the advertised count, and bound every job to this run, attempt and commit: 194 succeeded, 51 were skipped, none failed or remained pending. All three queue-slot jobs succeeded at attempt 3. The required aggregate also succeeded; its executed log confirms the complete same-attempt queue-evidence check finished successfully. This was the existing gate, including its original 100-job page size. Recovery therefore does not establish a page-size root cause or a reduction in API pressure.
This clears the previously pending recovery observation for that commit. It does not complete this issue's request-attribution, durable-budget repair or before/after measurement criteria, so #596 stays open.
Separately, #653 has since merged as
001dc6d9af20bb33c0980b50eb98762c72636917. Its new-main CI run39239227605was pending when collected. The successful earlier attempt is not validation of that newer revision. No additional retry or watcher was started here.devantler commentedon Oct 19, 2026
Correction to the 09:35Z recurrence note: the second attempt of main run 39225588155 passed every test job and failed only in the required-checks summary, with HTTP 502 on the run-jobs read. That read asks for 100 jobs per page, and on this 245-job run GitHub answered page 1 with 502 on every try, while 50 and 30 per page always succeeded. So this morning's attempt-2 red was a page-size failure, not installation quota. #653 (merged 10:41Z) reads 50 per page in both the summary and the Go cache check. The quota failures on attempt 1 remain this issue's scope.
chore: agent-claim #596 nonce=0f550907d02b8286807f046e3d4be29d
devantler commentedon Oct 19, 2026
Independent acceptance of merged #653: main CI run 39239227605, attempt 1 at
001dc6d9af20bb33c0980b50eb98762c72636917, completed successfully with 194 successful jobs and 51 skipped jobs.I collected all 245 unique jobs and verified their run, attempt and head identities. All three queue-slot tests passed, and CI - Required Checks passed. Its actual log shows the 50-jobs-per-page read and successful complete queue-evidence validation. The skill installer, package-cleanup fixture and native Go-cache policy jobs also passed.
This verifies the repaired job-page read on merged main. It does not establish the source of primary-quota pressure or a measured request-budget reduction; those acceptance criteria keep #596 open. The source change and merge belong to #653's delivering writer. This run dispatched no retry.