Three Repository resources never retry after an async update failure, so a fixed root cause cannot reach them

🤖 Generated by the Agentic Engineer

Evidence

#112 restored the repository write path (#136, merged 2026-08-06T01:37:50Z, artifact 1.22.1). Every resource that attempted an update afterwards succeeded:

Repository	`LastAsyncOperation` after the fix
`aws`	Success 2026-08-06T01:39:11Z
`doggy-countdown`	Success 01:39:09Z
`kyverno-policies`	Success 01:39:11Z
`provider-upjet-unifi`	Success 01:39:10Z

Three did not, and still report ReconcileError with the old 422 message:

Repository	last update attempt	age at 2026-08-06
`agent-skills`	2026-07-20T00:32:52Z	17 days
`platform-tenant-template`	2026-08-02T18:54:05Z	3 days
`agent-plugins`	2026-08-03T19:45:36Z	2 days

Problem

The Synced message on these three is the cached result of that old attempt, not a fresh rejection — the underlying cause it names was fixed and demonstrably no longer rejects anyone else. They are not being rejected; they are not trying.

That matters beyond these three repositories: it means a resource can enter a state where fixing the root cause is not enough to recover it, and the stale condition message actively misleads whoever looks — it points at a cause that is already resolved. Any future repository-settings failure can strand a resource the same way.

⚠️ Synced.lastTransitionTime is rewritten roughly every 60s whether or not anything happened, so it cannot be used to tell a fresh failure from a stale one. Use LastAsyncOperation.lastTransitionTime, which only moves when an operation actually completes.

Observation worth checking first

The three carry only the crossplane.io/external-name annotation. Healed resources such as kyverno-policies also carry crossplane.io/external-create-pending, crossplane.io/external-create-succeeded and crossplane.io/external-create-failed. Whether that is the cause or a side effect of the same history is unverified — it is the first thing to test, not a conclusion.

Suggested approach

Establish why no reconcile issues an update: a stuck async-operation record, an exponential backoff that has grown unbounded, or the resource being considered up to date while the condition says otherwise.
Prefer a fix that recovers automatically. A manual nudge (annotation bump, provider restart) may confirm the diagnosis, but a resource that cannot self-recover from a transient provider error is the actual defect.
Establish whether a bounded retry or a staleness check on LastAsyncOperation belongs in the repository-update contract, so a wedged resource is visible rather than silently frozen behind a stale message.

Acceptance criteria

All 20 Repository resources report ReconcileSuccess.
The mechanism that stopped the three from retrying is identified and recorded here.
A wedged resource is either recovered automatically or surfaced as a distinct signal, rather than presenting a stale message naming an already-fixed cause.

Blocks #112, which stays open until 20/20 is observed.

Evidence gathered live against admin@prod; no cluster mutation was performed, deliberately, so the stuck state remains available for diagnosis.

🔗 Related PRs

#88 - feat: archived-repo pattern + reusable-workflows archival groundwork [merged]
#90 - feat: archive the reusable-workflows repository (phase 2) [merged]
#125 - fix(repositories): pin hasDownloads so reconciliation stops 422-ing [merged]
#127 - fix(repositories): keep org-enforced signoff out of update payloads [merged]
#128 - fix(repositories): stop declaring org-enforced signoff so updates can apply [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

Independent verification from a separate run, plus two hypotheses I tested and refuted — recording both so the next run doesn't re-derive them.

Confirms the diagnosis. For agent-plugins, declared forProvider and live GitHub state are identical on every comparable field (allowAutoMerge, allowMergeCommit, allowRebaseMerge, allowSquashMerge, allowUpdateBranch, deleteBranchOnMerge, hasIssues, hasProjects, hasWiki, visibility, webCommitSignoffRequired). There is nothing left to PATCH, so the surfaced 422 cannot be describing a write that is still being attempted — it is a stored result from before #136.

Timing. #136 merged 2026-08-06T01:37:50Z. The three LastAsyncOperation transitions are frozen at 2026-07-20T00:32:52Z, 2026-08-02T18:54:05Z, 2026-08-03T19:45:36Z — all pre-fix — while Synced=ReconcileError is rewritten every ~60s (observed 03:22:08Z against a 03:22:17Z clock). The pairing of a live-refreshed Synced with a frozen LastAsyncOperation is the signature of this stuck state, and it is why Synced.lastTransitionTime must not be used to judge staleness here.

Refuted — archived presence. All three failing resources omit forProvider.archived, which looks like the same absent-optional-bool shape as the signoff bug. It is not the discriminator: aws, dotnet-template, go-template, homebrew-tap, kyverno-policies, platform-template, provider-upjet-unifi, unifi and world-at-ruin also omit it and all reconcile Success. A full field-presence comparison found no key present in every working resource and missing from every failing one.

Refuted — provider logs as evidence. provider-upjet-github shows zero occurrences of cannot be disabled in its entire retained log and only 2 lines in 20 minutes, so it is not logging reconcile activity at this verbosity. A zero here is not evidence of "no retry" in either direction — I nearly drew that inference before running the control.

Net: nothing in this run contradicts the issue as written; the remaining action is still to force one fresh reconcile per resource. I hold read-only cluster access, so I could not apply or verify that step.

🤖 Generated by the Agentic Engineer

Two measurements, one of which refutes this issue's own lead

Re-measured the fleet live. Still 17/20 synced; the same three are stuck.

1. The annotation lead in this issue is REFUTED

This issue recorded that the three stuck repos carry only crossplane.io/external-name while the
healed ones also carry crossplane.io/external-create-{pending,succeeded,failed}, and proposed that
as the discriminator.

That is not the discriminator. aws is healed and carries only crossplane.io/external-name —
the identical annotation set to all three stuck repos:

repo	annotations	LastAsyncOperation	Synced
agent-plugins	`external-name` only	`AsyncUpdateFailure` @ 2026-08-03	ReconcileError
agent-skills	`external-name` only	`AsyncUpdateFailure` @ 2026-07-20	ReconcileError
platform-tenant-template	`external-name` only	`AsyncUpdateFailure` @ 2026-08-02	ReconcileError
aws (healed control)	`external-name` only	`Success` @ 2026-08-06 01:39	ReconcileSuccess
kyverno-policies (healed)	all four	`Success` @ 2026-08-06 01:39	ReconcileSuccess

Please don't spend a run on that path.

2. There is no remaining desired-state work — the reported error is stale

Field-by-field comparison of spec.forProvider against status.atProvider returns zero
differing fields for all 20 repositories, stuck and healed alike. Every repo has
spec=true, atProvider=true for webCommitSignoffRequired. So .github#112's fix is complete and
correct; nothing about the three repos' desired state is still wrong.

The resource spec shape is also effectively identical between stuck and healed (same
managementPolicies: [Observe, Create, Update], same providerConfigRef, same finalizer, same
forProvider key set bar an unrelated topics).

Working hypothesis — a self-sustaining deadlock, not a retry failure. The stored async result is
only replaced by running a new async operation; upjet only launches one when there is a diff; there
is no diff. So the failure from days ago can never be cleared by reconciliation, which matches
Synced being rewritten every ~60s with a message quoting a PATCH that is no longer being attempted.

⚠️ This reframes the issue: LastAsyncOperation.lastTransitionTime being frozen is not by itself
proof that nothing is retrying — Kubernetes conditions only re-stamp that field when the reason
changes, so continuous identical failures look frozen too. The zero-diff measurement is what
distinguishes the two, not the timestamp.

Not yet established

I could not corroborate from provider-upjet-github logs: the pod emitted 0 lines in a 20-minute
window, and my healed-repo control also returned 0, so the empty result says nothing either way.
Whoever picks this up should raise the provider log level rather than read that silence as evidence.

I deliberately did not force a reconcile — cluster mutation is outside my grant, and it would destroy
the very evidence that distinguishes these three.

🤖 Generated by the Agentic Engineer

Closing: all three stranded repositories have recovered

Live read from prod, 2026-09-19 (read-only):

Repository	`Synced`	`LastAsyncOperation`
`agent-plugins`	True / ReconcileSuccess	Success 2026-08-16T18:09:25Z
`agent-skills`	True / ReconcileSuccess	Success 2026-08-16T18:09:25Z
`platform-tenant-template`	True / ReconcileSuccess (since at least 2026-09-06)	AsyncUpdateFailure 2026-08-16T00:53:34Z, the old signoff 422

The rest of the fleet is also Synced=True (20/20). So no resource is stranded and none reports a stale error through Synced, which was this issue's problem.

One residue, recorded rather than chased: platform-tenant-template still stores the old 422 in LastAsyncOperation. That matches the "the stored async result is only replaced by a new async operation, and none runs without a diff" mechanism described above. Since Synced and Ready are both clean, it misleads only someone reading that one condition, so it isn't worth a forced update. Any change to that repository's declared settings will overwrite it.

If a Repository shows Synced=False with a cause that has already been fixed, reopen this with that resource's conditions.

Navigation Menu

Repository navigation

Three Repository resources never retry after an async update failure, so a fixed root cause cannot reach them #137

Evidence

Problem

Observation worth checking first

Suggested approach

Acceptance criteria

Two measurements, one of which refutes this issue's own lead

1. The annotation lead in this issue is REFUTED

2. There is no remaining desired-state work — the reported error is stale

Not yet established

Closing: all three stranded repositories have recovered

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Three Repository resources never retry after an async update failure, so a fixed root cause cannot reach them #137

Description

Evidence

Problem

Observation worth checking first

Suggested approach

Acceptance criteria

Activity

coderabbitai commented on Aug 6, 2026

devantler commented on Aug 6, 2026

devantler commented on Aug 6, 2026

Two measurements, one of which refutes this issue's own lead

1. The annotation lead in this issue is REFUTED

2. There is no remaining desired-state work — the reported error is stale

Not yet established

devantler commented on Sep 19, 2026

Closing: all three stranded repositories have recovered

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions