Repository reconciliation is wedged on three resources, so everything they declare is advisory

🤖 Generated by the Agentic Engineer

Blocker: crossplane-contrib/provider-upjet-github release newer than v0.19.1 | upstream | last-verified 2026-09-22: not shipped (GHCR tags end at v0.19.1)

Evidence

Measured on prod, 2026-08-07. Three of twenty Repository managed resources report Synced=ReconcileError, each with the same message:

update failed: async update failed: failed to update the resource: PATCH …: 422 Commit signoff is enforced by the organization and cannot be disabled

agent-plugins, agent-skills, platform-tenant-template. The other 17 are ReconcileSuccess — because a resource with no pending diff issues no update PATCH at all, so it never reaches this error. A healthy reading here means "nothing to write", not "writes succeed".

kustomization.yaml already patches webCommitSignoffRequired: true onto every Repository specifically to avoid this, reasoning that the error "names disabling, not presence". Live data contradicts that: agent-skills declares true, the repository reports true, and the PATCH is still rejected.

Problem

While the update path is wedged, every field those resources declare is advisory — GitOps reports the intent and the cluster cannot apply it. The visible instance today is discovery metadata: agent-skills declares 11 topics and agent-plugins 10, and both repositories have zero live. Repositories that reconcile cleanly match exactly (ksail 20/20, platform 1/1, kyverno-policies 0/0).

platform-tenant-template shows a second shape of the same failure: every field it declares already matches live, and its LastAsyncOperation failure is dated 2026-08-02 while Synced is rewritten each reconcile. With no diff left to drive a new update, nothing can clear the cached error — it reports a failure it cannot retry out of.

Suggested approach

Settle the mechanism first. A no-op PATCH /repos/{owner}/{repo} with, and then without, web_commit_signoff_required distinguishes presence is rejected from a false reaches the API from provider state. The two have different fixes, so this comes before any change.
If presence is rejected, the field cannot be declared at all while the organization enforces it — drop it from the shared patch and record why, so the next reader does not re-add it.
If a false is leaking from provider state, correct the source of that value rather than the declaration.
Re-observe the three resources, and confirm the declared topics actually land.

Acceptance criteria

The mechanism is established by test and recorded here.
All 20 Repository resources report Synced=ReconcileSuccess.
Declared topics are live on agent-skills and agent-plugins.
platform-tenant-template's stale async failure is cleared, and the way a no-diff resource can hold an unclearable error is addressed rather than worked around once.
A signal exists for Synced != ReconcileSuccess on any Repository, so a wedged resource is noticed when it wedges rather than when someone spots a missing field.

Surfaced while closing acceptance criterion 3 of #123.

🔗 Related PRs

#88 - feat: archived-repo pattern + reusable-workflows archival groundwork [merged]
#125 - fix(repositories): pin hasDownloads so reconciliation stops 422-ing [merged]
#127 - fix(repositories): keep org-enforced signoff out of update payloads [merged]
#128 - fix(repositories): stop declaring org-enforced signoff so updates can apply [merged]
#136 - fix(repositories): declare org-enforced signoff so updates can apply [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

This and #137 look like the same issue but split the three resources two ways

Both were filed against the same symptom — three Repository resources, one 422 — and they read as
duplicates. They are not, and the drift check in #140 supplies the datum that separates them:

Repository	pending diff	which failure this is
`agent-skills`	yes — 11 declared topics, live `[]`	#139: the PATCH is genuinely attempted and genuinely rejected
`agent-plugins`	yes — 10 declared topics, live `[]`	#139: same
`platform-tenant-template`	no — every declared field matches live	#137: a cached failure with nothing left to retry it out of

So the two issues describe different causes that happen to share a symptom, and each needs its own
fix:

For the two with a real diff, declaring webCommitSignoffRequired: true demonstrably does not stop
the 422 — agent-skills declares it, live reports it, and the PATCH is still refused. That is
Repository reconciliation is wedged on three resources, so everything they declare is advisory #139's finding and it stands.
For platform-tenant-template, there is nothing to write, so no attempt is made and the stale
ReconcileError from 2026-08-02 simply persists. That is Three Repository resources never retry after an async update failure, so a fixed root cause cannot reach them #137's finding and it also stands.

Closing either as a duplicate of the other would lose one of the two causes. Cross-linking instead;
both are now children of #56.

The still-undecided question in #123 applies to the first group only: whether GitHub rejects the
field's presence in the PATCH or a false value reaching it from provider state. The
deciding test is a no-op PATCH /repos/{owner}/{repo} with and without
web_commit_signoff_required — a write outside GitOps, so it needs the maintainer or a deliberate
decision, not a drive-by.

🤖 Generated by the Agentic Engineer

Re-measured 2026-08-11: still 3/20, and the sequencing against #137 is now decided

Same three resources, same 422, four days on — agent-plugins, agent-skills,
platform-tenant-template — with the other 17 ReconcileSuccess. I reached this independently
before finding this issue, and every observation here reproduces what it already records, including
that the declared topics are still zero live on both skills repositories.

One thing does change, and it changes what to do first. LastAsyncOperation has not moved since
#137 recorded it on 2026-08-06:

Repository	`LastAsyncOperation` (2026-08-06, #137)	(2026-08-11)	age now
`agent-skills`	2026-07-20T00:32:52Z	2026-07-20T00:32:52Z	22 days
`platform-tenant-template`	2026-08-02T18:54:05Z	2026-08-02T18:54:05Z	9 days
`agent-plugins`	2026-08-03T19:45:36Z	2026-08-03T19:45:36Z	8 days

Contrast kyverno-policies, which healed: True/Success at 2026-08-06T01:39:11Z.

So the premise this issue reads from is not what the data supports. The text says the PATCH "is
still rejected", and treats agent-skills declaring true while being rejected as live evidence
that presence itself is refused. That inference needs a current rejection, and there is none: the
Synced message on agent-skills is the cached result of an attempt made on 2026-07-20, before
#136 restored the write path. Nothing has called the API for that resource since. Synced is
rewritten every ~60s regardless, which is exactly the trap #137 flagged, and it is what makes the
stale message read as fresh.

That does not weaken the case for step 1 — settle the mechanism by an explicit no-op PATCH with
and then without the field. It strengthens it, because that test is now the only way to learn the
answer: these three resources cannot supply it, since they are not issuing the request. Reading their
condition as an answer would be reasoning from a 22-day-old cache.

Suggested sequencing: #137 (they are not retrying) is the blocker, not a parallel finding. Even a
correct fix to the shared patch cannot reach a resource that never attempts an update — which is
#137's whole point, and it has now held across five more days and a merged root-cause fix. Worth
making that a real blocked-by link rather than two issues describing one wedge from opposite ends.

Also worth noting for whoever picks this up: platform-tenant-template is not a third instance of
the same shape. It has no pending diff at all — every declared field matches live — so unlike the
other two there is nothing left that could drive a new update even after the mechanism is fixed. It
needs the recovery path in #137 specifically, not a corrected declaration.

Detection remains unbuilt: devantler-tech/platform#2820 is the alert for exactly this condition, and
this is now its fourth measured occurrence. Its acceptance criterion — key on Synced, never on
Ready — is confirmed again here: all three wedged resources report Ready=True/Available.

🤖 Generated by the Agentic Engineer

Re-measured on prod 2026-09-22: the wedge described above is gone, and one fenced resource remains

All 20 Repository resources report Synced=True/ReconcileSuccess. agent-skills and agent-plugins completed a successful update on 2026-08-16, after fix(github): avoid signoff-blocked repository updates #158 and fix(github): stop signoff-blocked update retries #159. So they are no longer advisory.
14 resources run with Update. Their updates succeed because none of them has a diff that reaches the rejected field.
platform-tenant-template is deliberately Observe/Create only (fix(github): stop signoff-blocked update retries #159). Its last update attempt, on 2026-08-16, still failed with the same 422 Commit signoff is enforced by the organization. It now sends no updates, so that failure is cached, not retried.
The topic diffs quoted in the body no longer exist: fix(github): avoid signoff-blocked repository updates #158 removed the declared topics.

What is left is the root cause #158 named. The deployed GitHub provider (provider-upjet-github v0.19.1) sends the signoff field on every repository update, and GitHub rejects it while the organization enforces signoff. The fix is a provider release built on a newer Terraform GitHub provider. The registry has no release after v0.19.1, so this issue is now blocked upstream. Once a newer release exists: bump the platform's provider pin, restore Update on platform-tenant-template, and confirm its next update succeeds.

chore: agent-claim #139 nonce=5882de2c2a0ac91c8f83b7e0316d1233

Navigation Menu

Repository navigation

Repository reconciliation is wedged on three resources, so everything they declare is advisory #139

Evidence

Problem

Suggested approach

Acceptance criteria

This and #137 look like the same issue but split the three resources two ways

Re-measured 2026-08-11: still 3/20, and the sequencing against #137 is now decided

Re-measured on prod 2026-09-22: the wedge described above is gone, and one fenced resource remains

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Repository reconciliation is wedged on three resources, so everything they declare is advisory #139

Description

Evidence

Problem

Suggested approach

Acceptance criteria

Activity

coderabbitai commented on Aug 7, 2026

devantler commented on Aug 7, 2026

This and #137 look like the same issue but split the three resources two ways

devantler commented on Aug 11, 2026

Re-measured 2026-08-11: still 3/20, and the sequencing against #137 is now decided

devantler commented on Sep 22, 2026

Re-measured on prod 2026-09-22: the wedge described above is gone, and one fenced resource remains

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions