fix: declarative repo-settings updates are write-blocked (422 on every update PATCH)

🤖 Generated by the Daily AI Engineer

Summary

No managed repository setting can currently be changed declaratively. Every update PATCH to GitHub fails, so config merged into this repo never reaches the actual repos. This has been true the whole time but was invisible, because until 2026-07-18 nothing had ever changed a managed field — settings were only ever imported.

Evidence

#110 declared topics on agent-skills / agent-plugins. Both MRs went Synced=False ~2 minutes after merge and the topics never applied:

update failed: async update failed: failed to update the resource:
PATCH https://api.github.com/repos/devantler-tech/agent-skills: 422
Commit signoff is enforced by the organization and cannot be disabled

Every other repo MR is Synced=True — not because they work, but because they have no pending diff, so no PATCH is attempted.

Why this is the first sighting: every repo adopted its settings via LateInitialize, so Crossplane had only ever observed repo settings. ksail's 20 topics were imported, never written. #110 is the first commit in this repo's history to change a managed field on an already-adopted repo — i.e. the first real exercise of the update path.

Why the obvious fix doesn't work

The shared patch pins webCommitSignoffRequired: true, with a comment reasoning the provider "must always send true" to avoid this exact 422. That reasoning was never validated, and live state disproves it — spec, repo, all siblings, and the org are already true, and the PATCH is still rejected. The field is org-controlled, so GitHub refuses a repo-level PATCH carrying it at all; the value is irrelevant.

I attempted the naive fix in #111 (drop it from the patch). Codex correctly flagged it as a no-op (P1), confirmed live:

status.atProvider.webCommitSignoffRequired = true
spec.managementPolicies = Observe,Create,Update,LateInitialize

The field is observed and LateInitialize is on, so removing it from the patch just lets Crossplane write it back from the observed value. #111 is closed.

The real scope

For agent-skills, git declares 3 fields (name, description, topics), the shared patch adds 7, and the live spec has 21 — so 11 fields exist in spec purely via LateInitialize. Any org-enforced-but-repo-observable field will be absorbed into spec and then sent on update.

Candidate approaches (needs a deliberate decision, not a guess)

Drop LateInitialize from managementPolicies. Matches "declare what you manage" and stops absorbing drift. Caveat: does not retroactively strip the 11 already-written fields — those would need removing from the live specs too, and Flux/SSA will not prune fields owned by the Crossplane field manager.
Provider-side field exclusion, if provider-upjet-github supports skipping a field on late-init/update. Needs verifying against the provider's actual capabilities.
Upstream fix in provider-upjet-github / terraform-provider-github so the update payload omits org-controlled fields.

Acceptance criteria

agent-skills and agent-plugins return to Synced=True.
Their declared topics and descriptions from #110 appear on GitHub.
A subsequent settings change on some other repo also applies — proving the update path generally, not just these two.
The kustomization comment is corrected; the current one states a mitigation that does not hold.

Blocks devantler-tech/monorepo#2229.

🔗 Related PRs

#78 - feat(repositories): declare provider-upjet-unifi (net-new Library repo) [merged]
#82 - feat: declare the devantler-tech/aws config-tenant repo [merged]
#86 - fix: gate repository-permissions/ until provider-upjet-github supports shaPinningRequired [merged]
#88 - feat: archived-repo pattern + reusable-workflows archival groundwork [merged]
#103 - feat(teams): activate declarative Admins policy [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

fix(repositories): keep org-enforced signoff out of update payloads (#…

🤖 Generated by the Agentic Engineer

Reopening: #127 merged and deployed, and the defect is not fixed. Verified against the live cluster.

The change did what it said at the manifest layer — spec.forProvider.webCommitSignoffRequired is now
gone from every repository, and LateInitialize is off — but the PATCH still carries the field:

agent-plugins   spec.forProvider.webCommitSignoffRequired  ABSENT
                spec.initProvider.webCommitSignoffRequired true
                Synced=False @11:46:34Z
                422 Commit signoff is enforced by the organization and cannot be disabled

That reconcile is after the deploy (CD published v1.21.3 at 11:44) and against the updated spec,
so it is not a stale in-flight attempt. agent-plugins and agent-skills still fail, and their topics
are still empty on GitHub.

Two candidate causes, not yet distinguished:

initProvider is reaching the update payload. Moving the field there assumes upjet uses it only at
creation; if it is merged into the plan on every reconcile, the fix relocated the field without
removing it from the PATCH.
The Terraform provider sends the full resource on update, built from its own state rather than from
forProvider, so dropping a field from the spec never removes it from the request.

Distinguishing them is the next step: if (1), removing the initProvider seed should clear the 422 and
the cost is that a newly created repository no longer requests signoff at creation — which the org
enforces anyway, so it may be an acceptable trade. If (2), no manifest-level change can fix this and it
needs a provider-level answer.

What #127 did achieve, and is worth keeping: LateInitialize no longer copies live-only values into
forProvider, so the class of unresolvable diffs stops growing; ascoachingogvaner and wedding-app
are parked read-only and now reconcile cleanly without their declared-private setting being enacted
against two live public sites; and the update contract is pinned by a CI test.

Fleet state: 18/20 Synced=True, up from 16 — but two of that gain are the parked repos going
quiet, not the write path working. The write path is still broken for every repository that has
something to apply.

🤖 Generated by the Agentic Engineer

Root-caused against the live cluster. #136 is the fix.

Why the two previous attempts failed. Both were built on the premise that GitHub rejects the update whenever the field appears, so #127 moved it to initProvider and #128 removed it entirely. The premise is wrong in a specific way: forProvider is not the update payload, it is the Terraform configuration. Terraform sends a field only when it differs from observed state. So the causality is inverted — leaving the field unset is what puts it in the payload:

`forProvider`	TF config	vs live `true`	payload	result
`webCommitSignoffRequired: true`	`true`	no diff	field omitted	update applies
unset	zero value `false`	permanent diff	`web_commit_signoff_required: false`	422

The error text agrees: "cannot be disabled" — it is objecting to a false, not to the field's presence.

Cluster evidence, at two-minute resolution. LastAsyncOperation records completed async updates:

2026-07-27T04:33Z — fix(repositories): pin hasDownloads so reconciliation stops 422-ing #125 merges; the shared patch still declares the field.
2026-07-27T04:34:5x–04:35:03Z — nine write-enabled repositories record LastAsyncOperation=Success. Real update PATCHes, succeeding, with the field declared.
2026-07-27T14:42Z — fix(repositories): stop declaring org-enforced signoff so updates can apply #128 removes the declaration.
2026-07-27T14:44:4xZ — the same repositories begin recording AsyncUpdateFailure carrying the 422.

Current state (2026-08-06). 7 of 20 resources failing. The correlation across all 20 is exact:

declared true + write-enabled → Synced=True, 10/10
unset + write-enabled → ReconcileError 422, 7/7
unset + Observe-only (world-at-ruin) → fine, because it never PATCHes — which isolates the write path as the co-factor

Six of the seven failing resources differ from a working one by exactly this one field.

The part that matters beyond the 422. The ten that still work do so only because the provider had late-initialized the value into their spec before the declaration was removed — field manager provider, not Flux — so Flux's removal could not reach it. The write path therefore depends on undeclared residue rather than on anything in this repository, and a newly declared repository gets none of that residue and is broken from creation. That is exactly how this surfaced with doggy-countdown, and it is why declaring the value is the durable fix rather than a workaround.

platform-tenant-template is the one active resource holding no adopted values at all, so it also has no observed visibility to fall back on once updates resume; #136 pins its current value explicitly so restoring the write path cannot move it.

Closing this on merge alone would repeat the two earlier reopenings, so #136 verifies against the live sync state afterwards instead.

🤖 Generated by the Agentic Engineer

Reopened. #136 merged at 01:37:50Z (eaaa64b66d) and this issue auto-closed two seconds later — the Fixes #112 keyword was in the commit message, which a PR-body edit does not remove. That was my error: the PR says in two places that it should close on a live read rather than on merge, because closing without one is exactly what caused the two previous reopenings.

So it stays open until the fix is observed working.

What has to be true to close it:

The release tag publishes a new OCI artifact and the platform picks it up — the github-config OCIRepository revision advances past 1.22.0.
The seven repositories that were failing (agent-plugins, agent-skills, aws, doggy-countdown, kyverno-policies, platform-tenant-template, provider-upjet-unifi) report Synced=True / ReconcileSuccess instead of the 422.

kubectl --context admin@prod get repository.repo.github.m.upbound.io -A -o json \
  | jq -r ".items[]|\"\(.metadata.name) \((.status.conditions[]?|select(.type==\"Synced\")|.reason))\""

Filter on reason, not lastTransitionTime — that field is rewritten roughly every 60s, so it cannot distinguish fresh from stale here.

Expected end state: 20/20 ReconcileSuccess, with platform-tenant-template still visibility: public (it is the one resource holding no adopted values, so #136 pinned that explicitly).

🤖 Generated by the Agentic Engineer

Post-merge verification — the fix works; three resources need a separate nudge

Rollout completed: github-config OCIRepository advanced 1.22.0 → 1.22.1, Flux applied it (Ready=True), and all 20 resources now declare the field (signoff-declared=20/20).

Behavioural proof. Every resource that actually attempted an update after the change succeeded:

Repository	`LastAsyncOperation` before	after
`aws`	AsyncUpdateFailure 07-27	Success 2026-08-06T01:39:11Z
`doggy-countdown`	AsyncUpdateFailure 07-27	Success 01:39:09Z
`kyverno-policies`	AsyncUpdateFailure 07-27	Success 01:39:11Z
`provider-upjet-unifi`	AsyncUpdateFailure 08-03	Success 01:39:10Z

Those are real update PATCHes completing for the first time in over a week — the write path is restored. Sync state went 13/20 → 17/20, and none of the thirteen previously-healthy resources regressed, which is what the render-delta check predicted (their rendered spec was byte-identical; only field ownership moved).

What is NOT yet resolved. Three resources — agent-plugins, agent-skills, platform-tenant-template — still report ReconcileError carrying the old 422 text. But they have not attempted a new update at all: their LastAsyncOperation timestamps are unchanged at 2026-08-03T19:45:36Z, 2026-07-20T00:32:52Z and 2026-08-02T18:54:05Z respectively. The Synced message is the cached result of that last attempt, not a fresh failure — note Synced.lastTransitionTime is rewritten roughly every 60s regardless, so it cannot be read as evidence of a retry.

So this is a different condition from the one this issue tracks: those three are not retrying, rather than retrying and being rejected. agent-skills has not attempted an update since 2026-07-20 — 17 days — which is well beyond any ordinary backoff. One observable difference: the three carry only crossplane.io/external-name, while healed resources such as kyverno-policies also carry the crossplane.io/external-create-{pending,succeeded,failed} annotations.

Leaving this issue open until those three either retry on their own or the cause is understood — closing now would repeat the pattern that had it reopened twice. I have not forced a reconcile: that is a live-cluster mutation outside what I apply directly, and it would also destroy the evidence of why they are stuck.

🤖 Generated by the Agentic Engineer

Still reproducing 14 hours after #136 merged

#136 merged 2026-08-06T01:37:50Z. Measured against admin@prod at ~16:3xZ today, three
repositories are still failing with the original error:

update failed: async update failed: failed to update the resource:
PATCH https://api.github.com/repos/devantler-tech/<repo>:
422 Commit signoff is enforced by the organization and cannot be disabled

agent-plugins, agent-skills, platform-tenant-template.

The fix is deployed — this is not GitOps lag. All three live specs carry
spec.forProvider.webCommitSignoffRequired: true, and declared matches live everywhere I sampled
(web_commit_signoff_required is true on GitHub, and status.atProvider reflects true, for both
failing and succeeding repositories alike). So the signoff field is not what distinguishes them.

What actually distinguishes the 17 that pass from the 3 that fail

A pending diff on some other field:

Repo	declared topics	live topics (GitHub API)	Synced
`ksail`	20	20	ReconcileSuccess
`agent-skills`	11	0	ReconcileError
`agent-plugins`	10	0	ReconcileError

agent-skills and agent-plugins declare topics that were never applied, so Crossplane has a real
change to push, issues an update PATCH — and that PATCH carries the signoff field, which GitHub
rejects. The topics can therefore never be applied, and the failure is a permanent loop.
(platform-tenant-template declares no topics and status.atProvider.topics is absent for all three;
its pending field is something else and is worth identifying separately.)

The 17 "successes" are not evidence the write path works — they have nothing to push. A
repository with no pending diff issues no PATCH, so nothing can 422.

Consequence for the diagnosis

This matches the earlier API-level measurement on this issue: PATCHing
web_commit_signoff_required at its own current value of true still returns 422. The error names
disabling, but the trigger is the field's presence in the payload, not its value.

Declaring the value does not remove it from the payload of an update that has to happen for some
other reason. That is why the fix appeared to work — every repository that had nothing to update
stopped erroring, because it stopped issuing PATCHes at all.

So #112 is not resolved, and I do not think it can be resolved by choosing where to declare the
field. What is needed is for the field to be excluded from update payloads entirely, which is a
provider-level concern (an ignore/lifecycle exclusion) rather than a matter of declaring it in
forProvider vs initProvider — the two placements #127 and #136 tried between them.

Falsifiable: if this is right, applying the pending topics to agent-skills out-of-band should
make its error stop without any config change, and re-introducing any pending diff on a currently-green
repository should make it start. I have not performed either — the first mutates a live repository and
the second deliberately breaks a working one, so both are the maintainer's call rather than mine.

I have not reopened or re-labelled anything; this is evidence for whoever picks it up next.

Navigation Menu

Repository navigation

fix: declarative repo-settings updates are write-blocked (422 on every update PATCH) #112

Summary

Evidence

Why the obvious fix doesn't work

The real scope

Candidate approaches (needs a deliberate decision, not a guess)

Acceptance criteria

16 remaining items

Post-merge verification — the fix works; three resources need a separate nudge

Still reproducing 14 hours after #136 merged

What actually distinguishes the 17 that pass from the 3 that fail

Consequence for the diagnosis

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

fix: declarative repo-settings updates are write-blocked (422 on every update PATCH) #112

Description

Summary

Evidence

Why the obvious fix doesn't work

The real scope

Candidate approaches (needs a deliberate decision, not a guess)

Acceptance criteria

Activity

coderabbitai commented on Jul 18, 2026

devantler commented on Jul 27, 2026

16 remaining items

devantler commented on Aug 6, 2026

devantler commented on Aug 6, 2026

devantler commented on Aug 6, 2026

Post-merge verification — the fix works; three resources need a separate nudge

devantler commented on Aug 6, 2026

Still reproducing 14 hours after #136 merged

What actually distinguishes the 17 that pass from the 3 that fail

Consequence for the diagnosis

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions