Repository reconciliation is fully broken: every PATCH 422s on org-enforced commit signoff

🤖 Generated by the Agentic Engineer

Evidence

Measured 2026-07-26 while adding a new repository to deploy/repositories/:

Repository	Declared in `deploy/repositories/`	Live
`ascoachingogvaner`	`visibility: private`	public (`private=false`)
`wedding-app`	`visibility: private`	public (`private=false`)

Both Repository resources carry managementPolicies: [Observe, Create, Update, LateInitialize] and declare visibility explicitly in forProvider — the config comments state this is deliberate, so that "management can never expose it". They are also grouped under a # Private repos. heading in deploy/repositories/kustomization.yaml.

Problem

Declared intent and live state disagree on a field whose whole stated purpose is to prevent accidental exposure, and Update is in the management policy — so this should not be able to persist. Exactly one of these is true, and both are worth knowing:

Reconciliation is not applying visibility. Then the guarantee the comment describes does not exist: the config would not have prevented exposure, and it would not correct it either. Any other field these resources pin is equally suspect.
The repos were made public deliberately and the config was never updated. Then the declarative source of truth is stale and actively misleading — it says private, a reader believes private, and the next person to reconcile from it may flip two public sites private without meaning to.

The second is the more likely reading (the portfolio map was separately found stale on the same two repos), but it should be confirmed rather than assumed, because the first would be a much larger problem.

Suggested approach

Read the two Repository managed resources' status (Synced / Ready conditions and any LastAsyncOperation error) to establish whether reconciliation is succeeding at all.
If reconciliation is healthy, the config is simply stale: set visibility: public on both, move them out of the # Private repos. group, and correct the comments that justify the private pin.
If reconciliation is not applying the field, that is the real defect — fix it before touching the declared values, and check what else these resources are silently failing to enforce.

Acceptance criteria

Declared and live visibility agree for every repository in deploy/repositories/.
The kustomization's private/public grouping and the per-file comments match reality.
It is established, with evidence, whether visibility is actually enforced by reconciliation — recorded on this issue either way.
A check exists that would catch declared-vs-live divergence on this field, so it cannot drift silently again.

Notes

Surfaced while declaring doggy-countdown (#122), which was written against the ascoachingogvaner file as its template — so the same possibly-unenforced pin was copied forward.

⚠️ Possible Duplicate Issue(s)

fix: declarative repo-settings updates are write-blocked (422 on every update PATCH) #112

🔗 Related PRs

#67 - feat: require actions pinned to a full-length commit SHA on all repos but actions [merged]
#78 - feat(repositories): declare provider-upjet-unifi (net-new Library repo) [merged]
#82 - feat: declare the devantler-tech/aws config-tenant repo [merged]
#88 - feat: archived-repo pattern + reusable-workflows archival groundwork [merged]
#103 - feat(teams): activate declarative Admins policy [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

Root-caused — and it is the worse of the two branches

This issue asked which was true: reconciliation not applying visibility, or stale config. It is the first, and it is not limited to visibility or to those two repositories.

Every repository PATCH is being rejected by GitHub:

update failed: async update failed: failed to update the resource:
PATCH https://api.github.com/repos/devantler-tech/<repo>:
422 Commit signoff is enforced by the organization and cannot be disabled

The org sets web_commit_signoff_required: true. The provider sends that field as false (its zero value — the CRD field is unset in every one of our Repository specs), GitHub refuses, and the entire PATCH is rejected. Not the one field — the whole update. So no declared setting reaches GitHub: not visibility, not description, not archived.

Measured scope — 18 of 20 Repository managed resources are Synced: False, all with this same message (confirmed individually on ascoachingogvaner, wedding-app, agent-skills, doggy-countdown).

So the comment in each of those files — "visibility is pinned private explicitly … so management can never expose it" — describes a guarantee that does not currently exist. Nothing has been enforcing it.

Live reproduction, found the hard way

While setting up doggy-countdown (#122) I created it private — confirmed "private": true in the creation response, chosen deliberately because the repository contains personal photographs. Some time later it was public. Its managed resource showed:

Field	Value
`spec.forProvider.visibility`	`private`
`status.atProvider.visibility`	`public`
`Synced`	False

I set it back to private immediately. It is private now, and it will stay that way precisely because the provider cannot successfully PATCH it.

I have not established what flipped it — the failing PATCH should be atomic, so it should not have been the provider. That gap is worth closing, because "a repository declared private became public and we cannot say why" is its own problem. But it is downstream of the fact that nothing would have corrected it either way.

What to fix

Stop sending the field the org enforces. Either set webCommitSignoffRequired: true in every Repository spec so it matches the org, or determine whether the provider version can omit an unset boolean rather than sending false. The first is a config change we control today; the second is an upstream question.
Re-drive reconciliation and confirm Synced: True across all 20 before trusting any declared value again.
Reconcile declared-vs-live for every repository, not just the three named here — with reconciliation broken, any field could have drifted and nothing would have reported it.
Alert on Synced: False. 18 resources sat un-synced with no signal. A control plane that silently stops applying configuration is worse than not having one, because the config file reads as the truth.

Acceptance criteria

Synced: True on all Repository managed resources in github-config.
Declared and live agree for every field these resources pin — verified after sync is restored, not before.
A Synced: False managed resource raises an alert rather than sitting quietly.
RED-proved: changing a declared value actually moves the live repository.

Raising this to Security type — the failure silently voids an access-control guarantee that three repositories' comments claim to provide, and one of them demonstrably went public while declared private.

🤖 Generated by the Agentic Engineer

⚠️ Do not fix the signoff field first — the order matters

The obvious fix is to set webCommitSignoffRequired: true across the Repository specs so the PATCH stops 422-ing. Doing that alone would cause an outage, and I want that written down before someone reaches for it.

The moment reconciliation starts succeeding, every declared value is applied — including the ones that have been quietly wrong for as long as sync has been down:

Repository	Declared	Live today
`ascoachingogvaner`	`visibility: private`	public
`wedding-app`	`visibility: private`	public

So restoring sync would take two live public websites private in the same reconcile. Both are real deployed sites; ascoachingogvaner is a business site.

That is not an argument against fixing it — it is an argument for sequencing:

First, establish the intended visibility for every repository and correct the declarations. This is a maintainer decision, not something to infer: the config says private, the world says public, and there is no way to tell from here which one is the mistake.
Then fix webCommitSignoffRequired and let reconciliation resume.
Then verify Synced: True and that no live setting moved unexpectedly.

I did not correct the declarations myself for exactly that reason — picking a value would be guessing at intent on two production sites.

Same caution applies to any other field these resources pin. Sync has been down long enough that atProvider is the only trustworthy record of current state; the spec is a record of intent that has not been enforced. Reconcile the two deliberately before turning enforcement back on.

I have deliberately left doggy-countdown declared private and set it live-private, so those two agree and it is safe under either order.

🤖 Generated by the Agentic Engineer

The stated root cause is falsified — `webCommitSignoffRequired` is already correct

The diagnosis above says the field "is unset in every one of our Repository specs" so the provider sends false. Measured on the live cluster just now, that is not the case:

	value
`spec.forProvider.webCommitSignoffRequired`	`true` on 20 / 20
`status.atProvider.webCommitSignoffRequired`	`true` on 20 / 20

It has been pinned in the shared kustomize patch since faca686 (2026-06-18), with a comment describing this exact 422. So the obvious fix — "set the signoff field" — is already in place and would change nothing. Anyone reaching for it next would spend the run and move nothing.

What actually correlates: `hasDownloads`

GitHub retired the downloads feature and no longer reports the field. The Terraform provider still defaults it to true, so repositories that adopted that default through LateInitialize hold a value the API can never echo back.

Across all 20 managed resources the split is exact:

`hasDownloads` spec vs live	Synced
`true` vs absent	False — 17/17
absent vs absent	True — 2/2 (`doggy-countdown`, `world-at-ruin`)
`true` vs `true`	True — 1/1 (`reusable-workflows`, archived)

No other setting divides those two groups. The two newest adoptions never picked the default up, which is why they are the healthy ones.

PR #125 pins it to false in the shared patch — scoped to deploy/repositories/ only, since reusable-workflows is the one repo where the value genuinely agrees and pinning it there would create the mismatch.

⚠️ `status.atProvider` is NOT a reliable drift oracle — this affects AC2 and AC4

Worth recording before anyone reconciles declared-vs-live from it, as AC2 asks. world-at-ruin reports allowMergeCommit/allowRebaseMerge as false in spec and true live — while Synced=True. A genuine pending change cannot coexist with a successful reconcile, so at least one of those two readings is stale.

Concretely: comparing spec.forProvider against status.atProvider measures what the mirror happens to carry, not what reconciliation will act on. A declared-vs-live audit built on it will report drift that is not real and can miss drift that is. AC2 and AC4 should compare against the GitHub API directly instead.

That is also why #125 states its own confidence as inferred: the correlation is exact, but the mechanism cannot be observed from the mirror. It is cheap and reversible, and a no-op if wrong.

Still yours to decide — the two visibility declarations

Unchanged from the sequencing note above, and #125 deliberately does not touch it: ascoachingogvaner and wedding-app are declared private and are live public. Both keep a mismatch either way, so #125 cannot flip them — but it does not resolve them either.

Which value is the mistake is a judgement about intent on two live public sites, so it is not one to infer. Once you say which way each should go, the rest is mechanical.

Not blocking, but noted

world-at-ruin is the one actively-managed repository whose declared squash-only merge policy does not match live, so that policy is currently unenforced there regardless of this fix.

🤖 Generated by the Agentic Engineer

Verified — #125 deployed and reconciliation recovered

v1.21.2 published at 04:33:49Z (CD green) and the change reached the cluster. Measured against the live cluster after the deploy:

	before	after
`Synced: True`	3	16
`Synced: False`	17	4

13 repositories recovered. The inferred mechanism in #125 is now confirmed by outcome: removing the unresolvable mismatch stopped the rejected update, and the declared configuration is reaching GitHub again. The stale value is gone from the specs.

The two visibility declarations did not flip, as intended. ascoachingogvaner and wedding-app remain Synced: False and remain live-public — they retain their own mismatch, so nothing was applied to them. That was the sequencing risk this issue warned about, and it did not materialise. Deciding their intended visibility is still yours; nothing else in this fix depends on it.

Remaining 4, and why:

Repository	Why still failing
`ascoachingogvaner`, `wedding-app`	the visibility declaration — your call
`agent-plugins`, `agent-skills`	a `topics` mismatch, unrelated to this fix

The last two are ordinary drift and I will file them separately rather than widen this issue.

A correction worth keeping

ksail and platform both showed a pages mismatch when compared through status.atProvider, and both are now healthy without anything being done about it. That is direct confirmation of the warning above: the mirror reported drift that reconciliation did not act on. AC2 and AC4 should compare against the GitHub API, not that field — an audit built on it would have chased two mismatches that were never real.

Where this leaves the acceptance criteria

Sync restored for every repository not blocked on a separate decision — done.
Declared-vs-live reconciled for all 20 — done, with the caveat above about which source to trust.
Alerting on a stuck resource — not done, filed as Alert when a Crossplane managed resource stops reconciling (Ready=True hides Synced=False) platform#2820 and linked as a sub-issue. Worth its own item: the cluster has no PrometheusRule surface any more, so this needs a deliberate mechanism rather than a rule.
RED-proof that a declared change moves the live repository — now possible for the 16 healthy ones, where it was not before.

fix(repositories): park the visibility-drifted repos on Observe

fix(repositories): correct two stale private declarations on public r…

🤖 Generated by the Agentic Engineer

Measured live, 2026-08-07 — the headline is overstated, and the real defect is narrower and worse

Read all 20 Repository managed resources on prod, plus the GitHub API for each, before touching anything.

Reconciliation is not "fully broken": 17 of 20 report Synced=ReconcileSuccess. Three do not — agent-plugins, agent-skills, platform-tenant-template — each with the same message:

update failed: async update failed: failed to update the resource: PATCH …: 422 Commit signoff is enforced by the organization and cannot be disabled

The 422 is a consequence, not the cause. A managed resource with no pending diff never issues an update PATCH at all, so it never meets this error — which is why 17 look healthy. The three that fail are the three that have something to write.

What answers acceptance criterion 3

kustomization.yaml already patches webCommitSignoffRequired: true onto every Repository, and its comment explains this is done precisely to avoid the 422, on the reasoning that "the error names disabling, not presence".

That reasoning does not hold up against the live data. agent-skills declares webCommitSignoffRequired: true, the live repository reports web_commit_signoff_required: true, and its PATCH is still rejected with "cannot be disabled". So declaring the live value does not stop the error; something in the update path still presents the field in a form GitHub refuses.

Verdict: the pin is not enforced by reconciliation on these three, and neither is anything else they declare. Their update path is wedged, so every field on them is advisory — which is the larger of the two possibilities this issue set out to distinguish, on a subset rather than portfolio-wide.

The visible cost, today

agent-skills declares 11 topics and agent-plugins 10. Live, both have zero. Repos that reconcile cleanly match exactly (ksail 20/20, platform 1/1, kyverno-policies 0/0). The declared discovery metadata has never been applied, and cannot be while the PATCH fails.

platform-tenant-template is different again: every field it declares already matches live, and its LastAsyncOperation failure is dated 2026-08-02 while Synced is rewritten every reconcile. With no diff left to drive a fresh update, that cached failure has nothing to clear it — it reports an error it can no longer retry out of.

Not yet established

Whether GitHub rejects the field's presence in the PATCH or a false value reaching it from provider state. The two have different fixes, and the deciding test is a no-op PATCH /repos/{owner}/{repo} with and without web_commit_signoff_required. I did not run it: it writes repository configuration outside GitOps, and prod delivery is currently halted (devantler-tech/platform#3005), so this is the wrong hour to hand-modify repository state. Filed as its own issue with that test as step one.

Criteria status

1 — closed by fix(repositories): correct two stale private declarations on public repos #138 (both declarations corrected, verified against live after merge).
2 — closed by fix(repositories): correct two stale private declarations on public repos #138 (grouping and comments corrected).
3 — answered above: not enforced on the three wedged resources, enforced on the other 17.
4 — a drift check. Still open, and now better specified: the signal that catches this early is Synced != ReconcileSuccess on any Repository, which fires the moment a resource wedges instead of waiting for a human to notice a field never landed.

ci: check declared repository settings against live GitHub

🤖 Generated by the Agentic Engineer

Criterion 4 — the drift check exists, and its first run found a third divergence

#140 adds scripts/check-repository-drift.sh: it compares every forProvider field of every
declared Repository against the live repository object, and fails naming the repository and field.
It runs daily (repository-drift-check.yaml), read-only, and its comparison logic is pinned by
tests/repository-drift.sh in CI against fixtures, so the check itself stays offline and PR-safe.

First run against live GitHub, 2026-08-07 ~10:2xZ:

DRIFT agent-plugins.topics:          declared=[10 topics]  live=[]
DRIFT agent-skills.topics:           declared=[11 topics]  live=[]
DRIFT world-at-ruin.allowMergeCommit: declared=false        live=true
DRIFT world-at-ruin.allowRebaseMerge: declared=false        live=true

The first two are the wedged-PATCH class already tracked in #137. The third was not known, and it
is a second, structurally different way for a declaration to never land — filed as #141.

world-at-ruin is Observe-only, so it writes nothing at all, yet the shared kustomize patch stamps
the squash-only merge policy onto its forProvider. It reports Synced=ReconcileSuccess because a
resource with no write path never accumulates a diff. Live, merge commits and rebase merges are both
enabled — against the policy the release contract depends on.

That is worth stating plainly, because it sharpens this issue's own conclusion: the earlier answer to
criterion 3 was "not enforced on the three wedged resources, enforced on the other 17". It is
16. Observe-only resources are a third state — neither wedged nor enforcing — and there are
three of them (world-at-ruin, ascoachingogvaner, wedding-app). The latter two show no drift only
because #138 corrected their declarations to match live; the gap is structural, not fixed.

Verified live at the same moment:

`managementPolicies`	count
`[Observe, Create, Update]`	16
`[Observe]`	3
`[Observe, Update, LateInitialize]`	1

Criteria status

1 — closed by fix(repositories): correct two stale private declarations on public repos #138.
2 — closed by fix(repositories): correct two stale private declarations on public repos #138.
3 — answered, and corrected above: enforced on 16, wedged on 3, structurally inert on 3.
4 — closed by ci: check declared repository settings against live GitHub #140, which found a divergence on its first run rather than only asserting it could.

Closing once #140 merges. The two divergence classes it surfaced stay open on their own issues:
#137 (wedged PATCH) and #141 (Observe-only with a stamped policy).

fix(drift): read live state by external name, and compare topics as a…

fix(drift): scope the App token to the declared repositories

fix(drift): withhold a private repository's values from the public log

ci: check declared repository settings against live GitHub (#140)

🤖 Generated by the Agentic Engineer

Closing: #140 merged (45fb229a, 2026-08-07T10:46:02Z), which was the last open criterion.

All four criteria are met, and criterion 3's answer is the corrected one — enforced on 16 of 20,
wedged on 3, structurally inert on 3 (Observe-only, which no earlier count had separated out).

The two divergence classes this issue surfaced stay open on their own issues, with the drift check
now watching both: #137 and #139 (the wedged PATCH — and they are not duplicates of each other;
see the partition recorded on #139), and #141 (Observe-only resources carrying a stamped policy
they cannot apply).

Navigation Menu

Repository navigation

Repository reconciliation is fully broken: every PATCH 422s on org-enforced commit signoff #123

Evidence

Problem

Suggested approach

Acceptance criteria

Notes

Root-caused — and it is the worse of the two branches

Live reproduction, found the hard way

What to fix

Acceptance criteria

⚠️ Do not fix the signoff field first — the order matters

The stated root cause is falsified — `webCommitSignoffRequired` is already correct

What actually correlates: `hasDownloads`

⚠️ `status.atProvider` is NOT a reliable drift oracle — this affects AC2 and AC4

Still yours to decide — the two visibility declarations

Not blocking, but noted

Verified — #125 deployed and reconciliation recovered

A correction worth keeping

Where this leaves the acceptance criteria

12 remaining items

Measured live, 2026-08-07 — the headline is overstated, and the real defect is narrower and worse

What answers acceptance criterion 3

The visible cost, today

Not yet established

Criteria status

Criterion 4 — the drift check exists, and its first run found a third divergence

Criteria status

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Repository reconciliation is fully broken: every PATCH 422s on org-enforced commit signoff #123

Description

Evidence

Problem

Suggested approach

Acceptance criteria

Notes

Activity

coderabbitai commented on Jul 26, 2026

devantler commented on Jul 26, 2026

Root-caused — and it is the worse of the two branches

Live reproduction, found the hard way

What to fix

Acceptance criteria

devantler commented on Jul 26, 2026

⚠️ Do not fix the signoff field first — the order matters

devantler commented on Jul 27, 2026

The stated root cause is falsified — webCommitSignoffRequired is already correct

What actually correlates: hasDownloads

⚠️ status.atProvider is NOT a reliable drift oracle — this affects AC2 and AC4

Still yours to decide — the two visibility declarations

Not blocking, but noted

devantler commented on Jul 27, 2026

Verified — #125 deployed and reconciliation recovered

A correction worth keeping

Where this leaves the acceptance criteria

12 remaining items

devantler commented on Aug 7, 2026

Measured live, 2026-08-07 — the headline is overstated, and the real defect is narrower and worse

What answers acceptance criterion 3

The visible cost, today

Not yet established

Criteria status

devantler commented on Aug 7, 2026

Criterion 4 — the drift check exists, and its first run found a third divergence

Criteria status

devantler commented on Aug 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

The stated root cause is falsified — `webCommitSignoffRequired` is already correct

What actually correlates: `hasDownloads`

⚠️ `status.atProvider` is NOT a reliable drift oracle — this affects AC2 and AC4