GitOps Reverser is a Kubernetes operator that observes cluster mutations and writes the resulting desired object state to Git. It reverses the traditional GitOps direction: instead of Git driving the cluster, the Kubernetes API drives Git. The repository becomes a continuously updated mirror of live cluster state.
This document describes how the operator works today. If you are reading it for the first time, keep one sentence in mind: Kubernetes watch supplies state, Git stores the mirror, and Redis keeps the operator's short-lived coordination state. Read the Ground Rules, Mental Model, and Data Sources for the shape of the system, then the Configuration Model and Common Flows for how the pieces move together. The later sections give the reference detail behind each piece. If a detail here ever disagrees with the source, the source wins; deeper design records live under docs/design/. For the inventory rather than the mechanics, which package holds which responsibility and which components observe a Kubernetes API server, see components.md.
Both connection kinds are namespaced, for one reason: a connection carries a credential, and the
credential belongs in the namespace of whoever owns it. A namespaced GitProvider is a team's Git
write boundary: its credential, branch policy, and targets usually belong together. A namespaced
KubeProvider is one logical source identity: one client, discovery surface, watch state, and
attribution partition, shared by every target that names it. Its kubeconfig Secret lives in its own
namespace, so a tenant's connection keeps its credential with the tenant. Its grants
(spec.allowedGitTargets, spec.allowedApplierClaims) admit control-cluster namespaces and bound
what they may reach; omitted, each admits the provider's own namespace, and none grants
source-cluster RBAC. The source identity is the provider's
namespace and name, with no API-server identity probe: two providers configured for the same server
remain separate source partitions, and so do two providers with one name in two namespaces.
The operator's own identity is the one credential no namespace owns, so it is not an ordinary
secretRef. A KubeProvider asks for it with spec.inCluster: {}, and the installation decides:
only a provider listed by exact namespace/name in --in-cluster-kube-providers is honored. Any
other reports Stalled=True (InClusterNotAuthorized) and is never used, so no omitted, malformed,
or forbidden credential falls through to the operator's identity.
Keeping a repository's write credential in the namespace of whoever owns that repository is what
makes a team able to start mirroring without a platform admin. Each branch of that repository is a
GitBranch, in the credential's namespace or in one the GitProvider's spec.allowedGitBranches
admit for that branch, and one GitBranch holds a repository branch, because one branch worker
writes the branch, with one credential: the oldest admitted GitBranch whose credential opens a
push session on the remote holds it, and any other reports BranchConflict. A repository written
from several namespaces needs no copy of the credential in each: their GitTargets name the shared
GitBranch with spec.gitBranchRef.namespace, admitted by that GitBranch's
spec.allowedGitTargets. Omitted, that list admits the GitBranch's own namespace, and a written
list is the whole policy
(configuration.md). Neither kind can change scope later: a
CRD's spec.scope is immutable once the definition is Established, so the change would mean
deleting the definition (and every object of the kind with it) and recreating both.
These are the design decisions to keep in your head while reading or changing the code:
GitOps Reverser writes Git; Flux or Argo CD deploys. The output is commits that reflect the cluster's state. Bringing a change from Git back into the cluster is the job of a GitOps tool running alongside. GitOps Reverser only decides when that tool applies a folder it also writes, through an applier claim (see "GitOps Reverser decides when Git is applied" below).
The Kubernetes API is the source of truth for what GitOps Reverser writes. Git is a
materialized mirror of desired state from the API, ingested by watch; GitOps Reverser never
reads Git as object state. An edit made in Git reaches the cluster only through Flux or Argo CD.
When a GitTarget names an applier claim, those applies are coordinated: GitOps Reverser first
publishes the API edits it has decided, then asks the applier to apply. The watch observes the
result, and the applier's own writes during that apply (its echoes) are not mirrored, because Git
already holds them. When a push conflicts
with a newer remote commit, the operator fetches the new remote state, resets its local clone, and
replays its retained writes from the API. So the API comes first without being the only author: other
authors can push to the same branch and their commits stay, and only where an API edit and a Git
commit touch the same object does the API's version win.
Watch is the only object-state source. Each GitTarget names one KubeProvider and opens one
Kubernetes watch per claimed (GVR, scope) against that source, with sendInitialEvents=true. For a
namespaced type, a scope is one concrete source namespace; cluster-scoped types use the cluster-wide
scope. Every Git write derives from persisted state the watch observed. Audit never defines what
changed; it only, optionally, explains who caused it.
Sensitive resources are never written in plaintext. Core Secrets and configured sensitive resource types must be encrypted before they touch the Git worktree. If encryption cannot be configured, the write fails; there is no plaintext opt-out.
Writes are serialized per Git branch. One BranchWorker owns each
repository branch: the repository's URL, normalized, and the branch. Multiple GitTargets may share
one branch, from any namespace, through the one GitBranch that holds it. Every write to that
branch goes through the worker's single event loop and commit window.
The worktree sits at the remote tip, or the worker knows it does not. The write path used to
fetch and hard-reset before the first commit of every publication cycle. It no longer does: the push
session reads the remote's ref advertisement on a connection the cycle was making anyway, and a cycle
that commits nothing still reaches it, so a base the worker can vouch for needs no fetch to plan
against. Two pieces of state on the worker carry that claim, and they are separate on purpose:
baseTrusted is dropped by anything that can no longer prove where the remote is, while
checkoutApplied counts how many retained writes the checkout holds on top of its root. A write
that failed part-way sets it to unknown, which only a reset clears; a reset sets it to zero,
which with writes retained says their local commits are gone and must be replayed. Collapsing the
two would let one kind of doubt clear another. Either one makes the next cycle fetch. The
failure direction is safe by construction: a stale "untrusted" costs one fetch, while a stale "trusted"
is caught by the compare-and-swap on the next push. For a new write branch, the worker checks the
parent's advertised tip before sending the push. A moved parent costs a fetch, a reset, and a replay.
The server's compare-and-swap protects creation of the write branch with Old = zero; it does not
lock the parent. The parent can still move after the advertisement, so what is guaranteed is the
parent's tip as that advertisement showed it. The corollary is the rule to keep in your head
when adding code here: anything that resolves, commits, or concludes without reaching a push
advertisement must either fetch, or refuse to conclude. See
how the branch worker reads the remote.
That trust belongs to one repository, and so does the worker holding it. A worker is keyed by
the repository branch, and spec.url is immutable and repointed by deleting and recreating the
GitProvider. The key names the repository but not the object behind it, so the worker also
carries the repository identity it was created for: the provider's UID paired with its URL.
Every repository operation uses that URL rather than whatever the provider names when the cycle
reads it, and the reconcile compares the identity on each tick. A provider recreated under its name,
for the same repository or another, does not move the worker; it replaces it, and the replaced
worker takes its clone, its trust and its queued work with it. Nothing has to remember to invalidate
a field.
An idle target re-proves where its branch is, periodically. A target that is writing learns it
for free: the push session reads the remote's advertisement and the server names the hash it
accepted, so an active branch never schedules a look of its own. One that has gone quiet is asked
on its 5-minute reconcile tick, and spends one ref advertisement per --git-refresh-interval
(10m by default); only a branch that moved costs a fetch after that. What it learns is published
as the GitBranch's status.remote (the commit, when it was last proved, and whether a push or a
fetch proved it), and a folder somebody changed in Git is re-read and republished as status.placement.
A refresh never writes content. It plans nothing and commits nothing of its own. What it does
publish, besides status.remote and status.placement, is whether the folder can be written at
all: a structural refusal that arrived in somebody else's push sets GitPathAccepted=False, and a
later read that finds the folder fixed clears it. Both directions matter, because neither event
(breaking a folder in Git, or fixing it) produces a Kubernetes event to notice it by.
That condition puts the target on the ten-second re-check loop, which is the recovery path rather
than a side effect, and reconcile.configbutler.ai/requestedAt still forces a full re-check for a
human who wants one now. A refusal a write found is left alone: a structural read cannot see a
write-boundary precondition, so only the resync that writes the affected type clears that one.
Seconds-fresh is the push receiver's job: a GitReceiver accepts the Git host's push webhook, and
wakes the branch worker of every GitProvider whose spec.url names the repository the delivery
names, for the branch it names. A delivery is only a hint: it makes the branch worker read the ref advertisement
with its own provider's credential, and what the remote says decides whether the branch moved. It never triggers a
cluster snapshot: the cluster still holds the values from before the push, so a snapshot would
publish them over the incoming change.
GitOps Reverser decides when Git is applied to the cluster. An apply must not revert an edit
that is live and not yet in Git, and only the worker writing the branch can see those edits. So a
GitTarget names an applier claim, which holds its applier so that it applies nothing on
its own (a Flux Kustomization is held suspended, an Argo CD Application held with automated sync off),
and the branch worker sends apply requests instead. A request is sent only after a flush:
when an apply comes due, the worker closes the open commit window, publishes every write decided so
far at once, and sends the request for the tip that publication produced. An edit decided after
that is recorded in the apply window, so an apply that overwrites it is withheld and repaired.
Continuous editing therefore never holds an apply back. A request goes out:
- for a commit somebody else pushed, after replaying the pending writes onto it, so the applied commit carries the API side's answer;
- for a refused edit the target opted to revert;
- for an API edit that an apply reverted on the cluster after the edit was made, to put it back;
- to catch up (
CatchUp), for a FluxKustomizationand an Argo CDApplicationalike: to adopt an object the API created in the folder, because an applier prunes only what it applied and a later Git-side delete would otherwise leave the object running; and to converge, once when the worker starts and once when aGitTargetfirst names the applier claim, unless the applier already applied the tip. A push of GitOps Reverser's own asks for nothing: the cluster already runs the edits it holds.
No applier is asked for anything before every initial snapshot of the GitTargets it covers is
published, so the edits made while the operator was down reach Git before any apply. The worker
keeps each reason pending until an apply operation answers it, and decides every retry the same
way. A request runs off the publication path and cannot fail a publication; the applier claim's
status.operation records the operation it started. Built:
the Flux Kustomization applier and the Argo CD Application applier, its automated sync held off.
The aim: a webhook for any other tool. See the applier claim design.
Redis/Valkey is optional but advised. The default configured-author mode runs without it: a plain
helm install comes up healthy and watches initialize from current state on startup. When an endpoint
is configured, Redis stores cursors for reconnecting running watches. A CommitRequest's author comes from
the audit fact of the write that created it, like any write's, and needs no Redis record.
Attribution does not require it on its own: its facts travel on a
selectable transport, Redis Streams by default and an in-process ring with
--author-attribution-transport=memory, which is refused with more than one replica. HA will require
Redis as the shared store across replicas.
Audit is an optional attribution lookup. When attribution is enabled, kube-apiserver posts audit
events to /audit-webhook/<audit-route> (or a configured annotation-routed shared endpoint). The route
is KubeProvider.spec.attribution.auditRoute, which defaults to <namespace>.<name>.
The operator appends a minimal fact to that route's per-type fact stream; every process watching the
type follows that stream into a bounded, TTL'd in-process index, and a watch event joins against the
index within a bounded grace window. A missing, late, or absent fact never blocks state capture; it
only changes the author.
Behavior is deterministic and proven by tests. Given the same observed Kubernetes state, configuration, and Git base, the operator makes the same materialization decisions. Ordering, attribution fallbacks, conflict replay, path refusal, encryption behavior, and commit semantics are pinned by unit and e2e tests.
The easy part is "write YAML to Git." The hard parts shape the whole architecture:
- Ordering. Two updates to the same object must land in Git in cluster order.
- Not losing changes. A dropped or late event must never leave Git permanently wrong. In particular, a delete that happens while no watch is running must still be reconciled.
- Secrets. Sensitive objects must never touch disk in plaintext.
- Scale. A cluster holds thousands of objects across hundreds of types; you cannot keep a live watch open on all of them.
The solution, in the vocabulary used throughout this document:
- Watch is the only object-state source. Each
GitTargetselects eligible resource collections. The planner chooses one served API version per group/resource/namespace collection and opens its watch. Delivery order within one watch does not order arrivals across overlapping watches. - Watch cost follows selected collections. The planner opens watches for selected, eligible collections rather than every type in the cluster. All-namespace and named-namespace collections remain separate when rules select both.
- Initialization and reconnect differ. The first attempt after startup or watch replacement requests initial state for the new render-fidelity revision. Later attempts in the same watch loop can use a stored resume RV; expired history requires fresh initialization. A completed snapshot permits cleanup only within its observed collection and pruning policy. Cursors use target UID, group/resource, namespace, and label selector, with a TTL refreshed by events and bookmarks. Unsupported initial-event requests use LIST with a buffered watch. Snapshot reconciliation is triggered by initialization, never by a timer.
- Audit, when enabled, only names the author. It is an optional attribution lookup; a missing or
late fact costs author fidelity, never correctness. With attribution disabled the product commits as the
configured committer. With attribution enabled, an unresolved live change is visibly authored as
unknown (attribution unresolved), so an installation that expects attribution can investigate its audit policy, ingress, Redis connection, or source identity instead of mistaking the result for configured-author mode. - One
BranchWorkerper Git branch serializes all writes. Every write to a branch funnels through a single worker and a single commit window, which keeps concurrent GitTargets and authors from racing each other into a corrupt tree.
GitOps Reverser reads Kubernetes through four mechanisms. Two are required to function; the other two are optional and only add who did something:
| Source | Answers | Required | What it carries |
|---|---|---|---|
| Discovery (CRD/APIService) | what types exist | yes | the API surface: served GVRs, scope, preferred version, subresources; rules resolve against it |
Watch (per claimed (GVR, scope)) |
what changed | yes | the object body, ordered per type; the only object-state source, with deletes reconciled via sendInitialEvents replay + mark-and-sweep |
Audit webhook (/audit-webhook/<audit-route>) |
who changed mirrored state, and who created a CommitRequest |
no | a post-persist attribution fact, partitioned by source provider and joined to the watch event by resourceVersion, or to the CommitRequest by the UID it was created with (Optional Attribution) |
With the optional source off, the product still mirrors state correctly, and every commit is authored by the committer.
For a mirrored change, taking the author from a validating admission webhook is tempting, because the request
already carries userInfo. We tried it; the edge cases are not fixable, because admission runs before
the write reaches etcd:
- Admission sees attempts, not persistence. A request that passes our webhook can still be rejected by a later webhook or fail an optimistic-concurrency conflict at storage, so the change never becomes state, yet we would already have recorded an author for it (a dry-run is the same shape: admission fires, nothing persists).
- There is nothing to join on yet. Mirrored attribution matches an author to a persisted watch event
by identity + resourceVersion; at admission the resourceVersion does not exist, and for a
generateNamecreate even the final name/UID is unassigned, so an admission record cannot be coupled to the watch or audit stream.
Audit avoids both: kube-apiserver posts a ResponseComplete event after the write persisted, carrying
the resourceVersion that joins it to the watch event.
A CommitRequest's author comes from audit too, for the same reasons: the ResponseComplete event that
created it, joined by its UID, from the same index and with the same grace as the edits it saves. A later
write to the request, such as a label patch, never names its author. So the request and its edits are
attributed by one pipeline, on the cluster both were made on, and a request on a source cluster needs
nothing from the control cluster. A CommitRequest is never mirrored, whatever a WatchRule selects. See
CommitRequest Finalize.
How Kubernetes emits these structures is not something to guess at. A separate, standalone project. The mutation-capture lab (design, cmd/mutation-capture-lab/) records the exact watch, audit, and admission output a real apiserver produces for each interesting scenario and commits it as normalized example YAML (a versioned corpus). It doubles as a regression harness: point it at a new Kubernetes release, regenerate, and any change in verb naming, event ordering, body presence, or deletecollection fan-out surfaces as a reviewable corpus diff before it can surprise the operator. The "admission sees attempts, not persistence" claim above is read straight out of that corpus: its dry-run, record-and-reject, and conflict scenarios capture audit and admission with no resulting watch event.
You configure GitOps Reverser through ten CRDs (group configbutler.ai, version v1alpha3).
WatchRule and ClusterWatchRule choose which Kubernetes resources enter the pipeline.
CommitRequest can ask for the current window to be saved. GitTarget joins one source cluster to one
Git destination. KubeProvider supplies the source connection and authorization boundary;
GitProvider supplies the repository, credentials, commit settings, and push policy, and a
GitBranch holds one branch of that repository for every GitTarget that writes it. A
GitReceiver is the endpoint the Git host's push webhook calls; nothing references it, because a
delivery is routed by the repository it names. FluxClaim and ArgoCDClaim hold the Flux
Kustomization or Argo CD Application that applies a folder back, so GitOps Reverser decides
when it applies.
graph LR
subgraph CONTROL["Control plane: operator cluster"]
WR[WatchRule] -->|gitTargetRef| GT[GitTarget]
CWR[ClusterWatchRule] -->|gitTargetRef| GT
CR[CommitRequest] -->|gitTargetRef| GT
GT -->|applierClaimRef| TR["FluxClaim /\nArgoCDClaim"]
GT -->|kubeProviderRef| CP[KubeProvider]
GT -->|gitBranchRef| GB[GitBranch]
GB -->|gitProviderRef| GP[GitProvider]
GR[GitReceiver] -.->|"wakes by spec.url"| GP
KCFG[("Remote kubeconfig Secret\nprovider's namespace")]
WM["Watch manager\nper-provider source context"]
end
subgraph SOURCE["Source cluster selected by KubeProvider"]
API["Kubernetes API\ndiscovery · watches"]
APPLIER["Applier\nFlux Kustomization\nor Argo CD Application"]
end
HOST[("Git host")]
CP -. "secretRef (or inCluster)" .-> KCFG
CP -->|"source identity + config"| WM
WM -->|"source client"| API
TR -. "kustomizationRef / applicationRef" .-> APPLIER
HOST -. "push webhook to status.webhookPath" .-> GR
style GP fill:#e8f4fd,stroke:#2196f3
style CP fill:#e8f4fd,stroke:#2196f3
style GT fill:#e8f4fd,stroke:#2196f3
style WR fill:#fff3e0,stroke:#ff9800
style CWR fill:#fff3e0,stroke:#ff9800
style CR fill:#f3e5f5,stroke:#8e24aa
The diagram follows one GitTarget. A provider with spec.secretRef selects a remote cluster through a
kubeconfig Secret in the provider's own namespace; one with spec.inCluster selects the operator's own
cluster, when the installation lists it. Each referenced KubeProvider gets an independent source
context: client, discovery catalog, followability registry, and reachability state. Its GitTargets
open their source watches through that context. The contexts are keyed by the provider's
namespace/name and are released when no GitTarget references them, so a provider deleted and
recreated under the same name is a new connection.
The applier is not one of the ten kinds. It belongs to Flux or Argo CD, lives in the cluster the
applier claim's spec.kubeProviderRef names (the source cluster, or a management cluster), and is
reached through that KubeProvider's identity. The applier claim is ours: it holds the applier, and
records the apply operation it started.
| CRD | Scope | One line role |
|---|---|---|
WatchRule |
namespaced | which NAMESPACED resources, in which source namespaces, route to a GitTarget |
ClusterWatchRule |
cluster | which CLUSTER SCOPED resources route to a GitTarget |
CommitRequest |
namespaced | a one shot "save the open window now" signal |
GitTarget |
namespaced | one materialization from a KubeProvider to (GitBranch, path) |
KubeProvider |
namespaced | one source-cluster connection plus namespace access policy |
GitBranch |
namespaced | one repository branch, held for every GitTarget that writes it |
GitProvider |
namespaced | a Git repo + credentials + commit/signing config |
GitReceiver |
namespaced | the endpoint a Git host notifies on a push, for every GitProvider whose spec.url a delivery names |
FluxClaim |
namespaced | the Flux Kustomization GitOps Reverser asks to apply, and the apply in flight |
ArgoCDClaim |
namespaced | the Argo CD Application GitOps Reverser asks to sync, and the sync in flight |
- Sources: watchrule_types.go, clusterwatchrule_types.go
- Controllers: watchrule_controller.go, clusterwatchrule_controller.go
Scope is carried by the rule KIND. A WatchRule selects NAMESPACED resources and routes matching
events to a same namespace GitTarget; each of its rule items names the source namespace it watches.
A ClusterWatchRule selects CLUSTER SCOPED resources, with an explicit namespace gitTargetRef and no
namespace selection of its own. Both share the rule model:
spec.rules[]: OR resource rules (MinItems=1). A rule selects whole resource collections: every selected collection is observed through creates, updates, and deletions, and theGitTarget'sspec.prune.modedecides whether an observed removal deletes the Git document.rules[].apiGroups: omitted resolves the named resource across served groups;""is the core group;*is all.rules[].apiVersions: omitted means the preferred served version.rules[].resources: plural resource names or*.rules[].objectSelector: an optional label selector the API server applies to every list and watch the rule opens. An object that stops matching leaves the collection as a deleted one does.WatchRuleaddsrules[].sourceNamespace: omitted for the rule's own namespace, an exact name, or*for every namespace the source credential can read. Anything but the rule's own namespace passes the source-namespace gate (SourceNamespaceAuthorized). A*compiles to ONE cluster-wide collection (the empty namespace, which for a namespaced GVR is the all-namespaces collection), so it opens one stream and one list per matched type however large the cluster is. That collection is a peer of any named-namespace collection on the same type, never a replacement.ClusterWatchRulehas no scope or namespace choice of its own: the kind is the choice.
Subresources are rejected in rule resources. Mirroring operates on top level resources: a scale
made through the /scale subresource reaches Git as the parent's own watch event, and its audit
event serves only to name who scaled the parent.
One KubeProvider grant answers two questions in different planes:
consumerNamespaces(from, withnamesorselector) authorizes control-cluster namespaces to reference the provider. It is a tenant/export boundary; it neither selects nor grants access to source namespaces.sourceNamespacesandclusterResourcesbound what theGitTargets it admits may watch. They grant nothing: what may actually be read from the source cluster is bounded by that credential's own Kubernetes RBAC. The operator reads noNamespaceobjects in a source cluster for authorization.
A one shot "save now" signal that finalizes the open commit window for a GitTarget instead of waiting
for the silence timer. It is created on the cluster the GitTarget mirrors: the control cluster, or a
source cluster whose KubeProvider sets spec.commitRequests. The entire spec is immutable. Key
fields:
spec.gitTargetRef: target whose window the request attaches to, admitted by itsspec.allowedCommitRequests. On a source cluster the namespace is required.spec.message: optional verbatim commit message (1–1024 chars, no control characters).spec.window:attach(CurrentOrNextorNext),attachTimeout(default5s, the wait for a window), and the timers that replace the target's for the attached window:idleTimeout(none by default) andmaxDuration(default2s). A request never opens a window; only writes do.spec.whenNothingToCommit:Resolve(default) orCommitEmpty, which records the message in an empty commit when the request ends with nothing to commit.status.author: the creator, as the request's audit event names them; empty when it names nobody, and then the request can attach only to an unnamed window.status.conditions: kstatus-compatible. Ready is the summary (Unknown while the save runs, True once the request reached a terminal outcome that is not an error: a pushed commit, or a benign no-commit); Reconciling/Stalled are the kstatus progress/blocked pair. Pushed reports whether the commit reached the remote. TheReadycondition'sreasoncarriesCommitted,NoWindow,WindowMismatch,AlreadyPresent, orFinalizeFailed. A benign no-commit (e.g.NoWindow) isReady=True,Stalled=False, a correct non-error outcome, whereas aFinalizeFailedisReady=False,Stalled=True.status.branch/status.commit: set when the commit was pushed (Pushed=True).
How attribution and finalization interact is described under CommitRequest Finalize.
- Source: api/v1alpha3/gittarget_types.go
- Controller: internal/controller/gittarget_controller.go
One materialization from a source provider to a Git destination: (KubeProvider, GitBranch, path).
Key fields:
spec.gitBranchRef: required; aGitBranchby name, in theGitTarget's own namespace unlessnamespacenames another one. Itsspec.allowedGitTargetsmust admit theGitTarget's namespace.spec.kubeProviderRef: required; the sourceKubeProvider, in theGitTarget's own namespace unlessnamespacenames another one. Itsspec.allowedGitTargetsmust admit theGitTarget's namespace. There is no default provider: a missing one reportsValidated=False(KubeProviderNotFound).spec.path: immutable, required path under the repo (MinLength=1;.means repo root and must be chosen explicitly).spec.encryption: optional SOPS/age encryption settings for sensitive resources.spec.commit.window: when this target's commit windows close:idleTimeout(default5s) of silence, ormaxDuration(default1m) after the window opened.spec.commit.message:liveTemplate/reconcileTemplateGo templates.spec.applierClaimRef: optional; theFluxClaim(a FluxKustomizationGitOps Reverser holds suspended and asks to apply) orArgoCDClaim(an Argo CDApplicationwhose automated sync GitOps Reverser holds off, and whose syncs it starts) in this namespace. In a coordinated folder, an object another applier owns (by Flux's owner labels or Argo CD's tracking annotation) is not mirrored.spec.onRefusal:Ignore(default) orReapply, which asks the applier to reapply the tip when an edit to an existing document is refused.Reapplyrequiresspec.applierClaimRef.
spec.commit describes the folder, not the connection, so two GitTargets sharing one GitBranch
may batch and phrase their commits differently. A branch worker serves a repository branch, through
its GitBranch's GitProvider, and resolves the window per open window, since a window is bound to
exactly one target.
gitBranchRef, kubeProviderRef, and path are immutable so a target cannot silently orphan an
old materialization or change its source cluster. Among GitTargets on one GitBranch, from any
namespace, the controller rejects overlapping paths.
A GitTarget whose GitBranch does not exist reports Validated=False (GitBranchNotFound); one
whose GitBranch does not hold its branch, or is not Ready, reports GitBranchReady=False and
waits as progress (GitBranchNotReady). References are checked in status, not at admission, so Flux
and Argo CD can apply a folder of manifests in any order.
Status has a kstatus-compatible summary layer plus domain conditions:
Ready,Reconciling, andStalledare the generic conditions used by GitOps tooling.Ready=Truemeans the latest observed generation is valid, the Git path is accepted, and source streams are running. Initial replay reportsReconciling=True. A human-fixable block reportsStalled=True.ValidatedandEncryptionConfiguredexplain control-plane health.StreamsRunningexplains the source side: every tracked type is past initial replay and routing live events.KubeProviderReadyandSourceClusterReachabledistinguish valid source configuration from a source API the data plane can currently reach.GitPathAcceptedexplains the target side: the selected Git path is safe for the operator to materialize.GitBranchReadysays theGitBranchholds its branch and isReady.RenderMatchesLivesays the folder, as rendered, agrees with the live objects; while it is notTrue, live writes are held or dropped (see The render-vs-live gate).ApplierClaimReady, present only withspec.applierClaimRef, says the applier claim isReady(its applier exists, can be written, and is held: aKustomizationsuspended, anApplication's automated sync off) and that the applier applies this repository and branch. It is a gate.status.streamsis a bounded count summary, not a per-type list.
An applier claim has Configured, InControl and the kstatus trio, status.source
(what its applier applies, as last read), and status.operation and status.repairPending, which
the branch worker writes at every move of the operation it started.
WatchRule and ClusterWatchRule add ResourcesResolved and GitTargetReady. ResourcesResolved explains
the source selector. GitTargetReady mirrors the referenced GitTarget's write readiness. This keeps
StreamsRunning honest: it only says source watches are running, not that Git writes can succeed.
- Source: api/v1alpha3/gitprovider_types.go
- Controller: internal/controller/gitprovider_controller.go
Represents a Git repository and the credentials/configuration used to write it. Key fields:
spec.url: immutable repository URL.spec.secretRef: optional Secret in the same namespace for HTTP/SSH authentication.spec.allowedGitBranches: which namespaces'GitBranches may claim which branches with this credential, as grants with afromand branch patterns. It has no default: its own namespace needs a grant too. AGitBranchit no longer admits gives its branch up, keeping its finalizer.spec.knownHostsRef: optional SSH known hosts source.spec.commit.committer: committer identity (defaults toGitOps Reverser/noreply@configbutler.ai).spec.commit.signing: SSH signing key reference and optional key generation.status.signingPublicKey: populated when signing is configured and key material is available.
The controller verifies repository reachability and manages the signing key lifecycle. It generates an
ed25519 keypair when signing.generateWhenMissing is set. The portable artifact across GitOps
ecosystems is the credentials Secret, not a foreign repository object. The credentials reader accepts the
Kubernetes native, Flux, and Argo CD Secret key dialects (see
finished/git-credentials-interop.md).
- Source: api/v1alpha3/gitbranch_types.go
- Controller: internal/controller/gitbranch_controller.go
One branch of the repository a GitProvider names, holding it for every GitTarget that references
it. Every field but spec.allowedGitTargets is immutable:
spec.gitProviderRef: theGitProvider, in theGitBranch's own namespace unlessnamespacenames another. Itsspec.allowedGitBranchesmust admit this namespace and the branch (NamespaceNotAuthorized,BranchNotAllowed), checked before any push session.spec.branch: the branch it claims.spec.parentBranch: optional branch the write branch is created from and compared against while it does not exist. Omitted means the remote's default branch. An explicit parent must exist when creating the write branch (ParentBranchNotFound). The grants need not allow it, because it is only read.spec.allowedGitTargets: which namespaces'GitTargets may write through it. Omitted, its own namespace's; a written list replaces that default, and[]admits none.status.repository: the normalized repository URL, half of the key oneGitBranchholds.status.lastPushNoticeAt: when aGitReceiverlast delivered a push to it. It records receipt, so it is published even when the look the delivery caused could not read the remote.status.remote: where the branch is on the remote, and when that was last proved.status.gitTargetCount: how manyGitTargets reference it, from any namespace.
The controller proves the provider's credential by opening a receive-pack session (at most once per steady interval while it succeeds), keeps the branch claim, and publishes the worker's observation. See Git write architecture for the branch claim.
- Source: api/v1alpha3/gitreceiver_types.go
- Controller: internal/controller/gitreceiver_controller.go
- Handler: internal/webhook/receiver.go
The endpoint a Git host notifies on a push. No GitProvider names it: a delivery is routed by the
repository it names, to providers in any namespace.
spec.type:github,gitlab,giteaorgeneric-hmac, named after Flux'sReceiver.spec.secretRef: a Secret in the receiver's own namespace whosetokenkey is the webhook secret.status.webhookPath:/git-push/<hex HMAC of namespace/name, keyed by the token>, once the token Secret holds a token.ReadyisFalsewithListenerDisabledwhile the operator serves no listener.
A delivery is authenticated against the token before its body is read. The handler then reads the
repository URLs and the branch the payload names, compares the URLs with every GitProvider's
spec.url (normalized by internal/repourl), and wakes the branch worker writing that branch for
each one that matches, once. The worker reads the remote with its own credential, so nothing in the
payload chooses a URL, a credential or a commit, and it holds at most one pending look, so a run of
deliveries costs one ref advertisement at a time and never crowds writes out of the queue. It does
not bound the rate: deliveries arriving during successive reads keep a worker reading
continuously. That availability cost of letting any namespace's receiver wake any matching worker
is accepted, with no per-worker cooldown (see the security model).
The 202 carries no body: a count of what was woken would tell the token holder whether another
namespace tracks a repository.
- Source: api/v1alpha3/kubeprovider_types.go
- Controller: internal/controller/kubeprovider_controller.go
KubeProvider is the Kubernetes-side peer of GitProvider: a connection to one cluster and the
identity used there. What it's used for depends on who names it: a GitTarget reads its source
cluster through it, and an applier claim reaches its Kustomization or Application through it,
often in a management cluster. A GitTarget, FluxClaim or ArgoCDClaim references it by the
required, immutable spec.kubeProviderRef; an omitted namespace means the
referrer's own, and nothing falls back to a default provider. Its credential is exactly one of two
immutable arms (delete and recreate the provider to repoint it):
spec.secretRef: a kubeconfig Secret in the provider's own namespace (keyvalue, thenvalue.yaml, unlesskeynames one). The kubeconfig safety checks refuse credentials named by file path, and exec auth and insecure TLS unless the operator opts in through its flags.spec.inCluster: {}: the operator's own identity, honored only for a provider listed in--in-cluster-kube-providers; any other isStalled=True(InClusterNotAuthorized) and never used.spec.clientis refused beside it, because the operator's own client is not throttled per provider.
Several namespaces may mirror through one provider, but its source identity must not vary by target
because it keys source clients, discovery, watches, and attribution. The identity is the provider's
namespace/name rather than a deduplicated physical-cluster identity, so two providers pointing at the
same API server still have separate contexts and authorization boundaries. A provider admits only
the namespaces its grants admit; omitted, the grants admit its own namespace, and a written list
replaces that default. The check runs on every reconcile
before watches start (NamespaceNotAuthorized), and a grant or Namespace label change triggers that
reconcile, so tightening it also stops an already-existing GitTarget, which an admission-time check
could not do. It guards which tenant may cause the operator to export a source; it does not expand that source
credential's Kubernetes permissions.
Its audit route, spec.attribution.auditRoute, defaults to <namespace>.<name> and is held by one
namespace: the namespace the route's first dot-segment names when a provider there carries it,
otherwise the namespace of the oldest carrier. Providers in the holding namespace may share a route
(several connections to one API server); one in another namespace reports Stalled=True
(AuditRouteConflict) and is not used. AuditFactsReceived reports whether the route has ever
delivered a fact: Unknown until one arrives, and it is never part of Ready, because a quiet
cluster and a broken audit policy look the same from here
(configuration.md). The KubeProvider
validates its connection inputs, while a GitTarget projects provider readiness and live source
reachability.
Credentials are re-resolved on the catalog-refresh cadence, not on every watch reconnect, inCluster
ones included. A rotated Secret, a spec.client change, or a provider deleted and recreated under the
same name rebuilds that provider's clients and restarts only the watches using that provider; a
definitively invalid or missing credential, or an inCluster provider the installation stops
listing, fails closed and stops those watches. An unreachable
remote similarly makes only its dependent targets unready; it does not block discovery or watches for
other source contexts.
Say a GitTarget in control-cluster namespace team-a watches ConfigMaps, and a user runs kubectl apply
to edit the ConfigMap team-a/app-config in that target's selected source cluster. Here is the path that
change takes.
flowchart LR
subgraph CONTROL["Control plane: operator cluster"]
CP[KubeProvider]
KCFG[("Remote kubeconfig Secret\nprovider's namespace")]
OWN["GitTarget watch set\nclaimed ∩ followable (GVR, scope)"]
RELEVANCE["Relevance filter\nsanitize · followability · no-op diff"]
RESOLVE["Resolver\nbounded grace window"]
GTES[GitTargetEventStream]
AFACTS["Audit fact extractor"]
ASTREAM[("Per-type fact stream\nRedis Streams or in-process ring")]
AINDEX[("In-process fact index\nbounded, TTL, keyed for join")]
end
subgraph SOURCE["Source cluster selected by KubeProvider\n(the control cluster for spec.inCluster)"]
WATCH["WATCH + sendInitialEvents replay<br/>(per claimed GVR + scope)"]
DISC["Discovery: CRDs / APIServices"]
AUDIT["/audit-webhook/<route> (optional)"]
end
subgraph GIT["Git writes: internal/git"]
BW[BranchWorker]
PLAN["Manifest aware plan/flush"]
PUSH["Atomic push"]
end
CP -. "secretRef" .-> KCFG
CP -->|"source client"| DISC
CP -->|"source client"| WATCH
DISC --> OWN
WATCH --> OWN
OWN --> RELEVANCE
RELEVANCE --> RESOLVE
RESOLVE --> GTES
GTES --> BW
BW --> PLAN
PLAN --> PUSH
AUDIT -. configured .-> AFACTS
AFACTS -->|append| ASTREAM
ASTREAM -->|follow| AINDEX
AINDEX -. lookup .-> RESOLVE
Following the ConfigMap edit:
- Watch delivers it. The API server applies the change and bumps the object's
resourceVersion. TheGitTarget's watch oncore/configmapsinteam-adelivers aMODIFIEDevent carrying the new object body. (On a cold start or after410 Gone, the same object instead arrives as anADDEDduring thesendInitialEventsreplay.) - Desired-state change filter. The event is sanitized (status, managedFields, and volatile
metadata stripped) and compared with what the branch worker last accepted for this object on this
stream, from a live event or the stream's replay. A no-op (e.g. a
*/statusbump whose desired-state projection is unchanged) is dropped here. - Resolve the author. When attribution is enabled, the resolver waits a bounded grace window
(
--author-attribution-grace, default3s) for a matching audit fact in the attribution index, joining by resourceVersion/UID. On a strong match the real user or named service account becomes the author. With attribution disabled, the configured committer is the author; with attribution enabled but no usable fact, the explicitunknown (attribution unresolved)author marks the unresolved change. This wait is per-event and never reorders a watch. See Watch Event Ordering. - Route + window. The event flows into the
GitTargetEventStream, and the
BranchWorker appends it to the open commit window for
(author, GitTarget). - Commit + push. When the window closes (5s of silence, a
CommitRequest, or a buffer limit), the manifest aware writer patchesteam-a/.../app-config.yamlin place, commits as the resolved author, and pushes via PushAtomic (retrying with fetch/reset/replay if the remote moved).
Separately, the audit path (only when attribution is enabled): kube-apiserver POSTs audit events to
/audit-webhook/<audit-route>; AuditHandler extracts the minimal
attribution facts and appends them to that route's per-type fact stream. A follower on the watch side
reads the streams for the types it watches into a bounded, TTL'd in-process index. That index is read
only by the resolver in step 3; it never creates or repairs object state.
And if the watch had been lost? A reconnect can recover a retained deletion event from its
resume RV. When a fresh snapshot is needed, reconciliation can infer absence within its observed
collection; Always permits that cleanup, while Never and OnEvent retain snapshot orphans.
See State Ingestion and Not Losing Deletes.
Most writes are closed by the commit window timer. A CommitRequest is the explicit "save now" path: it
does not create new state, and it does not bypass watch. It only asks the branch worker to close a matching
open window early.
flowchart TD
CHANGE[Persisted Kubernetes change] --> WATCH[Per-GitTarget watch event]
WATCH --> FILTER[Sanitize + relevance filter]
FILTER --> AUTHOR{Attribution enabled?}
AUTHOR -->|yes| AUDIT[Wait bounded grace for audit fact]
AUTHOR -->|no| COMMITTER[Use configured committer as author]
AUDIT -->|fact matched| RESOLVED[Use resolved actor]
AUDIT -->|no usable fact| UNRESOLVED[Use explicit unresolved author]
COMMITTER --> WINDOW[BranchWorker open window<br/>one author + one GitTarget]
RESOLVED --> WINDOW
UNRESOLVED --> WINDOW
WINDOW --> TIMER{What closes it?}
TIMER -->|silence timer| BASE
TIMER -->|buffer limit or resync boundary| BASE
CR[CommitRequest persisted] --> AUTHOR[Author from its audit fact<br/>named or unnamed]
AUTHOR --> ATTACH[Attach to matching open window]
ATTACH -->|same author + GitTarget| BASE
ATTACH -->|no matching window| BENIGN[Ready=True, Pushed=False]
BASE{Base trusted<br/>and worktree clean?} -->|yes| FLUSH
BASE -->|no| FETCH[Fetch and reset first]
FETCH --> FLUSH[Plan YAML edits + commit]
FLUSH --> PUSH[PushAtomic<br/>reads the advertisement]
PUSH --> DONE[Remote branch updated]
The important boundary is the open window: it accepts only the same author and GitTarget. A
CommitRequest from another author never closes someone else's window; it resolves as a benign no-commit
instead.
If someone else pushes to the same remote branch before GitOps Reverser's push lands, or, for a branch not created yet, if its parent branch moved, the operator does not treat its local clone as authoritative. It keeps the finalized pending writes, fetches the new remote tip, resets the local clone, replays those writes, and pushes again.
flowchart TD
LOCAL[Window finalized<br/>pending write retained] --> CHECK[PushAtomic checks remote ref]
CHECK --> MATCH{Write branch, or while it is absent its parent,<br/>still at the expected commit?}
MATCH -->|yes| ACCEPT[Push accepted]
ACCEPT --> CLEAR[Clear retained pending writes]
MATCH -->|no| FETCH[Fetch latest remote tip]
FETCH --> RESET[Hard reset local clone to remote]
RESET --> REPLAY[Apply each retained pending write's<br/>decided actions onto the new tip]
REPLAY --> REFRESH[Create fresh commit hashes]
REFRESH --> CHECK
CHECK -->|attempts exhausted or fetch fails| RETAIN[Keep pending writes for later retry]
For a CommitRequest, success is reported only after the push reaches the remote. If the push had to
replay on top of someone else's commit, status.commit is the refreshed post-replay commit, not the stale
local commit from before the retry.
A rejected push is one of four ways the branch worker learns the branch gained a commit it did
not push. The others are a push-receiver delivery, the periodic refresh, and a Flux
GitRepository that fetched another commit than the one an apply asked for, which the worker
takes as a hint and checks the way it checks a delivery. All four set the same piece of worker
state, the applier is behind, and the worker clears it by sending an apply request after a
flush.
flowchart TD
HINT["Receiver delivery, or a source<br/>that fetched another commit"] --> ADV["Read the ref advertisement"]
REJECT["Rejected compare-and-swap push"] --> SEEN
REFRESH["Periodic refresh fetch"] --> SEEN
ADV --> OURS{"Tip is a commit<br/>we pushed?"}
OURS -->|yes| DROP["Nothing to do"]
OURS -->|"no, or cannot tell"| SEEN["Applier is behind"]
SEEN --> PENDING{"Open window or<br/>pending writes?"}
PENDING -->|yes| FLUSH["Close the window, publish the pending writes at once:<br/>a rejected push fetches and replays them onto the foreign commit"]
FLUSH --> FLUSHED
PENDING -->|no| FLUSHED["Flushed: every write decided<br/>before the apply came due is published"]
FLUSHED --> REQ["Apply request to every applier on the branch,<br/>naming the tip, reason ForeignCommit"]
style DROP fill:#eeeeee,stroke:#9e9e9e
style REQ fill:#fff3e0,stroke:#ff9800
A delivery is handled on the worker's own queue, so it is never judged while a push is in flight:
the push has recorded its commit before the delivery is read. A rollback onto one of the worker's
older commits is foreign, because the advertisement names a tip other than the last one observed.
A delivery also closes the open commit window, unless a CommitRequest holds it, so its
publication finds the move.
A Git-side push usually reaches the worker through the flush itself: the push of the pending writes is rejected, the fetch finds a commit the worker did not push, the pending writes' decided actions are applied onto it, and the applier is asked once, for the tip that holds both sides. Applying the foreign commit on its own first would revert the pending API edits on the cluster and need a second apply to put them back.
The same flush comes before three other requests: Refusal, to the refusing target's applier
when spec.onRefusal: Reapply is set; Repair, to an applier whose late write over a newer API
edit the apply window withheld; and CatchUp, to a Kustomization or an Application alike, to
adopt an object the API created in a folder that applier applies (a live ADDED event whose newest
managers are not an applier's, or an object a resync added to Git) and to converge, once when the
worker starts and once when a GitTarget first names the applier claim. A converge of the commit the
applier last applied asks nothing; an adopt is sent even then, because applying the same commit
again is what stamps the owner labels or tracking annotation on the new object. A push of the
worker's own leaves nothing pending.
Three states still hold an apply back, because in them no flush can be made: a registered
CommitRequest (a save defines its own commit boundary, and the apply waits for it), a parent
recovery, and a checkout a failed write left dirty. An apply also waits until every initial
snapshot of the GitTargets its applier covers is published (a GitTarget with no planned
collection yet, or with a blocked watch, holds its applier the same way), and an empty tip (a
remote that does not carry the branch) asks nothing until there is a commit to apply.
One worker writes a repository branch, so the flush has every unpublished edit on the branch in
hand, whichever namespace its GitTarget is in.
Which apply is due lives on the worker, per applier: one attempt out at a time, and its own retry
clock. The dispatcher in internal/applier carries out one attempt off the worker's queue and
reports back. It reads each GitTarget again first (the same incarnation by uid, still naming this
applier through its applier claim on this branch, still admitted, still consenting to a Refusal), checks that the
Kustomization's source fetches this repository and branch, has the GitRepository fetch the
commit when its artifact lacks it, asks the worker whether its decision still holds (the commit is
still the tip), writes the operation through to the applier claim's
status.operation, and only then resumes the Kustomization the applier claim holds suspended, with a
requestedAt token. It follows the apply until that token is handled, suspends the Kustomization
again with a second token, and reports the operation complete once that one is handled
(apply-trigger.md). For an Argo CD
Application, whose automated sync the claim holds off, the dispatcher checks that its single
spec.source is this repository and branch, then sets the Application's operation to a sync
pinned to that commit, never while another operation is set; the operation is complete when Argo CD
clears it (the Argo CD applier). It never retries:
an attempt that does not deliver gives the worker back what it carried, a refusal only while it still
stands, and the worker decides the retry after a flush, once that applier's backoff has run out. A delivered
request records an ApplyRequested Event, and the worker records the operation on the applier claim's
status.operation. A failed request never reaches back
into the publication that caused it.
A GitTarget owns one subtree (spec.path) on one branch. A new object follows the layout the folder
already has (or a declared policy); once a document exists it is edited in place wherever it already
lives. A populated target seeded from empty looks like:
team-a-config/ # GitTarget spec.path
├── README.md # operator-managed bootstrap file
├── .gittargetignore # operator-managed bootstrap file
├── .sops.yaml # present only when encryption is configured
├── team-a/
│ ├── configmaps/app-config.yaml
│ ├── secrets/db-creds.sops.yaml # sensitive types are SOPS/age encrypted
│ └── apps/deployments/api.yaml
The built-in default path is {spec.path}/{namespace}/{group}/{resource}/{name}.yaml, namespace
first, the API group omitted for core resources, no version segment, and a .sops.yaml suffix for
sensitive resources; a cluster-scoped resource uses the literal _cluster/ in place of the namespace
(an illegal Kubernetes namespace name, so it can never clash with a real one).
That default is what a new resource gets unless something more specific applies: a GitTarget's own
declared placement policy, or a folder that kustomize builds from a single root. Details are in
File Placement.
Object state arrives through Kubernetes watches. Selected, relevant changes feed the writer; recovery establishes current state when event history is unavailable. Git commit grouping and pruning policy determine which intermediate states and removals appear in the repository.
One watch delivers live changes in order. Independent watch handlers can still interleave observations of the same object; the ordering contract states that boundary. The manager derives resource identity and operation from the source event, including deletion intent, then sanitizes content before routing it.
Initial events establish current state, so their object RVs need not arrive in mutation order.
The annotated initial-events-end bookmark marks completion. Ordinary bookmarks report progress,
and an error such as 410 Gone requires rebuilding from current state.
The first attempt after process startup or watch replacement requests initial state, even when
Redis holds a cursor. runTargetWatch passes resumeFromCursor=false until this watch's own
replay has completed: the new render-fidelity revision needs fresh observation. Attempts after that,
in the same loop, may open a normal watch from the stored RV. If history has expired, or no
usable cursor exists, initialization establishes
current state again.
Streaming initialization uses sendInitialEvents=true and resourceVersionMatch=NotOlderThan:
- The manager collects current objects until the annotated initial-events-end bookmark.
- It enqueues a snapshot reconciliation for that observed collection. Managed documents absent
from the desired set are removed only when
prune.mode: Alwayspermits inferred deletion;NeverandOnEventretain them. Terminating objects are excluded from desired state by the deletion-as-intent rule. - Subsequent live events enter the branch worker after the accepted snapshot request. This queue order does not mean the snapshot's Git write has already completed.
Reconciliation edits managed documents, which can share a YAML file. A snapshot for one namespace cannot establish absence in another. A failed observation cannot establish absence anywhere. See Mark and Sweep Resync.
There is no periodic object LIST or timer-driven object-drift sweep. If the API server rejects
streaming initialization, targetWatchListAndStream first opens and buffers a watch, then takes
a LIST snapshot. It enqueues the same scoped reconciliation and processes buffered events newer
than the snapshot's RV. This ordering is the compatibility path for APIs without initial events.
Watch has no audit policy, so it delivers every persisted MODIFIED, including controller status churn.
The relevance filter reproduces that filter in product code, on the hot path:
- Sanitization (internal/sanitize) strips status, managedFields, and volatile metadata before diffing, so runtime churn never masquerades as a desired-state change.
- Followability (internal/typeset) encodes "controller-owned → don't mirror". A type that is not followable never gets a watch.
- No-op suppression. A
*/statuswrite bumpsresourceVersionbut its sanitized projection equals the prior commit; the writer diffs it to a no-op and discards it.
Watch carries only the versions it observes. While connected it sees each MODIFIED; across a replay
after 410, a compaction, or downtime it collapses every intermediate version into current state.
The product is therefore a state mirror with opportunistic per-mutation history, not a guaranteed
per-mutation change log.
- Handler / fact extractor: internal/webhook/audit_handler.go
- Fact transport (per-type stream): internal/queue/fact_stream.go
- In-process fact index: internal/queue/fact_index.go
- Resolver (grace window join): internal/watch/author_resolver.go
- Spec: spec/attribution.md (the single reference)
- Design record: attribution-fact-stream.md
Attribution runs only when --author-attribution=true. A normal
source posts audit EventList payloads to /audit-webhook/<audit-route>, where the route is
KubeProvider.spec.attribution.auditRoute and defaults to <namespace>.<name>. The bare
/audit-webhook endpoint is enabled only with --author-attribution-audit-route-annotation-key, for a
trusted control plane that puts an audit route in each event. There is no supplementary body endpoint or
body joiner, because watch (not audit) carries the object body. The handler applies an intrinsic accept
gate (StageResponseComplete, a mutating verb, success, non-dry-run, a changed resourceVersion, and the
/scale subresource only), then appends the accepted events' facts to that route's per-type fact
stream: one append per type per request, not one per event.
The two halves never call each other. The receiver publishes and returns; the watch side follows the streams for the types it watches and keeps their facts in a bounded, TTL'd in-process index. They meet only through the keys a fact was filed under. That split is what lets the audit endpoint answer fast during a rollout, and what lets a fact published by one replica serve a watch running on another once the streams are shared (the Redis transport below).
The transport is selectable, because the index is what does the work and the stream is only how facts travel:
--author-attribution-transport |
Store | When |
|---|---|---|
redis (default) |
Redis Streams, per type, with a retention window a restarting process replays | any install; required for more than one replica |
memory |
an in-process ring | single replica, no Redis; every fact is lost on restart, by design |
memory removes attribution's need for Redis. It does not on its own remove the install's:
--redis-addr also holds watch resume cursors, independently of attribution. Without that cursor
store, reconnect attempts initialize from current state. The first attempt after a process restart
requests initial state with either configuration.
| Endpoint | Role |
|---|---|
/audit-webhook/<audit-route> |
One audit route's stream; some KubeProvider must carry that route |
/audit-webhook |
Shared stream only when annotation routing is configured; each event names its route |
The handler accepts a client certificate signed by the audit CA, but it does not yet bind that certificate to a named provider. Do not treat named routes as an isolation boundary for independently administered remote sources that share a client credential; provider-bound ingress authentication remains outstanding.
Attribution being optional does not make it casual or take-it-or-leave-it. When it is enabled the operator does everything it can to name the real actor, but it values high certainty over a plausible guess. Attaching a name to a change is a consequential, sometimes politically charged claim ("this person did that"), so it must not be made lightly. The design therefore fails toward honesty: a weak, conflicting, late, or absent fact produces an explicit unresolved author rather than a guessed person or a misleading committer identity. It is better to say that attribution did not resolve than to assert an actor we are not sure of. In an installation that expects live mutations to be attributable, this outcome is a concrete signal to check audit delivery and attribution configuration.
Two engineering choices follow directly from that stance:
- mTLS on the audit ingress is on by default. The audit server requires and verifies a client
certificate (
tls.RequireAndVerifyClientCertagainst a configured CA;--audit-insecuredefaults tofalse). This authenticates membership in that CA's client set. It is not yet a sender-to-provider binding, so multi-source deployments must not share a credential across independently trusted sources. - Tests pin the behavior. Because a misattribution is a real harm, the attribution and resolver paths carry unit and e2e tests that prove the concrete cases: strong match, weak/last-key match, deletes whose audit RV differs from the watch RV, a collection delete joined by uid set and by scope, an aggregated API whose facts carry only a name, missing/late/expired facts, service-account vs human actor, impersonation, and the explicit unresolved outcome, so these certainty guarantees cannot silently regress.
The fact is the smallest thing needed to name an author, not an object log:
| Field | Purpose |
|---|---|
auditID |
diagnostics / dedupe |
user / impersonatedUser |
author candidate (human or service account) |
verb, subresource |
explain the write |
responseStatus.code, dryRun |
reject failures and non-persistent requests (at the handler gate) |
| source provider, GVR, namespace, name, UID | source partition plus exact join keys |
| response object resourceVersion | exact watch-event match |
| stage timestamp | recency |
A deletecollection produces one collection fact instead: it names no object, so it keeps the
request's selector and whatever uid set the API server returned, and every removal in its scope joins
against it.
The index files each fact under the strongest key it has, first match wins, within a scope of
(audit route, group/resource): (uid, rv) and (uid) together when it has both, else (uid),
else (rv), else (namespace, name). A fact keeps every field it recovered but is not filed under
weaker keys it could never be read by. A watch event always knows its object's uid, so a uid-keyed
fact is never looked up by name, and a second copy would cost memory on every replica for the whole
TTL and answer nothing. The one branch that files twice is (uid, rv) plus (uid): the first serves
creates and updates, the second serves removals, one fact answering two different questions.
Entries carry a short TTL (minutes, not hours) and are bounded per type and in total; expiry is checked on read, so an aged-out fact is never joined merely because the sweep has not run. Old facts are never needed for correctness, because watch owns state.
A watch event waits a bounded grace window (--author-attribution-grace, default 3s) for
matching evidence, then ships regardless. It does not poll: it registers a waiter under every key its
query could match, looks once, and then sleeps until either a fact that could answer it is applied or
the grace expires. Registering before the first lookup is what closes the race a fact landing in the
gap would otherwise win. There is no Redis call on this path: the fast case is a map read.
The resolver takes the strongest evidence available, and the tier it used is the tier label on
attribution_resolutions_total. Who that evidence named is the separate actor_kind label, in the
same user/serviceaccount/none vocabulary git_commits_total{author_kind} uses:
| Tier | Key | What it asserts |
|---|---|---|
delete_sticky |
uid (a delete fact, sticky) | who asked for this object's deletion |
exact |
uid + rv | this actor produced this exact version |
deletecollection_body_uid |
uid in a collection fact's set | the API server said this request deleted this object |
latest |
uid (the object's own delete fact) | who removed it, when the sticky slot was evicted |
name |
namespace + name | the same, for a fact with no uid (an aggregated API's usual shape) |
deletecollection_scope |
namespace + selector + window | a collection request covering it was made |
resource_version |
rv (the escape hatch for a fact with no uid) | a fact for this exact version, unidentified |
absent |
none | nothing usable arrived in time |
Three rules carry most of the behavior. A fact about a deletion may not be replaced by a fact about
a write: a removal fact takes a sticky, uid-keyed slot that the ordinary last-writer-wins
structures cannot reach, and a removal of a terminating object asks that slot first. That slot is
the one structure the fact TTL does not bound: a uid is unique across space and time, so the
statement cannot be superseded, and its horizon is the index's caps instead. It is still in-memory:
a restart re-warms the index from one TTL of stream retention like everything else. Without the
slot, a finalizer patch's fact overwrites the deleter's, because both carry the resourceVersion the
deletion stamped (spec/attribution.md). A removal is credited only to
the request that caused it: a write at its own resourceVersion (an object relabeled out of a
rule's objectSelector) or a fact about the deletion. The object's last edit says nothing about
who removed it, so it never answers, and a removal without such a fact is unresolved. None of this
reads the watch's label selector. And an exact-capable event may not fall through to the removal
tiers: a create or update presents the resourceVersion its own write produced, so if the exact
tier misses, the uid pointer may name an older, different author.
No branch on this path depends on the type. Every decision is made on the verb, on whether the event is a removal, and on which fields the event happens to carry. That is why an aggregated API needed no special case to attribute.
On a resolved tier the actor becomes the author: a human or a service account alike, always named by
its own username (e.g. system:serviceaccount:flux-system:kustomize-controller). An absent
resolution produces unknown (attribution unresolved) <attribution-unresolved@gitops-reverser.invalid>
instead of a guessed actor. A late fact that arrives after a commit has shipped never rewrites it.
With attribution disabled the resolver is absent and every commit is committer-authored.
The wait is head-of-line on its watch shard, so a resolution that sits out the grace delays the events
queued behind it on the same (GitTarget, GVR, scope) goroutine. That cost, and what it once broke,
is in Watch Event Ordering and
attribution-removal-wait-options.md.
A CommitRequest's author is read from this index too, on the route of its GitTarget's KubeProvider,
with the same grace (see CommitRequest Finalize).
Git natively carries two identities on every commit, the committer (who created the commit object) and the author (who wrote the change), and GitOps Reverser uses both on purpose (commit.go):
-
The committer is always the operator: the configured
GitProvider.spec.commit.committer, defaulting toGitOps Reverser <noreply@configbutler.ai>. Every commit, attributed or not, is committed by the operator, because the operator is what wrote it to Git. -
The author is the real actor, but only when we are sure. On a strong attribution match the author is set to that actor. Git always carries an author, so it is never left blank, and it is never a guessed person. What fills it when we are not sure depends on WHY:
- Attribution is switched off (configured-author mode, and reconcile/resync writes that have no actor at all): the author is the operator, identical to the committer. That is honest, because the operator is the author.
- Attribution ran and did not resolve an actor: the author is the explicit sentinel
unknown (attribution unresolved) <attribution-unresolved@gitops-reverser.invalid>. It is deliberately NOT the operator, because authoring it as the operator made a lost actor byte-identical to a configured-author commit, so the gap was invisible in Git history and only countable in a metric.
A commit whose author is a person is therefore a positive statement that the operator is confident who made the change; one authored by the sentinel is a positive statement that it tried and could not tell.
The author identity is never fabricated as a person: a real author is always taken from the authenticated
request, and an unresolved one is labeled as unresolved rather than attributed to anybody. The name and email come
from the OIDC claims when the apiserver maps them; otherwise they come from the actor's own
Kubernetes identity: the username, which for a controller is its service account
(system:serviceaccount:<namespace>:<name>), with a stable derived email under noreply.cluster.local.
The two user.extra keys the operator reads for the OIDC name and email, and the apiserver config that
fills them, are in Wiring OIDC author claims.
A confidently attributed commit shows distinct author and committer lines, which git surfaces with
--format=fuller. A human carries the OIDC display name and email:
$ git show --no-patch --format=fuller HEAD
Author: alice <alice@example.com>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit: GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000A service-account actor (no OIDC claims) is named by its own Kubernetes identity, with a complete derived email and the operator still the committer:
$ git show --no-patch --format=fuller HEAD
Author: system:serviceaccount:flux-system:kustomize-controller <systemserviceaccountflux-systemkustomize-controller@noreply.cluster.local>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit: GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000When attribution is off, or when a reconcile/resync write has no actor because attribution was never attempted, the author is set to the operator as well, so both lines are identical. A Git UI will show this as a single "author". This is configured-author mode: the operator created the Git commit and makes no claim about a Kubernetes actor.
$ git show --no-patch --format=fuller HEAD
Author: GitOps Reverser <noreply@configbutler.ai>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit: GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000- Audit-path extractor: internal/queue/audit_event_parsing.go
- Signature construction: internal/git/commit.go (
authorName/authorEmail)
The operator never talks to the identity provider. It reads two fixed keys from the request's user.extra
map, from the audit event that names the actor:
user.extra key |
OIDC claim it carries | Used for |
|---|---|---|
configbutler.ai/claims/display-name |
name |
git author Name |
configbutler.ai/claims/email |
email |
git author Email |
So the ID token the provider issues must carry the standard name and email claims:
{
"iss": "https://idp.example.com",
"sub": "248289761001",
"name": "alice",
"email": "alice@example.com",
"aud": "kubernetes",
"exp": 1751284805
}The cluster operator maps those claims into the two keys with a structured
AuthenticationConfiguration
(apiserver.config.k8s.io, beta from Kubernetes 1.30):
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://idp.example.com
audiences: [kubernetes]
claimMappings:
username:
claim: email # the Kubernetes username; `sub` also works
extra:
- key: "configbutler.ai/claims/display-name"
valueExpression: "claims.name"
- key: "configbutler.ai/claims/email"
valueExpression: "claims.email"Both mappings are optional: a missing or unusable name falls back to the username, and a missing or
invalid email falls back to a derived address under noreply.cluster.local.
The grace window is a per-event wait on a single-threaded watch goroutine. Each running
collection watch uses one chosen GVR and namespace scope. The downstream
GitTargetEventStream → BranchWorker path preserves enqueue order in a FIFO.
- Order within one watch is preserved. The loop resolves attribution and routes one event before reading the next. Later events on that watch cannot overtake the attribution wait.
- Independent watches can overlap. An all-namespace watch and a named-namespace watch can
deliver the same object. Their concurrent handlers establish no common mutation order, and
the worker's FIFO does not sort their arrivals by
resourceVersion. The target watch plan records this limitation. - The wait affects throughput and grouping. Each unresolved event can delay its watch by the grace window. Accumulated queueing can add further delay, and different authors can split commits. Intermediate object states can also collapse into one commit window.
- Branch writes remain serialized. Different objects can share a multi-document YAML file; the branch worker edits their documents sequentially. That serialization prevents concurrent file edits but does not establish source ordering across overlapping watches. RVs also do not provide a common clock across resource types.
The current processing model and worked examples are in Watch event ordering under the attribution grace window. For server-side selectors, initial state, bookmarks, and resume options, see Kubernetes watch options.
How a user's WatchRule becomes "this GitTarget follows these concrete types in these namespaces."
- Source: internal/rulestore/store.go
An in memory cache populated through the shared compile paths (watch.CompileWatchRule /
watch.CompileClusterWatchRule) that both the controllers and the startup bootstrap call, so a
restart cannot seed an ungated rule. Compiled rules carry the full chain from rule to GitTarget,
GitProvider, branch, and path, plus each WatchRule item's RESOLVED source-namespace set. It is
read by the watch manager (to build watched type tables for GitTargets) and the rule change
reconcile path.
A thin normalizer for each scan: it turns one discovery result into a policy annotated typeset.Scan and
keeps only mechanical bookkeeping. All judgement across scans lives in the typeset registry
(Registry.UpdateFromScan): a failed group/version keeps serving last known facts instead of looking
like an empty API surface, and a group/version that vanishes from a complete scan rides a removal grace
rather than being pruned. Both protect against accidental Git deletions on a discovery blink (see
typeset-owns-discovery-grace.md). Every active source context has
its own catalog and registry, so a CRD served only by prod-eu-1 is never resolved for a target mirroring
prod-us-1. Catalogs refresh at startup and on the 30-second reconciliation cadence. The
CustomResourceDefinition and APIService informers run only in the control plane, where the
operator's own CRDs live; a remote source's API-surface change is discovered on that periodic
refresh. A target's plan pass never calls discovery
itself; for a source nothing has observed yet, it asks for a refresh at once.
- Source: internal/typeset/
- Design: type-followability.md
internal/typeset is the single decision surface for "can this type be followed?" Each TypeRecord
carries GVK/GVR identity, scope and preferred version facts, origin classification, subresource facts
(including usable /scale bindings), sensitivity policy, and one Followability verdict. Manifest
analysis and delete/scale resolution with only GVR all read it. Demand is applied one layer up: the
watched type table intersects the followable set with what each GitTarget claims, and only that
intersection gets a watch.
A projection for each GitTarget from the type registry, filtered by that target's rules, recording
resolved GVK/GVR/scope plus namespace coverage. This is where rule matching effectively
happens: it resolves the set of (GVR, scope) a GitTarget claims, so the watch manager opens one
watch per claimed ∩ followable (GVR, scope) and scopes each watch's events back to that GitTarget's
source namespaces. A sourceNamespace: "*" rule resolves here to ONE cluster-wide collection (the
empty namespace, which for a namespaced GVR is the all-namespaces collection), so it has a single
stream and a single mark-and-sweep boundary however many namespaces it covers. That collection is a
peer of any named-namespace collection on the same type, never a replacement for one.
Desired state comes from one raw watch per (GitTarget, GVR, scope). Each event is sanitized, diffed
against current Git content, and applied. There is no separate per-type object store to reconstruct,
because Git already holds current state. This section is the authoritative contract; the
watch-first design record is historical context.
- Manager: internal/watch/manager.go
- Watch / replay / sweep: internal/watch/target_watch.go
- Author resolver (attribution join + grace window): internal/watch/author_resolver.go
- Event router: internal/watch/event_router.go
- Worker resync apply (mark-and-sweep): internal/git/resync_flush.go
The watch manager is a controller runtime Runnable (NeedLeaderElection). It owns type level
discovery, the per-GitTarget watch sets, and the watched type tables for GitTargets. Its object-state
intake is the watches themselves. Every active KubeProvider has an independent source context; its
catalog, registry, dynamic client, and reachability can fail or recover without borrowing facts from another
source. Discovery informers (on CustomResourceDefinition and APIService) run only in the control
plane and accelerate its catalog refresh; remote source catalogs use the periodic refresh.
On Start it bootstraps the RuleStore, refreshes the API catalog and TypeRegistry, resolves rule
selection, and builds the collection watch plan.
On each GitTarget reconcile, rule resolution combines selection with discovery and mirroring
eligibility. targetWatchStreams groups selected served versions by group/resource/namespace,
and chooses one version per collection. The plan diff keeps unaffected watches running, starts
new collections, and replaces a collection only when its served version changes.
Each managed watch has one goroutine. Its first attempt requests initial state, enqueues scoped snapshot reconciliation, and then routes every live object event through its desired-state change filter and author resolution. Later attempts may resume a cursor; see Recovery: resume, replay, or list plus mark-and-sweep.
There is no periodic object LIST or timer-driven object-drift sweep. Source-namespace selection
uses delegated rule configuration and the source credential's permissions; it does not require
reading source Namespace labels. Control-cluster Namespace labels can govern a Selector grant,
which is a separate provider-reference authorization boundary.
A rule or target change and an API-surface change arrive on two different paths.
A WatchRule / ClusterWatchRule / GitTarget change reaches a GitTarget through the GitTarget
controller, which Watches those objects (generation change predicates) and queues the affected
GitTarget again. On reconcile the GitTarget declares its watch set again, and the watch plane's
next plan pass resolves its claimed (GVR, scope) set against the source's current catalog; a type
a new rule starts watching gets a new watch opened (with a
sendInitialEvents replay) and a type no longer claimed has its watch closed.
A control-plane CustomResourceDefinition or APIService change reaches the same
re-resolution through the watch manager instead. Its informers wake a catalog refresh, which
re-resolves the watched type tables and brings the running watches into line. The GitTarget
controller does not watch those objects. A remote CustomResourceDefinition or APIService change
has no informer at all; the next periodic source-catalog refresh performs the same re-resolution.
The BranchWorker applies a reconcile by scanning the GitTarget subtree and building a manifest plan (this write side is shared with live writes):
- before anything is planned, a structure-only acceptance gate runs over the scanned subtree; if it
finds content the operator cannot safely manage: a kustomization using an unsupported feature
(
secretGeneratoror a hash-suffixed or mergingconfigMapGenerator/ inline or JSON6902 patches / components / helm / replacements / transformers / name(pre|suf)fix / remote bases) or malformedimages:/replicas:overrides, a duplicate manifest identity, an impure managed file, or a standalone non-KRM / invalid YAML; (a path-based strategic-mergepatches:entry is tolerated as read-only build context, and an overlay reading../../baseis rendered by reading that base, and neither refuses the folder) the whole apply is refused: nothing is committed,GitPathAccepted=False,Stalled=True, andReady=Falsewith reasonUnsupportedContentuntil a human cleans the path; - desired resources are upserted through the same content derived path as live writes;
- existing managed documents that are watched but absent from the desired set are deleted, under
spec.prune.mode: Alwaysonly; - in a coordinated folder, an object another applier owns (by Flux's owner labels or
Argo CD's tracking annotation, read before sanitize strips them) is neither upserted nor treated
as missing, so its document is left alone and counted in
apply_ownership_conflicts_total; - the operator's own build directives (
kustomization.yaml,.sops.yaml) and other allowlisted auxiliary YAML are retained, not materialised and not refused; - nothing is committed if the apply cannot complete safely.
A write-boundary refusal is a different thing and is scoped accordingly: the folder is
accepted, and one edit had nowhere to land in it. Only those can earn the apply request a target
opts into with
spec.onRefusal: Reapply, and only
when the edit is to a document the folder already holds. A folder-level refusal above is excluded
deliberately: reapplying Git cannot repair a folder, and only a human can.
The acceptance gate is structure-only on purpose: it never refuses on a discovery-derived followability fact (unwatched / out-of-scope), which can blink on a discovery wobble; only facts that are true from the path's structure alone block a GitTarget. The same gate runs on the live write path, so a refused path is never written into by a racing live event either. See unsupported-folder-refusal-plan.md.
A reconcile is collection scoped (ResyncScope): the sweep is restricted to the
(group, resource, namespace) the snapshot was gathered over, so anchoring one collection again
never disturbs another's manifests. The desired set for the sweep is the sendInitialEvents replay
(everything marked ADDED up to initial-events-end), so a delete is reconciled only after the
replay completes.
The snapshot decides once. The upserts and deletes its first commit decided are what a replay
onto a newer tip applies, and the sweep never runs again there. On a branch an applier applies, a
snapshot collected while Git was ahead of the cluster (a commit pushed and not applied yet),
collected across an apply, or arriving while Git is ahead is discarded instead of decided, and a
fresh one is asked for once Git is no longer ahead; it would otherwise revert the unapplied commit. A
GitTarget's initial snapshots are exempt, and after 5 minutes of discards the next snapshot is
used anyway. See
stale snapshots.
- Gate state, per collection: internal/watch/render_fidelity_gate.go
- Writer side and the hold: internal/git/render_fidelity_gate.go, internal/git/render_fidelity_hold.go
- Design: render-fidelity.md
The operator renders a folder with its own kustomize build, but Flux or Argo CD may apply it with
context the folder does not hold: a Flux postBuild.substitute value for a ${var} token, or an
Argo CD spec.source.kustomize override. Mirroring a live value over such a token would replace
the token in Git with whatever the applier substituted. So every GitTarget carries a gate:
each collection's replay compares the rendered ${...} values of the documents it governs with
the sanitized live objects, and the gate opens only once every collection of the target has
reported a clean replay under its current revision. The gate is per GitTarget, so one collection
that is still replaying closes live writes for all of them. RenderMatchesLive publishes it:
Unknown(Rechecking): a collection is new, restarted, or still replaying, which is every collection at operator start. A live event that arrives is judged at once and then held; an atomic write is dropped at once. When the gate opens, the held events are released in arrival order, before any later event, and committed with their authors. An applied snapshot of a collection supersedes the events of that collection held before it. The hold is bounded, at 10 minutes from the first held event and by the branch's retained-byte budget. Past either, the held events are dropped and a fresh snapshot is asked for, which commits the current state as the operator. A delete among the dropped events comes back only underprune.mode: Always.False(RenderDoesNotMatchLive): a rendered token disagrees with live. Live and atomic writes are dropped and the target isStalled. Scoped resyncs still run, so a fresh replay can measure the folder again. A repair in Git does not reopen the gate on its own; a replay of the target's collections that measures them clean does.
Every dropped write is counted in git_writes_dropped_total{reason} (render_fidelity_pending for
a hold that gave up, render_fidelity_diverged for a divergence). Nothing replays a dropped write.
Three objects meet at a write, and each answers a different question. Keeping them apart is what stops a fact being published by something that cannot actually prove it.
| Question | Proved by | Published on |
|---|---|---|
| Can this repository be reached with this credential? | the GitProvider controller's connectivity check |
GitProvider.status: Ready, lastVerifiedAt |
| Which repository is this? | the GitProvider object's identity (metadata.uid with spec.url) |
nothing: it is carried on the branch worker and compared, not reported |
May this GitBranch write branch B? |
the GitBranch controller, opening a push session with the GitProvider's credential |
the GitBranch's Ready (BranchConflict, PushSessionFailed) |
| Where is branch B on the remote, and when was that proved? | the branch worker, on its queue | GitBranch.status.remote, written at most once a minute |
| What is this repository being used for? | the GitBranches that name its GitProvider |
kubectl get gitbranch, and each one's status.gitTargetCount |
| Is this folder accepted, streaming and matching live? | the watch plane, per GitTarget |
the GitTarget's conditions and status.placement |
| Does the applier need to apply, and is now safe? | the branch worker, on its queue | the applier claim's status.operation and an ApplyRequested Event |
| Can the applier be asked at all? | the applier claim controller, reading the applier in its cluster | the applier claim's Configured, and the GitTarget's ApplierClaimReady |
A branch worker is keyed by the repository branch, normalized as internal/repourl compares
URLs, and writes for the GitBranch that holds it, with that GitBranch's GitProvider
credential, signing key and metric labels. The branch claim (internal/git/branch_owner.go) lives
in the operator's memory:
- A
GitBranchbecomes a candidate once its credential opens a receive-pack session on the remote. One that cannot claims nothing, so creating it first blocks nobody. - Among the candidates for one repository branch, the oldest by
creationTimestamp(then UID) holds it. - The holder stays a candidate through a later credential or network failure, so it keeps the branch and its pending writes; any other candidate whose proof fails stops being one.
- After a restart the claim is decided again by 1 and 2.
The GitTarget reconciler, the worker sweep and EnsureWorker all read the claim, so a GitBranch
that does not hold its branch gets no worker, however close together two were created. A worker
serves exactly the GitTargets whose spec.gitBranchRef names its GitBranch, which is how the
event router finds it with no read on the event path. A provider recreated under its name does not
move a live worker; it replaces it, and the replacement starts with no clone, no base trust and no
observation.
A GitBranch carries a finalizer. Deleting it releases the claim, then stops its worker, which
closes the worker's intake, pushes or refuses every pending write and settles every CommitRequest
it holds; only then does the finalizer go. Nothing deletes the branch on the remote.
GitBranch.status.remote is a branch fact, and it is worth being exact about that, because the
reverse reading is tempting and wrong. A fresh lastVerifiedAt says the branch tip was recently
proved. It does not say that any one target's folder was scanned, that its documents were
published, or that its streams are running. Those are the GitTarget conditions' job.
Delivery is not publication, and the split is deliberate. The worker wakes its GitBranch every
time it proves something, a push or a fetch. Publication is a status write (an etcd write that
invalidates every watcher's cached copy of the type), so the GitBranch writes the stanza at most
once a minute, and comes back when that floor lifts.
Reading or writing a branch's repository state is the branch worker's job, and the worker does it on
one FIFO queue, on one goroutine. Live events, resyncs, CommitRequest attaches, periodic
refreshes and push-receiver deliveries all enter the same queue and are served in order. Which
writes are decided and which of them are published is therefore a fact one goroutine holds, which
is what lets the worker flush before an apply request goes out. Nothing else opens a connection on a
branch's behalf, and nothing else touches its clone. That is what makes "the base is trusted" a
statement one goroutine can hold.
The GitProvider controller's connectivity check is the one thing outside that rule, and it is
outside it deliberately: it answers a question about the repository that has to be answerable when
no branch worker exists at all, including before any GitTarget has been created.
- Source: internal/git/branch_worker.go
- Worker manager: internal/git/worker_manager.go
BranchWorker owns a local clone and a single FIFO event loop for its repository branch. Events
accumulate in one open commit window, which accepts only one (author, GitTarget) pair at a time:
- same author + same GitTarget: append to the window;
- different author or GitTarget: finalize the current window first;
- repeated writes to the same Git path inside a window use last write wins.
The window finalizes when its idleTimeout passes with no new matching event, its maxDuration runs
out, the retained buffer reaches --branch-buffer-max-size (default 8Mi), an attach: Next
CommitRequest closes it, a resync, an atomic batch, a push-receiver delivery or shutdown arrives,
or an apply comes due for an applier on the branch. The timers
are the GitTarget's spec.commit.window until a CommitRequest attaches and replaces them. Successful local commits
are retained until a fixed push cooldown (5s) allows a push, which prevents remote push storms during
bursts. The one exception is an apply's flush: the writes decided before the apply came due are
pushed at once, cooldown or not, because the apply waits for them.
Local clones live under /tmp/gitops-reverser-workers/{GitBranch UID}: one directory per
GitBranch, named by its metadata.uid. A GitBranch pins its branch (spec.branch is immutable),
the branch claim gives a repository branch to one GitBranch at a time, and a UID is minted fresh for
every object, so no two live workers share a directory and a GitBranch recreated under its name starts
in a new one. The branch name is not part of the path, so its slashes cannot nest one checkout inside
another. The repository is the one thing the UID does not pin: a GitProvider recreated against
another URL keeps the GitBranch, so its replacement worker reuses the directory, after the old worker
has removed the clone in it. A worker never trusts a checkout it finds on disk: its first cycle repoints
origin, fetches and resets.
A retiring worker removes its own directory. The worker sweep, which runs every minute and after
every GitTarget delete, also removes every directory that no live worker uses and no existing
GitBranch names. That covers a GitBranch deleted while no worker ran for it, which happens across
a restart that keeps the disk. The root is an emptyDir by default, so the checkouts survive a
container restart but not a new pod. The
chart README shows how to
mount a volume there instead.
PushAtomic checks the remote ref before pushing. If the
remote diverged it smart fetches the latest tip, hard resets the local clone, replays the retained pending
writes against the fresh tip (refreshing commit hashes), and retries up to the attempt limit. This is valid
because every pending write is rebuilt from sanitized API state; nothing depends on locally edited files.
A write carries the actions its first commit decided, and a replay applies those and plans nothing
else. An object whose plan changed nothing is no action, so its old content can never revert what
the newer tip changed, and a write with no action left settles at once without a push. A resync
keeps the upserts that changed the tree and the documents its sweep removed: a replay plans each
upsert onto the new tree and removes each document if it is still there and the prune mode still
sweeps, and never sweeps again, so a document the newer tip added survives. A save is the exception:
a write carrying a CommitRequest keeps every object it captured, and the push decides its outcome.
See the pending-writes design.
A branch whose remote has been deleted takes the same path. The advertisement does not carry the branch at all, which is reported as a moved remote with a zero hash, so the replay re-roots on the remote's default branch and the retry re-creates the branch with the retained writes on top.
A branch that does not exist yet is created from its parent (the GitBranch's spec.parentBranch, or the
remote's default branch when that is omitted) only when a write has something to commit, so an
idle or in-sync target never creates it. Until then the worker compares the folder against the
parent. Periodic refresh, ten minutes by default, follows the
parent from the same advertisement and updates the idle view independently of the publication
check. The push records the parent and the commit it started from as its root, and treats a moved
parent like a moved branch: fetch, reset, replay, check again. A branch somebody else created in the
meantime is a moved remote too, so the replay builds on their commit instead of overwriting it; that
holds when the server refuses our upload too, because the fallback reads both refs from one
advertisement. A cycle planned on an empty repository that has since gained refs is refused at the
advertisement (RepositoryNotEmptyError) and replayed, never pushed as an orphan. A parent that is
missing, or a default branch that does not resolve, opens parent recovery on the worker: it
probes on its own backoff and publishes what it held back once the parent exists (see
docs/design/gittarget-parent-branch.md). Proved end to end by the e2e spec "New write branch
starts from its parent"; that publication alone finds a moved parent is proved by worker tests
with no refresher.
When the cycle fetches at all. Only the first commit of a cycle may fetch, and only when the base
is untrusted or the worktree is dirty (see the ground rule above); a healthy publishing target plans
straight onto its own last push. Once writes are retained the guard flips off entirely, because a
reset would destroy the local commits those writes already produced. Anything that needs a fresh
tree with work in hand therefore resets and replays rather than resetting alone, applying the
retained writes' decided actions onto the new tip. One function on the event loop does that,
materialize: every commit and push the loop makes runs it
first, and the loop's commit refuses a checkout that is not the projection of the retained writes.
A worktree a failed write left dirty, retained writes whose commits a reset discarded, a parent
change, and the snapshot a resync judges against all reach it.
What a reset has to leave behind, now that it is the only cleanup. A hard reset restores tracked
files and stops there, so a document a failed write created, and the placement directory it created
for it, both survive one. While every cycle fetched, the next one swept them up by accident; a cycle
that plans on a trusted base would instead commit them under whoever wrote next. So the reset
explicitly discards untracked and newly staged leftovers and prunes the directories that held them
(discardWorktreeLeftovers), and a write that fails while creating
directories removes the ones it made on its way out. Only then is worktreeDirty cleared.
A decided write is kept until a push lands. When a commit or a push cannot reach the remote, the
worker keeps every decided write pending (pending_writes_loop.go), with
its author, its message, and any save riding it, and tries again on one schedule: 10 seconds,
doubling to 5 minutes (retry.go). The retry does not wait for another
edit. Until it is due nothing else on the branch spends a connection: a write that needs one stays
pending, and a resync is answered with the failure the retry is waiting out and kept. Only a
failure of the write itself, such as a refused plan, drops an entry. Every call to a Git server is
bounded, two minutes per call and five per push cycle, so a server that stops answering cannot hold
the branch.
The kept work is bounded at intake (intake.go). While a retry is
pending, --branch-buffer-max-size counts everything accepted and not yet published: queued items,
the open window, the pending writes and waiting saves, each with a 1 KiB per-item charge.
The first payload that does not fit pauses the branch's intake until a push lands; room under the
budget again, or a remote that can be read but refuses the push, keeps it paused. A paused branch's
watch streams wait to be woken instead of reconnecting, and their cursors stay where the last
accepted event left them. Withdrawals, refreshes, and shutdown are lifecycle work and are never
paused. The budget counts serialized bytes; a queued object costs several times that in memory.
Every GitTarget on the branch reports the outage from one worker report
(publication.go): Ready=False, Reconciling=True,
Stalled=False under Progressing, naming since when, what failed, and whether intake is paused. A
held save says the same in its WaitingForPush message, and the git_retained_*,
git_intake_paused and git_next_retry_timestamp_seconds metrics describe the backlog. What this
recovers, and what watch history and a restart cannot, is the
recovery contract; the
event pipeline overview draws it.
A BranchWorker's queue (the open commit window's retained writes plus any local commits not yet pushed) lives only in process memory today, bounded as described above. Persisting it to Redis is planned, for two reasons:
- High availability. Multi-pod HA needs a durable, cross-pod write queue so a branch's accepted-but-unpushed work survives a failover: the pod that takes the branch-shard lease resumes that queue instead of starting from an empty buffer. This is part of the HA / GitTarget distribution plan.
- Crash safety that watch replay cannot guarantee. A crash drops the in-memory queue, and those writes
cannot be reliably re-derived by replaying the watch, because Kubernetes does not guarantee it can
replay events from every
resourceVersion. The apiserver compacts old versions, so resuming from a stored cursor can return410 Gone; recovery then falls back to a fullsendInitialEventsreplay plus mark-and-sweep, which rebuilds current state but collapses the intermediate versions (see History granularity). Persisting the queue lets accepted-but-unpushed writes survive a restart on their own, independent of whether the watch can resume from where it left off.
- Steady state: internal/git/plan_flush.go
- Resync apply: internal/git/resync_flush.go
- YAML editor: internal/git/manifestedit/
- Analyzer/planner: internal/manifestanalyzer/
- Object projection: internal/manifestreport/
For each commit the writer scans YAML files under the GitTarget path, builds a manifest store without bytes keyed by resource identity, resolves each event (or each desired resource, for resync) to one action, hydrates only touched files into buffers for the commit, and flushes only changed or deleted files.
- Upserts: if a managed document for the resource already exists, patch it in place (preserving siblings in a multi document file); if it is sensitive, encrypt the whole document again at its existing path; if no document exists, place a new file per File Placement (declared policy, then the folder's one kustomize root, then the built-in path).
- Kustomize override edit-through: a live value produced by a well-formed
images:orreplicas:entry in the document's kustomization chain is written back to that entry (comment-preserving, only fields the entry already declares); the source manifest keeps its bytes. Anything the inversion cannot express falls back to the plain in-place patch. See images-and-replicas edit-through design. - Deletes: use the manifest identity index, so a moved manifest can still be deleted even when it is not at the built-in path.
Placement runs only for a resource with no existing document in the target. Existing resources are match first: once a document exists in Git, updates and deletes use its current location, found by manifest identity rather than by path, instead of recomputing placement. So a change to how new files are placed never moves a file already in Git. A new resource is placed by the first of these that applies (internal/manifestanalyzer/placement.go, design):
- Declared policy (
spec.placement). AGitTargetcan declare abyTypemap (exact[group/]version/resource→ path template) plus adefaulttemplate, rendered from a small brace-variable path language ({namespace},{group},{resource},{name}, …). - The folder's one kustomize root. When the whole writable subtree is governed by exactly one
supported
kustomization.yaml, the file lands beside it and gets aresources:entry in the same commit. This step is a structural fact rather than a reading of the folder's conventions: the built-in path below is a tree aresources:graph cannot reach, so a file written there would never be rendered. Two supported kustomizations is ambiguous and declines. - Built-in fallback. Otherwise the built-in default
{spec.path}/{namespace}/{group}/{resource}/{name}.yaml: namespace-first, group omitted for core, no version,_cluster/for cluster-scoped,.sops.yamlfor sensitive.
The layout of the folder's other documents is not an input. Where a new file goes is decided by the
three rungs above and nothing else, so a human's edit to the repository never changes where the
operator writes next. A layout the ladder cannot derive is declared in spec.placement, and
gitopsreverser_placements_total{source="builtin"} names the target and type that needs the line.
Sensitivity is a write-safety classifier, not a placement input: whatever path is chosen, a sensitive
resource is written encrypted, is never appended to an existing file, and is never co-mingled with a
plaintext document. When those guarantees cannot be honoured (e.g. a bundling default would route a
sensitive resource into a shared file), the resource is refused fail-safe rather than written
unsafely: logged per-resource, counted in the resync summary as placementSkipped, and counted by
gitopsreverser_placement_refusals_total{reason}.
- Bootstrap (bootstrapped_repo_template.go): the
first write to a GitTarget path stages a
README.md, a.gittargetignoreand, when encryption is configured, a.sops.yamlwith age recipient rules. Existing files are preserved. For aGitTargetthat names an applier claim, a resync into a folder the branch does not hold yet commits these files even with no object to write, because the folder has to exist before it is applied: Argo CD retries a sync of a missing path for minutes, and Flux fails it. The bootstrap files are never deleted, so the folder outlives its last resource. - Encryption (encryption.go,
sops_encryptor.go,
sensitivity policy): core Secrets are sensitive by default;
--additional-sensitive-resourcesadds more. Sensitive resources are never written in plaintext. If encryption is required and unavailable, the write fails before any plaintext file is created. Encrypted output is cached by metadata + plaintext digest to avoid redundant SOPS work. - Signing (signing.go, sshsig/): commits use
OpenSSH signatures, with the key read from
GitProvider.spec.commit.signing.secretRefor generated when configured.
- Core: internal/controller/commitrequest_core.go
- Control-cluster path: internal/controller/commitrequest_controller.go
- Source-cluster path: internal/controller/commitrequest_server.go
A CommitRequest finalizes the open commit window for its GitTarget. It is picked up in one of two
ways: by the CommitRequestReconciler on the control cluster, or by the CommitRequestServer's watch
of a source cluster, through the KubeProvider whose spec.commitRequests reconciles it. Both hand it
to one core, which takes one step at a time, each decided again from the object as it is stored
(contract):
- The owner. The
GitTarget'sKubeProvideralone picks the path that serves the request: the control-cluster path for aninClusterprovider, that provider's watch for a kubeconfig. Any other path writes nothing butGitTargetNotFound, onceattachTimeouthas passed with noGitTarget, and only into an empty status. Every status write is conditional on the resourceVersion it was decided from. - Admission and the author. A grant in the
GitTarget'sspec.allowedCommitRequestsmust admit the request:consumerNamespaceson the control cluster,sourceNamespaceson a source cluster. The request's author comes from its own audit fact, on the route of theGitTarget'sKubeProvider, awaited with the attribution grace; a request no fact names, on a provider that has received audit facts, isAuthorUnresolvedand saves nothing. - The claim. The owner writes the first status,
status.authorandReady=Unknown(Saving), and only then attaches the request to the worker (AttachCommitRequest), which waits for a window untilregistration + attachTimeout. The worker binds it to an open window only when the author state and GitTarget match, first come first served, and replaces the window's timers with the request's own, so time spent waiting never shortens the collection. It never finalizes another author's window; a window carries at most one request. The owner reports the phase the worker reports:WaitingForWindow,CollectingWindow,WaitingForPush. - The window finalizes when the request's timers close it (or when it closes for any other reason). If a finalize closes an open window, the worker always schedules a push, so a window closed by an otherwise no-op resync is not stranded.
- Outcomes resolve on push and are reported as conditions: a pushed commit sets
Ready=True/Pushed=Truewithbranch/commit; a benign no-commit setsReady=Truewith the reason onReadyandPushed=False; a failure setsReady=False/Stalled=Truewith a message. Between the window's finalize and that push the request is still in flight, which matters because the owner keeps re-sending its attach until it reads an outcome: the worker marks the request committed rather than forgetting it, so a re-send is recognized as the same request and cannot resolve early or claim a later window. A worker that stops without pushing fails the request instead of leaving it to time out.AlreadyPresenttherefore means the remote confirmed there was nothing to add, not that a local plan found no diff. A conflict on the outcome's write re-applies the outcome onto the fresh object; the save is never restarted. - Retention. A finished request is stamped
configbutler.ai/delete-afterand deleted once it has passed: always on the control cluster, and on a source cluster only withcommitRequests: ReconcileAndDeleteFinished.
A request whose audit event names nobody claims no actor. The final Git author remains the attached watch window's author: configured committer when attribution was disabled, or the explicit unresolved author when live attribution ran but could not resolve.
Controllers watch their dependencies so dependents reconcile quickly after spec changes:
GitTargetReconcilerwatchesGitBranch,GitProvider,KubeProvider,Namespace,WatchRule, andClusterWatchRule. Provider changes re-check connection and authorization inputs; control-cluster Namespace label changes re-check provider-reference access. Rule changes re-resolve selected collections. It deliberately does not watch encryption Secrets, so their recovery is picked up by periodic reconciliation without retaining every Secret value in the control-plane cache. It watches the applier claims itsGitTargets name, forApplierClaimReady.ApplierClaimReconciler(one per applier claim kind) watches its applier claims, theGitTargets naming them,GitBranches,GitProviders,KubeProviders andNamespacelabels. It does not watch appliers: it reads the FluxKustomizationor Argo CDApplicationon its tick to setConfigured, so a cluster without the Flux or Argo CD CRDs needs no informer for them.WatchRuleReconciler/ClusterWatchRuleReconcilerwatchGitTarget,KubeProviderandNamespacelabels, plus the watch manager's stream-state events forStreamsRunning, populate the RuleStore, and start the rule-change reconcile.GitProviderReconcilervalidates reachability and manages the signing key lifecycle. It watches nothing but its own objects.GitBranchReconcilerwatchesGitProviders,Namespacelabels and theGitTargets naming it. It proves the credential with a push session, keeps the branch claim, and publishes the worker'sstatus.remote(see Git write architecture).GitReceiverReconcilerwatches only its own objects and publishesstatus.webhookPathonce the tokenSecretholds a token.KubeProviderReconcilervalidates source-connection inputs, decides whether the provider may be used at all (the installation'sinClusterconsent, and which namespace holds its audit route), and projects its readiness to dependent targets. A provider created, deleted or moved to another route wakes the others carrying that route, so anAuditRouteConflictclears without waiting for a requeue.CommitRequestReconcilerserves the control cluster'sCommitRequests through the shared core, four at a time, so one request's author wait does not hold up the others. It reads through the uncached API reader, and shares the watch plane's author resolver and fact streams, which are nil with attribution off. TheCommitRequestServerserves the source clusters' requests through the same core, one watch perKubeProviderthat reconciles them, and reports each to theKubeProviderReconciler, which writes it asCommitRequestsServed.
Dependency watches use narrow predicates to avoid status-only heartbeat churn: a KubeProvider also
admits a Ready transition, and a Namespace admits only label changes. GitProvider, GitBranch,
GitTarget, and CommitRequest carry immutability constraints where a spec change would orphan a
materialized subtree or invalidate an in-flight finalize.
Defined in cmd/main.go:
flowchart TD
A[Parse flags + logger + build info] --> B[Init telemetry / metrics server]
B --> C[Create controller runtime manager]
C --> D[Create RuleStore]
D --> E[Create WorkerManager + register Runnable]
E --> R{receiver-bind-address set?}
R -->|yes| Ri[Add push receiver listener; route deliveries to branch workers]
R -->|no| Rj[No receiver; foreign pushes found by rejection and refresh]
Ri --> F
Rj --> F
F[Create Watch Manager + EventRouter; inject TypeRegistry] --> P[Create apply dispatcher + applier claim status writer; wire them to the workers]
P --> G[Register WatchRule + ClusterWatchRule controllers]
G --> H{redis-addr set?}
H -->|yes| Hi[Create Redis cursor store + wire WatchCursorStore + readiness gate]
H -->|no| Hj[Skip cursor store; reconnects initialize again]
Hi --> Hq{author-attribution?}
Hj --> Hq
Hq -->|yes| I[Select fact transport; start fact index + follower;<br/>wire audit fact extractor + audit HTTP server + resolver]
Hq -->|no| J[Configured-author: no fact streams or index; audit webhook skipped]
I --> K[Setup + register Watch Manager]
J --> K
K --> L[Register GitProvider + GitReceiver + KubeProvider + GitBranch + GitTarget<br/>+ FluxClaim + ArgoCDClaim + CommitRequest controllers; CommitRequest server]
L --> M[Add cert watchers + health checks]
M --> N[mgr.Start]
Redis is optional in configured-author mode. When --redis-addr is set, the cursor store is wired and a
Redis readiness gate keeps the pod not-ready until Redis is reachable. Later reconnect attempts
can resume stored RVs. When --redis-addr is empty, the cursor store is skipped and reconnects
initialize again. The first attempt after startup or watch replacement requests initial state in
both cases. The binary's --author-attribution flag defaults to
on: the fact transport is constructed, the fact index and its follower are started, the audit HTTP
handler is wired with the fact extractor, the watch manager gets the author resolver, and the audit
ingress is added to /readyz. Which transport it builds is --author-attribution-transport: redis
(the default) requires a non-empty --redis-addr, while memory runs the streams in process and is
refused unless --replica-count is 1. The Helm chart deliberately passes --author-attribution=false
by default, so a first install runs configured-author with no fact streams or audit webhook and every
commit is committer-authored.
- Instruments: internal/telemetry/exporter.go
- Reader's guide: interpreting-metrics.md, the one place the live instrument list, label vocabularies, queries and suggested alerts are maintained
- Why the surface has this shape: metrics-observability-plan.md
Instruments are OpenTelemetry instruments, exported through the SDK's Prometheus exporter onto the
controller-runtime registry, so they are scraped from the metrics server's /metrics with every
controller-runtime series beside them. Nothing is pushed over OTLP. Every name starts with
gitopsreverser_.
The families follow the architecture, which is why this section is a map rather than a list:
| Family | Follows | In the guide |
|---|---|---|
resource_condition |
the kstatus trio of every object a human declares | Configuration state |
watch_* |
ingest: events, sessions, replays, recoveries, resolved types, open streams | Pipeline |
git_*, commit_requests_total |
the branch worker: documents, windows, commits, pushes, fetches, refusals, losses, the retained backlog | Pipeline, Tuning commit windows |
placement* |
where a new document lands, and the ones refused | New-file placement |
apply_*, receiver_deliveries_total |
apply requests, the apply window and the push receiver | Apply requests |
audit_*, attribution_* |
the optional author lookup: the door, the fact pipeline, the join | Audit attribution |
api_catalog_* |
discovery, per source cluster | API resource catalog |
watch_plan_* |
the queue that plans each GitTarget's watches |
Watch plane owner |
secret_encryptions_total |
the path that keeps Secrets out of Git in plaintext | Secret encryption |
Six rules hold across all of them, and a new instrument follows them too:
- The two ingestion paths are measured apart. Watch carries object state and decides whether an object reaches Git; audit carries only the author's name. They meet at the resolver and nowhere else, so no metric adds one to the other.
- Each stage counts once, with a bounded
outcome. The pipeline reads as a funnel, and its loss paths (git_commit_failures_total,git_writes_dropped_total,placement_refusals_total,git_documents_total{outcome="refused"}) as one selector. - Two identities, and one join between them. Watch and document families are labeled by the
GitTarget(gittarget_namespace,gittarget_name); branch-worker families by theGitBranchthe worker writes for and itsGitProvider(gitbranch_*,provider_*), because one worker serves everyGitTargeton it.git_branch_targetsmaps one to the other.source_clusteris theKubeProvideras<namespace>/<name>. Keys are prefixed because a pod scrape withhonor_labels=falserenames a barenamespacetoexported_namespaceand puts the scraped pod's own in its place. - Configuration objects are labels; mirrored resources never are. A
GitTarget,GitBranch, applier claim orGitReceivername is bounded by how many someone wrote, so it may be a label; a mirrored resource is data and is never one, nor is a name a client supplies before it resolves. A family that grows past its per-instrument cap (15000 points) collapses into oneotel_metric_overflow="true"series. - A gauge about a work loop is read at scrape time. One published from inside the loop reports
its last healthy moment for as long as the loop is stuck, so the queue, backlog, watch-plane and
apply gauges are observable. "How long has this been waiting" is a timestamp, read as
time() - <gauge>. The few recorded at the step they describe (api_catalog_*, the fact index size, the attribution follower's last success, the transport) record a step that completed. - A metric needs an interpretation. A new instrument lands with its recording site and its row in the guide in the same change (adding a metric).
Renamed series are listed in UPGRADING.md. See Operational boundaries for what is not measured.
Current limitations:
- Single active replica. The watch manager and worker manager declare
NeedLeaderElection, so all object-state work runs on one elected pod; multi-pod HA is not finished. The target design is the HA / GitTarget distribution plan, which needs Redis for resume cursors, branch-shard leases, and durable write queues. - Resume cursors are best-effort. Each watch shard stores its last processed resourceVersion in Redis,
so short reconnects resume a normal watch from that cursor. Kubernetes does not guarantee replay from an
arbitrary resourceVersion, so if the apiserver has expired the cursor (
410 Gone) recovery falls back tosendInitialEventsreplay or LIST + mark-and-sweep. - Watch queue DELAY is not measurable. Per-type watch volume, session ends, replay cost and
recovery mode are all emitted now (see Observability), and
watch_event_handling_secondsreports how long a stream was busy. How long an event waited before being picked up is a different number, and measuring it needs an arrival timestamp stamped before the blocking consumer, and the events arrive on a client-go watch channel this process does not fill, so there is nowhere honest to stamp one. Occupancy is the available proxy: a stream that is busy is a stream nothing else is being read from. - The in-process attribution transport is single-replica.
--author-attribution-transport=memoryis refused with more than one replica, and it loses every unjoined fact on restart by design; a multi-replica install must use the Redis transport. - No pull request creation; the operator writes directly to branches.
- GitOps Reverser is the only path to a deployment. With an applier claim, the applier applies
nothing on its own (a
Kustomizationis held suspended, anApplicationhas automated sync off), so while the operator is down a Git-side push reaches the cluster only through a manual sync orflux resume, which the operator suspends again when it starts. - Only Flux and Argo CD can be appliers yet, and an Argo CD
Applicationonly with automated sync off: one that syncs on its own is refused, not driven. The applier may live in any cluster aKubeProviderreaches (the applier claim'sspec.kubeProviderRef), such as a management cluster. - An apply request cannot reserve time. An edit made while the requested apply runs, to a field the applier sets, is reverted on the cluster. Its value stays in Git, because the applier's write over it is not mirrored, and the next apply restores it; until then the cluster runs the older value. See the apply window.
- Audit ingress uses a shared-CA trust boundary. Named
/audit-webhook/<audit-route>routes and the annotation-routed shared endpoint both require a client certificate signed by the audit CA, but that certificate is not bound to one audit route. This is an accepted privileged-control-plane assumption, not tenant isolation; see SECURITY.md. deletecollectionis reconciled by the watch (each item arrives as its ownDELETED, or the mark-and-sweep reconciles them on replay).- A fail-safe placement skip has no dedicated status condition. A resource the writer refuses to
place unsafely is logged per-resource and counted in the resync summary (
placementSkipped), but is not (yet) surfaced as a distinct GitTarget status condition.
| Package | Role |
|---|---|
| api/v1alpha3/ | CRD types |
| internal/applier/ | apply requests: the Flux and Argo CD clients, the dispatcher that carries out each attempt, the applier claim status writer, and the Configured check |
| cmd/ | operator entry point and server setup |
| internal/auditutil/ | audit identity, objectRef, and subresource helpers feeding attribution facts |
| internal/controller/ | Kubernetes reconcilers |
| internal/git/ | branch workers, Git ops, commit/signing/encryption, manifest writer |
| internal/git/manifestedit/ | YAML document editor |
| internal/giteaclient/ | Gitea helper client |
| internal/manifestanalyzer/ | manifest inventory, acceptance, and resync planning |
| internal/manifestreport/ | projection of Kubernetes objects into comparable manifest reports |
| internal/queue/ | attribution fact streams (Redis or in-process), the in-process fact index and its follower, and per-watch resume cursors |
| internal/reconcile/ | per-GitTarget event stream (watch event → branch worker) |
| internal/rulestore/ | compiled rule cache |
| internal/sanitize/ | Kubernetes object sanitization and stable YAML marshal |
| internal/ssh/ | SSH authentication helpers |
| internal/sshsig/ | SSH signature implementation |
| internal/telemetry/ | metrics and OTLP setup |
| internal/types/ | shared resource identity/reference and sensitivity policy |
| internal/typeset/ | type followability registry, lookup model, and the relevance filter (controller-owned → don't mirror) |
| internal/watch/ | discovery catalog, watch manager, per-(GVR, scope) raw watches with sendInitialEvents replay + mark-and-sweep, the author resolver, watched type tables, event router |
| internal/webhook/ | /audit-webhook ingress and attribution fact extraction (no body joiner), and the /git-push receiver |
Deeper dives live under docs/design/:
Ingestion, attribution, and reconcile:
- Watch-first ingestion design record: historical context for the current watch-only object-state model and optional audit attribution.
- Attribution: how a commit gets its author: the two halves, the tier ladder, and the wait.
- Attribution facts as a stream, not a keyspace: the shipped transport seam, the in-process index, and what running without Redis costs.
- Watch event ordering under the attribution grace window
- Reconcile via watchlist mark and sweep
- CommitRequest design
Types, discovery, status, and the Git write side:
- Type followability
- Typeset owns discovery grace
- Kubernetes API resource catalog
- Status and conditions guide
- The support contract: what we edit, what we refuse, and why
- The applier claim
- Git credentials interop
- SOPS/age key management
- Commit signing
- Interpreting metrics
Future direction: