Skip to content

Latest commit

 

History

History
2063 lines (1725 loc) · 136 KB

architecture.md

File metadata and controls

2063 lines (1725 loc) · 136 KB

GitOps Reverser architecture

GitOps Reverser is a Kubernetes operator that observes cluster mutations and writes the resulting desired object state to Git. It reverses the traditional GitOps direction: instead of Git driving the cluster, the Kubernetes API drives Git. The repository becomes a continuously updated mirror of live cluster state.

This document describes how the operator works today. If you are reading it for the first time, keep one sentence in mind: Kubernetes watch supplies state, Git stores the mirror, and Redis keeps the operator's short-lived coordination state. Read the Ground Rules, Mental Model, and Data Sources for the shape of the system, then the Configuration Model and Common Flows for how the pieces move together. The later sections give the reference detail behind each piece. If a detail here ever disagrees with the source, the source wins; deeper design records live under docs/design/. For the inventory rather than the mechanics, which package holds which responsibility and which components observe a Kubernetes API server, see components.md.

Both connection kinds are namespaced, for one reason: a connection carries a credential, and the credential belongs in the namespace of whoever owns it. A namespaced GitProvider is a team's Git write boundary: its credential, branch policy, and targets usually belong together. A namespaced KubeProvider is one logical source identity: one client, discovery surface, watch state, and attribution partition, shared by every target that names it. Its kubeconfig Secret lives in its own namespace, so a tenant's connection keeps its credential with the tenant. Its grants (spec.allowedGitTargets, spec.allowedApplierClaims) admit control-cluster namespaces and bound what they may reach; omitted, each admits the provider's own namespace, and none grants source-cluster RBAC. The source identity is the provider's namespace and name, with no API-server identity probe: two providers configured for the same server remain separate source partitions, and so do two providers with one name in two namespaces.

The operator's own identity is the one credential no namespace owns, so it is not an ordinary secretRef. A KubeProvider asks for it with spec.inCluster: {}, and the installation decides: only a provider listed by exact namespace/name in --in-cluster-kube-providers is honored. Any other reports Stalled=True (InClusterNotAuthorized) and is never used, so no omitted, malformed, or forbidden credential falls through to the operator's identity.

Keeping a repository's write credential in the namespace of whoever owns that repository is what makes a team able to start mirroring without a platform admin. Each branch of that repository is a GitBranch, in the credential's namespace or in one the GitProvider's spec.allowedGitBranches admit for that branch, and one GitBranch holds a repository branch, because one branch worker writes the branch, with one credential: the oldest admitted GitBranch whose credential opens a push session on the remote holds it, and any other reports BranchConflict. A repository written from several namespaces needs no copy of the credential in each: their GitTargets name the shared GitBranch with spec.gitBranchRef.namespace, admitted by that GitBranch's spec.allowedGitTargets. Omitted, that list admits the GitBranch's own namespace, and a written list is the whole policy (configuration.md). Neither kind can change scope later: a CRD's spec.scope is immutable once the definition is Established, so the change would mean deleting the definition (and every object of the kind with it) and recreating both.


Ground rules

These are the design decisions to keep in your head while reading or changing the code:

GitOps Reverser writes Git; Flux or Argo CD deploys. The output is commits that reflect the cluster's state. Bringing a change from Git back into the cluster is the job of a GitOps tool running alongside. GitOps Reverser only decides when that tool applies a folder it also writes, through an applier claim (see "GitOps Reverser decides when Git is applied" below).

The Kubernetes API is the source of truth for what GitOps Reverser writes. Git is a materialized mirror of desired state from the API, ingested by watch; GitOps Reverser never reads Git as object state. An edit made in Git reaches the cluster only through Flux or Argo CD. When a GitTarget names an applier claim, those applies are coordinated: GitOps Reverser first publishes the API edits it has decided, then asks the applier to apply. The watch observes the result, and the applier's own writes during that apply (its echoes) are not mirrored, because Git already holds them. When a push conflicts with a newer remote commit, the operator fetches the new remote state, resets its local clone, and replays its retained writes from the API. So the API comes first without being the only author: other authors can push to the same branch and their commits stay, and only where an API edit and a Git commit touch the same object does the API's version win.

Watch is the only object-state source. Each GitTarget names one KubeProvider and opens one Kubernetes watch per claimed (GVR, scope) against that source, with sendInitialEvents=true. For a namespaced type, a scope is one concrete source namespace; cluster-scoped types use the cluster-wide scope. Every Git write derives from persisted state the watch observed. Audit never defines what changed; it only, optionally, explains who caused it.

Sensitive resources are never written in plaintext. Core Secrets and configured sensitive resource types must be encrypted before they touch the Git worktree. If encryption cannot be configured, the write fails; there is no plaintext opt-out.

Writes are serialized per Git branch. One BranchWorker owns each repository branch: the repository's URL, normalized, and the branch. Multiple GitTargets may share one branch, from any namespace, through the one GitBranch that holds it. Every write to that branch goes through the worker's single event loop and commit window.

The worktree sits at the remote tip, or the worker knows it does not. The write path used to fetch and hard-reset before the first commit of every publication cycle. It no longer does: the push session reads the remote's ref advertisement on a connection the cycle was making anyway, and a cycle that commits nothing still reaches it, so a base the worker can vouch for needs no fetch to plan against. Two pieces of state on the worker carry that claim, and they are separate on purpose: baseTrusted is dropped by anything that can no longer prove where the remote is, while checkoutApplied counts how many retained writes the checkout holds on top of its root. A write that failed part-way sets it to unknown, which only a reset clears; a reset sets it to zero, which with writes retained says their local commits are gone and must be replayed. Collapsing the two would let one kind of doubt clear another. Either one makes the next cycle fetch. The failure direction is safe by construction: a stale "untrusted" costs one fetch, while a stale "trusted" is caught by the compare-and-swap on the next push. For a new write branch, the worker checks the parent's advertised tip before sending the push. A moved parent costs a fetch, a reset, and a replay. The server's compare-and-swap protects creation of the write branch with Old = zero; it does not lock the parent. The parent can still move after the advertisement, so what is guaranteed is the parent's tip as that advertisement showed it. The corollary is the rule to keep in your head when adding code here: anything that resolves, commits, or concludes without reaching a push advertisement must either fetch, or refuse to conclude. See how the branch worker reads the remote.

That trust belongs to one repository, and so does the worker holding it. A worker is keyed by the repository branch, and spec.url is immutable and repointed by deleting and recreating the GitProvider. The key names the repository but not the object behind it, so the worker also carries the repository identity it was created for: the provider's UID paired with its URL. Every repository operation uses that URL rather than whatever the provider names when the cycle reads it, and the reconcile compares the identity on each tick. A provider recreated under its name, for the same repository or another, does not move the worker; it replaces it, and the replaced worker takes its clone, its trust and its queued work with it. Nothing has to remember to invalidate a field.

An idle target re-proves where its branch is, periodically. A target that is writing learns it for free: the push session reads the remote's advertisement and the server names the hash it accepted, so an active branch never schedules a look of its own. One that has gone quiet is asked on its 5-minute reconcile tick, and spends one ref advertisement per --git-refresh-interval (10m by default); only a branch that moved costs a fetch after that. What it learns is published as the GitBranch's status.remote (the commit, when it was last proved, and whether a push or a fetch proved it), and a folder somebody changed in Git is re-read and republished as status.placement.

A refresh never writes content. It plans nothing and commits nothing of its own. What it does publish, besides status.remote and status.placement, is whether the folder can be written at all: a structural refusal that arrived in somebody else's push sets GitPathAccepted=False, and a later read that finds the folder fixed clears it. Both directions matter, because neither event (breaking a folder in Git, or fixing it) produces a Kubernetes event to notice it by.

That condition puts the target on the ten-second re-check loop, which is the recovery path rather than a side effect, and reconcile.configbutler.ai/requestedAt still forces a full re-check for a human who wants one now. A refusal a write found is left alone: a structural read cannot see a write-boundary precondition, so only the resync that writes the affected type clears that one.

Seconds-fresh is the push receiver's job: a GitReceiver accepts the Git host's push webhook, and wakes the branch worker of every GitProvider whose spec.url names the repository the delivery names, for the branch it names. A delivery is only a hint: it makes the branch worker read the ref advertisement with its own provider's credential, and what the remote says decides whether the branch moved. It never triggers a cluster snapshot: the cluster still holds the values from before the push, so a snapshot would publish them over the incoming change.

GitOps Reverser decides when Git is applied to the cluster. An apply must not revert an edit that is live and not yet in Git, and only the worker writing the branch can see those edits. So a GitTarget names an applier claim, which holds its applier so that it applies nothing on its own (a Flux Kustomization is held suspended, an Argo CD Application held with automated sync off), and the branch worker sends apply requests instead. A request is sent only after a flush: when an apply comes due, the worker closes the open commit window, publishes every write decided so far at once, and sends the request for the tip that publication produced. An edit decided after that is recorded in the apply window, so an apply that overwrites it is withheld and repaired. Continuous editing therefore never holds an apply back. A request goes out:

  • for a commit somebody else pushed, after replaying the pending writes onto it, so the applied commit carries the API side's answer;
  • for a refused edit the target opted to revert;
  • for an API edit that an apply reverted on the cluster after the edit was made, to put it back;
  • to catch up (CatchUp), for a Flux Kustomization and an Argo CD Application alike: to adopt an object the API created in the folder, because an applier prunes only what it applied and a later Git-side delete would otherwise leave the object running; and to converge, once when the worker starts and once when a GitTarget first names the applier claim, unless the applier already applied the tip. A push of GitOps Reverser's own asks for nothing: the cluster already runs the edits it holds.

No applier is asked for anything before every initial snapshot of the GitTargets it covers is published, so the edits made while the operator was down reach Git before any apply. The worker keeps each reason pending until an apply operation answers it, and decides every retry the same way. A request runs off the publication path and cannot fail a publication; the applier claim's status.operation records the operation it started. Built: the Flux Kustomization applier and the Argo CD Application applier, its automated sync held off. The aim: a webhook for any other tool. See the applier claim design.

Redis/Valkey is optional but advised. The default configured-author mode runs without it: a plain helm install comes up healthy and watches initialize from current state on startup. When an endpoint is configured, Redis stores cursors for reconnecting running watches. A CommitRequest's author comes from the audit fact of the write that created it, like any write's, and needs no Redis record. Attribution does not require it on its own: its facts travel on a selectable transport, Redis Streams by default and an in-process ring with --author-attribution-transport=memory, which is refused with more than one replica. HA will require Redis as the shared store across replicas.

Audit is an optional attribution lookup. When attribution is enabled, kube-apiserver posts audit events to /audit-webhook/<audit-route> (or a configured annotation-routed shared endpoint). The route is KubeProvider.spec.attribution.auditRoute, which defaults to <namespace>.<name>. The operator appends a minimal fact to that route's per-type fact stream; every process watching the type follows that stream into a bounded, TTL'd in-process index, and a watch event joins against the index within a bounded grace window. A missing, late, or absent fact never blocks state capture; it only changes the author.

Behavior is deterministic and proven by tests. Given the same observed Kubernetes state, configuration, and Git base, the operator makes the same materialization decisions. Ordering, attribution fallbacks, conflict replay, path refusal, encryption behavior, and commit semantics are pinned by unit and e2e tests.


Mental model

The easy part is "write YAML to Git." The hard parts shape the whole architecture:

  • Ordering. Two updates to the same object must land in Git in cluster order.
  • Not losing changes. A dropped or late event must never leave Git permanently wrong. In particular, a delete that happens while no watch is running must still be reconciled.
  • Secrets. Sensitive objects must never touch disk in plaintext.
  • Scale. A cluster holds thousands of objects across hundreds of types; you cannot keep a live watch open on all of them.

The solution, in the vocabulary used throughout this document:

  • Watch is the only object-state source. Each GitTarget selects eligible resource collections. The planner chooses one served API version per group/resource/namespace collection and opens its watch. Delivery order within one watch does not order arrivals across overlapping watches.
  • Watch cost follows selected collections. The planner opens watches for selected, eligible collections rather than every type in the cluster. All-namespace and named-namespace collections remain separate when rules select both.
  • Initialization and reconnect differ. The first attempt after startup or watch replacement requests initial state for the new render-fidelity revision. Later attempts in the same watch loop can use a stored resume RV; expired history requires fresh initialization. A completed snapshot permits cleanup only within its observed collection and pruning policy. Cursors use target UID, group/resource, namespace, and label selector, with a TTL refreshed by events and bookmarks. Unsupported initial-event requests use LIST with a buffered watch. Snapshot reconciliation is triggered by initialization, never by a timer.
  • Audit, when enabled, only names the author. It is an optional attribution lookup; a missing or late fact costs author fidelity, never correctness. With attribution disabled the product commits as the configured committer. With attribution enabled, an unresolved live change is visibly authored as unknown (attribution unresolved), so an installation that expects attribution can investigate its audit policy, ingress, Redis connection, or source identity instead of mistaking the result for configured-author mode.
  • One BranchWorker per Git branch serializes all writes. Every write to a branch funnels through a single worker and a single commit window, which keeps concurrent GitTargets and authors from racing each other into a corrupt tree.

Data sources

GitOps Reverser reads Kubernetes through four mechanisms. Two are required to function; the other two are optional and only add who did something:

Source Answers Required What it carries
Discovery (CRD/APIService) what types exist yes the API surface: served GVRs, scope, preferred version, subresources; rules resolve against it
Watch (per claimed (GVR, scope)) what changed yes the object body, ordered per type; the only object-state source, with deletes reconciled via sendInitialEvents replay + mark-and-sweep
Audit webhook (/audit-webhook/<audit-route>) who changed mirrored state, and who created a CommitRequest no a post-persist attribution fact, partitioned by source provider and joined to the watch event by resourceVersion, or to the CommitRequest by the UID it was created with (Optional Attribution)

With the optional source off, the product still mirrors state correctly, and every commit is authored by the committer.

Why audit, not admission, attributes a change

For a mirrored change, taking the author from a validating admission webhook is tempting, because the request already carries userInfo. We tried it; the edge cases are not fixable, because admission runs before the write reaches etcd:

  • Admission sees attempts, not persistence. A request that passes our webhook can still be rejected by a later webhook or fail an optimistic-concurrency conflict at storage, so the change never becomes state, yet we would already have recorded an author for it (a dry-run is the same shape: admission fires, nothing persists).
  • There is nothing to join on yet. Mirrored attribution matches an author to a persisted watch event by identity + resourceVersion; at admission the resourceVersion does not exist, and for a generateName create even the final name/UID is unassigned, so an admission record cannot be coupled to the watch or audit stream.

Audit avoids both: kube-apiserver posts a ResponseComplete event after the write persisted, carrying the resourceVersion that joins it to the watch event.

A CommitRequest's author comes from audit too, for the same reasons: the ResponseComplete event that created it, joined by its UID, from the same index and with the same grace as the edits it saves. A later write to the request, such as a label patch, never names its author. So the request and its edits are attributed by one pipeline, on the cluster both were made on, and a request on a source cluster needs nothing from the control cluster. A CommitRequest is never mirrored, whatever a WatchRule selects. See CommitRequest Finalize.

Proving it on new Kubernetes versions

How Kubernetes emits these structures is not something to guess at. A separate, standalone project. The mutation-capture lab (design, cmd/mutation-capture-lab/) records the exact watch, audit, and admission output a real apiserver produces for each interesting scenario and commits it as normalized example YAML (a versioned corpus). It doubles as a regression harness: point it at a new Kubernetes release, regenerate, and any change in verb naming, event ordering, body presence, or deletecollection fan-out surfaces as a reviewable corpus diff before it can surprise the operator. The "admission sees attempts, not persistence" claim above is read straight out of that corpus: its dry-run, record-and-reject, and conflict scenarios capture audit and admission with no resulting watch event.


Configuration model

You configure GitOps Reverser through ten CRDs (group configbutler.ai, version v1alpha3). WatchRule and ClusterWatchRule choose which Kubernetes resources enter the pipeline. CommitRequest can ask for the current window to be saved. GitTarget joins one source cluster to one Git destination. KubeProvider supplies the source connection and authorization boundary; GitProvider supplies the repository, credentials, commit settings, and push policy, and a GitBranch holds one branch of that repository for every GitTarget that writes it. A GitReceiver is the endpoint the Git host's push webhook calls; nothing references it, because a delivery is routed by the repository it names. FluxClaim and ArgoCDClaim hold the Flux Kustomization or Argo CD Application that applies a folder back, so GitOps Reverser decides when it applies.

graph LR
    subgraph CONTROL["Control plane: operator cluster"]
        WR[WatchRule] -->|gitTargetRef| GT[GitTarget]
        CWR[ClusterWatchRule] -->|gitTargetRef| GT
        CR[CommitRequest] -->|gitTargetRef| GT
        GT -->|applierClaimRef| TR["FluxClaim /\nArgoCDClaim"]
        GT -->|kubeProviderRef| CP[KubeProvider]
        GT -->|gitBranchRef| GB[GitBranch]
        GB -->|gitProviderRef| GP[GitProvider]
        GR[GitReceiver] -.->|"wakes by spec.url"| GP
        KCFG[("Remote kubeconfig Secret\nprovider's namespace")]
        WM["Watch manager\nper-provider source context"]
    end

    subgraph SOURCE["Source cluster selected by KubeProvider"]
        API["Kubernetes API\ndiscovery · watches"]
        APPLIER["Applier\nFlux Kustomization\nor Argo CD Application"]
    end

    HOST[("Git host")]

    CP -. "secretRef (or inCluster)" .-> KCFG
    CP -->|"source identity + config"| WM
    WM -->|"source client"| API
    TR -. "kustomizationRef / applicationRef" .-> APPLIER
    HOST -. "push webhook to status.webhookPath" .-> GR

    style GP fill:#e8f4fd,stroke:#2196f3
    style CP fill:#e8f4fd,stroke:#2196f3
    style GT fill:#e8f4fd,stroke:#2196f3
    style WR fill:#fff3e0,stroke:#ff9800
    style CWR fill:#fff3e0,stroke:#ff9800
    style CR fill:#f3e5f5,stroke:#8e24aa
Loading

The diagram follows one GitTarget. A provider with spec.secretRef selects a remote cluster through a kubeconfig Secret in the provider's own namespace; one with spec.inCluster selects the operator's own cluster, when the installation lists it. Each referenced KubeProvider gets an independent source context: client, discovery catalog, followability registry, and reachability state. Its GitTargets open their source watches through that context. The contexts are keyed by the provider's namespace/name and are released when no GitTarget references them, so a provider deleted and recreated under the same name is a new connection.

The applier is not one of the ten kinds. It belongs to Flux or Argo CD, lives in the cluster the applier claim's spec.kubeProviderRef names (the source cluster, or a management cluster), and is reached through that KubeProvider's identity. The applier claim is ours: it holds the applier, and records the apply operation it started.

CRD Scope One line role
WatchRule namespaced which NAMESPACED resources, in which source namespaces, route to a GitTarget
ClusterWatchRule cluster which CLUSTER SCOPED resources route to a GitTarget
CommitRequest namespaced a one shot "save the open window now" signal
GitTarget namespaced one materialization from a KubeProvider to (GitBranch, path)
KubeProvider namespaced one source-cluster connection plus namespace access policy
GitBranch namespaced one repository branch, held for every GitTarget that writes it
GitProvider namespaced a Git repo + credentials + commit/signing config
GitReceiver namespaced the endpoint a Git host notifies on a push, for every GitProvider whose spec.url a delivery names
FluxClaim namespaced the Flux Kustomization GitOps Reverser asks to apply, and the apply in flight
ArgoCDClaim namespaced the Argo CD Application GitOps Reverser asks to sync, and the sync in flight

WatchRule / ClusterWatchRule

Scope is carried by the rule KIND. A WatchRule selects NAMESPACED resources and routes matching events to a same namespace GitTarget; each of its rule items names the source namespace it watches. A ClusterWatchRule selects CLUSTER SCOPED resources, with an explicit namespace gitTargetRef and no namespace selection of its own. Both share the rule model:

  • spec.rules[]: OR resource rules (MinItems=1). A rule selects whole resource collections: every selected collection is observed through creates, updates, and deletions, and the GitTarget's spec.prune.mode decides whether an observed removal deletes the Git document.
  • rules[].apiGroups: omitted resolves the named resource across served groups; "" is the core group; * is all.
  • rules[].apiVersions: omitted means the preferred served version.
  • rules[].resources: plural resource names or *.
  • rules[].objectSelector: an optional label selector the API server applies to every list and watch the rule opens. An object that stops matching leaves the collection as a deleted one does.
  • WatchRule adds rules[].sourceNamespace: omitted for the rule's own namespace, an exact name, or * for every namespace the source credential can read. Anything but the rule's own namespace passes the source-namespace gate (SourceNamespaceAuthorized). A * compiles to ONE cluster-wide collection (the empty namespace, which for a namespaced GVR is the all-namespaces collection), so it opens one stream and one list per matched type however large the cluster is. That collection is a peer of any named-namespace collection on the same type, never a replacement.
  • ClusterWatchRule has no scope or namespace choice of its own: the kind is the choice.

Subresources are rejected in rule resources. Mirroring operates on top level resources: a scale made through the /scale subresource reaches Git as the parent's own watch event, and its audit event serves only to name who scaled the parent.

One KubeProvider grant answers two questions in different planes:

  • consumerNamespaces (from, with names or selector) authorizes control-cluster namespaces to reference the provider. It is a tenant/export boundary; it neither selects nor grants access to source namespaces.
  • sourceNamespaces and clusterResources bound what the GitTargets it admits may watch. They grant nothing: what may actually be read from the source cluster is bounded by that credential's own Kubernetes RBAC. The operator reads no Namespace objects in a source cluster for authorization.

CommitRequest

A one shot "save now" signal that finalizes the open commit window for a GitTarget instead of waiting for the silence timer. It is created on the cluster the GitTarget mirrors: the control cluster, or a source cluster whose KubeProvider sets spec.commitRequests. The entire spec is immutable. Key fields:

  • spec.gitTargetRef: target whose window the request attaches to, admitted by its spec.allowedCommitRequests. On a source cluster the namespace is required.
  • spec.message: optional verbatim commit message (1–1024 chars, no control characters).
  • spec.window: attach (CurrentOrNext or Next), attachTimeout (default 5s, the wait for a window), and the timers that replace the target's for the attached window: idleTimeout (none by default) and maxDuration (default 2s). A request never opens a window; only writes do.
  • spec.whenNothingToCommit: Resolve (default) or CommitEmpty, which records the message in an empty commit when the request ends with nothing to commit.
  • status.author: the creator, as the request's audit event names them; empty when it names nobody, and then the request can attach only to an unnamed window.
  • status.conditions: kstatus-compatible. Ready is the summary (Unknown while the save runs, True once the request reached a terminal outcome that is not an error: a pushed commit, or a benign no-commit); Reconciling/Stalled are the kstatus progress/blocked pair. Pushed reports whether the commit reached the remote. The Ready condition's reason carries Committed, NoWindow, WindowMismatch, AlreadyPresent, or FinalizeFailed. A benign no-commit (e.g. NoWindow) is Ready=True, Stalled=False, a correct non-error outcome, whereas a FinalizeFailed is Ready=False, Stalled=True.
  • status.branch / status.commit: set when the commit was pushed (Pushed=True).

How attribution and finalization interact is described under CommitRequest Finalize.

GitTarget

One materialization from a source provider to a Git destination: (KubeProvider, GitBranch, path). Key fields:

  • spec.gitBranchRef: required; a GitBranch by name, in the GitTarget's own namespace unless namespace names another one. Its spec.allowedGitTargets must admit the GitTarget's namespace.
  • spec.kubeProviderRef: required; the source KubeProvider, in the GitTarget's own namespace unless namespace names another one. Its spec.allowedGitTargets must admit the GitTarget's namespace. There is no default provider: a missing one reports Validated=False (KubeProviderNotFound).
  • spec.path: immutable, required path under the repo (MinLength=1; . means repo root and must be chosen explicitly).
  • spec.encryption: optional SOPS/age encryption settings for sensitive resources.
  • spec.commit.window: when this target's commit windows close: idleTimeout (default 5s) of silence, or maxDuration (default 1m) after the window opened.
  • spec.commit.message: liveTemplate / reconcileTemplate Go templates.
  • spec.applierClaimRef: optional; the FluxClaim (a Flux Kustomization GitOps Reverser holds suspended and asks to apply) or ArgoCDClaim (an Argo CD Application whose automated sync GitOps Reverser holds off, and whose syncs it starts) in this namespace. In a coordinated folder, an object another applier owns (by Flux's owner labels or Argo CD's tracking annotation) is not mirrored.
  • spec.onRefusal: Ignore (default) or Reapply, which asks the applier to reapply the tip when an edit to an existing document is refused. Reapply requires spec.applierClaimRef.

spec.commit describes the folder, not the connection, so two GitTargets sharing one GitBranch may batch and phrase their commits differently. A branch worker serves a repository branch, through its GitBranch's GitProvider, and resolves the window per open window, since a window is bound to exactly one target.

gitBranchRef, kubeProviderRef, and path are immutable so a target cannot silently orphan an old materialization or change its source cluster. Among GitTargets on one GitBranch, from any namespace, the controller rejects overlapping paths.

A GitTarget whose GitBranch does not exist reports Validated=False (GitBranchNotFound); one whose GitBranch does not hold its branch, or is not Ready, reports GitBranchReady=False and waits as progress (GitBranchNotReady). References are checked in status, not at admission, so Flux and Argo CD can apply a folder of manifests in any order.

Status has a kstatus-compatible summary layer plus domain conditions:

  • Ready, Reconciling, and Stalled are the generic conditions used by GitOps tooling. Ready=True means the latest observed generation is valid, the Git path is accepted, and source streams are running. Initial replay reports Reconciling=True. A human-fixable block reports Stalled=True.
  • Validated and EncryptionConfigured explain control-plane health.
  • StreamsRunning explains the source side: every tracked type is past initial replay and routing live events.
  • KubeProviderReady and SourceClusterReachable distinguish valid source configuration from a source API the data plane can currently reach.
  • GitPathAccepted explains the target side: the selected Git path is safe for the operator to materialize.
  • GitBranchReady says the GitBranch holds its branch and is Ready.
  • RenderMatchesLive says the folder, as rendered, agrees with the live objects; while it is not True, live writes are held or dropped (see The render-vs-live gate).
  • ApplierClaimReady, present only with spec.applierClaimRef, says the applier claim is Ready (its applier exists, can be written, and is held: a Kustomization suspended, an Application's automated sync off) and that the applier applies this repository and branch. It is a gate.
  • status.streams is a bounded count summary, not a per-type list.

An applier claim has Configured, InControl and the kstatus trio, status.source (what its applier applies, as last read), and status.operation and status.repairPending, which the branch worker writes at every move of the operation it started.

WatchRule and ClusterWatchRule add ResourcesResolved and GitTargetReady. ResourcesResolved explains the source selector. GitTargetReady mirrors the referenced GitTarget's write readiness. This keeps StreamsRunning honest: it only says source watches are running, not that Git writes can succeed.

GitProvider

Represents a Git repository and the credentials/configuration used to write it. Key fields:

  • spec.url: immutable repository URL.
  • spec.secretRef: optional Secret in the same namespace for HTTP/SSH authentication.
  • spec.allowedGitBranches: which namespaces' GitBranches may claim which branches with this credential, as grants with a from and branch patterns. It has no default: its own namespace needs a grant too. A GitBranch it no longer admits gives its branch up, keeping its finalizer.
  • spec.knownHostsRef: optional SSH known hosts source.
  • spec.commit.committer: committer identity (defaults to GitOps Reverser / noreply@configbutler.ai).
  • spec.commit.signing: SSH signing key reference and optional key generation.
  • status.signingPublicKey: populated when signing is configured and key material is available.

The controller verifies repository reachability and manages the signing key lifecycle. It generates an ed25519 keypair when signing.generateWhenMissing is set. The portable artifact across GitOps ecosystems is the credentials Secret, not a foreign repository object. The credentials reader accepts the Kubernetes native, Flux, and Argo CD Secret key dialects (see finished/git-credentials-interop.md).

GitBranch

One branch of the repository a GitProvider names, holding it for every GitTarget that references it. Every field but spec.allowedGitTargets is immutable:

  • spec.gitProviderRef: the GitProvider, in the GitBranch's own namespace unless namespace names another. Its spec.allowedGitBranches must admit this namespace and the branch (NamespaceNotAuthorized, BranchNotAllowed), checked before any push session.
  • spec.branch: the branch it claims.
  • spec.parentBranch: optional branch the write branch is created from and compared against while it does not exist. Omitted means the remote's default branch. An explicit parent must exist when creating the write branch (ParentBranchNotFound). The grants need not allow it, because it is only read.
  • spec.allowedGitTargets: which namespaces' GitTargets may write through it. Omitted, its own namespace's; a written list replaces that default, and [] admits none.
  • status.repository: the normalized repository URL, half of the key one GitBranch holds.
  • status.lastPushNoticeAt: when a GitReceiver last delivered a push to it. It records receipt, so it is published even when the look the delivery caused could not read the remote.
  • status.remote: where the branch is on the remote, and when that was last proved.
  • status.gitTargetCount: how many GitTargets reference it, from any namespace.

The controller proves the provider's credential by opening a receive-pack session (at most once per steady interval while it succeeds), keeps the branch claim, and publishes the worker's observation. See Git write architecture for the branch claim.

GitReceiver

The endpoint a Git host notifies on a push. No GitProvider names it: a delivery is routed by the repository it names, to providers in any namespace.

  • spec.type: github, gitlab, gitea or generic-hmac, named after Flux's Receiver.
  • spec.secretRef: a Secret in the receiver's own namespace whose token key is the webhook secret.
  • status.webhookPath: /git-push/<hex HMAC of namespace/name, keyed by the token>, once the token Secret holds a token. Ready is False with ListenerDisabled while the operator serves no listener.

A delivery is authenticated against the token before its body is read. The handler then reads the repository URLs and the branch the payload names, compares the URLs with every GitProvider's spec.url (normalized by internal/repourl), and wakes the branch worker writing that branch for each one that matches, once. The worker reads the remote with its own credential, so nothing in the payload chooses a URL, a credential or a commit, and it holds at most one pending look, so a run of deliveries costs one ref advertisement at a time and never crowds writes out of the queue. It does not bound the rate: deliveries arriving during successive reads keep a worker reading continuously. That availability cost of letting any namespace's receiver wake any matching worker is accepted, with no per-worker cooldown (see the security model). The 202 carries no body: a count of what was woken would tell the token holder whether another namespace tracks a repository.

KubeProvider

KubeProvider is the Kubernetes-side peer of GitProvider: a connection to one cluster and the identity used there. What it's used for depends on who names it: a GitTarget reads its source cluster through it, and an applier claim reaches its Kustomization or Application through it, often in a management cluster. A GitTarget, FluxClaim or ArgoCDClaim references it by the required, immutable spec.kubeProviderRef; an omitted namespace means the referrer's own, and nothing falls back to a default provider. Its credential is exactly one of two immutable arms (delete and recreate the provider to repoint it):

  • spec.secretRef: a kubeconfig Secret in the provider's own namespace (key value, then value.yaml, unless key names one). The kubeconfig safety checks refuse credentials named by file path, and exec auth and insecure TLS unless the operator opts in through its flags.
  • spec.inCluster: {}: the operator's own identity, honored only for a provider listed in --in-cluster-kube-providers; any other is Stalled=True (InClusterNotAuthorized) and never used. spec.client is refused beside it, because the operator's own client is not throttled per provider.

Several namespaces may mirror through one provider, but its source identity must not vary by target because it keys source clients, discovery, watches, and attribution. The identity is the provider's namespace/name rather than a deduplicated physical-cluster identity, so two providers pointing at the same API server still have separate contexts and authorization boundaries. A provider admits only the namespaces its grants admit; omitted, the grants admit its own namespace, and a written list replaces that default. The check runs on every reconcile before watches start (NamespaceNotAuthorized), and a grant or Namespace label change triggers that reconcile, so tightening it also stops an already-existing GitTarget, which an admission-time check could not do. It guards which tenant may cause the operator to export a source; it does not expand that source credential's Kubernetes permissions.

Its audit route, spec.attribution.auditRoute, defaults to <namespace>.<name> and is held by one namespace: the namespace the route's first dot-segment names when a provider there carries it, otherwise the namespace of the oldest carrier. Providers in the holding namespace may share a route (several connections to one API server); one in another namespace reports Stalled=True (AuditRouteConflict) and is not used. AuditFactsReceived reports whether the route has ever delivered a fact: Unknown until one arrives, and it is never part of Ready, because a quiet cluster and a broken audit policy look the same from here (configuration.md). The KubeProvider validates its connection inputs, while a GitTarget projects provider readiness and live source reachability.

Credentials are re-resolved on the catalog-refresh cadence, not on every watch reconnect, inCluster ones included. A rotated Secret, a spec.client change, or a provider deleted and recreated under the same name rebuilds that provider's clients and restarts only the watches using that provider; a definitively invalid or missing credential, or an inCluster provider the installation stops listing, fails closed and stops those watches. An unreachable remote similarly makes only its dependent targets unready; it does not block discovery or watches for other source contexts.


Common flows

Say a GitTarget in control-cluster namespace team-a watches ConfigMaps, and a user runs kubectl apply to edit the ConfigMap team-a/app-config in that target's selected source cluster. Here is the path that change takes.

flowchart LR
    subgraph CONTROL["Control plane: operator cluster"]
        CP[KubeProvider]
        KCFG[("Remote kubeconfig Secret\nprovider's namespace")]
        OWN["GitTarget watch set\nclaimed ∩ followable (GVR, scope)"]
        RELEVANCE["Relevance filter\nsanitize · followability · no-op diff"]
        RESOLVE["Resolver\nbounded grace window"]
        GTES[GitTargetEventStream]
        AFACTS["Audit fact extractor"]
        ASTREAM[("Per-type fact stream\nRedis Streams or in-process ring")]
        AINDEX[("In-process fact index\nbounded, TTL, keyed for join")]
    end

    subgraph SOURCE["Source cluster selected by KubeProvider\n(the control cluster for spec.inCluster)"]
        WATCH["WATCH + sendInitialEvents replay<br/>(per claimed GVR + scope)"]
        DISC["Discovery: CRDs / APIServices"]
        AUDIT["/audit-webhook/&lt;route&gt; (optional)"]
    end

    subgraph GIT["Git writes: internal/git"]
        BW[BranchWorker]
        PLAN["Manifest aware plan/flush"]
        PUSH["Atomic push"]
    end

    CP -. "secretRef" .-> KCFG
    CP -->|"source client"| DISC
    CP -->|"source client"| WATCH
    DISC --> OWN
    WATCH --> OWN
    OWN --> RELEVANCE
    RELEVANCE --> RESOLVE
    RESOLVE --> GTES
    GTES --> BW
    BW --> PLAN
    PLAN --> PUSH

    AUDIT -. configured .-> AFACTS
    AFACTS -->|append| ASTREAM
    ASTREAM -->|follow| AINDEX
    AINDEX -. lookup .-> RESOLVE
Loading

Following the ConfigMap edit:

  1. Watch delivers it. The API server applies the change and bumps the object's resourceVersion. The GitTarget's watch on core/configmaps in team-a delivers a MODIFIED event carrying the new object body. (On a cold start or after 410 Gone, the same object instead arrives as an ADDED during the sendInitialEvents replay.)
  2. Desired-state change filter. The event is sanitized (status, managedFields, and volatile metadata stripped) and compared with what the branch worker last accepted for this object on this stream, from a live event or the stream's replay. A no-op (e.g. a */status bump whose desired-state projection is unchanged) is dropped here.
  3. Resolve the author. When attribution is enabled, the resolver waits a bounded grace window (--author-attribution-grace, default 3s) for a matching audit fact in the attribution index, joining by resourceVersion/UID. On a strong match the real user or named service account becomes the author. With attribution disabled, the configured committer is the author; with attribution enabled but no usable fact, the explicit unknown (attribution unresolved) author marks the unresolved change. This wait is per-event and never reorders a watch. See Watch Event Ordering.
  4. Route + window. The event flows into the GitTargetEventStream, and the BranchWorker appends it to the open commit window for (author, GitTarget).
  5. Commit + push. When the window closes (5s of silence, a CommitRequest, or a buffer limit), the manifest aware writer patches team-a/.../app-config.yaml in place, commits as the resolved author, and pushes via PushAtomic (retrying with fetch/reset/replay if the remote moved).

Separately, the audit path (only when attribution is enabled): kube-apiserver POSTs audit events to /audit-webhook/<audit-route>; AuditHandler extracts the minimal attribution facts and appends them to that route's per-type fact stream. A follower on the watch side reads the streams for the types it watches into a bounded, TTL'd in-process index. That index is read only by the resolver in step 3; it never creates or repairs object state.

And if the watch had been lost? A reconnect can recover a retained deletion event from its resume RV. When a fresh snapshot is needed, reconciliation can infer absence within its observed collection; Always permits that cleanup, while Never and OnEvent retain snapshot orphans. See State Ingestion and Not Losing Deletes.

Normal commit flow

Most writes are closed by the commit window timer. A CommitRequest is the explicit "save now" path: it does not create new state, and it does not bypass watch. It only asks the branch worker to close a matching open window early.

flowchart TD
    CHANGE[Persisted Kubernetes change] --> WATCH[Per-GitTarget watch event]
    WATCH --> FILTER[Sanitize + relevance filter]
    FILTER --> AUTHOR{Attribution enabled?}
    AUTHOR -->|yes| AUDIT[Wait bounded grace for audit fact]
    AUTHOR -->|no| COMMITTER[Use configured committer as author]
    AUDIT -->|fact matched| RESOLVED[Use resolved actor]
    AUDIT -->|no usable fact| UNRESOLVED[Use explicit unresolved author]
    COMMITTER --> WINDOW[BranchWorker open window<br/>one author + one GitTarget]
    RESOLVED --> WINDOW
    UNRESOLVED --> WINDOW

    WINDOW --> TIMER{What closes it?}
    TIMER -->|silence timer| BASE
    TIMER -->|buffer limit or resync boundary| BASE

    CR[CommitRequest persisted] --> AUTHOR[Author from its audit fact<br/>named or unnamed]
    AUTHOR --> ATTACH[Attach to matching open window]
    ATTACH -->|same author + GitTarget| BASE
    ATTACH -->|no matching window| BENIGN[Ready=True, Pushed=False]

    BASE{Base trusted<br/>and worktree clean?} -->|yes| FLUSH
    BASE -->|no| FETCH[Fetch and reset first]
    FETCH --> FLUSH[Plan YAML edits + commit]
    FLUSH --> PUSH[PushAtomic<br/>reads the advertisement]
    PUSH --> DONE[Remote branch updated]
Loading

The important boundary is the open window: it accepts only the same author and GitTarget. A CommitRequest from another author never closes someone else's window; it resolves as a benign no-commit instead.

Remote moved while we were writing

If someone else pushes to the same remote branch before GitOps Reverser's push lands, or, for a branch not created yet, if its parent branch moved, the operator does not treat its local clone as authoritative. It keeps the finalized pending writes, fetches the new remote tip, resets the local clone, replays those writes, and pushes again.

flowchart TD
    LOCAL[Window finalized<br/>pending write retained] --> CHECK[PushAtomic checks remote ref]
    CHECK --> MATCH{Write branch, or while it is absent its parent,<br/>still at the expected commit?}
    MATCH -->|yes| ACCEPT[Push accepted]
    ACCEPT --> CLEAR[Clear retained pending writes]

    MATCH -->|no| FETCH[Fetch latest remote tip]
    FETCH --> RESET[Hard reset local clone to remote]
    RESET --> REPLAY[Apply each retained pending write's<br/>decided actions onto the new tip]
    REPLAY --> REFRESH[Create fresh commit hashes]
    REFRESH --> CHECK

    CHECK -->|attempts exhausted or fetch fails| RETAIN[Keep pending writes for later retry]
Loading

For a CommitRequest, success is reported only after the push reaches the remote. If the push had to replay on top of someone else's commit, status.commit is the refreshed post-replay commit, not the stale local commit from before the retry.

Somebody else pushed, and the cluster has to follow

A rejected push is one of four ways the branch worker learns the branch gained a commit it did not push. The others are a push-receiver delivery, the periodic refresh, and a Flux GitRepository that fetched another commit than the one an apply asked for, which the worker takes as a hint and checks the way it checks a delivery. All four set the same piece of worker state, the applier is behind, and the worker clears it by sending an apply request after a flush.

flowchart TD
    HINT["Receiver delivery, or a source<br/>that fetched another commit"] --> ADV["Read the ref advertisement"]
    REJECT["Rejected compare-and-swap push"] --> SEEN
    REFRESH["Periodic refresh fetch"] --> SEEN
    ADV --> OURS{"Tip is a commit<br/>we pushed?"}
    OURS -->|yes| DROP["Nothing to do"]
    OURS -->|"no, or cannot tell"| SEEN["Applier is behind"]
    SEEN --> PENDING{"Open window or<br/>pending writes?"}
    PENDING -->|yes| FLUSH["Close the window, publish the pending writes at once:<br/>a rejected push fetches and replays them onto the foreign commit"]
    FLUSH --> FLUSHED
    PENDING -->|no| FLUSHED["Flushed: every write decided<br/>before the apply came due is published"]
    FLUSHED --> REQ["Apply request to every applier on the branch,<br/>naming the tip, reason ForeignCommit"]

    style DROP fill:#eeeeee,stroke:#9e9e9e
    style REQ fill:#fff3e0,stroke:#ff9800
Loading

A delivery is handled on the worker's own queue, so it is never judged while a push is in flight: the push has recorded its commit before the delivery is read. A rollback onto one of the worker's older commits is foreign, because the advertisement names a tip other than the last one observed. A delivery also closes the open commit window, unless a CommitRequest holds it, so its publication finds the move.

A Git-side push usually reaches the worker through the flush itself: the push of the pending writes is rejected, the fetch finds a commit the worker did not push, the pending writes' decided actions are applied onto it, and the applier is asked once, for the tip that holds both sides. Applying the foreign commit on its own first would revert the pending API edits on the cluster and need a second apply to put them back.

The same flush comes before three other requests: Refusal, to the refusing target's applier when spec.onRefusal: Reapply is set; Repair, to an applier whose late write over a newer API edit the apply window withheld; and CatchUp, to a Kustomization or an Application alike, to adopt an object the API created in a folder that applier applies (a live ADDED event whose newest managers are not an applier's, or an object a resync added to Git) and to converge, once when the worker starts and once when a GitTarget first names the applier claim. A converge of the commit the applier last applied asks nothing; an adopt is sent even then, because applying the same commit again is what stamps the owner labels or tracking annotation on the new object. A push of the worker's own leaves nothing pending.

Three states still hold an apply back, because in them no flush can be made: a registered CommitRequest (a save defines its own commit boundary, and the apply waits for it), a parent recovery, and a checkout a failed write left dirty. An apply also waits until every initial snapshot of the GitTargets its applier covers is published (a GitTarget with no planned collection yet, or with a blocked watch, holds its applier the same way), and an empty tip (a remote that does not carry the branch) asks nothing until there is a commit to apply.

One worker writes a repository branch, so the flush has every unpublished edit on the branch in hand, whichever namespace its GitTarget is in.

Which apply is due lives on the worker, per applier: one attempt out at a time, and its own retry clock. The dispatcher in internal/applier carries out one attempt off the worker's queue and reports back. It reads each GitTarget again first (the same incarnation by uid, still naming this applier through its applier claim on this branch, still admitted, still consenting to a Refusal), checks that the Kustomization's source fetches this repository and branch, has the GitRepository fetch the commit when its artifact lacks it, asks the worker whether its decision still holds (the commit is still the tip), writes the operation through to the applier claim's status.operation, and only then resumes the Kustomization the applier claim holds suspended, with a requestedAt token. It follows the apply until that token is handled, suspends the Kustomization again with a second token, and reports the operation complete once that one is handled (apply-trigger.md). For an Argo CD Application, whose automated sync the claim holds off, the dispatcher checks that its single spec.source is this repository and branch, then sets the Application's operation to a sync pinned to that commit, never while another operation is set; the operation is complete when Argo CD clears it (the Argo CD applier). It never retries: an attempt that does not deliver gives the worker back what it carried, a refusal only while it still stands, and the worker decides the retry after a flush, once that applier's backoff has run out. A delivered request records an ApplyRequested Event, and the worker records the operation on the applier claim's status.operation. A failed request never reaches back into the publication that caused it.


What it writes to Git

A GitTarget owns one subtree (spec.path) on one branch. A new object follows the layout the folder already has (or a declared policy); once a document exists it is edited in place wherever it already lives. A populated target seeded from empty looks like:

team-a-config/                              # GitTarget spec.path
├── README.md                               # operator-managed bootstrap file
├── .gittargetignore                        # operator-managed bootstrap file
├── .sops.yaml                              # present only when encryption is configured
├── team-a/
│   ├── configmaps/app-config.yaml
│   ├── secrets/db-creds.sops.yaml          # sensitive types are SOPS/age encrypted
│   └── apps/deployments/api.yaml

The built-in default path is {spec.path}/{namespace}/{group}/{resource}/{name}.yaml, namespace first, the API group omitted for core resources, no version segment, and a .sops.yaml suffix for sensitive resources; a cluster-scoped resource uses the literal _cluster/ in place of the namespace (an illegal Kubernetes namespace name, so it can never clash with a real one). That default is what a new resource gets unless something more specific applies: a GitTarget's own declared placement policy, or a folder that kustomize builds from a single root. Details are in File Placement.


State ingestion and not losing deletes

Object state arrives through Kubernetes watches. Selected, relevant changes feed the writer; recovery establishes current state when event history is unavailable. Git commit grouping and pruning policy determine which intermediate states and removals appear in the repository.

Ordering within one watch

One watch delivers live changes in order. Independent watch handlers can still interleave observations of the same object; the ordering contract states that boundary. The manager derives resource identity and operation from the source event, including deletion intent, then sanitizes content before routing it.

Initial events establish current state, so their object RVs need not arrive in mutation order. The annotated initial-events-end bookmark marks completion. Ordinary bookmarks report progress, and an error such as 410 Gone requires rebuilding from current state.

Recovery: resume, replay, or list plus mark-and-sweep

The first attempt after process startup or watch replacement requests initial state, even when Redis holds a cursor. runTargetWatch passes resumeFromCursor=false until this watch's own replay has completed: the new render-fidelity revision needs fresh observation. Attempts after that, in the same loop, may open a normal watch from the stored RV. If history has expired, or no usable cursor exists, initialization establishes current state again.

Streaming initialization uses sendInitialEvents=true and resourceVersionMatch=NotOlderThan:

  1. The manager collects current objects until the annotated initial-events-end bookmark.
  2. It enqueues a snapshot reconciliation for that observed collection. Managed documents absent from the desired set are removed only when prune.mode: Always permits inferred deletion; Never and OnEvent retain them. Terminating objects are excluded from desired state by the deletion-as-intent rule.
  3. Subsequent live events enter the branch worker after the accepted snapshot request. This queue order does not mean the snapshot's Git write has already completed.

Reconciliation edits managed documents, which can share a YAML file. A snapshot for one namespace cannot establish absence in another. A failed observation cannot establish absence anywhere. See Mark and Sweep Resync.

There is no periodic object LIST or timer-driven object-drift sweep. If the API server rejects streaming initialization, targetWatchListAndStream first opens and buffers a watch, then takes a LIST snapshot. It enqueues the same scoped reconciliation and processes buffered events newer than the snapshot's RV. This ordering is the compatibility path for APIs without initial events.

Relevance filtering is product code

Watch has no audit policy, so it delivers every persisted MODIFIED, including controller status churn. The relevance filter reproduces that filter in product code, on the hot path:

  • Sanitization (internal/sanitize) strips status, managedFields, and volatile metadata before diffing, so runtime churn never masquerades as a desired-state change.
  • Followability (internal/typeset) encodes "controller-owned → don't mirror". A type that is not followable never gets a watch.
  • No-op suppression. A */status write bumps resourceVersion but its sanitized projection equals the prior commit; the writer diffs it to a no-op and discards it.

History granularity

Watch carries only the versions it observes. While connected it sees each MODIFIED; across a replay after 410, a compaction, or downtime it collapses every intermediate version into current state. The product is therefore a state mirror with opportunistic per-mutation history, not a guaranteed per-mutation change log.


Optional attribution

Attribution runs only when --author-attribution=true. A normal source posts audit EventList payloads to /audit-webhook/<audit-route>, where the route is KubeProvider.spec.attribution.auditRoute and defaults to <namespace>.<name>. The bare /audit-webhook endpoint is enabled only with --author-attribution-audit-route-annotation-key, for a trusted control plane that puts an audit route in each event. There is no supplementary body endpoint or body joiner, because watch (not audit) carries the object body. The handler applies an intrinsic accept gate (StageResponseComplete, a mutating verb, success, non-dry-run, a changed resourceVersion, and the /scale subresource only), then appends the accepted events' facts to that route's per-type fact stream: one append per type per request, not one per event.

The two halves never call each other. The receiver publishes and returns; the watch side follows the streams for the types it watches and keeps their facts in a bounded, TTL'd in-process index. They meet only through the keys a fact was filed under. That split is what lets the audit endpoint answer fast during a rollout, and what lets a fact published by one replica serve a watch running on another once the streams are shared (the Redis transport below).

The transport is selectable, because the index is what does the work and the stream is only how facts travel:

--author-attribution-transport Store When
redis (default) Redis Streams, per type, with a retention window a restarting process replays any install; required for more than one replica
memory an in-process ring single replica, no Redis; every fact is lost on restart, by design

memory removes attribution's need for Redis. It does not on its own remove the install's: --redis-addr also holds watch resume cursors, independently of attribution. Without that cursor store, reconnect attempts initialize from current state. The first attempt after a process restart requests initial state with either configuration.

Endpoint Role
/audit-webhook/<audit-route> One audit route's stream; some KubeProvider must carry that route
/audit-webhook Shared stream only when annotation routing is configured; each event names its route

The handler accepts a client certificate signed by the audit CA, but it does not yet bind that certificate to a named provider. Do not treat named routes as an isolation boundary for independently administered remote sources that share a client credential; provider-bound ingress authentication remains outstanding.

Optional, but never casual

Attribution being optional does not make it casual or take-it-or-leave-it. When it is enabled the operator does everything it can to name the real actor, but it values high certainty over a plausible guess. Attaching a name to a change is a consequential, sometimes politically charged claim ("this person did that"), so it must not be made lightly. The design therefore fails toward honesty: a weak, conflicting, late, or absent fact produces an explicit unresolved author rather than a guessed person or a misleading committer identity. It is better to say that attribution did not resolve than to assert an actor we are not sure of. In an installation that expects live mutations to be attributable, this outcome is a concrete signal to check audit delivery and attribution configuration.

Two engineering choices follow directly from that stance:

  • mTLS on the audit ingress is on by default. The audit server requires and verifies a client certificate (tls.RequireAndVerifyClientCert against a configured CA; --audit-insecure defaults to false). This authenticates membership in that CA's client set. It is not yet a sender-to-provider binding, so multi-source deployments must not share a credential across independently trusted sources.
  • Tests pin the behavior. Because a misattribution is a real harm, the attribution and resolver paths carry unit and e2e tests that prove the concrete cases: strong match, weak/last-key match, deletes whose audit RV differs from the watch RV, a collection delete joined by uid set and by scope, an aggregated API whose facts carry only a name, missing/late/expired facts, service-account vs human actor, impersonation, and the explicit unresolved outcome, so these certainty guarantees cannot silently regress.

Attribution fact shape

The fact is the smallest thing needed to name an author, not an object log:

Field Purpose
auditID diagnostics / dedupe
user / impersonatedUser author candidate (human or service account)
verb, subresource explain the write
responseStatus.code, dryRun reject failures and non-persistent requests (at the handler gate)
source provider, GVR, namespace, name, UID source partition plus exact join keys
response object resourceVersion exact watch-event match
stage timestamp recency

A deletecollection produces one collection fact instead: it names no object, so it keeps the request's selector and whatever uid set the API server returned, and every removal in its scope joins against it.

The index files each fact under the strongest key it has, first match wins, within a scope of (audit route, group/resource): (uid, rv) and (uid) together when it has both, else (uid), else (rv), else (namespace, name). A fact keeps every field it recovered but is not filed under weaker keys it could never be read by. A watch event always knows its object's uid, so a uid-keyed fact is never looked up by name, and a second copy would cost memory on every replica for the whole TTL and answer nothing. The one branch that files twice is (uid, rv) plus (uid): the first serves creates and updates, the second serves removals, one fact answering two different questions.

Entries carry a short TTL (minutes, not hours) and are bounded per type and in total; expiry is checked on read, so an aged-out fact is never joined merely because the sweep has not run. Old facts are never needed for correctness, because watch owns state.

The resolver and its grace window

A watch event waits a bounded grace window (--author-attribution-grace, default 3s) for matching evidence, then ships regardless. It does not poll: it registers a waiter under every key its query could match, looks once, and then sleeps until either a fact that could answer it is applied or the grace expires. Registering before the first lookup is what closes the race a fact landing in the gap would otherwise win. There is no Redis call on this path: the fast case is a map read.

The resolver takes the strongest evidence available, and the tier it used is the tier label on attribution_resolutions_total. Who that evidence named is the separate actor_kind label, in the same user/serviceaccount/none vocabulary git_commits_total{author_kind} uses:

Tier Key What it asserts
delete_sticky uid (a delete fact, sticky) who asked for this object's deletion
exact uid + rv this actor produced this exact version
deletecollection_body_uid uid in a collection fact's set the API server said this request deleted this object
latest uid (the object's own delete fact) who removed it, when the sticky slot was evicted
name namespace + name the same, for a fact with no uid (an aggregated API's usual shape)
deletecollection_scope namespace + selector + window a collection request covering it was made
resource_version rv (the escape hatch for a fact with no uid) a fact for this exact version, unidentified
absent none nothing usable arrived in time

Three rules carry most of the behavior. A fact about a deletion may not be replaced by a fact about a write: a removal fact takes a sticky, uid-keyed slot that the ordinary last-writer-wins structures cannot reach, and a removal of a terminating object asks that slot first. That slot is the one structure the fact TTL does not bound: a uid is unique across space and time, so the statement cannot be superseded, and its horizon is the index's caps instead. It is still in-memory: a restart re-warms the index from one TTL of stream retention like everything else. Without the slot, a finalizer patch's fact overwrites the deleter's, because both carry the resourceVersion the deletion stamped (spec/attribution.md). A removal is credited only to the request that caused it: a write at its own resourceVersion (an object relabeled out of a rule's objectSelector) or a fact about the deletion. The object's last edit says nothing about who removed it, so it never answers, and a removal without such a fact is unresolved. None of this reads the watch's label selector. And an exact-capable event may not fall through to the removal tiers: a create or update presents the resourceVersion its own write produced, so if the exact tier misses, the uid pointer may name an older, different author.

No branch on this path depends on the type. Every decision is made on the verb, on whether the event is a removal, and on which fields the event happens to carry. That is why an aggregated API needed no special case to attribute.

On a resolved tier the actor becomes the author: a human or a service account alike, always named by its own username (e.g. system:serviceaccount:flux-system:kustomize-controller). An absent resolution produces unknown (attribution unresolved) <attribution-unresolved@gitops-reverser.invalid> instead of a guessed actor. A late fact that arrives after a commit has shipped never rewrites it. With attribution disabled the resolver is absent and every commit is committer-authored.

The wait is head-of-line on its watch shard, so a resolution that sits out the grace delays the events queued behind it on the same (GitTarget, GVR, scope) goroutine. That cost, and what it once broke, is in Watch Event Ordering and attribution-removal-wait-options.md.

A CommitRequest's author is read from this index too, on the route of its GitTarget's KubeProvider, with the same grace (see CommitRequest Finalize).

Author and committer identity in Git

Git natively carries two identities on every commit, the committer (who created the commit object) and the author (who wrote the change), and GitOps Reverser uses both on purpose (commit.go):

  • The committer is always the operator: the configured GitProvider.spec.commit.committer, defaulting to GitOps Reverser <noreply@configbutler.ai>. Every commit, attributed or not, is committed by the operator, because the operator is what wrote it to Git.

  • The author is the real actor, but only when we are sure. On a strong attribution match the author is set to that actor. Git always carries an author, so it is never left blank, and it is never a guessed person. What fills it when we are not sure depends on WHY:

    • Attribution is switched off (configured-author mode, and reconcile/resync writes that have no actor at all): the author is the operator, identical to the committer. That is honest, because the operator is the author.
    • Attribution ran and did not resolve an actor: the author is the explicit sentinel unknown (attribution unresolved) <attribution-unresolved@gitops-reverser.invalid>. It is deliberately NOT the operator, because authoring it as the operator made a lost actor byte-identical to a configured-author commit, so the gap was invisible in Git history and only countable in a metric.

    A commit whose author is a person is therefore a positive statement that the operator is confident who made the change; one authored by the sentinel is a positive statement that it tried and could not tell.

The author identity is never fabricated as a person: a real author is always taken from the authenticated request, and an unresolved one is labeled as unresolved rather than attributed to anybody. The name and email come from the OIDC claims when the apiserver maps them; otherwise they come from the actor's own Kubernetes identity: the username, which for a controller is its service account (system:serviceaccount:<namespace>:<name>), with a stable derived email under noreply.cluster.local. The two user.extra keys the operator reads for the OIDC name and email, and the apiserver config that fills them, are in Wiring OIDC author claims.

A confidently attributed commit shows distinct author and committer lines, which git surfaces with --format=fuller. A human carries the OIDC display name and email:

$ git show --no-patch --format=fuller HEAD
Author:     alice <alice@example.com>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit:     GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000

A service-account actor (no OIDC claims) is named by its own Kubernetes identity, with a complete derived email and the operator still the committer:

$ git show --no-patch --format=fuller HEAD
Author:     system:serviceaccount:flux-system:kustomize-controller <systemserviceaccountflux-systemkustomize-controller@noreply.cluster.local>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit:     GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000

When attribution is off, or when a reconcile/resync write has no actor because attribution was never attempted, the author is set to the operator as well, so both lines are identical. A Git UI will show this as a single "author". This is configured-author mode: the operator created the Git commit and makes no claim about a Kubernetes actor.

$ git show --no-patch --format=fuller HEAD
Author:     GitOps Reverser <noreply@configbutler.ai>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit:     GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000

Wiring OIDC author claims

The operator never talks to the identity provider. It reads two fixed keys from the request's user.extra map, from the audit event that names the actor:

user.extra key OIDC claim it carries Used for
configbutler.ai/claims/display-name name git author Name
configbutler.ai/claims/email email git author Email

So the ID token the provider issues must carry the standard name and email claims:

{
  "iss": "https://idp.example.com",
  "sub": "248289761001",
  "name": "alice",
  "email": "alice@example.com",
  "aud": "kubernetes",
  "exp": 1751284805
}

The cluster operator maps those claims into the two keys with a structured AuthenticationConfiguration (apiserver.config.k8s.io, beta from Kubernetes 1.30):

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: https://idp.example.com
      audiences: [kubernetes]
    claimMappings:
      username:
        claim: email          # the Kubernetes username; `sub` also works
      extra:
        - key: "configbutler.ai/claims/display-name"
          valueExpression: "claims.name"
        - key: "configbutler.ai/claims/email"
          valueExpression: "claims.email"

Both mappings are optional: a missing or unusable name falls back to the username, and a missing or invalid email falls back to a derived address under noreply.cluster.local.


Watch event ordering

The grace window is a per-event wait on a single-threaded watch goroutine. Each running collection watch uses one chosen GVR and namespace scope. The downstream GitTargetEventStream → BranchWorker path preserves enqueue order in a FIFO.

  • Order within one watch is preserved. The loop resolves attribution and routes one event before reading the next. Later events on that watch cannot overtake the attribution wait.
  • Independent watches can overlap. An all-namespace watch and a named-namespace watch can deliver the same object. Their concurrent handlers establish no common mutation order, and the worker's FIFO does not sort their arrivals by resourceVersion. The target watch plan records this limitation.
  • The wait affects throughput and grouping. Each unresolved event can delay its watch by the grace window. Accumulated queueing can add further delay, and different authors can split commits. Intermediate object states can also collapse into one commit window.
  • Branch writes remain serialized. Different objects can share a multi-document YAML file; the branch worker edits their documents sequentially. That serialization prevents concurrent file edits but does not establish source ordering across overlapping watches. RVs also do not provide a common clock across resource types.

The current processing model and worked examples are in Watch event ordering under the attribution grace window. For server-side selectors, initial state, bookmarks, and resume options, see Kubernetes watch options.


Rule and type resolution

How a user's WatchRule becomes "this GitTarget follows these concrete types in these namespaces."

RuleStore

An in memory cache populated through the shared compile paths (watch.CompileWatchRule / watch.CompileClusterWatchRule) that both the controllers and the startup bootstrap call, so a restart cannot seed an ungated rule. Compiled rules carry the full chain from rule to GitTarget, GitProvider, branch, and path, plus each WatchRule item's RESOLVED source-namespace set. It is read by the watch manager (to build watched type tables for GitTargets) and the rule change reconcile path.

APIResourceCatalog

A thin normalizer for each scan: it turns one discovery result into a policy annotated typeset.Scan and keeps only mechanical bookkeeping. All judgement across scans lives in the typeset registry (Registry.UpdateFromScan): a failed group/version keeps serving last known facts instead of looking like an empty API surface, and a group/version that vanishes from a complete scan rides a removal grace rather than being pruned. Both protect against accidental Git deletions on a discovery blink (see typeset-owns-discovery-grace.md). Every active source context has its own catalog and registry, so a CRD served only by prod-eu-1 is never resolved for a target mirroring prod-us-1. Catalogs refresh at startup and on the 30-second reconciliation cadence. The CustomResourceDefinition and APIService informers run only in the control plane, where the operator's own CRDs live; a remote source's API-surface change is discovered on that periodic refresh. A target's plan pass never calls discovery itself; for a source nothing has observed yet, it asks for a refresh at once.

TypeRegistry and followability

internal/typeset is the single decision surface for "can this type be followed?" Each TypeRecord carries GVK/GVR identity, scope and preferred version facts, origin classification, subresource facts (including usable /scale bindings), sensitivity policy, and one Followability verdict. Manifest analysis and delete/scale resolution with only GVR all read it. Demand is applied one layer up: the watched type table intersects the followable set with what each GitTarget claims, and only that intersection gets a watch.

WatchedTypeTable

A projection for each GitTarget from the type registry, filtered by that target's rules, recording resolved GVK/GVR/scope plus namespace coverage. This is where rule matching effectively happens: it resolves the set of (GVR, scope) a GitTarget claims, so the watch manager opens one watch per claimed ∩ followable (GVR, scope) and scopes each watch's events back to that GitTarget's source namespaces. A sourceNamespace: "*" rule resolves here to ONE cluster-wide collection (the empty namespace, which for a namespaced GVR is the all-namespaces collection), so it has a single stream and a single mark-and-sweep boundary however many namespaces it covers. That collection is a peer of any named-namespace collection on the same type, never a replacement for one.


Watch ingestion and reconcile

Desired state comes from one raw watch per (GitTarget, GVR, scope). Each event is sanitized, diffed against current Git content, and applied. There is no separate per-type object store to reconstruct, because Git already holds current state. This section is the authoritative contract; the watch-first design record is historical context.

The watch manager

The watch manager is a controller runtime Runnable (NeedLeaderElection). It owns type level discovery, the per-GitTarget watch sets, and the watched type tables for GitTargets. Its object-state intake is the watches themselves. Every active KubeProvider has an independent source context; its catalog, registry, dynamic client, and reachability can fail or recover without borrowing facts from another source. Discovery informers (on CustomResourceDefinition and APIService) run only in the control plane and accelerate its catalog refresh; remote source catalogs use the periodic refresh.

On Start it bootstraps the RuleStore, refreshes the API catalog and TypeRegistry, resolves rule selection, and builds the collection watch plan.

Opening watches

On each GitTarget reconcile, rule resolution combines selection with discovery and mirroring eligibility. targetWatchStreams groups selected served versions by group/resource/namespace, and chooses one version per collection. The plan diff keeps unaffected watches running, starts new collections, and replaces a collection only when its served version changes.

Each managed watch has one goroutine. Its first attempt requests initial state, enqueues scoped snapshot reconciliation, and then routes every live object event through its desired-state change filter and author resolution. Later attempts may resume a cursor; see Recovery: resume, replay, or list plus mark-and-sweep.

There is no periodic object LIST or timer-driven object-drift sweep. Source-namespace selection uses delegated rule configuration and the source credential's permissions; it does not require reading source Namespace labels. Control-cluster Namespace labels can govern a Selector grant, which is a separate provider-reference authorization boundary.

Rule change reconcile

A rule or target change and an API-surface change arrive on two different paths.

A WatchRule / ClusterWatchRule / GitTarget change reaches a GitTarget through the GitTarget controller, which Watches those objects (generation change predicates) and queues the affected GitTarget again. On reconcile the GitTarget declares its watch set again, and the watch plane's next plan pass resolves its claimed (GVR, scope) set against the source's current catalog; a type a new rule starts watching gets a new watch opened (with a sendInitialEvents replay) and a type no longer claimed has its watch closed.

A control-plane CustomResourceDefinition or APIService change reaches the same re-resolution through the watch manager instead. Its informers wake a catalog refresh, which re-resolves the watched type tables and brings the running watches into line. The GitTarget controller does not watch those objects. A remote CustomResourceDefinition or APIService change has no informer at all; the next periodic source-catalog refresh performs the same re-resolution.

Mark and sweep resync

The BranchWorker applies a reconcile by scanning the GitTarget subtree and building a manifest plan (this write side is shared with live writes):

  • before anything is planned, a structure-only acceptance gate runs over the scanned subtree; if it finds content the operator cannot safely manage: a kustomization using an unsupported feature (secretGenerator or a hash-suffixed or merging configMapGenerator / inline or JSON6902 patches / components / helm / replacements / transformers / name(pre|suf)fix / remote bases) or malformed images:/replicas: overrides, a duplicate manifest identity, an impure managed file, or a standalone non-KRM / invalid YAML; (a path-based strategic-merge patches: entry is tolerated as read-only build context, and an overlay reading ../../base is rendered by reading that base, and neither refuses the folder) the whole apply is refused: nothing is committed, GitPathAccepted=False, Stalled=True, and Ready=False with reason UnsupportedContent until a human cleans the path;
  • desired resources are upserted through the same content derived path as live writes;
  • existing managed documents that are watched but absent from the desired set are deleted, under spec.prune.mode: Always only;
  • in a coordinated folder, an object another applier owns (by Flux's owner labels or Argo CD's tracking annotation, read before sanitize strips them) is neither upserted nor treated as missing, so its document is left alone and counted in apply_ownership_conflicts_total;
  • the operator's own build directives (kustomization.yaml, .sops.yaml) and other allowlisted auxiliary YAML are retained, not materialised and not refused;
  • nothing is committed if the apply cannot complete safely.

A write-boundary refusal is a different thing and is scoped accordingly: the folder is accepted, and one edit had nowhere to land in it. Only those can earn the apply request a target opts into with spec.onRefusal: Reapply, and only when the edit is to a document the folder already holds. A folder-level refusal above is excluded deliberately: reapplying Git cannot repair a folder, and only a human can.

The acceptance gate is structure-only on purpose: it never refuses on a discovery-derived followability fact (unwatched / out-of-scope), which can blink on a discovery wobble; only facts that are true from the path's structure alone block a GitTarget. The same gate runs on the live write path, so a refused path is never written into by a racing live event either. See unsupported-folder-refusal-plan.md.

A reconcile is collection scoped (ResyncScope): the sweep is restricted to the (group, resource, namespace) the snapshot was gathered over, so anchoring one collection again never disturbs another's manifests. The desired set for the sweep is the sendInitialEvents replay (everything marked ADDED up to initial-events-end), so a delete is reconciled only after the replay completes.

The snapshot decides once. The upserts and deletes its first commit decided are what a replay onto a newer tip applies, and the sweep never runs again there. On a branch an applier applies, a snapshot collected while Git was ahead of the cluster (a commit pushed and not applied yet), collected across an apply, or arriving while Git is ahead is discarded instead of decided, and a fresh one is asked for once Git is no longer ahead; it would otherwise revert the unapplied commit. A GitTarget's initial snapshots are exempt, and after 5 minutes of discards the next snapshot is used anyway. See stale snapshots.

The render-vs-live gate

The operator renders a folder with its own kustomize build, but Flux or Argo CD may apply it with context the folder does not hold: a Flux postBuild.substitute value for a ${var} token, or an Argo CD spec.source.kustomize override. Mirroring a live value over such a token would replace the token in Git with whatever the applier substituted. So every GitTarget carries a gate: each collection's replay compares the rendered ${...} values of the documents it governs with the sanitized live objects, and the gate opens only once every collection of the target has reported a clean replay under its current revision. The gate is per GitTarget, so one collection that is still replaying closes live writes for all of them. RenderMatchesLive publishes it:

  • Unknown (Rechecking): a collection is new, restarted, or still replaying, which is every collection at operator start. A live event that arrives is judged at once and then held; an atomic write is dropped at once. When the gate opens, the held events are released in arrival order, before any later event, and committed with their authors. An applied snapshot of a collection supersedes the events of that collection held before it. The hold is bounded, at 10 minutes from the first held event and by the branch's retained-byte budget. Past either, the held events are dropped and a fresh snapshot is asked for, which commits the current state as the operator. A delete among the dropped events comes back only under prune.mode: Always.
  • False (RenderDoesNotMatchLive): a rendered token disagrees with live. Live and atomic writes are dropped and the target is Stalled. Scoped resyncs still run, so a fresh replay can measure the folder again. A repair in Git does not reopen the gate on its own; a replay of the target's collections that measures them clean does.

Every dropped write is counted in git_writes_dropped_total{reason} (render_fidelity_pending for a hold that gave up, render_fidelity_diverged for a divergence). Nothing replays a dropped write.


Git write architecture

Who owns which question

Three objects meet at a write, and each answers a different question. Keeping them apart is what stops a fact being published by something that cannot actually prove it.

Question Proved by Published on
Can this repository be reached with this credential? the GitProvider controller's connectivity check GitProvider.status: Ready, lastVerifiedAt
Which repository is this? the GitProvider object's identity (metadata.uid with spec.url) nothing: it is carried on the branch worker and compared, not reported
May this GitBranch write branch B? the GitBranch controller, opening a push session with the GitProvider's credential the GitBranch's Ready (BranchConflict, PushSessionFailed)
Where is branch B on the remote, and when was that proved? the branch worker, on its queue GitBranch.status.remote, written at most once a minute
What is this repository being used for? the GitBranches that name its GitProvider kubectl get gitbranch, and each one's status.gitTargetCount
Is this folder accepted, streaming and matching live? the watch plane, per GitTarget the GitTarget's conditions and status.placement
Does the applier need to apply, and is now safe? the branch worker, on its queue the applier claim's status.operation and an ApplyRequested Event
Can the applier be asked at all? the applier claim controller, reading the applier in its cluster the applier claim's Configured, and the GitTarget's ApplierClaimReady

A branch worker is keyed by the repository branch, normalized as internal/repourl compares URLs, and writes for the GitBranch that holds it, with that GitBranch's GitProvider credential, signing key and metric labels. The branch claim (internal/git/branch_owner.go) lives in the operator's memory:

  1. A GitBranch becomes a candidate once its credential opens a receive-pack session on the remote. One that cannot claims nothing, so creating it first blocks nobody.
  2. Among the candidates for one repository branch, the oldest by creationTimestamp (then UID) holds it.
  3. The holder stays a candidate through a later credential or network failure, so it keeps the branch and its pending writes; any other candidate whose proof fails stops being one.
  4. After a restart the claim is decided again by 1 and 2.

The GitTarget reconciler, the worker sweep and EnsureWorker all read the claim, so a GitBranch that does not hold its branch gets no worker, however close together two were created. A worker serves exactly the GitTargets whose spec.gitBranchRef names its GitBranch, which is how the event router finds it with no read on the event path. A provider recreated under its name does not move a live worker; it replaces it, and the replacement starts with no clone, no base trust and no observation.

A GitBranch carries a finalizer. Deleting it releases the claim, then stops its worker, which closes the worker's intake, pushes or refuses every pending write and settles every CommitRequest it holds; only then does the finalizer go. Nothing deletes the branch on the remote.

GitBranch.status.remote is a branch fact, and it is worth being exact about that, because the reverse reading is tempting and wrong. A fresh lastVerifiedAt says the branch tip was recently proved. It does not say that any one target's folder was scanned, that its documents were published, or that its streams are running. Those are the GitTarget conditions' job.

Delivery is not publication, and the split is deliberate. The worker wakes its GitBranch every time it proves something, a push or a fetch. Publication is a status write (an etcd write that invalidates every watcher's cached copy of the type), so the GitBranch writes the stanza at most once a minute, and comes back when that floor lifts.

Everything about a branch happens on that branch's queue

Reading or writing a branch's repository state is the branch worker's job, and the worker does it on one FIFO queue, on one goroutine. Live events, resyncs, CommitRequest attaches, periodic refreshes and push-receiver deliveries all enter the same queue and are served in order. Which writes are decided and which of them are published is therefore a fact one goroutine holds, which is what lets the worker flush before an apply request goes out. Nothing else opens a connection on a branch's behalf, and nothing else touches its clone. That is what makes "the base is trusted" a statement one goroutine can hold.

The GitProvider controller's connectivity check is the one thing outside that rule, and it is outside it deliberately: it answers a question about the repository that has to be answerable when no branch worker exists at all, including before any GitTarget has been created.

BranchWorker

BranchWorker owns a local clone and a single FIFO event loop for its repository branch. Events accumulate in one open commit window, which accepts only one (author, GitTarget) pair at a time:

  • same author + same GitTarget: append to the window;
  • different author or GitTarget: finalize the current window first;
  • repeated writes to the same Git path inside a window use last write wins.

The window finalizes when its idleTimeout passes with no new matching event, its maxDuration runs out, the retained buffer reaches --branch-buffer-max-size (default 8Mi), an attach: Next CommitRequest closes it, a resync, an atomic batch, a push-receiver delivery or shutdown arrives, or an apply comes due for an applier on the branch. The timers are the GitTarget's spec.commit.window until a CommitRequest attaches and replaces them. Successful local commits are retained until a fixed push cooldown (5s) allows a push, which prevents remote push storms during bursts. The one exception is an apply's flush: the writes decided before the apply came due are pushed at once, cooldown or not, because the apply waits for them.

Local clones and conflict retry

Local clones live under /tmp/gitops-reverser-workers/{GitBranch UID}: one directory per GitBranch, named by its metadata.uid. A GitBranch pins its branch (spec.branch is immutable), the branch claim gives a repository branch to one GitBranch at a time, and a UID is minted fresh for every object, so no two live workers share a directory and a GitBranch recreated under its name starts in a new one. The branch name is not part of the path, so its slashes cannot nest one checkout inside another. The repository is the one thing the UID does not pin: a GitProvider recreated against another URL keeps the GitBranch, so its replacement worker reuses the directory, after the old worker has removed the clone in it. A worker never trusts a checkout it finds on disk: its first cycle repoints origin, fetches and resets.

A retiring worker removes its own directory. The worker sweep, which runs every minute and after every GitTarget delete, also removes every directory that no live worker uses and no existing GitBranch names. That covers a GitBranch deleted while no worker ran for it, which happens across a restart that keeps the disk. The root is an emptyDir by default, so the checkouts survive a container restart but not a new pod. The chart README shows how to mount a volume there instead.

PushAtomic checks the remote ref before pushing. If the remote diverged it smart fetches the latest tip, hard resets the local clone, replays the retained pending writes against the fresh tip (refreshing commit hashes), and retries up to the attempt limit. This is valid because every pending write is rebuilt from sanitized API state; nothing depends on locally edited files. A write carries the actions its first commit decided, and a replay applies those and plans nothing else. An object whose plan changed nothing is no action, so its old content can never revert what the newer tip changed, and a write with no action left settles at once without a push. A resync keeps the upserts that changed the tree and the documents its sweep removed: a replay plans each upsert onto the new tree and removes each document if it is still there and the prune mode still sweeps, and never sweeps again, so a document the newer tip added survives. A save is the exception: a write carrying a CommitRequest keeps every object it captured, and the push decides its outcome. See the pending-writes design.

A branch whose remote has been deleted takes the same path. The advertisement does not carry the branch at all, which is reported as a moved remote with a zero hash, so the replay re-roots on the remote's default branch and the retry re-creates the branch with the retained writes on top.

A branch that does not exist yet is created from its parent (the GitBranch's spec.parentBranch, or the remote's default branch when that is omitted) only when a write has something to commit, so an idle or in-sync target never creates it. Until then the worker compares the folder against the parent. Periodic refresh, ten minutes by default, follows the parent from the same advertisement and updates the idle view independently of the publication check. The push records the parent and the commit it started from as its root, and treats a moved parent like a moved branch: fetch, reset, replay, check again. A branch somebody else created in the meantime is a moved remote too, so the replay builds on their commit instead of overwriting it; that holds when the server refuses our upload too, because the fallback reads both refs from one advertisement. A cycle planned on an empty repository that has since gained refs is refused at the advertisement (RepositoryNotEmptyError) and replayed, never pushed as an orphan. A parent that is missing, or a default branch that does not resolve, opens parent recovery on the worker: it probes on its own backoff and publishes what it held back once the parent exists (see docs/design/gittarget-parent-branch.md). Proved end to end by the e2e spec "New write branch starts from its parent"; that publication alone finds a moved parent is proved by worker tests with no refresher.

When the cycle fetches at all. Only the first commit of a cycle may fetch, and only when the base is untrusted or the worktree is dirty (see the ground rule above); a healthy publishing target plans straight onto its own last push. Once writes are retained the guard flips off entirely, because a reset would destroy the local commits those writes already produced. Anything that needs a fresh tree with work in hand therefore resets and replays rather than resetting alone, applying the retained writes' decided actions onto the new tip. One function on the event loop does that, materialize: every commit and push the loop makes runs it first, and the loop's commit refuses a checkout that is not the projection of the retained writes. A worktree a failed write left dirty, retained writes whose commits a reset discarded, a parent change, and the snapshot a resync judges against all reach it.

What a reset has to leave behind, now that it is the only cleanup. A hard reset restores tracked files and stops there, so a document a failed write created, and the placement directory it created for it, both survive one. While every cycle fetched, the next one swept them up by accident; a cycle that plans on a trusted base would instead commit them under whoever wrote next. So the reset explicitly discards untracked and newly staged leftovers and prunes the directories that held them (discardWorktreeLeftovers), and a write that fails while creating directories removes the ones it made on its way out. Only then is worktreeDirty cleared.

When the remote cannot be reached

A decided write is kept until a push lands. When a commit or a push cannot reach the remote, the worker keeps every decided write pending (pending_writes_loop.go), with its author, its message, and any save riding it, and tries again on one schedule: 10 seconds, doubling to 5 minutes (retry.go). The retry does not wait for another edit. Until it is due nothing else on the branch spends a connection: a write that needs one stays pending, and a resync is answered with the failure the retry is waiting out and kept. Only a failure of the write itself, such as a refused plan, drops an entry. Every call to a Git server is bounded, two minutes per call and five per push cycle, so a server that stops answering cannot hold the branch.

The kept work is bounded at intake (intake.go). While a retry is pending, --branch-buffer-max-size counts everything accepted and not yet published: queued items, the open window, the pending writes and waiting saves, each with a 1 KiB per-item charge. The first payload that does not fit pauses the branch's intake until a push lands; room under the budget again, or a remote that can be read but refuses the push, keeps it paused. A paused branch's watch streams wait to be woken instead of reconnecting, and their cursors stay where the last accepted event left them. Withdrawals, refreshes, and shutdown are lifecycle work and are never paused. The budget counts serialized bytes; a queued object costs several times that in memory.

Every GitTarget on the branch reports the outage from one worker report (publication.go): Ready=False, Reconciling=True, Stalled=False under Progressing, naming since when, what failed, and whether intake is paused. A held save says the same in its WaitingForPush message, and the git_retained_*, git_intake_paused and git_next_retry_timestamp_seconds metrics describe the backlog. What this recovers, and what watch history and a restart cannot, is the recovery contract; the event pipeline overview draws it.

Durability of the write queue (planned)

A BranchWorker's queue (the open commit window's retained writes plus any local commits not yet pushed) lives only in process memory today, bounded as described above. Persisting it to Redis is planned, for two reasons:

  • High availability. Multi-pod HA needs a durable, cross-pod write queue so a branch's accepted-but-unpushed work survives a failover: the pod that takes the branch-shard lease resumes that queue instead of starting from an empty buffer. This is part of the HA / GitTarget distribution plan.
  • Crash safety that watch replay cannot guarantee. A crash drops the in-memory queue, and those writes cannot be reliably re-derived by replaying the watch, because Kubernetes does not guarantee it can replay events from every resourceVersion. The apiserver compacts old versions, so resuming from a stored cursor can return 410 Gone; recovery then falls back to a full sendInitialEvents replay plus mark-and-sweep, which rebuilds current state but collapses the intermediate versions (see History granularity). Persisting the queue lets accepted-but-unpushed writes survive a restart on their own, independent of whether the watch can resume from where it left off.

Manifest aware writer

For each commit the writer scans YAML files under the GitTarget path, builds a manifest store without bytes keyed by resource identity, resolves each event (or each desired resource, for resync) to one action, hydrates only touched files into buffers for the commit, and flushes only changed or deleted files.

  • Upserts: if a managed document for the resource already exists, patch it in place (preserving siblings in a multi document file); if it is sensitive, encrypt the whole document again at its existing path; if no document exists, place a new file per File Placement (declared policy, then the folder's one kustomize root, then the built-in path).
  • Kustomize override edit-through: a live value produced by a well-formed images: or replicas: entry in the document's kustomization chain is written back to that entry (comment-preserving, only fields the entry already declares); the source manifest keeps its bytes. Anything the inversion cannot express falls back to the plain in-place patch. See images-and-replicas edit-through design.
  • Deletes: use the manifest identity index, so a moved manifest can still be deleted even when it is not at the built-in path.

File placement

Placement runs only for a resource with no existing document in the target. Existing resources are match first: once a document exists in Git, updates and deletes use its current location, found by manifest identity rather than by path, instead of recomputing placement. So a change to how new files are placed never moves a file already in Git. A new resource is placed by the first of these that applies (internal/manifestanalyzer/placement.go, design):

  1. Declared policy (spec.placement). A GitTarget can declare a byType map (exact [group/]version/resource → path template) plus a default template, rendered from a small brace-variable path language ({namespace}, {group}, {resource}, {name}, …).
  2. The folder's one kustomize root. When the whole writable subtree is governed by exactly one supported kustomization.yaml, the file lands beside it and gets a resources: entry in the same commit. This step is a structural fact rather than a reading of the folder's conventions: the built-in path below is a tree a resources: graph cannot reach, so a file written there would never be rendered. Two supported kustomizations is ambiguous and declines.
  3. Built-in fallback. Otherwise the built-in default {spec.path}/{namespace}/{group}/{resource}/{name}.yaml: namespace-first, group omitted for core, no version, _cluster/ for cluster-scoped, .sops.yaml for sensitive.

The layout of the folder's other documents is not an input. Where a new file goes is decided by the three rungs above and nothing else, so a human's edit to the repository never changes where the operator writes next. A layout the ladder cannot derive is declared in spec.placement, and gitopsreverser_placements_total{source="builtin"} names the target and type that needs the line.

Sensitivity is a write-safety classifier, not a placement input: whatever path is chosen, a sensitive resource is written encrypted, is never appended to an existing file, and is never co-mingled with a plaintext document. When those guarantees cannot be honoured (e.g. a bundling default would route a sensitive resource into a shared file), the resource is refused fail-safe rather than written unsafely: logged per-resource, counted in the resync summary as placementSkipped, and counted by gitopsreverser_placement_refusals_total{reason}.

Bootstrap, encryption, and signing

  • Bootstrap (bootstrapped_repo_template.go): the first write to a GitTarget path stages a README.md, a .gittargetignore and, when encryption is configured, a .sops.yaml with age recipient rules. Existing files are preserved. For a GitTarget that names an applier claim, a resync into a folder the branch does not hold yet commits these files even with no object to write, because the folder has to exist before it is applied: Argo CD retries a sync of a missing path for minutes, and Flux fails it. The bootstrap files are never deleted, so the folder outlives its last resource.
  • Encryption (encryption.go, sops_encryptor.go, sensitivity policy): core Secrets are sensitive by default; --additional-sensitive-resources adds more. Sensitive resources are never written in plaintext. If encryption is required and unavailable, the write fails before any plaintext file is created. Encrypted output is cached by metadata + plaintext digest to avoid redundant SOPS work.
  • Signing (signing.go, sshsig/): commits use OpenSSH signatures, with the key read from GitProvider.spec.commit.signing.secretRef or generated when configured.

CommitRequest finalize

A CommitRequest finalizes the open commit window for its GitTarget. It is picked up in one of two ways: by the CommitRequestReconciler on the control cluster, or by the CommitRequestServer's watch of a source cluster, through the KubeProvider whose spec.commitRequests reconciles it. Both hand it to one core, which takes one step at a time, each decided again from the object as it is stored (contract):

  1. The owner. The GitTarget's KubeProvider alone picks the path that serves the request: the control-cluster path for an inCluster provider, that provider's watch for a kubeconfig. Any other path writes nothing but GitTargetNotFound, once attachTimeout has passed with no GitTarget, and only into an empty status. Every status write is conditional on the resourceVersion it was decided from.
  2. Admission and the author. A grant in the GitTarget's spec.allowedCommitRequests must admit the request: consumerNamespaces on the control cluster, sourceNamespaces on a source cluster. The request's author comes from its own audit fact, on the route of the GitTarget's KubeProvider, awaited with the attribution grace; a request no fact names, on a provider that has received audit facts, is AuthorUnresolved and saves nothing.
  3. The claim. The owner writes the first status, status.author and Ready=Unknown (Saving), and only then attaches the request to the worker (AttachCommitRequest), which waits for a window until registration + attachTimeout. The worker binds it to an open window only when the author state and GitTarget match, first come first served, and replaces the window's timers with the request's own, so time spent waiting never shortens the collection. It never finalizes another author's window; a window carries at most one request. The owner reports the phase the worker reports: WaitingForWindow, CollectingWindow, WaitingForPush.
  4. The window finalizes when the request's timers close it (or when it closes for any other reason). If a finalize closes an open window, the worker always schedules a push, so a window closed by an otherwise no-op resync is not stranded.
  5. Outcomes resolve on push and are reported as conditions: a pushed commit sets Ready=True / Pushed=True with branch/commit; a benign no-commit sets Ready=True with the reason on Ready and Pushed=False; a failure sets Ready=False / Stalled=True with a message. Between the window's finalize and that push the request is still in flight, which matters because the owner keeps re-sending its attach until it reads an outcome: the worker marks the request committed rather than forgetting it, so a re-send is recognized as the same request and cannot resolve early or claim a later window. A worker that stops without pushing fails the request instead of leaving it to time out. AlreadyPresent therefore means the remote confirmed there was nothing to add, not that a local plan found no diff. A conflict on the outcome's write re-applies the outcome onto the fresh object; the save is never restarted.
  6. Retention. A finished request is stamped configbutler.ai/delete-after and deleted once it has passed: always on the control cluster, and on a source cluster only with commitRequests: ReconcileAndDeleteFinished.

A request whose audit event names nobody claims no actor. The final Git author remains the attached watch window's author: configured committer when attribution was disabled, or the explicit unresolved author when live attribution ran but could not resolve.


Controller wiring

Controllers watch their dependencies so dependents reconcile quickly after spec changes:

  • GitTargetReconciler watches GitBranch, GitProvider, KubeProvider, Namespace, WatchRule, and ClusterWatchRule. Provider changes re-check connection and authorization inputs; control-cluster Namespace label changes re-check provider-reference access. Rule changes re-resolve selected collections. It deliberately does not watch encryption Secrets, so their recovery is picked up by periodic reconciliation without retaining every Secret value in the control-plane cache. It watches the applier claims its GitTargets name, for ApplierClaimReady.
  • ApplierClaimReconciler (one per applier claim kind) watches its applier claims, the GitTargets naming them, GitBranches, GitProviders, KubeProviders and Namespace labels. It does not watch appliers: it reads the Flux Kustomization or Argo CD Application on its tick to set Configured, so a cluster without the Flux or Argo CD CRDs needs no informer for them.
  • WatchRuleReconciler / ClusterWatchRuleReconciler watch GitTarget, KubeProvider and Namespace labels, plus the watch manager's stream-state events for StreamsRunning, populate the RuleStore, and start the rule-change reconcile.
  • GitProviderReconciler validates reachability and manages the signing key lifecycle. It watches nothing but its own objects.
  • GitBranchReconciler watches GitProviders, Namespace labels and the GitTargets naming it. It proves the credential with a push session, keeps the branch claim, and publishes the worker's status.remote (see Git write architecture).
  • GitReceiverReconciler watches only its own objects and publishes status.webhookPath once the token Secret holds a token.
  • KubeProviderReconciler validates source-connection inputs, decides whether the provider may be used at all (the installation's inCluster consent, and which namespace holds its audit route), and projects its readiness to dependent targets. A provider created, deleted or moved to another route wakes the others carrying that route, so an AuditRouteConflict clears without waiting for a requeue.
  • CommitRequestReconciler serves the control cluster's CommitRequests through the shared core, four at a time, so one request's author wait does not hold up the others. It reads through the uncached API reader, and shares the watch plane's author resolver and fact streams, which are nil with attribution off. The CommitRequestServer serves the source clusters' requests through the same core, one watch per KubeProvider that reconciles them, and reports each to the KubeProviderReconciler, which writes it as CommitRequestsServed.

Dependency watches use narrow predicates to avoid status-only heartbeat churn: a KubeProvider also admits a Ready transition, and a Namespace admits only label changes. GitProvider, GitBranch, GitTarget, and CommitRequest carry immutability constraints where a spec change would orphan a materialized subtree or invalidate an in-flight finalize.


Startup sequence

Defined in cmd/main.go:

flowchart TD
    A[Parse flags + logger + build info] --> B[Init telemetry / metrics server]
    B --> C[Create controller runtime manager]
    C --> D[Create RuleStore]
    D --> E[Create WorkerManager + register Runnable]
    E --> R{receiver-bind-address set?}
    R -->|yes| Ri[Add push receiver listener; route deliveries to branch workers]
    R -->|no| Rj[No receiver; foreign pushes found by rejection and refresh]
    Ri --> F
    Rj --> F
    F[Create Watch Manager + EventRouter; inject TypeRegistry] --> P[Create apply dispatcher + applier claim status writer; wire them to the workers]
    P --> G[Register WatchRule + ClusterWatchRule controllers]
    G --> H{redis-addr set?}
    H -->|yes| Hi[Create Redis cursor store + wire WatchCursorStore + readiness gate]
    H -->|no| Hj[Skip cursor store; reconnects initialize again]
    Hi --> Hq{author-attribution?}
    Hj --> Hq
    Hq -->|yes| I[Select fact transport; start fact index + follower;<br/>wire audit fact extractor + audit HTTP server + resolver]
    Hq -->|no| J[Configured-author: no fact streams or index; audit webhook skipped]
    I --> K[Setup + register Watch Manager]
    J --> K
    K --> L[Register GitProvider + GitReceiver + KubeProvider + GitBranch + GitTarget<br/>+ FluxClaim + ArgoCDClaim + CommitRequest controllers; CommitRequest server]
    L --> M[Add cert watchers + health checks]
    M --> N[mgr.Start]
Loading

Redis is optional in configured-author mode. When --redis-addr is set, the cursor store is wired and a Redis readiness gate keeps the pod not-ready until Redis is reachable. Later reconnect attempts can resume stored RVs. When --redis-addr is empty, the cursor store is skipped and reconnects initialize again. The first attempt after startup or watch replacement requests initial state in both cases. The binary's --author-attribution flag defaults to on: the fact transport is constructed, the fact index and its follower are started, the audit HTTP handler is wired with the fact extractor, the watch manager gets the author resolver, and the audit ingress is added to /readyz. Which transport it builds is --author-attribution-transport: redis (the default) requires a non-empty --redis-addr, while memory runs the streams in process and is refused unless --replica-count is 1. The Helm chart deliberately passes --author-attribution=false by default, so a first install runs configured-author with no fact streams or audit webhook and every commit is committer-authored.


Observability

Instruments are OpenTelemetry instruments, exported through the SDK's Prometheus exporter onto the controller-runtime registry, so they are scraped from the metrics server's /metrics with every controller-runtime series beside them. Nothing is pushed over OTLP. Every name starts with gitopsreverser_.

The families follow the architecture, which is why this section is a map rather than a list:

Family Follows In the guide
resource_condition the kstatus trio of every object a human declares Configuration state
watch_* ingest: events, sessions, replays, recoveries, resolved types, open streams Pipeline
git_*, commit_requests_total the branch worker: documents, windows, commits, pushes, fetches, refusals, losses, the retained backlog Pipeline, Tuning commit windows
placement* where a new document lands, and the ones refused New-file placement
apply_*, receiver_deliveries_total apply requests, the apply window and the push receiver Apply requests
audit_*, attribution_* the optional author lookup: the door, the fact pipeline, the join Audit attribution
api_catalog_* discovery, per source cluster API resource catalog
watch_plan_* the queue that plans each GitTarget's watches Watch plane owner
secret_encryptions_total the path that keeps Secrets out of Git in plaintext Secret encryption

Six rules hold across all of them, and a new instrument follows them too:

  • The two ingestion paths are measured apart. Watch carries object state and decides whether an object reaches Git; audit carries only the author's name. They meet at the resolver and nowhere else, so no metric adds one to the other.
  • Each stage counts once, with a bounded outcome. The pipeline reads as a funnel, and its loss paths (git_commit_failures_total, git_writes_dropped_total, placement_refusals_total, git_documents_total{outcome="refused"}) as one selector.
  • Two identities, and one join between them. Watch and document families are labeled by the GitTarget (gittarget_namespace, gittarget_name); branch-worker families by the GitBranch the worker writes for and its GitProvider (gitbranch_*, provider_*), because one worker serves every GitTarget on it. git_branch_targets maps one to the other. source_cluster is the KubeProvider as <namespace>/<name>. Keys are prefixed because a pod scrape with honor_labels=false renames a bare namespace to exported_namespace and puts the scraped pod's own in its place.
  • Configuration objects are labels; mirrored resources never are. A GitTarget, GitBranch, applier claim or GitReceiver name is bounded by how many someone wrote, so it may be a label; a mirrored resource is data and is never one, nor is a name a client supplies before it resolves. A family that grows past its per-instrument cap (15000 points) collapses into one otel_metric_overflow="true" series.
  • A gauge about a work loop is read at scrape time. One published from inside the loop reports its last healthy moment for as long as the loop is stuck, so the queue, backlog, watch-plane and apply gauges are observable. "How long has this been waiting" is a timestamp, read as time() - <gauge>. The few recorded at the step they describe (api_catalog_*, the fact index size, the attribution follower's last success, the transport) record a step that completed.
  • A metric needs an interpretation. A new instrument lands with its recording site and its row in the guide in the same change (adding a metric).

Renamed series are listed in UPGRADING.md. See Operational boundaries for what is not measured.


Operational boundaries

Current limitations:

  • Single active replica. The watch manager and worker manager declare NeedLeaderElection, so all object-state work runs on one elected pod; multi-pod HA is not finished. The target design is the HA / GitTarget distribution plan, which needs Redis for resume cursors, branch-shard leases, and durable write queues.
  • Resume cursors are best-effort. Each watch shard stores its last processed resourceVersion in Redis, so short reconnects resume a normal watch from that cursor. Kubernetes does not guarantee replay from an arbitrary resourceVersion, so if the apiserver has expired the cursor (410 Gone) recovery falls back to sendInitialEvents replay or LIST + mark-and-sweep.
  • Watch queue DELAY is not measurable. Per-type watch volume, session ends, replay cost and recovery mode are all emitted now (see Observability), and watch_event_handling_seconds reports how long a stream was busy. How long an event waited before being picked up is a different number, and measuring it needs an arrival timestamp stamped before the blocking consumer, and the events arrive on a client-go watch channel this process does not fill, so there is nowhere honest to stamp one. Occupancy is the available proxy: a stream that is busy is a stream nothing else is being read from.
  • The in-process attribution transport is single-replica. --author-attribution-transport=memory is refused with more than one replica, and it loses every unjoined fact on restart by design; a multi-replica install must use the Redis transport.
  • No pull request creation; the operator writes directly to branches.
  • GitOps Reverser is the only path to a deployment. With an applier claim, the applier applies nothing on its own (a Kustomization is held suspended, an Application has automated sync off), so while the operator is down a Git-side push reaches the cluster only through a manual sync or flux resume, which the operator suspends again when it starts.
  • Only Flux and Argo CD can be appliers yet, and an Argo CD Application only with automated sync off: one that syncs on its own is refused, not driven. The applier may live in any cluster a KubeProvider reaches (the applier claim's spec.kubeProviderRef), such as a management cluster.
  • An apply request cannot reserve time. An edit made while the requested apply runs, to a field the applier sets, is reverted on the cluster. Its value stays in Git, because the applier's write over it is not mirrored, and the next apply restores it; until then the cluster runs the older value. See the apply window.
  • Audit ingress uses a shared-CA trust boundary. Named /audit-webhook/<audit-route> routes and the annotation-routed shared endpoint both require a client certificate signed by the audit CA, but that certificate is not bound to one audit route. This is an accepted privileged-control-plane assumption, not tenant isolation; see SECURITY.md.
  • deletecollection is reconciled by the watch (each item arrives as its own DELETED, or the mark-and-sweep reconciles them on replay).
  • A fail-safe placement skip has no dedicated status condition. A resource the writer refuses to place unsafely is logged per-resource and counted in the resync summary (placementSkipped), but is not (yet) surfaced as a distinct GitTarget status condition.

Package map

Package Role
api/v1alpha3/ CRD types
internal/applier/ apply requests: the Flux and Argo CD clients, the dispatcher that carries out each attempt, the applier claim status writer, and the Configured check
cmd/ operator entry point and server setup
internal/auditutil/ audit identity, objectRef, and subresource helpers feeding attribution facts
internal/controller/ Kubernetes reconcilers
internal/git/ branch workers, Git ops, commit/signing/encryption, manifest writer
internal/git/manifestedit/ YAML document editor
internal/giteaclient/ Gitea helper client
internal/manifestanalyzer/ manifest inventory, acceptance, and resync planning
internal/manifestreport/ projection of Kubernetes objects into comparable manifest reports
internal/queue/ attribution fact streams (Redis or in-process), the in-process fact index and its follower, and per-watch resume cursors
internal/reconcile/ per-GitTarget event stream (watch event → branch worker)
internal/rulestore/ compiled rule cache
internal/sanitize/ Kubernetes object sanitization and stable YAML marshal
internal/ssh/ SSH authentication helpers
internal/sshsig/ SSH signature implementation
internal/telemetry/ metrics and OTLP setup
internal/types/ shared resource identity/reference and sensitivity policy
internal/typeset/ type followability registry, lookup model, and the relevance filter (controller-owned → don't mirror)
internal/watch/ discovery catalog, watch manager, per-(GVR, scope) raw watches with sendInitialEvents replay + mark-and-sweep, the author resolver, watched type tables, event router
internal/webhook/ /audit-webhook ingress and attribution fact extraction (no body joiner), and the /git-push receiver

Design documents

Deeper dives live under docs/design/:

Ingestion, attribution, and reconcile:

Types, discovery, status, and the Git write side:

Future direction: