Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions docs/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,9 @@ should land before the next tag is cut (the last release is v0.55.0). Checked ag

## Before the next release

- [ ] **See the concurrent-authors outage e2e pass in CI** (`concurrent-authors-outage`,
`test/e2e/concurrent_authors_outage_e2e_test.go`): two authors save while every push is refused,
and the retry publishes both, one commit each, once Git takes pushes again. It is the last part of
stage 3 of [concurrent-author-windows.md](design/concurrent-author-windows.md#stages); the reference
model's part is done.
Nothing open. Concurrent commit windows, the last gate, passed: stage 3 of
[concurrent-author-windows.md](design/concurrent-author-windows.md#stages), outage e2e included, was
green in CI on PR #476.

## Applier claims and bi-directional editing

Expand Down
16 changes: 10 additions & 6 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@ fails; there is no plaintext opt-out.
**Writes are serialized per Git branch.** One [BranchWorker](../internal/git/branch_worker.go) owns each
repository branch: the repository's URL, normalized, and the branch. Multiple `GitTarget`s may share
one branch, from any namespace, through the one `GitBranch` that holds it. Every write to that
branch goes through the worker's single event loop and commit window.
branch goes through the worker's single event loop, which holds a commit window per author, side
by side only while it proves their files cannot overlap.

**The worktree sits at the remote tip, or the worker knows it does not.** The write path used to
fetch and hard-reset before the first commit of every publication cycle. It no longer does: the push
Expand Down Expand Up @@ -146,7 +147,7 @@ that is live and not yet in Git, and only the worker writing the branch can see
`GitTarget` names an **applier claim**, which holds its **applier** so that it applies nothing on
its own (a Flux `Kustomization` is held suspended, an Argo CD `Application` held with automated sync off),
and the branch worker sends **apply requests** instead. A request is sent only **after a flush**:
when an apply comes due, the worker closes the open commit window, publishes every write decided so
when an apply comes due, the worker closes every open commit window, publishes every write decided so
far at once, and sends the request for the tip that publication produced. An edit decided after
that is recorded in the apply window, so an apply that overwrites it is withheld and repaired.
Continuous editing therefore never holds an apply back. A request goes out:
Expand Down Expand Up @@ -707,7 +708,8 @@ Following the ConfigMap edit:
[GitTargetEventStream](../internal/reconcile/git_target_event_stream.go), and the
[BranchWorker](../internal/git/branch_worker.go) appends it to the open commit window for
`(author, GitTarget)`.
5. **Commit + push.** When the window closes (5s of silence, a `CommitRequest`, or a buffer limit), the
5. **Commit + push.** When the window closes (5s of silence, a `CommitRequest`, a buffer limit, or a
write to one of its files by another author, which closes it first), the
manifest aware writer patches `team-a/.../app-config.yaml` in place, commits as the resolved author,
and pushes via [PushAtomic](../internal/git/git_atomic_push.go) (retrying with fetch/reset/replay if
the remote moved).
Expand Down Expand Up @@ -745,9 +747,10 @@ flowchart TD
WINDOW --> TIMER{What closes it?}
TIMER -->|silence timer| BASE
TIMER -->|buffer limit or resync boundary| BASE
TIMER -->|another author writes one of its files| BASE

CR[CommitRequest persisted] --> AUTHOR[Author from its audit fact<br/>named or unnamed]
AUTHOR --> ATTACH[Attach to matching open window]
CR[CommitRequest persisted] --> CRAUTHOR[Author from its audit fact<br/>named or unnamed]
CRAUTHOR --> ATTACH[Attach to its author's open window]
ATTACH -->|same author + GitTarget| BASE
ATTACH -->|no matching window| BENIGN[Ready=True, Pushed=False]

Expand Down Expand Up @@ -1766,7 +1769,8 @@ unsafely: logged per-resource, counted in the resync summary as `placementSkippe
- **Control-cluster path**: [internal/controller/commitrequest_controller.go](../internal/controller/commitrequest_controller.go)
- **Source-cluster path**: [internal/controller/commitrequest_server.go](../internal/controller/commitrequest_server.go)

A `CommitRequest` finalizes the open commit window for its GitTarget. It is picked up in one of two
A `CommitRequest` finalizes its author's open commit window on its GitTarget; other authors' windows
stay open beside it. It is picked up in one of two
ways: by the `CommitRequestReconciler` on the control cluster, or by the `CommitRequestServer`'s watch
of a source cluster, through the `KubeProvider` whose `spec.commitRequests` reconciles it. Both hand it
to one core, which takes one step at a time, each decided again from the object as it is stored
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1894,7 +1894,7 @@ spec:
```

**When a request is made.** After a [flush](bi-directional.md#flush-then-apply): when an apply comes
due, the branch worker closes the open commit window, publishes the writes decided up to that
due, the branch worker closes every open commit window, publishes the writes decided up to that
moment at once, and asks for the tip that publication produced. An edit decided after that is
recorded in the apply window, so an apply that writes over it is withheld from Git and repaired. A
registered `CommitRequest` (a save sets its own commit boundary), a
Expand Down
69 changes: 41 additions & 28 deletions docs/design/concurrent-author-windows.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Design: one commit window per author, open at the same time

Status: **stages 1 and 2 built** (`internal/git/window_set.go`, `footprint.go`,
`window_admission.go`; see [what stage 2 built](#what-stage-2-built)); **stages 3 to 5 proposed**.
Stage 2 is on with no startup flag, so **stage 3 gates the next release**: it must land before the
next tag is cut (`docs/TODO.md`, "Before the next release"). Raised by
Status: **built and ready to release.** Stages 1 to 3 are built (`internal/git/window_set.go`,
`footprint.go`, `window_admission.go`), stage 4 was decided against, and stage 5 is proposed. Stage 2
is on with no startup flag, so stage 3 gated the release; its last part, the outage e2e spec, passed in
CI on PR #476. [How it is proven](#how-it-is-proven) lists the tests. Raised by
gitops-api's review of the source-side save (PR #456):
with saves created straight from a browser, a second person's edit splits the first person's change
into two commits. This page sets out the change that fixes that, the rules it must keep, how to test
Expand Down Expand Up @@ -197,7 +197,7 @@ unrelated push split a save that is protected today.
| `apply_due` (`flushed`) | every window, when the flush begins and takes the flush mark | the flush does not begin while any request is registered (`writeBusy`) |
| `resync_before_apply` | every window | none (as today) |
| `shutdown` | every window | none |
| `buffer_limit` | every window | none; closing only the largest is an open question |
| `buffer_limit` | every window | none |
| `path_conflict` | the windows whose footprint the new write overlaps | none |
| `footprint_fallback` | every window but the newest, when a commit outside its footprint makes the worker hold one window at a time | none |
| `identity_change` | every other window | stays for writes of another `GitTarget` and for unproven footprints |
Expand Down Expand Up @@ -226,7 +226,7 @@ map iteration or by time, so the order is deterministic and testable.
Commits land in the order their windows close, not the order their first writes arrived. With Alice
open since 10:00 and Bob since 10:01, Bob's commit can come first. Together with the footprint rule
above, no file ever goes back in time, but `git log` no longer reads as one timeline of edits. That
is the point of the change, and it needs saying in `docs/configuration.md` when it ships.
is the point of the change, and `docs/configuration.md` says so.

### Saves

Expand Down Expand Up @@ -295,8 +295,8 @@ window, so they are proven against today's behavior before they are used to judg

## Stages

Stages 1, 2 and 3 are built; stage 4 was decided against, and stage 5 is proposed. Stage 3 gates the
next release, because stage 2 shipped without a flag: its last part is the outage e2e passing in CI.
Stages 1, 2 and 3 are built; stage 4 was decided against, and stage 5 is proposed. Stage 3 gated the
release, because stage 2 shipped without a flag.

1. **Built. Extract window management without changing behavior.** Move the window bookkeeping
(`openWindow`, its timers, attach and finalize) behind one type that holds a set of windows, with
Expand All @@ -305,7 +305,7 @@ next release, because stage 2 shipped without a flag: its last part is the outag
2. **Built. Allow several windows for proven-independent destinations.** Writes keep whole-object commits
and apply behavior stays as it is. Writes of another `GitTarget` keep closing other windows
(`identity_change`), and every layout in the footprint table's "no" rows stays serial.
3. **Built; gates the next release. Test the failure boundaries.** Both close orders,
3. **Built. Test the failure boundaries.** Both close orders,
conflicting writes, delete and recreate, `attach_next`, zero-duration windows, resync, shutdown,
failed publication and replay, and buffer pressure, each against the reference model, which
keeps work pending across failed pushes and retries. See [What stage 3 built](#what-stage-3-built).
Expand Down Expand Up @@ -373,19 +373,31 @@ and the test driver delivers the push and retry timers too.
are the newest writes of their objects; a save that changes nothing commits nothing, so it adds no
bootstrap files; a push that lands makes the base trusted again.

## How it is proven

Three layers, each catching what the one below cannot:

| Layer | Where | What it proves |
| --- | --- | --- |
| Reference model | `internal/git/window_model_test.go` | Random runs of every input above, with several windows open, against an independent implementation: author, message and files of every commit, no older write landing after a newer conflicting one, each save settled once, the retained charge after every step. 20 seeds of 60 steps in CI, with the reach floors of stage 3. |
| Admission unit tests | `internal/git/window_admission_test.go` | Each rule on its own: disjoint authors keep a window each, a write to another window's file closes it first, unproven writes and another `GitTarget` stay serial, a footprint bug is refused and the worker falls back, a reset or a `GitTarget` change proves the open windows again, shutdown closes oldest first, the gauges. |
| e2e, label `concurrent-authors` | `test/e2e/concurrent_authors_e2e_test.go` | The whole path: kubectl as each person, audit attribution, watch, branch worker, Gitea, a `CommitRequest` per save. Two authors interleave and each save is one commit; three authors hold three windows at once (`git_commit_windows_open` reads 3), save in the reverse of the order they started, and Git's history is that close order; a write to a file in another author's open window closes that window first (`path_conflict`), its commit lands first with the older copy, and Git ends with the newer one. |
| e2e, label `concurrent-authors-outage` | `test/e2e/concurrent_authors_outage_e2e_test.go` | Two saves made while Git refuses every push stay pending, neither gives up, and the retry publishes both as one commit each once Git is back. |

## What it costs

- **The loop's state.** The single `openWindow` becomes a set with an opening order, the close timer
becomes the earliest of all windows' deadlines, and every path that reads `l.openWindow` today
(attach, finalize, the apply window, resync cut-off, metrics) has to say which window, or all.
Stage 1 does this refactor on its own, with no behavior change.
- **Footprints at arrival.** Deciding a footprint needs the worker's placement index of the
`GitTarget` subtree when a write arrives, not only when it plans a commit.
- **Memory.** One window per active author. The byte cap stays global, so the bound does not change.
- **Tests.** The commit-window, attach and apply-window suites assume one window. Most of them keep
their meaning with one author; the interleaving cases are new, and the reference model is new.
- **Metrics.** `git_commit_windows_total` gains `close_reason="path_conflict"`. A gauge of open
windows shows how often authors overlap at all.
- **The loop's state.** A set of windows with an opening order; the close timer is the earliest of
their deadlines, and every reader of the open window (attach, finalize, the apply window, resync
cut-off, metrics) names which window, or all.
- **Footprints at arrival.** The worker builds a placement index of the `GitTarget` subtree, rebuilt
whenever its revision changes, every own commit included: about 3.5 ms and 3.9 MB allocated for 100
small manifests, 33 ms and 37 MB for 1,000 (`docs/TODO.md` tracks measuring it on production-sized
repositories).
- **Memory.** One window per active author. The byte cap stays global, but it does not charge the
footprint indexes, the accepted versions or the withheld-field records.
- **Metrics.** `git_commit_windows_total` has `close_reason="path_conflict"` and
`"footprint_fallback"`; `git_commit_windows_open` shows how often authors overlap at all, and
`git_commit_windows_single` a worker that fell back to one window.

## Potential problems

Expand Down Expand Up @@ -431,12 +443,13 @@ the reference model checks every commit.

## Open questions

1. Does `buffer_limit` close every window, or only the one that crossed it?
2. After stage 4, do windows of different `GitTarget`s on one branch also stay open side by side?
The per-`GitTarget` encryption and bootstrap that made one window per target necessary are per
window already; the first release keeps them serial anyway.
3. Is a cap on open windows needed beyond the byte cap, such as one per `GitTarget`?
4. When a save's own window is closed early by `path_conflict`, it commits with the save's message
Settled by the build: `buffer_limit` closes every window; windows of different `GitTarget`s on one
branch stay serial; the proof at arrival is the placement index, with its cost under
[what it costs](#what-it-costs).

1. Do windows of different `GitTarget`s on one branch stay open side by side later? The
per-`GitTarget` encryption and bootstrap that made one window per target necessary are per window
already.
2. Is a cap on open windows needed beyond the byte cap, such as one per `GitTarget`?
3. When a save's own window is closed early by `path_conflict`, it commits with the save's message
and less than the person meant to save. Is that acceptable, or should the save report it?
5. How much of the proof can the worker's placement index answer at arrival without a plan, and
what does that lookup and the recomputation after each revision change cost?
4 changes: 2 additions & 2 deletions docs/spec/commit-window-refactor.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,8 @@ path order.

Resyncs and atomic writes are different: they bypass the live-event window, but
they do not jump ahead of already-windowed live events. If one arrives while the
branch worker has an open event window, the worker finalizes that window into a
commit pending write first, then appends its own pending write. This preserves
branch worker has open event windows, the worker finalizes them, oldest first, into
commit pending writes, then appends its own pending write. This preserves
arrival order while keeping a resync or an atomic write as one batch.

## Core Types
Expand Down
3 changes: 2 additions & 1 deletion test/e2e/commit_request_e2e_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -851,14 +851,15 @@ type windowedDeploymentPipeline struct {
gitProvider string
gitTarget string
watchRule string
basePath string
}

// setupWindowedDeploymentPipeline creates the pipeline, every name derived from name, and returns
// once its streams run: from then on a Deployment is a live, windowed event, and nothing is on main
// until a commit window is finalized.
func setupWindowedDeploymentPipeline(name, basePath, commitWindow string) windowedDeploymentPipeline {
By("creating " + name + " test namespace and applying git secrets")
p := windowedDeploymentPipeline{ns: testNamespaceFor(name)}
p := windowedDeploymentPipeline{ns: testNamespaceFor(name), basePath: basePath}
_, _ = kubectlRun("create", "namespace", p.ns)
p.repo = SetupRepo(resolveE2EContext(), p.ns, fmt.Sprintf("e2e-%s-%d", name, GinkgoRandomSeed()))
_, err := kubectlRunInNamespace(p.ns, "apply", "-f", p.repo.SecretsYAML)
Expand Down
Loading
Loading