test(git): keep work pending across wakes in the window model, and save through a Git outage

docs/TODO.md

@@ -7,30 +7,11 @@ should land before the next tag is cut (the last release is v0.55.0). Checked ag
 
 ## Before the next release
 
-- [ ] **Test the failure boundaries of concurrent commit windows** (stage 3 of
-  [concurrent-author-windows.md](design/concurrent-author-windows.md#stages)). Stage 2 ships with no
-  flag: a branch worker keeps a window per author whenever their footprints are proven disjoint.
-  Before it is released, the reference model (`internal/git/window_model_test.go`) must cover both
-  close orders, conflicting writes, delete and recreate, `attach_next`, zero-duration windows, a
-  resync, shutdown, failed publication and replay, and buffer pressure, each with several windows
-  open. The reviews of stages 1 and 2 add:
-  - **gate:** the model retains work across failed pushes and retries before several windows are
-    released. Today it publishes at once after every wake (`windowModelRun.wake`) and asserts no
-    pending writes after it (`drained`), so decided-but-uncommitted writes with windows open
-    (cooldown, failed push, replay) are never modeled;
-  - reach floors the model asserts: shutdown, flush and push hint with two or more windows open,
-    `attach_next` with others open, a reproof overlap, `path_conflict`; a concurrency-biased profile;
-    CI raised to about 20 seeds × 60 steps;
-  - model steps: a shutdown mid-run; a timer overdue but not yet delivered (the `!isOpen` guard in
-    `attachWaitingCommitRequests` survives its removal); a late stale applier write (the model
-    never withholds);
-  - the GitTarget-generation and checkout-reset parts of a footprint revision are untested;
-  - measure the footprint index on realistic repositories and commit rates before widening the
-    layouts it proves. It is rebuilt whenever its revision changes, every local commit included;
-    a review measured a rebuild at about 3.5 ms and 3.9 MB allocated for 100 small manifests, and
-    33 ms and 37 MB for 1,000 (allocation, not retained heap), against 19 µs for a reuse. The
-    retained-byte budget does not charge the indexes, the accepted versions or the withheld-field
-    records, so a full budget does not bound the worker's memory.
+- [ ] **See the concurrent-authors outage e2e pass in CI** (`concurrent-authors-outage`,
+  `test/e2e/concurrent_authors_outage_e2e_test.go`): two authors save while every push is refused,
+  and the retry publishes both, one commit each, once Git takes pushes again. It is the last part of
+  stage 3 of [concurrent-author-windows.md](design/concurrent-author-windows.md#stages); the reference
+  model's part is done.
 
 ## Applier claims and bi-directional editing
 
@@ -120,6 +101,19 @@ Their argument for each is in [gitops-api-open-asks.md](future/gitops-api-open-a
 
 ## Branch worker and CommitRequests
 
+- [ ] **Measure the footprint index** on realistic repositories and commit rates before widening the
+  layouts it proves. It is rebuilt whenever its revision changes, every local commit included; a
+  review measured a rebuild at about 3.5 ms and 3.9 MB allocated for 100 small manifests, and 33 ms
+  and 37 MB for 1,000 (allocation, not retained heap), against 19 µs for a reuse. The retained-byte
+  budget does not charge the indexes, the accepted versions or the withheld-field records, so a full
+  budget does not bound the worker's memory.
+- [ ] What the window reference model (`internal/git/window_model_test.go`) does not cover: a resync
+  waiting for its retry whose fetch finds a push from outside and discards it (the model makes no
+  push from outside while a resync waits); objects of more than one field, so a withheld apply's
+  record is only ever one value; an applier write after its apply window's grace (the window runs on
+  the wall clock, so the model sends each late write inside it). Two guards no input reaches are pinned by unit
+  tests instead: the outage ending when its writes settle with no push, and an attach that closes a
+  later window in the same pass.
 - [ ] `expireWaitingCommitRequests` ranges over a map, so empty commits for saves that expire in
   the same pass land in random order; sort by `seq` (`internal/git/commit_request_attach_loop.go`).
 - [ ] `buildRequestRecordWrite` keeps only the username (`UserInfo{Username: pcr.author}`); carry the

docs/design/concurrent-author-windows.md

@@ -295,8 +295,8 @@ window, so they are proven against today's behavior before they are used to judg
 
 ## Stages
 
-Stages 1 and 2 are built; stage 3 is in progress, stage 4 was decided against, and stage 5 is
-proposed. Stage 3 gates the next release, because stage 2 shipped without a flag.
+Stages 1, 2 and 3 are built; stage 4 was decided against, and stage 5 is proposed. Stage 3 gates the
+next release, because stage 2 shipped without a flag: its last part is the outage e2e passing in CI.
 
 1. **Built. Extract window management without changing behavior.** Move the window bookkeeping
    (`openWindow`, its timers, attach and finalize) behind one type that holds a set of windows, with
@@ -305,10 +305,10 @@ proposed. Stage 3 gates the next release, because stage 2 shipped without a flag
 2. **Built. Allow several windows for proven-independent destinations.** Writes keep whole-object commits
    and apply behavior stays as it is. Writes of another `GitTarget` keep closing other windows
    (`identity_change`), and every layout in the footprint table's "no" rows stays serial.
-3. **Proposed; gates the next release. Test the failure boundaries.** Both close orders,
+3. **Built; gates the next release. Test the failure boundaries.** Both close orders,
    conflicting writes, delete and recreate, `attach_next`, zero-duration windows, resync, shutdown,
    failed publication and replay, and buffer pressure, each against the reference model, which
-   first has to keep work pending across failed pushes and retries.
+   keeps work pending across failed pushes and retries. See [What stage 3 built](#what-stage-3-built).
 4. **Decided against (2026-10-28): no startup flag.** Stage 2 is always on, so stage 3's tests gate
    the release instead of a default-off rollout. What the flag's measurement period was for is read
    from the metrics: `git_commit_windows_open` shows windows open side by side,
@@ -346,6 +346,33 @@ The rules above, with these choices where the page left room:
   window is open records a mismatch only when no window of its own author is open.
 - **The window ID** (`window`) is in the open, close and finalized log lines, with the branch.
 
+### What stage 3 built
+
+The reference model keeps work across wakes. A decided write is committed at the end of the wake
+that decided it, unless it needs a connection while a failed attempt waits for its retry (a resync
+during an outage), and published by a push that the cooldown spaces after the last one that landed;
+a save and an apply's flush push at once. A failed push keeps every write for the retry, 10s doubling
+to 5m. The push cooldown runs on the worker's clock, as the window timers and the retry already did,
+and the test driver delivers the push and retry timers too.
+
+- **Checked after every step:** the pending writes in order (which carry a save, which are a resync,
+  which are committed), what each is charged, the total retained charge, and all four timers.
+- **Inputs added:** a Git outage that starts and ends (every push, fetch and sync fails), a restart
+  mid-run (a new worker on an empty checkout), a timer overdue while an item is handled first, an
+  applier's write that lands over a newer edit (withheld, and a Repair asked for), and a push from
+  outside that moves one author's document into another's file under their open windows.
+- **Profiles:** wide, concurrent (authors keep to their own files) and outage (short clock steps, so
+  windows stay open across the cooldown and the retries). CI runs 20 seeds of 60 steps, about 15 s.
+- **Reach floors:** the search must reach, across its seeds, each of shutdown, flush and push hint
+  with several windows open, `attach_next` beside another window, a reproof that finds windows
+  overlap, `path_conflict`, a failed push, the cooldown and a retry wait with several windows open, a
+  commit waiting for its retry, a replay with a window open, buffer pressure with several windows
+  open, an apply waiting for the retry, a restart, an overdue timer, and a withheld applier write.
+- **Found:** no fault of the worker. The model learned four rules it never met while it pushed at
+  once: a pending resync holds every applier back (the barrier) and ends the flush; a resync's edits
+  are the newest writes of their objects; a save that changes nothing commits nothing, so it adds no
+  bootstrap files; a push that lands makes the base trusted again.
+
 ## What it costs
 
 - **The loop's state.** The single `openWindow` becomes a set with an opening order, the close timer

internal/git/branch_worker.go

@@ -1093,8 +1093,11 @@ type branchWorkerEventLoop struct {
 	// commitTimerAt is the worker-clock time commitTimer is armed for, while it is armed. The timer
 	// itself runs on real time; tests that step the clock deliver its wake at this time.
 	commitTimerAt time.Time
-	// pushTimer fires the next push when the success cooldown ends.
-	pushTimer *time.Timer
+	// pushTimer fires the next push when the success cooldown ends, and pushTimerAt is the
+	// worker-clock time it is armed for, while it is armed. Like commitTimerAt, tests that step the
+	// clock deliver its wake at this time.
+	pushTimer   *time.Timer
+	pushTimerAt time.Time
 	// retry is the one schedule of the next attempt at work a failed attempt left pending. See
 	// retry.go.
 	retry retrySchedule
@@ -1778,13 +1781,14 @@ func (l *branchWorkerEventLoop) maybeSchedulePush() {
 		l.pushPending()
 		return
 	}
-	elapsed := time.Since(l.lastPushAt)
+	elapsed := l.w.now().Sub(l.lastPushAt)
 	if elapsed >= PushCooldown {
 		l.pushPending()
 		return
 	}
 	if l.pushTimer == nil {
 		l.pushTimer = time.NewTimer(PushCooldown - elapsed)
+		l.pushTimerAt = l.lastPushAt.Add(PushCooldown)
 	}
 }
 
@@ -1850,7 +1854,7 @@ func (l *branchWorkerEventLoop) pushPending() {
 
 	l.pendingWrites = nil
 	l.pendingWritesBytes = 0
-	l.lastPushAt = time.Now()
+	l.lastPushAt = l.w.now()
 	l.stopPushTimer()
 	l.endOutage()
 	l.settleReplayRefusals(refused)
@@ -1902,6 +1906,7 @@ func (l *branchWorkerEventLoop) resolvePushedCommitRequests() {
 }
 
 func (l *branchWorkerEventLoop) stopPushTimer() {
+	l.pushTimerAt = time.Time{}
 	if l.pushTimer == nil {
 		return
 	}

internal/git/commit_request_attach_test.go

@@ -69,10 +69,11 @@ func forceDue(loop *branchWorkerEventLoop) {
 	}
 }
 
-// holdPushes starts the push cooldown now and keeps every push behind it, a save's too, so the local
-// commits stay inspectable until the test pushes.
+// holdPushes keeps every push behind the cooldown, a save's too, so the local commits stay
+// inspectable until the test pushes. The cooldown runs on the worker's clock, so it starts an hour
+// ahead: a test that steps the clock still holds.
 func holdPushes(loop *branchWorkerEventLoop) {
-	loop.lastPushAt = time.Now()
+	loop.lastPushAt = loop.w.now().Add(time.Hour)
 	loop.cooldownHoldsSaves = true
 }
 

internal/git/parent_recovery.go

@@ -52,9 +52,9 @@ type parentRecovery struct {
 	gen uint64
 }
 
-// now is the loop's clock, injectable so the retry schedule and the commit windows' timers can be
-// tested without sleeping. The timers themselves are real: a test that moves the clock past a
-// deadline delivers the wake the timer would have, as run does.
+// now is the loop's clock, injectable so the retry schedule, the push cooldown and the commit
+// windows' timers can be tested without sleeping. The timers themselves are real: a test that moves
+// the clock past a deadline delivers the wake the timer would have, as run does.
 func (w *BranchWorker) now() time.Time {
 	if w.clock != nil {
 		return w.clock()

internal/git/retry_test.go

@@ -20,6 +20,8 @@ import (
 	"github.com/go-logr/logr"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+
+	"github.com/ConfigButler/gitops-reverser/internal/manifestanalyzer"
 )
 
 // failNextPushes makes the next n push attempts fail as an unreachable remote would, and counts every
@@ -224,3 +226,28 @@ func TestPublicationRetry_AHeldCommitRequestIsCommittedWithoutAnotherWrite(t *te
 	assert.Equal(t, f.ref("feature").String(), res.Commit)
 	assert.Equal(t, int32(2), attempts.Load(), "one failed push, then the retry")
 }
+
+// An outage can end with nothing to publish: a resync decided while Git is down waits for the retry,
+// and committed then it changes nothing, so no write is left to push. The outage is over all the same:
+// the retry closes, and the next failure waits the first backoff again, not twice the last.
+func TestPublicationRetry_AnOutageEndsWhenItsWritesSettleWithoutAPush(t *testing.T) {
+	f := newAdmissionFixture(t, "retry-settles", nil)
+	f.loop.pushPending() // the bootstrap write
+	clock := useTestClock(f.worker)
+	outage := injectOutage(t)
+	outage.down = true
+	req := &ResyncRequest{Desired: []manifestanalyzer.DesiredResource{echoDesired("x", "x1")},
+		GitTargetName: echoTarget, GitTargetNamespace: "default", Result: make(chan ResyncResult, 1)}
+	f.loop.handleQueueItem(resyncItem(req))
+	require.Error(t, (<-req.Result).Err, "precondition: the resync waits for the retry")
+	require.Len(t, f.loop.pendingWrites, 1)
+	require.True(t, f.loop.retry.pending())
+
+	outage.down = false
+	clock.advance(retryInitialBackoff)
+	fireRetry(f.loop)
+
+	require.Empty(t, f.loop.pendingWrites, "the resync changed nothing")
+	assert.False(t, f.loop.retry.pending(), "the outage is over")
+	assert.Zero(t, f.loop.retry.backoff, "and the next one starts from the first backoff")
+}

internal/git/window_admission_test.go

@@ -5,6 +5,8 @@ package git
 import (
 	"fmt"
 	"maps"
+	"os"
+	"path/filepath"
 	"strings"
 	"testing"
 	"time"
@@ -570,3 +572,94 @@ func TestFootprint_OverlapUnionAndOutside(t *testing.T) {
 	assert.Equal(t, "unknown: why", unknown.String())
 	assert.Equal(t, "unproven", footprintString(nil))
 }
+
+// TestWindowAdmission_AGitTargetChangeProvesTheOpenWindowsAgain pins the generation in a footprint's
+// revision: a GitTarget that now places by a template proves nothing, so two windows proven apart
+// before the change close once the next write proves them again. With the generation left out, the
+// index of the old spec was reused, and they stayed open side by side.
+func TestWindowAdmission_AGitTargetChangeProvesTheOpenWindowsAgain(t *testing.T) {
+	f := newAdmissionFixture(t, "admission-generation", nil)
+	f.writeBy("alice", "x", "x2")
+	f.writeBy("bob", "z", "z2")
+	require.Equal(t, []string{"alice", "bob"}, authorsOf(f.loop.windows.all()))
+
+	var target configv1alpha3.GitTarget
+	require.NoError(t, f.worker.Client.Get(f.worker.ctx, keyOf(echoTarget), &target))
+	target.Spec.Placement = &configv1alpha3.GitTargetPlacementSpec{Default: "all.yaml"}
+	target.Generation++
+	require.NoError(t, f.worker.Client.Update(f.worker.ctx, &target))
+	f.writeBy("carol", "q", "q3")
+
+	assert.Equal(t, []string{"carol"}, authorsOf(f.loop.windows.all()),
+		"proven again under the new spec, the open windows are unknown, and close before carol's write")
+}
+
+// TestWindowAdmission_AResetProvesTheOpenWindowsAgain pins the resets in a footprint's revision. A push
+// from outside makes carol's held write a no-op and moves z's document in with x's; the replay onto
+// it resets the checkout and commits nothing, so only the reset says the tree changed. The next write
+// proves alice's and bob's windows again, finds both in one file, and closes them. With the resets left
+// out, the index of the tree before the reset was reused, and both stayed open on one file.
+func TestWindowAdmission_AResetProvesTheOpenWindowsAgain(t *testing.T) {
+	f := newAdmissionFixture(t, "admission-reset", nil)
+	f.loop.pushPending() // the bootstrap write; the cooldown holds every push after it
+	f.writeBy("alice", "x", "x2")
+	f.writeBy("bob", "z", "z2")
+	f.writeBy("carol", "q", "q3")
+	open := f.loop.windows.all()
+	require.Equal(t, []string{"alice", "bob", "carol"}, authorsOf(open))
+	f.loop.finalizeWindow(open[2], windowFinalizeReasonIdleTimeout)
+	f.loop.endWake(0)
+	require.Len(t, f.loop.pendingWrites, 1, "precondition: the cooldown holds carol's commit")
+	// Proven on the tree carol's commit made, so only the reset below can make these proofs stale.
+	f.writeBy("alice", "x", "x3")
+
+	gitIn(t, f.clone, "pull", "--quiet", "--ff-only", "origin", "main")
+	write := func(path, content string) {
+		require.NoError(t, os.WriteFile(filepath.Join(f.clone, path), []byte(content), 0o600))
+	}
+	write(echoFile("q"), echoDocument("q", "q3"))
+	write(echoFile("x"), echoDocument("x", "x1")+"---\n"+echoDocument("z", "z1"))
+	gitIn(t, f.clone, "rm", "--quiet", echoFile("z"))
+	gitIn(t, f.clone, "add", "-A")
+	gitIn(t, f.clone, "commit", "--quiet", "-m", "outside: q3, and z in with x")
+	gitIn(t, f.clone, "push", "--quiet", "origin", "HEAD:main")
+	resets, commits := f.worker.checkoutResets.Load(), f.worker.checkoutCommits.Load()
+	f.loop.pushPending()
+	require.Empty(t, f.loop.pendingWrites)
+	require.Greater(t, f.worker.checkoutResets.Load(), resets, "precondition: the replay reset the checkout")
+	require.Equal(t, commits, f.worker.checkoutCommits.Load(), "precondition: and committed nothing")
+
+	f.writeBy("dave", "q", "q4")
+	assert.Equal(t, []string{"dave"}, authorsOf(f.loop.windows.all()),
+		"proven again on the tree the reset brought, alice's and bob's windows share a file")
+}
+
+// TestWindowAdmission_AWindowAnAttachClosedTakesNoRequest pins the guard in
+// attachWaitingCommitRequests: an attach closes every window that is due, so a window later in the
+// pass may be gone, and a request that would have attached to it keeps waiting for the next. The
+// state is built by hand: a request waits only while its author has no window to take, so the model
+// never meets a second window with a request waiting for it in the pass.
+func TestWindowAdmission_AWindowAnAttachClosedTakesNoRequest(t *testing.T) {
+	f := newAdmissionFixture(t, "admission-attach-closed", nil)
+	f.writeBy("alice", "x", "x2")
+	f.writeBy("bob", "z", "z2")
+	open := f.loop.windows.all()
+	require.Equal(t, []string{"alice", "bob"}, authorsOf(open))
+	open[1].timers.maxAt = f.worker.now().Add(-time.Second) // bob's window is due, its timer not delivered
+
+	waiting := func(author string, seq uint64) *pendingCommitRequest {
+		return &pendingCommitRequest{
+			id:     commitRequestID{Namespace: "default", Name: "save-" + author, UID: "uid-" + author},
+			author: author, gitTargetName: echoTarget, gitTargetNamespace: "default", seq: seq,
+			message: "feat: " + author, maxDuration: time.Hour, attachDeadline: f.worker.now().Add(time.Minute),
+		}
+	}
+	alice, bob := waiting("alice", 1), waiting("bob", 2)
+	f.loop.pendingCRs = map[commitRequestID]*pendingCommitRequest{alice.id: alice, bob.id: bob}
+
+	f.loop.attachWaitingCommitRequests()
+
+	assert.True(t, alice.attached, "alice's request takes her window")
+	assert.Equal(t, []string{"alice"}, authorsOf(f.loop.windows.all()), "and the attach closed bob's, which was due")
+	assert.False(t, bob.attached, "bob's request takes no window that is gone: it waits for his next")
+}