From 42a13dfb629b2917ac4aecbbfb19368968e97022 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 11:12:54 +0100 Subject: [PATCH 01/13] :page_facing_up: docs(spec): propose sink-breaker-terminal-isolation Terminal export errors trip the shared per-sink circuit breaker, so one misconfigured inventory opens it for every inventory on the sink. Spec set for excluding them (gobreaker IsExcluded), with test-first tasks. --- .../sink-breaker-terminal-isolation/design.md | 53 ++++++++++++ .../sink-breaker-terminal-isolation/loop.md | 70 +++++++++++++++ .../proposal.md | 85 +++++++++++++++++++ .../specs/sink-circuit-breaker/spec.md | 60 +++++++++++++ .../sink-breaker-terminal-isolation/tasks.md | 81 ++++++++++++++++++ 5 files changed, 349 insertions(+) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/design.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/loop.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/proposal.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/tasks.md diff --git a/openspec/changes/sink-breaker-terminal-isolation/design.md b/openspec/changes/sink-breaker-terminal-isolation/design.md new file mode 100644 index 00000000..f2de0a95 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/design.md @@ -0,0 +1,53 @@ +# Design + +## Context + +`exportThroughBreaker(key, fn)` (`internal/sink/circuit_breaker.go:26`) wraps the export plan run in +`RunExportEnvelope` (`internal/sink/export.go:200`). The key is `req.SinkNamespace + "/" + +req.SinkName`. `breakerFor` lazily creates one `gobreaker.CircuitBreaker[struct{}]` per key in a +process-wide `sync.Map` with `MaxRequests: 1`, `Interval: 60s`, `Timeout: 30s` and `ReadyToTrip: +ConsecutiveFailures >= 5`. + +Errors reaching the breaker are whatever `plan.run` returns. Some are already classified +`kollecterrors.Terminal` inside the export (ownership, identity, payload, CA/TLS configuration, +backend-specific Terminal classification in `bigquery/errors.go`, `mongodb/errors.go`). Others are +classified only after the breaker by `git.ClassifyExportError` and `classifyExportFailure` +(`export.go:207-212`), e.g. Git auth failures; those count as failures today and keep counting. + +## Option chosen + +A: `Settings.IsExcluded = kollecterrors.IsTerminal`. Rejected: B (tag only inventory-scoped Terminal +errors with a new marker; a forgotten tag fails back to today's defect and protects nothing extra), +C (breaker per sink and inventory; a dead backend is then hit 5×N times before any back-off and the +registry needs eviction). Decision record: `agent-context/decisions.md` 2026-11-05. + +## Behaviour (gobreaker v2.4.0, read in `gobreaker.go:237-302`, `counter.go:16-41`) + +- **Closed.** An excluded error increments `TotalExclusions` only. `ConsecutiveFailures` is neither + incremented nor reset, so T, X, T, T, T, T (T Transient, X Terminal) trips on the fifth T. A success + still resets the streak. +- **Open.** Unchanged: requests fail fast with `ErrOpenState`, mapped to a Transient "sink circuit + breaker open" error (`circuit_breaker.go:39-41`). The export function does not run, so no + classification happens. +- **Half-open** (`MaxRequests: 1`). `beforeRequest` admits a request while `Requests - + TotalExclusions < MaxRequests`. An excluded trial result leaves the state half-open and frees the + slot; the next request is admitted as the trial. A success closes the breaker; a counted failure + re-opens it. +- `ErrTooManyRequests` (a second concurrent request in half-open) is returned unwrapped from + `exportThroughBreaker` and becomes Transient in `classifyExportFailure`. Unchanged; not in scope. + +## Testing approach + +Unit tests drive `exportThroughBreaker` directly with unique keys per test (the registry is +process-wide; unique keys keep `t.Parallel()` safe) and one boundary test drives `RunExportEnvelope` +with a stub backend, as `TestRunExportEnvelope_circuitBreakerTripsAfterRepeatedFailures` does +(`circuit_breaker_test.go:26`). Half-open cases need the 30 s timeout to elapse; the test must not +sleep 30 s. The implementation may make the timeout overridable for tests through an unexported +package variable or a constructor parameter, read once at breaker creation; it must not change the +production value. + +## Risks + +- Mislabelled Terminal errors hide an outage from the breaker (see proposal *Assumptions*). +- A test that relies on Terminal errors tripping the breaker would now fail; none is known + (`rg -n 'circuit breaker open|ErrOpenState' internal` at G1 records the search). diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md new file mode 100644 index 00000000..304299b7 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -0,0 +1,70 @@ +# Loop — sink-breaker-terminal-isolation (spec-loop-duo run state) + +Repo: `/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation` · Branch: `fix/sink-breaker-terminal-isolation` · Base: `origin/main` @ `96c41b0d0` · Started: 2026-11-05 · Reviewers: two-model duo +Next: R spec-set review +Task prompt: local, `agent-context/loops/sink-breaker-terminal-isolation/task-prompt.md` (harness, not committed) · decisions by model alignment, logged in agent-context/decisions.md +Budget: paid review legs 0/12 (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 + +Backlog SINK-BREAKER-TENANT-01 (P2) and RACE-COUNT-BREAKER-ISOLATION (P3). Spec set: proposal.md, +design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `sink-breaker-terminal` +(both legs A, pending operator validation). + +## Stages + +- [x] 0 orient — new change; file:line references measured against `origin/main` @ `96c41b0d0` (2026-11-05). Open PRs touching these files: none (#523 mongo-driver v2 touches `internal/sink/mongodb/` only; #520 helm v4 CI only) +- [x] P plan + tasks — written; `task spec:validate` green (22 items); spec set committed: (this commit) +- [ ] R spec-set review — `reviews/R-spec-set/` +- [ ] L task loop — see *Tasks* +- [ ] B branch review — `reviews/B-branch/round-N/` +- [ ] Hand-off — `pr-description.md` + +## Fitness functions + +| Characteristic | Command | Kind | Baseline (number, list, allow-list) | Per task / at B | +|---|---|---|---|---| +| Unit tests, changed packages | `go test -v ./internal/sink/ ./internal/controller/ -skip TestControllers` | triggered | green | per task | +| Race | `go test -race -count=3 ./internal/sink/ ./internal/controller/` | triggered | RACE-COUNT-BREAKER-ISOLATION flake on base (4/6 runs, 2026-10) | per task; G2 makes it green | +| Architecture | `task lint` (golangci-lint v2 and go-arch-lint) | triggered | no new findings | per task | +| Markdown | `task lint:markdown` (tracked files only; stage new files first) | triggered | 0 issues | per task + every orchestrator state edit | +| OpenSpec strict | `task spec:validate` | triggered | green | per spec-set edit | +| Full local gate | `task check` | holistic | green on base, minus known environment reds | at B | + +Holistic run at B: not yet run. + +## Triage + +| Stage | Finding (one line) | Sev | Reviewers | Disposition (fix / reject / defer) | Where it went / why | Needs user? | +|---|---|---|---|---|---|---| + +## Tasks + +| Task | Verdict | Review (legs, rounds, paid leg?) | Gaps / decision request | +|---|---|---|---| + +## Known red + +Tests a test task left red for a later task; excluded from "green" at close, nothing else is. + +| Test (file:name) | Story | Written in | Cleared in | +|---|---|---|---| + +## Test changes + +Tests from a test task changed by a later task (close-task `TEST` class); each one changes what green means. + +| Test (file:name) | Written in | Changed in | Evidence it was wrong | +|---|---|---|---| + +## Lessons + +## Where a human should look first + +## Proposed harness changes + +| Lesson | Seen in | Proposed change (AGENTS.md line / sensor / close-task.yml) | +|---|---|---| + +## Owner tasks (skipped by the loop) + +| Task | Command sheet | +|---|---| diff --git a/openspec/changes/sink-breaker-terminal-isolation/proposal.md b/openspec/changes/sink-breaker-terminal-isolation/proposal.md new file mode 100644 index 00000000..26ad0d28 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/proposal.md @@ -0,0 +1,85 @@ +# Proposal + +## Why + +Every export to a sink runs through one process-wide circuit breaker keyed +`/` (`internal/sink/export.go:200`, `internal/sink/circuit_breaker.go:26-62`, +sony/gobreaker v2.4.0). It trips after 5 consecutive failures (`circuitBreakerTripAt`), stays open +30 s (`circuitBreakerTimeout`), and while open every export to that sink, from every inventory, fails +fast with a Transient "sink circuit breaker open" error. + +The breaker counts **every** non-nil error from the export as a failure, Terminal ones included +(gobreaker's default `IsSuccessful` returns false for any non-nil error; `Settings` sets neither +`IsSuccessful` nor `IsExcluded`). Terminal errors are raised inside the export for problems that +belong to one inventory: a Git ownership rejection (`internal/sink/git/prune_owned.go:255`), an +identity/path mismatch, an oversized payload (`internal/sink/export.go:117-173`). Five such +rejections from one misconfigured inventory open the breaker for every healthy inventory on the same +sink for 30 s (backlog SINK-BREAKER-TENANT-01, found by the #412 review 2026-10-04). + +This contradicts the documented design: ADR-0602 says the breaker is "per sink/doc backend for +`ErrTransient` storms" (`docs/adr/0602-error-taxonomy.md:38`), and the troubleshooting page says it +opens after "5 consecutive transient failures per sink key" +(`docs/operator-manual/troubleshooting.md:214`). + +The same root cause makes `go test -race -count=3 -skip TestControllers ./internal/controller/` +flaky (backlog RACE-COUNT-BREAKER-ISOLATION, P3): repeated runs of +`TestKollectInventoryReconciler_terminalTotalExportFailureDoesNotRequeue` record terminal failures +against the shared `default/postgres-demo` breaker until it opens, and the next test that exports to +that key gets "breaker open" instead of its own outcome (4 of 6 base runs failed, no DATA RACE). + +A Terminal error is never retried, so counting it protects no backend; it only widens one +inventory's failure to the whole sink. + +## What Changes + +- The sink circuit breaker **excludes** errors classified Terminal (`kollecterrors.IsTerminal`) at + the breaker boundary: they count neither as failures nor as successes (gobreaker `IsExcluded`). + Transient and unclassified errors keep counting, so the breaker still trips after 5 consecutive + retryable failures. +- An excluded error during half-open neither closes nor re-opens the breaker and releases the trial + slot, so the next export is admitted as the trial. +- The four reconciler tests named in RACE-COUNT-BREAKER-ISOLATION pass under `-race -count=3`; + where a test still shares a breaker key through Transient failures, it is isolated (its own sink + key or a breaker reset), not retried. +- Docs say which errors count. + +Decision: option A of three (exclude all Terminal / tag inventory-scoped Terminal only / key per +inventory), chosen by two independent legs, both A (`agent-context/decisions.md`, 2026-11-05 +`sink-breaker-terminal`, pending operator validation). + +## Capabilities + +### New Capabilities + +- `sink-circuit-breaker`: which export errors a sink's shared circuit breaker counts, and how it + behaves closed, open and half-open. + +### Modified Capabilities + +None. + +## Impact + +- Code: `internal/sink/circuit_breaker.go` (breaker settings). No change to `export.go`'s + classification after the breaker. +- Tests: `internal/sink/circuit_breaker_test.go` (new cases), `internal/controller` tests named in + RACE-COUNT-BREAKER-ISOLATION (isolation only if the fix alone does not make them pass). +- Docs: `docs/operator-manual/troubleshooting.md:214`, `docs/adr/0602-error-taxonomy.md:38` + (clarifying note, not a new ADR). +- Behaviour visible to users: a misconfigured inventory no longer opens the breaker for other + inventories on its sink. A sink-wide Terminal error (e.g. a CA misconfiguration) no longer opens + the breaker either; it was never retried, and it stays visible in each inventory's status. +- Release note: v0.22.x / next minor, under fixes. + +## Non-goals + +- Changing the breaker key, trip threshold, interval or timeout. +- Reclassifying any error (e.g. making Git auth failures Terminal before the breaker). +- A breaker-state metric or condition. + +## Assumptions + +- Terminal classification at the breaker boundary is correct for the backends: an error labelled + Terminal is not a symptom of an outage. A backend that mislabels an outage as Terminal makes the + breaker blind to it; those exports are still not requeued, so the load on the backend stays bounded + (main risk named by both decision legs). diff --git a/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md b/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md new file mode 100644 index 00000000..fa11cd33 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md @@ -0,0 +1,60 @@ +# Spec Delta + +## ADDED Requirements + +### Requirement: SCB-1 Terminal export errors do not count toward the sink breaker + +The shared per-sink circuit breaker SHALL exclude an export error classified Terminal: it SHALL count +neither as a failure nor as a success, and SHALL NOT reset the consecutive-failure count. + +#### Scenario: One inventory's terminal rejections do not open the breaker + +- **GIVEN** a sink with a closed breaker +- **WHEN** one inventory's export to that sink fails with a Terminal error 10 times in a row +- **THEN** the breaker stays closed +- **AND** another inventory's next export to the same sink runs against the backend and succeeds + +#### Scenario: An excluded error does not break a streak of retryable failures + +- **GIVEN** a sink with a closed breaker +- **WHEN** exports fail Transient, Terminal, Transient, Transient, Transient, Transient +- **THEN** the breaker opens after the fifth Transient failure + +### Requirement: SCB-2 Retryable export errors still open the sink breaker + +The breaker SHALL count Transient and unclassified export errors as failures and SHALL open after 5 +consecutive ones. While open, an export to the sink SHALL NOT reach the backend and SHALL fail with a +Transient error naming the open breaker. + +#### Scenario: Five transient failures open the breaker + +- **GIVEN** a sink with a closed breaker +- **WHEN** 5 consecutive exports fail with a Transient error +- **THEN** the sixth export does not call the backend +- **AND** it fails with a Transient error whose message contains "circuit breaker open" + +#### Scenario: Unclassified errors count + +- **GIVEN** a sink with a closed breaker +- **WHEN** 5 consecutive exports fail with an error that carries no class +- **THEN** the sixth export does not call the backend + +### Requirement: SCB-3 A terminal result in half-open neither closes nor wedges the breaker + +When the breaker is half-open, an export whose result is a Terminal error SHALL leave the breaker +half-open and SHALL release the trial slot, so that the next export is admitted as the trial. + +#### Scenario: Terminal trial, then a successful trial + +- **GIVEN** a breaker that opened on Transient failures and whose open timeout has elapsed +- **WHEN** the trial export fails with a Terminal error +- **AND** the next export succeeds +- **THEN** the next export reaches the backend +- **AND** the breaker is closed afterwards + +#### Scenario: Terminal trial, then a failing trial + +- **GIVEN** a breaker that opened on Transient failures and whose open timeout has elapsed +- **WHEN** the trial export fails with a Terminal error +- **AND** the next export fails with a Transient error +- **THEN** the breaker is open afterwards diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md new file mode 100644 index 00000000..786ebca5 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -0,0 +1,81 @@ +# Tasks + +Red first. Each group's test task comes before its implementation and records its red in +`evidence/G.md`: the named test runs (`=== RUN `) and fails on the target assertion, not on +a compile error or a skip. "Red on base" means red on `origin/main` for G1 and on the G1 commit for +G2. A stub with a `not implemented` (no-op) body is in scope so tests compile. Each group closes in +one commit (tests and implementation), so the tree is green at every commit. A case marked *control* +is expected green on its base and is recorded as a control, not a red. + +Each case reads: given / when / then · the failure it exposes. + +Test rules for every group: + +- Breaker unit tests live in `internal/sink/circuit_breaker_test.go`, use a key unique to the test + (the registry is process-wide) and `t.Parallel()`. No test sleeps for the production 30 s timeout; + see `design.md` *Testing approach*. +- Each group records `go test -race -count=3 ./internal/sink/ ./internal/controller/` (G2 adds + `-skip TestControllers` six times, see 2.1) and `task lint`. +- Mutations are judged by the test command's exit status, with one no-op control that must survive; + mutant trees live outside the repository. + +## 0. Probe + +- [ ] 0.1 Verify the load-bearing claims in `proposal.md` and `design.md` on `origin/main` and record + each with file:line or command output in `evidence/T0.1.md`: the breaker settings and key; that + gobreaker v2.4.0 `IsExcluded` neither counts nor resets `ConsecutiveFailures` and frees the + half-open slot; that ownership rejections reach the breaker already Terminal; that Git auth + failures are classified only after the breaker; that RACE-COUNT-BREAKER-ISOLATION reproduces + (`go test -race -count=3 -skip TestControllers ./internal/controller/`, six runs, failures counted, + failing test names). A claim that does not hold stops the loop with a decision request. + +## 1. Breaker excludes Terminal errors (SCB-1, SCB-2, SCB-3) + +- [ ] 1.1 Tests (red on base unless marked *control*): + - SCB-1a: given a fresh key, when `exportThroughBreaker` returns a Terminal error 10 times, then an + 11th call with a succeeding function runs it and returns nil · exposes Terminal errors counting + as failures. + - SCB-1b (boundary): given a stub backend that fails Terminal for inventory path A and succeeds for + path B on one sink, when `RunExportEnvelope` runs A 10 times, then B's export reaches the backend + and succeeds · exposes the cross-inventory blast radius at the real call site. + - SCB-1c: given a fresh key, when calls fail T, X, T, T, T (T Transient, X Terminal), then the + sixth call still runs its function (four counted failures); when that sixth call fails T, the + seventh does not run and returns a Transient error containing "circuit breaker open" · on base + the breaker opens after the fifth call (X counted), so the sixth call's assertion is the red; + after the fix it exposes an implementation that resets the streak on exclusion (`IsSuccessful` + instead of `IsExcluded`), which would let the seventh call run. + - SCB-2a (*control*): 5 Transient failures, then the 6th call's function does not run and the + error is Transient with "circuit breaker open" · guards against over-excluding. + - SCB-2b (*control*): same with an unclassified `errors.New` error · guards against excluding + everything that is not Transient. + - SCB-3a: given a breaker opened by 5 Transient failures and its timeout elapsed, when the trial + returns Terminal and the next call succeeds, then the next call's function runs and a following + call also runs (closed) · exposes an excluded trial closing the breaker or wedging the slot. + - SCB-3b: as 3a, but the call after the Terminal trial fails Transient, then the call after that + does not run (re-opened) · exposes an excluded trial closing the breaker. +- [ ] 1.2 Set `IsExcluded: kollecterrors.IsTerminal` in `breakerFor`, plus the test seam for the + open timeout that `design.md` allows. Mutations: drop `IsExcluded`; replace it with + `IsSuccessful`; exclude every non-nil error. Each must be killed by a named 1.1 case. + +## 2. Reconciler tests stop sharing a tripped breaker (RACE-COUNT-BREAKER-ISOLATION) + +- [ ] 2.1 On the G1 commit run `go test -race -count=3 -skip TestControllers ./internal/controller/` + six times and record each exit status. All six green → record it and close 2.2 as not needed. Any + red → name the failing tests and the breaker key they share; that red is the group's red. +- [ ] 2.2 Only if 2.1 is red: isolate the failing tests from each other's breaker state (distinct + sink names per test, or `t.Cleanup(sink.ResetBreakersForTest)` where the test is not parallel), + never a retry. Six green runs of the 2.1 command afterwards. + +## 3. Docs + +- [ ] 3.1 `docs/operator-manual/troubleshooting.md:214`: the breaker counts Transient and unclassified + export errors, not Terminal ones; a Terminal failure stays in that inventory's status. + `docs/adr/0602-error-taxonomy.md:38`: add that Terminal errors are excluded (dated note, ADR not + re-decided). `task lint:markdown` 0 issues; `rg -n 'circuit breaker' docs` reviewed for any other + claim the change contradicts, the list recorded in `evidence/G3.md`. + +## 4. Integration + +- [ ] 4.1 Integrator (orchestrator): branch review (stage B), PR with the release-note line, merge on + review + green CI, post-merge evidence, archive the change, update BACKLOG (SINK-BREAKER-TENANT-01, + RACE-COUNT-BREAKER-ISOLATION). From dbfdaef52caf7d8038adbc7fb6ecca075b2d93b9 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 11:17:38 +0100 Subject: [PATCH 02/13] :page_facing_up: docs(spec): address spec-set review round 1 Test isolation for the process-wide breaker registry, a constructor-only timeout seam, the before/at/after-breaker classification boundary, Forbidden and inferred-Terminal controls, half-open state assertions and a concurrent-trial case, and a corrected flake mechanism. --- .../sink-breaker-terminal-isolation/design.md | 43 ++++++++---- .../sink-breaker-terminal-isolation/loop.md | 13 +++- .../proposal.md | 28 +++++--- .../reviews/R-spec-set/brief-reviewer-a.md | 28 ++++++++ .../reviews/R-spec-set/leg-reviewer-a-spec.md | 33 ++++++++++ .../reviews/R-spec-set/leg-reviewer-b-spec.md | 35 ++++++++++ .../reviews/R-spec-set/verification.md | 18 +++++ .../specs/sink-circuit-breaker/spec.md | 44 ++++++++++--- .../sink-breaker-terminal-isolation/tasks.md | 66 ++++++++++++------- 9 files changed, 254 insertions(+), 54 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/leg-reviewer-a-spec.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/leg-reviewer-b-spec.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/verification.md diff --git a/openspec/changes/sink-breaker-terminal-isolation/design.md b/openspec/changes/sink-breaker-terminal-isolation/design.md index f2de0a95..5b16f5a8 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/design.md +++ b/openspec/changes/sink-breaker-terminal-isolation/design.md @@ -8,11 +8,20 @@ req.SinkName`. `breakerFor` lazily creates one `gobreaker.CircuitBreaker[struct{ process-wide `sync.Map` with `MaxRequests: 1`, `Interval: 60s`, `Timeout: 30s` and `ReadyToTrip: ConsecutiveFailures >= 5`. -Errors reaching the breaker are whatever `plan.run` returns. Some are already classified -`kollecterrors.Terminal` inside the export (ownership, identity, payload, CA/TLS configuration, -backend-specific Terminal classification in `bigquery/errors.go`, `mongodb/errors.go`). Others are -classified only after the breaker by `git.ClassifyExportError` and `classifyExportFailure` -(`export.go:207-212`), e.g. Git auth failures; those count as failures today and keep counting. +Three zones (`internal/sink/export.go`): + +- **Before the breaker** (`:79-190`): registry/spec checks, inventory identity (`:86`), backend + acquire incl. construction and CA/TLS config (`:92-110`), payload decode, size and layout. These + return early and never touch the breaker. Unchanged. +- **At the breaker** (`:200`): whatever `plan.run` returns. The class is + `kollecterrors.ClassOf(err)` (`internal/errors/errors.go:86-108`): an explicit `ClassError`, else a + Kubernetes API status (NotFound/Invalid/BadRequest → Terminal, Forbidden → Forbidden), else + Transient. Terminal results here include the Git ownership/prune rejection + (`git/prune_owned.go:255`) and backend-classified write errors (`bigquery/errors.go`, + `mongodb/errors.go`). +- **After the breaker** (`:207-212`): `git.ClassifyExportError` and `classifyExportFailure`. Errors + that become Terminal only here (e.g. Git auth failures, `git/errors.go:35-36`) were not Terminal at + the breaker and keep counting. ## Option chosen @@ -38,13 +47,25 @@ registry needs eviction). Decision record: `agent-context/decisions.md` 2026-11- ## Testing approach -Unit tests drive `exportThroughBreaker` directly with unique keys per test (the registry is -process-wide; unique keys keep `t.Parallel()` safe) and one boundary test drives `RunExportEnvelope` +Unit tests drive `exportThroughBreaker` directly and one boundary test drives `RunExportEnvelope` with a stub backend, as `TestRunExportEnvelope_circuitBreakerTripsAfterRepeatedFailures` does -(`circuit_breaker_test.go:26`). Half-open cases need the 30 s timeout to elapse; the test must not -sleep 30 s. The implementation may make the timeout overridable for tests through an unexported -package variable or a constructor parameter, read once at breaker creation; it must not change the -production value. +(`circuit_breaker_test.go:26`). + +Isolation. The registry is process-wide and `ResetBreakersForTest` clears every key +(`circuit_breaker.go:68-70`). Rules: (1) a key is unique per test *invocation*, not per test name, +because `-count=N` reruns a test against the breakers its earlier run left behind (e.g. `t.Name()` +plus a package-level atomic counter); (2) no `t.Parallel()` test calls `ResetBreakersForTest`, so +the existing `TestResetBreakersForTest_clearsOpenBreaker` (`circuit_breaker_test.go:95-96`, parallel, +resets at `:155`) becomes non-parallel in G1. The other callers (`release_on_delete_test.go`, +`inventory_identity_test.go`, `git/owned_prune_fuzz_test.go`) are checked in G1 for the same rule and +listed in the evidence. + +Timeout seam. Half-open cases must not sleep 30 s. The only allowed seam is a constructor: +`newBreaker(key string, timeout time.Duration)`, which `breakerFor` calls with +`circuitBreakerTimeout`; a test builds a short-timeout breaker and stores it under its own unique key +with `breakerRegistry.LoadOrStore` before the first call. No mutable package variable (a write from +one parallel test is a data race with `breakerFor` in another and would leak into their breakers). +State assertions read `breakerFor(key).State()`. ## Risks diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index 304299b7..8635d048 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -1,9 +1,9 @@ # Loop — sink-breaker-terminal-isolation (spec-loop-duo run state) Repo: `/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation` · Branch: `fix/sink-breaker-terminal-isolation` · Base: `origin/main` @ `96c41b0d0` · Started: 2026-11-05 · Reviewers: two-model duo -Next: R spec-set review +Next: R spec-set review round 2 Task prompt: local, `agent-context/loops/sink-breaker-terminal-isolation/task-prompt.md` (harness, not committed) · decisions by model alignment, logged in agent-context/decisions.md -Budget: paid review legs 0/12 (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 +Budget: paid review legs 1/12 (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 Backlog SINK-BREAKER-TENANT-01 (P2) and RACE-COUNT-BREAKER-ISOLATION (P3). Spec set: proposal.md, design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `sink-breaker-terminal` @@ -13,7 +13,7 @@ design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `si - [x] 0 orient — new change; file:line references measured against `origin/main` @ `96c41b0d0` (2026-11-05). Open PRs touching these files: none (#523 mongo-driver v2 touches `internal/sink/mongodb/` only; #520 helm v4 CI only) - [x] P plan + tasks — written; `task spec:validate` green (22 items); spec set committed: (this commit) -- [ ] R spec-set review — `reviews/R-spec-set/` +- [ ] R spec-set review — round 1 on `a505029da`: reviewer-a REQUEST_CHANGES (1 CRITICAL), reviewer-b CONCERNS; all 8 findings verified and fixed (`reviews/R-spec-set/verification.md`); CRITICAL had a mechanical fix both legs named → round 2 (last) on the fix commit - [ ] L task loop — see *Tasks* - [ ] B branch review — `reviews/B-branch/round-N/` - [ ] Hand-off — `pr-description.md` @@ -35,6 +35,13 @@ Holistic run at B: not yet run. | Stage | Finding (one line) | Sev | Reviewers | Disposition (fix / reject / defer) | Where it went / why | Needs user? | |---|---|---|---|---|---|---| +| R1 | Parallel test resets every breaker key; `-count` reuses keys | CRITICAL | reviewer-a, reviewer-b | fix | design isolation rules, tasks rules, 1.2 | no | +| R1 | Mutable timeout seam races | WARNING | reviewer-a, reviewer-b | fix | design: constructor seam only | no | +| R1 | Cited Terminal errors return before the breaker | WARNING | reviewer-a, reviewer-b | fix | proposal, design three zones, 0.1 | no | +| R1 | Classification boundary, Forbidden, inferred Terminal unspecified | WARNING | reviewer-a, reviewer-b | fix | SCB-1/SCB-2 text + scenarios, cases 1d/2c/2d | no | +| R1 | SCB-3a proves no state; no concurrent trial | WARNING | reviewer-a | fix | SCB-3a/3b `State()`, SCB-3c | no | +| R1 | Flake mechanism overstated | WARNING | reviewer-a, reviewer-b | fix | proposal, 2.1 ten runs, 2.2 always | no | +| R1 | Docs "self-closes after timeout"; Terminal trial keeps half-open | NOTE | reviewer-a, reviewer-b | fix | 3.1 wording; SCB-3 accepted consequence | no | ## Tasks diff --git a/openspec/changes/sink-breaker-terminal-isolation/proposal.md b/openspec/changes/sink-breaker-terminal-isolation/proposal.md index 26ad0d28..faea8112 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/proposal.md +++ b/openspec/changes/sink-breaker-terminal-isolation/proposal.md @@ -11,8 +11,10 @@ fast with a Transient "sink circuit breaker open" error. The breaker counts **every** non-nil error from the export as a failure, Terminal ones included (gobreaker's default `IsSuccessful` returns false for any non-nil error; `Settings` sets neither `IsSuccessful` nor `IsExcluded`). Terminal errors are raised inside the export for problems that -belong to one inventory: a Git ownership rejection (`internal/sink/git/prune_owned.go:255`), an -identity/path mismatch, an oversized payload (`internal/sink/export.go:117-173`). Five such +belong to one inventory, most visibly the Git ownership/prune rejection +(`internal/sink/git/prune_owned.go:255`), raised from inside the export plan. (Identity, payload, +layout and backend-acquire errors return before the breaker, `internal/sink/export.go:86-190`, and +never reach it; only errors returned by `plan.run` at `export.go:200` do.) Five such rejections from one misconfigured inventory open the breaker for every healthy inventory on the same sink for 30 s (backlog SINK-BREAKER-TENANT-01, found by the #412 review 2026-10-04). @@ -21,11 +23,14 @@ This contradicts the documented design: ADR-0602 says the breaker is "per sink/d opens after "5 consecutive transient failures per sink key" (`docs/operator-manual/troubleshooting.md:214`). -The same root cause makes `go test -race -count=3 -skip TestControllers ./internal/controller/` -flaky (backlog RACE-COUNT-BREAKER-ISOLATION, P3): repeated runs of -`TestKollectInventoryReconciler_terminalTotalExportFailureDoesNotRequeue` record terminal failures -against the shared `default/postgres-demo` breaker until it opens, and the next test that exports to -that key gets "breaker open" instead of its own outcome (4 of 6 base runs failed, no DATA RACE). +A related flake: `go test -race -count=3 -skip TestControllers ./internal/controller/` fails in four +reconciler tests (backlog RACE-COUNT-BREAKER-ISOLATION, P3; 4 of 6 base runs failed 2026-10, no DATA +RACE, not yet re-measured). Those tests share the `default/postgres-demo` breaker through +`newExportFailureReconciler` (`internal/controller/kollectinventory_export_test.go:339-375`): the +Terminal-failure test, the Transient-failure test and successful exports all record against one key, +so whether it opens depends on the mix and order of outcomes. Excluding Terminal results removes one +contributor; the Transient test still contributes, so this change re-measures the flake and isolates +the tests if it persists (tasks group 2). A Terminal error is never retried, so counting it protects no backend; it only widens one inventory's failure to the whole sink. @@ -67,8 +72,13 @@ None. - Docs: `docs/operator-manual/troubleshooting.md:214`, `docs/adr/0602-error-taxonomy.md:38` (clarifying note, not a new ADR). - Behaviour visible to users: a misconfigured inventory no longer opens the breaker for other - inventories on its sink. A sink-wide Terminal error (e.g. a CA misconfiguration) no longer opens - the breaker either; it was never retried, and it stays visible in each inventory's status. + inventories on its sink. A sink-wide Terminal error returned during the export (e.g. a BigQuery + authorization failure classified Terminal in `internal/sink/bigquery/errors.go`) no longer opens + the breaker either; it was never retried, and it stays visible in each inventory's status. Export + error metrics are recorded after the breaker (`export.go:203-212`) and are unchanged. A Terminal + result on the half-open trial leaves the breaker half-open, so one misconfigured inventory can keep + a recovering sink admitting one trial at a time until another export's trial succeeds or fails + retryably; bounded and stated in SCB-3. - Release note: v0.22.x / next minor, under fixes. ## Non-goals diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/brief-reviewer-a.md new file mode 100644 index 00000000..c06a91fe --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/brief-reviewer-a.md @@ -0,0 +1,28 @@ +# Cold review: spec set `sink-breaker-terminal-isolation` (lens: spec) + +You are an independent reviewer. You have read-only access to this repository (Go Kubernetes +operator "kollect"). Review the OpenSpec change at +`openspec/changes/sink-breaker-terminal-isolation/`: `proposal.md`, `design.md`, +`specs/sink-circuit-breaker/spec.md`, `tasks.md` (commit `a505029da`). Ignore `loop.md` except as context. + +Question: **would executing these tasks as written deliver the spec, safely?** + +Check, against the actual code (read it; do not trust the spec's file:line claims): + +1. Every factual claim in proposal.md/design.md: breaker settings and key + (`internal/sink/circuit_breaker.go`, `internal/sink/export.go`), which errors reach the breaker + already Terminal vs classified after it, gobreaker v2.4.0 `IsExcluded` semantics in closed and + half-open state (module source under the Go module cache, `github.com/sony/gobreaker/v2@v2.4.0`), + and the RACE-COUNT-BREAKER-ISOLATION root cause (`internal/controller/*_test.go`). +2. Requirements SCB-1..3: are they testable, unambiguous, and complete? Missing scenarios (e.g. + concurrency in half-open, `ErrTooManyRequests`, Forbidden-class errors, a Terminal error that is + in fact sink-wide)? +3. Tasks: missing, misordered or unverifiable tasks; a test case that cannot be red on base for its + stated reason; a case whose stated failure it would not actually expose; mutations a case would + not kill; a test seam for the 30 s timeout that could change production behaviour or race. +4. Anything the change would break elsewhere (other tests or docs relying on Terminal errors + tripping the breaker; metrics; docs claims). + +Output: findings as a list, each with severity CRITICAL / WARNING / NOTE, file:line evidence, and a +concrete fix. CRITICAL only for something that makes the tasks fail to deliver the spec or ship a +defect. End with `## Verdict` and one of APPROVE / CONCERNS / REQUEST_CHANGES / BLOCK. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/leg-reviewer-a-spec.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/leg-reviewer-a-spec.md new file mode 100644 index 00000000..929d8925 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/leg-reviewer-a-spec.md @@ -0,0 +1,33 @@ +Reviewed commit `a505029da` and the local gobreaker v2.4.0 source. The proposed `IsExcluded` setting has the intended closed/half-open behavior. I did not execute tests in this read-only review; the historical “4 of 6” reproduction remains unverified. + +1. **CRITICAL — Parallel tests remain vulnerable to global breaker resets.** + `tasks.md:14–16` and `design.md:41–42` assume unique keys make parallel breaker tests safe. However, `internal/sink/circuit_breaker_test.go:95–96,155` runs a parallel test that calls `ResetBreakersForTest()`, which clears **all** keys (`internal/sink/circuit_breaker.go:68–69`). It can erase a new test’s breaker between failures, or between opening and probing it. This introduces intermittent failures into the required validation and can invalidate mutation evidence. + **Fix:** Make the existing global-reset test nonparallel, audit other reset callers, and require keys unique per test **invocation**, including `-count` repetitions. + +2. **WARNING — The permitted timeout seam can race and affect unrelated breakers.** + `design.md:45–47` permits a mutable package variable while `tasks.md:14–16` requires parallel tests. Reading that variable once during construction does not synchronize it with test writes; unrelated breakers can also inherit the shortened timeout. + **Fix:** Require a constructor parameter with production explicitly supplying 30 seconds. Install the shortened-timeout breaker only under the current test’s unique key. Prohibit a mutable global override. + +3. **WARNING — Several claimed examples never reach the breaker.** + `proposal.md:11–17` and `design.md:11–15` include identity, payload and CA/TLS configuration errors among failures counted by the breaker. Identity validation returns at `internal/sink/export.go:86–89`; payload failures return at lines 115–120 and 168–177; backend construction failures return at lines 92–110. For example, Git configuration faults are Terminal during construction (`internal/sink/git/backend.go:20–33`). The breaker starts only at `export.go:200`. + **Fix:** Separate errors occurring before, inside and after the breaker. Use ownership rejection and backend write classification as affected examples. Replace the CA example in `proposal.md:70` with an actual sink-wide Terminal returned during export, such as BigQuery authorization failure (`internal/sink/bigquery/errors.go:67–71`). + +4. **WARNING — The requirements omit the classification boundary and leave Forbidden behavior unspecified.** + `specs/sink-circuit-breaker/spec.md:7–8,25–27` does not limit Terminal classification to the breaker boundary. Git auth errors become Terminal afterwards (`internal/sink/export.go:208–212`; `internal/sink/git/errors.go:35–36`) and will still trip it. Also, `IsTerminal` derives Terminal from raw Kubernetes status errors; an error need not carry an explicit class (`internal/errors/errors.go:99–104,115–117`). Forbidden errors remain counted, but SCB-2 does not say so. + **Fix:** Define exclusion as `IsTerminal(err)` evaluated on `plan.run`’s result; count every other non-nil result. Add controls for Forbidden, inferred Terminal, sink-wide Terminal, and an error classified Terminal only after the breaker. + +5. **WARNING — SCB-3a does not prove the state transitions it claims.** + `tasks.md:51–53` says admitting two subsequent successful calls exposes premature closing and proves closure. Both calls also run if the Terminal trial incorrectly closes the breaker. They can also run if successful trials incorrectly leave it half-open while releasing slots. SCB-3b covers premature closing, but neither specified assertion proves successful recovery closes it. + **Fix:** Assert `StateHalfOpen` immediately after the Terminal trial and `StateClosed` after success. Add a channel-controlled concurrent probe: while a trial is blocked, a second call must not execute and must return `ErrTooManyRequests`; after Terminal completion, one new trial must be admitted. + +6. **WARNING — The claimed controller root cause overstates what the named test can accumulate.** + `proposal.md:25–28` attributes opening to repeated runs of the Terminal test. That test performs one reconciliation (`internal/controller/kollectinventory_export_test.go:405–413`), so `-count=3` contributes only three failures. The Transient test shares its `default/postgres-demo` key through the same helper (`:362,439–443`), contributing additional failures; successful tests also share the key and can reset the streak (`:263,291–312`). The mechanism is mixed outcomes and scheduling, not the Terminal test alone. + **Fix:** Describe the shared-key interaction accurately. In task 0.1, distinguish failure to reproduce a probabilistic flake from disproving the mechanism; record the historical reproduction separately from current evidence. + +7. **NOTE — Correct the recovery documentation while editing it.** + `docs/operator-manual/troubleshooting.md:214`, already targeted by task 3.1, says the breaker “self-closes after timeout.” gobreaker instead enters half-open and closes after a successful trial (`gobreaker.go:276–284,312–315`); excluded Terminal trials can keep it half-open indefinitely. + **Fix:** State that timeout permits a trial, Terminal releases the slot without changing state, success closes, and counted failure reopens. Existing export-error metrics remain recorded after exclusion (`internal/sink/export.go:203–212`). + +## Verdict + +REQUEST_CHANGES diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/leg-reviewer-b-spec.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/leg-reviewer-b-spec.md new file mode 100644 index 00000000..ebb60a7a --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/leg-reviewer-b-spec.md @@ -0,0 +1,35 @@ +## Verdict: CONCERNS + +## Findings +- [WARNING] The test rules say parallel breaker tests with unique keys are safe. They are not, because a sibling parallel test clears every key. — `internal/sink/circuit_breaker_test.go:96,155`, `internal/sink/circuit_breaker.go:68-70`, `openspec/changes/sink-breaker-terminal-isolation/tasks.md:14-15`, `design.md:41-42` + Failure: `TestResetBreakersForTest_clearsOpenBreaker` is `t.Parallel()` and calls `ResetBreakersForTest()`, which runs `breakerRegistry.Clear()` on all keys. If that reset lands between SCB-2a's fifth Transient failure and its sixth call, or after SCB-3's trip, the "does not run" or "re-opened" assertion fails at random. The file says as much at lines 165-166 ("must not wipe the breakers of sibling parallel tests mid-trip"). This is the same flake class that group 2 is meant to remove. + Fix: make `TestResetBreakersForTest_clearsOpenBreaker` non-parallel, or replace it with a per-key reset helper, and add a test rule: "no `t.Parallel()` test may call `ResetBreakersForTest`". + Confidence: 80 +- [WARNING] The design lets the timeout test seam be an unexported package variable. With the required `t.Parallel()`, that is a data race under `-race`. — `design.md:45-47`, `tasks.md:14,17,56` + Failure: SCB-3a and 3b write the variable to shorten the timeout while other parallel tests read it in `breakerFor`. `go test -race` reports DATA RACE, and both groups record their gates under `-race`. Restoring the value in `t.Cleanup` also changes the timeout for breakers that other tests create at the same moment. + Fix: allow only the constructor-parameter form, e.g. a `newBreaker(key, timeout)` used by `breakerFor` with the const, and have tests register a breaker for their own key. Or make the half-open tests serial. + Confidence: 75 +- [WARNING] Some of the proposal's examples of Terminal errors that reach the breaker are wrong: they return before `exportThroughBreaker` runs. — `proposal.md:14-15`, `design.md:12-13`, `internal/sink/export.go:86,117-190` versus `:200` + Failure: the identity/path check (line 86), the oversized-payload error (173), payload decode/marshal (117/136/161) and layout resolve (186) all return early, so they never count today. Only errors from `plan.run` reach the breaker: the ownership/prune rejection at `git/prune_owned.go:255` and backend-classified Terminal errors. The proposal's Impact line about CA errors is probably wrong the same way (backend acquire happens at ~:107, also before the breaker). Probe 0.1 checks only the ownership claim, so the overstated motivation would pass it. + Fix: in 0.1, list for each cited Terminal site whether it is inside `plan.run`, and correct the proposal and design. + Confidence: 85 +- [NOTE] SCB-2 says only "Transient and unclassified" errors count. Forbidden-classified errors also count under `IsExcluded = IsTerminal`, and the spec says nothing about them. — `internal/errors/errors.go:86-108`, `spec.md:25` + Failure: a backend error classified Forbidden, or a raw K8s 403, still trips the breaker. That holds in the code, but it is neither required nor forbidden, so a later change could exclude it without breaking any test. + Fix: one sentence in SCB-2 that says Forbidden counts, or that it is excluded, plus one test case for it. + Confidence: 60 +- [NOTE] "Unclassified" in the spec is not the same as `ClassOf`. A bare `apierrors` NotFound/Invalid/BadRequest carries no `ClassError`, yet `IsTerminal` treats it as Terminal, so it is excluded. — `internal/errors/errors.go:99-101`, `spec.md:39-40` + Failure: an error that "carries no class" per the spec's wording, returned by `plan.run`, would not count. SCB-2b uses only `errors.New`, so this goes unnoticed. + Fix: say in the spec that classification means `kollecterrors.ClassOf`. + Confidence: 55 + +**Requirement status:** +- **SCB-1 holds** if the tasks are executed. In gobreaker v2.4.0, `afterRequest` checks `isExcluded` before `isSuccessful`/`isFailure` (`gobreaker.go:264-267`), and `onExclusion` only increments `TotalExclusions` (`counter.go:32-34`), so the streak is neither reset nor incremented. The 1.1 cases (1a, 1b, 1c) cover the scenarios. +- **SCB-2 holds.** The message "sink circuit breaker open for %q" contains "circuit breaker open" (`circuit_breaker.go:40`). SCB-2a and 2b are valid controls. +- **SCB-3 holds.** Half-open admits a request while `validRequests() < maxRequests`, with `validRequests = Requests - TotalExclusions` (`gobreaker.go:246`, `counter.go:36-41`), so an excluded trial frees the slot. Success closes the breaker (`:282`) and a counted failure re-opens it (`:296`). The design's line citations match the module source. +- **The three mutations in 1.2 are each killed.** Dropping `IsExcluded` is caught by SCB-1a, swapping in `IsSuccessful` by SCB-1c, and excluding every error by SCB-2a/2b. +- **Behaviour no requirement mentions:** a Terminal result from the trial request leaves the breaker half-open, so one misconfigured inventory can keep a recovering sink half-open, serialising exports one at a time, for as long as its requests keep winning the trial. This is bounded but not stated. Also, `TestKollectInventoryReconciler_transientTotalExportFailureRequeues` (`internal/controller/kollectinventory_export_test.go:439`) still sends Transient failures to the shared `default/postgres-demo` key. Task 2.1's six-run sample (base fails 4 of 6) gives weak evidence that the flake is gone. + +## Could not check +- The breaker results of the four tests named in backlog RACE-COUNT-BREAKER-ISOLATION; the backlog and `agent-context/decisions.md` are outside the change. +- `git.ClassifyExportError` and the backend Terminal mappings (`bigquery/errors.go`, `mongodb/errors.go`), to confirm whether the claim that Git auth failures are classified only after the breaker is true. +- `loop.md` and the untracked `reviews/` directory. No tests were run. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/verification.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/verification.md new file mode 100644 index 00000000..323f56bd --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/verification.md @@ -0,0 +1,18 @@ +# R round 1 — verification and triage + +Spec set `a505029da`. Reviewer A (spec lens): REQUEST_CHANGES. Reviewer B (spec lens): CONCERNS. +Each finding was checked against the code by the orchestrator on 2026-11-05 before triage. + +| # | Finding | Sev | Reviewers | Verified | Disposition | +|---|---|---|---|---|---| +| 1 | A parallel test (`TestResetBreakersForTest_clearsOpenBreaker`, `circuit_breaker_test.go:95-96`) calls `ResetBreakersForTest` (`:155`), which clears every key; unique keys do not make new tests safe; `-count` reuses keys | CRITICAL (A) / WARNING (B) | A, B | yes: `t.Parallel()` at `:96`, reset at `:155`, `breakerRegistry.Clear()` at `circuit_breaker.go:69` | fix: design *Testing approach* isolation rules; tasks test rules; 1.2 makes the test non-parallel; 0.1 lists every caller | +| 2 | Mutable package-variable timeout seam races under `-race` and leaks into other breakers | WARNING | A, B | yes (design allowed it) | fix: constructor `newBreaker(key, timeout)` only | +| 3 | Identity, payload, layout, acquire and CA/TLS errors return before the breaker | WARNING | A, B | yes: `export.go:86-190` return before `:200` | fix: proposal *Why*/*Impact*, design *Context* (three zones); 0.1 classifies each site | +| 4 | Classification boundary unspecified; Forbidden and `ClassOf`-inferred Terminal not specified | WARNING (A) / NOTE (B) | A, B | yes: `errors.go:86-108` | fix: SCB-1 boundary text and two scenarios; SCB-2 names Forbidden; cases SCB-1d, SCB-2c, SCB-2d | +| 5 | SCB-3a does not prove state transitions; no concurrent-trial case | WARNING | A | yes: two runs pass if an excluded trial closes the breaker | fix: `State()` assertions in SCB-3a/3b; SCB-3c concurrent probe; spec scenario added | +| 6 | Flake root cause overstated: the Terminal test runs once per count; the Transient and success tests share the key | WARNING | A, B | yes: `newExportFailureReconciler` (`kollectinventory_export_test.go:339-375`) used by both | fix: proposal reworded; 2.1 ten runs; 2.2 isolates regardless | +| 7 | Troubleshooting "self-closes after timeout" is wrong for gobreaker | NOTE | A | yes: half-open then trial | fix: task 3.1 wording | +| 8 | Terminal trial can keep a recovering sink half-open | NOTE | B | yes (follows from `validRequests`) | fix: stated as accepted consequence in SCB-3 and proposal *Impact* | + +The CRITICAL (1) has a mechanical fix both reviewers named and that the code confirms; it was fixed +in the spec set and goes to a second and last spec-set round instead of stopping (spec-loop R rule). diff --git a/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md b/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md index fa11cd33..f24b2548 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md +++ b/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md @@ -4,8 +4,10 @@ ### Requirement: SCB-1 Terminal export errors do not count toward the sink breaker -The shared per-sink circuit breaker SHALL exclude an export error classified Terminal: it SHALL count -neither as a failure nor as a success, and SHALL NOT reset the consecutive-failure count. +The shared per-sink circuit breaker SHALL exclude an export plan result for which +`kollecterrors.ClassOf` is Terminal: it SHALL count neither as a failure nor as a success, and SHALL +NOT reset the consecutive-failure count. Classification is evaluated on the result at the breaker, +not after it. #### Scenario: One inventory's terminal rejections do not open the breaker @@ -14,6 +16,18 @@ neither as a failure nor as a success, and SHALL NOT reset the consecutive-failu - **THEN** the breaker stays closed - **AND** another inventory's next export to the same sink runs against the backend and succeeds +#### Scenario: An inferred Terminal status is excluded + +- **GIVEN** a sink with a closed breaker +- **WHEN** the export plan returns an unwrapped Kubernetes NotFound status 10 times in a row +- **THEN** the breaker stays closed + +#### Scenario: An error classified Terminal only after the breaker still counts + +- **GIVEN** a sink with a closed breaker +- **WHEN** the export plan returns 5 times an unclassified error that the post-breaker Git classifier would mark Terminal (an authentication failure) +- **THEN** the sixth export does not call the backend + #### Scenario: An excluded error does not break a streak of retryable failures - **GIVEN** a sink with a closed breaker @@ -22,8 +36,9 @@ neither as a failure nor as a success, and SHALL NOT reset the consecutive-failu ### Requirement: SCB-2 Retryable export errors still open the sink breaker -The breaker SHALL count Transient and unclassified export errors as failures and SHALL open after 5 -consecutive ones. While open, an export to the sink SHALL NOT reach the backend and SHALL fail with a +The breaker SHALL count every other non-nil export result as a failure (Transient, Forbidden, and +errors without a class, including errors that become Terminal only after the breaker) and SHALL open +after 5 consecutive ones. While open, an export to the sink SHALL NOT reach the backend and SHALL fail with a Transient error naming the open breaker. #### Scenario: Five transient failures open the breaker @@ -33,23 +48,27 @@ Transient error naming the open breaker. - **THEN** the sixth export does not call the backend - **AND** it fails with a Transient error whose message contains "circuit breaker open" -#### Scenario: Unclassified errors count +#### Scenario: Unclassified and Forbidden errors count - **GIVEN** a sink with a closed breaker -- **WHEN** 5 consecutive exports fail with an error that carries no class +- **WHEN** 5 consecutive exports fail with an error that carries no class, or with a Forbidden error - **THEN** the sixth export does not call the backend ### Requirement: SCB-3 A terminal result in half-open neither closes nor wedges the breaker When the breaker is half-open, an export whose result is a Terminal error SHALL leave the breaker -half-open and SHALL release the trial slot, so that the next export is admitted as the trial. +half-open and SHALL release the trial slot, so that the next export is admitted as the trial. While a +trial is in flight, a concurrent export SHALL NOT reach the backend. A successful trial SHALL close the +breaker; a counted failure on the trial SHALL re-open it. Consequence (accepted): while Terminal +results keep winning the trial, the sink stays half-open and admits one export at a time. #### Scenario: Terminal trial, then a successful trial - **GIVEN** a breaker that opened on Transient failures and whose open timeout has elapsed - **WHEN** the trial export fails with a Terminal error -- **AND** the next export succeeds -- **THEN** the next export reaches the backend +- **THEN** the breaker is half-open +- **WHEN** the next export succeeds +- **THEN** that export reached the backend - **AND** the breaker is closed afterwards #### Scenario: Terminal trial, then a failing trial @@ -58,3 +77,10 @@ half-open and SHALL release the trial slot, so that the next export is admitted - **WHEN** the trial export fails with a Terminal error - **AND** the next export fails with a Transient error - **THEN** the breaker is open afterwards + +#### Scenario: Concurrent export during a trial + +- **GIVEN** a half-open breaker whose trial export is in flight +- **WHEN** a second export starts before the trial finishes +- **THEN** the second export does not reach the backend +- **AND** after the trial finishes with a Terminal error, the next export is admitted diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index 786ebca5..cad6db80 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -11,9 +11,10 @@ Each case reads: given / when / then · the failure it exposes. Test rules for every group: -- Breaker unit tests live in `internal/sink/circuit_breaker_test.go`, use a key unique to the test - (the registry is process-wide) and `t.Parallel()`. No test sleeps for the production 30 s timeout; - see `design.md` *Testing approach*. +- Breaker unit tests live in `internal/sink/circuit_breaker_test.go` and use `t.Parallel()` with a + key unique per test invocation (`-count` safe). No parallel test calls `ResetBreakersForTest`. The + only timeout seam is the `newBreaker(key, timeout)` constructor; no test sleeps the production 30 s + timeout and none writes a package variable. See `design.md` *Testing approach*. - Each group records `go test -race -count=3 ./internal/sink/ ./internal/controller/` (G2 adds `-skip TestControllers` six times, see 2.1) and `task lint`. - Mutations are judged by the test command's exit status, with one no-op control that must survive; @@ -24,10 +25,13 @@ Test rules for every group: - [ ] 0.1 Verify the load-bearing claims in `proposal.md` and `design.md` on `origin/main` and record each with file:line or command output in `evidence/T0.1.md`: the breaker settings and key; that gobreaker v2.4.0 `IsExcluded` neither counts nor resets `ConsecutiveFailures` and frees the - half-open slot; that ownership rejections reach the breaker already Terminal; that Git auth - failures are classified only after the breaker; that RACE-COUNT-BREAKER-ISOLATION reproduces - (`go test -race -count=3 -skip TestControllers ./internal/controller/`, six runs, failures counted, - failing test names). A claim that does not hold stops the loop with a decision request. + half-open slot; the three zones in `design.md` *Context* — for each Terminal site cited there, is + it before, at or after the breaker; that Git auth failures are classified only after the breaker; + every caller of `ResetBreakersForTest` and whether its test is parallel; RACE-COUNT-BREAKER-ISOLATION + on base (`go test -race -count=3 -skip TestControllers ./internal/controller/`, ten runs, exit + status each, failing test names). A flake that does not reproduce in ten runs is recorded as "not + reproduced", not as "mechanism disproved". A claim that does not hold stops the loop with a + decision request. ## 1. Breaker excludes Terminal errors (SCB-1, SCB-2, SCB-3) @@ -48,28 +52,46 @@ Test rules for every group: error is Transient with "circuit breaker open" · guards against over-excluding. - SCB-2b (*control*): same with an unclassified `errors.New` error · guards against excluding everything that is not Transient. - - SCB-3a: given a breaker opened by 5 Transient failures and its timeout elapsed, when the trial - returns Terminal and the next call succeeds, then the next call's function runs and a following - call also runs (closed) · exposes an excluded trial closing the breaker or wedging the slot. - - SCB-3b: as 3a, but the call after the Terminal trial fails Transient, then the call after that - does not run (re-opened) · exposes an excluded trial closing the breaker. -- [ ] 1.2 Set `IsExcluded: kollecterrors.IsTerminal` in `breakerFor`, plus the test seam for the - open timeout that `design.md` allows. Mutations: drop `IsExcluded`; replace it with - `IsSuccessful`; exclude every non-nil error. Each must be killed by a named 1.1 case. + - SCB-2c (*control*): same with `kollecterrors.Forbidden(...)` · guards against excluding Forbidden. + - SCB-2d (*control*): same with a raw error whose text a later classifier would call a Git auth + failure (e.g. `errors.New("authentication required")`) · guards against classifying at the + breaker with the after-breaker classifier. + - SCB-1d: an unwrapped `apierrors.NewNotFound(...)` returned 10 times leaves the breaker closed (an + 11th succeeding call runs) · documents that `ClassOf`-inferred Terminal is excluded (red on base). + - SCB-3a: given a short-timeout breaker (constructor seam) opened by 5 Transient failures and its + timeout elapsed, when the trial returns Terminal, then `State()` is half-open; when the next call + succeeds, its function ran and `State()` is closed · exposes an excluded trial closing the + breaker (state after the trial) or wedging the slot (next call not run). + - SCB-3b: as 3a, but the call after the Terminal trial fails Transient, then `State()` is open and + the following call does not run · exposes an excluded trial closing the breaker. + - SCB-3c: half-open breaker; a trial blocks on a channel; a second concurrent call does not run its + function and returns an error (gobreaker `ErrTooManyRequests`); release the trial with a Terminal + result; a third call runs · exposes a Terminal trial wedging the slot. *Control* for the + concurrency half on base. +- [ ] 1.2 Add `newBreaker(key, timeout)` and set `IsExcluded: kollecterrors.IsTerminal` in it; + `breakerFor` passes `circuitBreakerTimeout`. Make `TestResetBreakersForTest_clearsOpenBreaker` + non-parallel and fix any other parallel caller of `ResetBreakersForTest` found in 0.1. Mutations: + drop `IsExcluded`; replace it with `IsSuccessful`; exclude every non-nil error; exclude + `!IsTransient` (would exclude Forbidden). Each must be killed by a named 1.1 case. ## 2. Reconciler tests stop sharing a tripped breaker (RACE-COUNT-BREAKER-ISOLATION) - [ ] 2.1 On the G1 commit run `go test -race -count=3 -skip TestControllers ./internal/controller/` - six times and record each exit status. All six green → record it and close 2.2 as not needed. Any - red → name the failing tests and the breaker key they share; that red is the group's red. -- [ ] 2.2 Only if 2.1 is red: isolate the failing tests from each other's breaker state (distinct - sink names per test, or `t.Cleanup(sink.ResetBreakersForTest)` where the test is not parallel), - never a retry. Six green runs of the 2.1 command afterwards. + ten times and record each exit status, against T0.1's base count. Any red → name the failing tests + and the breaker key they share; that red is the group's red. All ten green is weak evidence (the + Transient-failure test still shares `default/postgres-demo`), so 2.2 runs either way. +- [ ] 2.2 Isolate the reconciler tests built on `newExportFailureReconciler` and the other tests that + share the `default/postgres-demo` key from each other's breaker state, without changing what they + assert (a per-invocation sink name where the assertion does not name the sink; otherwise a + non-parallel test with `t.Cleanup(sink.ResetBreakersForTest)`), never a retry. Ten green runs of + the 2.1 command afterwards. ## 3. Docs -- [ ] 3.1 `docs/operator-manual/troubleshooting.md:214`: the breaker counts Transient and unclassified - export errors, not Terminal ones; a Terminal failure stays in that inventory's status. +- [ ] 3.1 `docs/operator-manual/troubleshooting.md:214`: the breaker counts retryable export errors + (Transient, Forbidden, unclassified), not Terminal ones; a Terminal failure stays in that + inventory's status. Replace "self-closes after timeout": after 30 s one trial export is admitted, + success closes it, a counted failure re-opens it, a Terminal result leaves it half-open. `docs/adr/0602-error-taxonomy.md:38`: add that Terminal errors are excluded (dated note, ADR not re-decided). `task lint:markdown` 0 issues; `rg -n 'circuit breaker' docs` reviewed for any other claim the change contradicts, the list recorded in `evidence/G3.md`. From 34b39b11aa1f54e9f08604cfa61cb4c4a3b00f92 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 11:25:11 +0100 Subject: [PATCH 03/13] :page_facing_up: docs(spec): exclude only inventory-scoped results from the breaker Spec-set review round 2 showed that Git auth and protected-branch rejections reach the breaker already Terminal and that Terminal exports repeat on every event, so excluding all Terminal results would drop load shedding for sink-wide failures. Re-decided: mark ownership rejections as inventory-scoped and exclude only those. --- .../sink-breaker-terminal-isolation/design.md | 111 +++++++++----- .../sink-breaker-terminal-isolation/loop.md | 14 +- .../proposal.md | 108 ++++++------- .../R-spec-set/round-2/brief-reviewer-a.md | 28 ++++ .../R-spec-set/round-2/leg-reviewer-a-spec.md | 33 ++++ .../R-spec-set/round-2/leg-reviewer-b-spec.md | 34 ++++ .../R-spec-set/round-2/verification.md | 20 +++ .../specs/sink-circuit-breaker/spec.md | 71 +++++---- .../sink-breaker-terminal-isolation/tasks.md | 145 ++++++++++-------- 9 files changed, 368 insertions(+), 196 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/leg-reviewer-a-spec.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/leg-reviewer-b-spec.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/verification.md diff --git a/openspec/changes/sink-breaker-terminal-isolation/design.md b/openspec/changes/sink-breaker-terminal-isolation/design.md index 5b16f5a8..c0608240 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/design.md +++ b/openspec/changes/sink-breaker-terminal-isolation/design.md @@ -13,62 +13,95 @@ Three zones (`internal/sink/export.go`): - **Before the breaker** (`:79-190`): registry/spec checks, inventory identity (`:86`), backend acquire incl. construction and CA/TLS config (`:92-110`), payload decode, size and layout. These return early and never touch the breaker. Unchanged. -- **At the breaker** (`:200`): whatever `plan.run` returns. The class is - `kollecterrors.ClassOf(err)` (`internal/errors/errors.go:86-108`): an explicit `ClassError`, else a - Kubernetes API status (NotFound/Invalid/BadRequest → Terminal, Forbidden → Forbidden), else - Transient. Terminal results here include the Git ownership/prune rejection - (`git/prune_owned.go:255`) and backend-classified write errors (`bigquery/errors.go`, - `mongodb/errors.go`). -- **After the breaker** (`:207-212`): `git.ClassifyExportError` and `classifyExportFailure`. Errors - that become Terminal only here (e.g. Git auth failures, `git/errors.go:35-36`) were not Terminal at - the breaker and keep counting. +- **At the breaker** (`:200`): whatever `plan.run` returns. Backends classify inside the plan: the + Git backend returns `ClassifyExportError(...)` (`git/export.go:128-149`), so Git auth (401/403), + protected-branch and pre-receive rejections arrive Terminal (`git/errors.go:35-42`); BigQuery and + MongoDB classify some write errors Terminal (`bigquery/errors.go`, `mongodb/errors.go`). The + ownership/prune rejection arrives Terminal through `invalidPrune` (`git/prune_owned.go:254-256`, + 26 call sites). +- **After the breaker** (`:207-212`): `git.ClassifyExportError` again and `classifyExportFailure`, + for errors the plan returned unclassified. + +Inventories reconcile on collected-resource events, and a failed export is retried on the next +reconcile, so a sink-wide failure repeats across every inventory on the sink regardless of class. ## Option chosen -A: `Settings.IsExcluded = kollecterrors.IsTerminal`. Rejected: B (tag only inventory-scoped Terminal -errors with a new marker; a forgotten tag fails back to today's defect and protects nothing extra), -C (breaker per sink and inventory; a dead backend is then hit 5×N times before any back-off and the -registry needs eviction). Decision record: `agent-context/decisions.md` 2026-11-05. +B: mark inventory-scoped Terminal results and exclude only those. Rejected: A (exclude every Terminal; +drops load-shedding for sink-wide Terminal failures such as a revoked Git token, which repeat on every +event), C (breaker per sink and inventory; a dead backend is hit 5×N times before any back-off and the +registry needs eviction). A was chosen first on a false premise and reversed; decision record +`agent-context/decisions.md` 2026-11-05 `sink-breaker-terminal`. + +## Marker + +In `internal/errors`: + +- `var ErrInventoryScoped = errors.New("inventory-scoped")`. +- `func InventoryScoped(err error) error`: nil for nil; otherwise an error that is Terminal + (`ClassOf` = Terminal), satisfies `errors.Is(_, ErrInventoryScoped)`, keeps `errors.Is`/`errors.As` + to everything `err` wrapped, and has exactly `err.Error()` as its text (status messages and logs do + not change). +- `func IsInventoryScoped(err error) bool { return errors.Is(err, ErrInventoryScoped) }`. + +At the raise sites: T0.1 classifies the 26 `invalidPrune` calls and every other Terminal raised inside +an export plan as inventory-scoped (about the inventory's own paths or its ownership claim) or +sink-wide/shared (e.g. corrupt or unreadable prune metadata, repository state). G1 routes the +inventory-scoped ones through a new helper (e.g. `ownershipConflict(format, args...)` returning +`kollecterrors.InventoryScoped(...)`) and leaves the rest on `invalidPrune`. A site whose class T0.1 +cannot settle stays unmarked (falls back to today's behaviour) and is listed in the evidence. + +Breaker: `IsExcluded: kollecterrors.IsInventoryScoped`. ## Behaviour (gobreaker v2.4.0, read in `gobreaker.go:237-302`, `counter.go:16-41`) -- **Closed.** An excluded error increments `TotalExclusions` only. `ConsecutiveFailures` is neither - incremented nor reset, so T, X, T, T, T, T (T Transient, X Terminal) trips on the fifth T. A success - still resets the streak. +- **Closed.** An excluded result increments `TotalExclusions` only. `ConsecutiveFailures` is neither + incremented nor reset, so T, X, T, T, T, T (T Transient, X marked) trips on the fifth T. A success + still resets the streak. The closed-state counts reset when the 60 s `Interval` expires + (`gobreaker.go:304-309`), independent of exclusions; unchanged. - **Open.** Unchanged: requests fail fast with `ErrOpenState`, mapped to a Transient "sink circuit - breaker open" error (`circuit_breaker.go:39-41`). The export function does not run, so no - classification happens. + breaker open" error (`circuit_breaker.go:39-41`). The export function does not run. - **Half-open** (`MaxRequests: 1`). `beforeRequest` admits a request while `Requests - - TotalExclusions < MaxRequests`. An excluded trial result leaves the state half-open and frees the - slot; the next request is admitted as the trial. A success closes the breaker; a counted failure - re-opens it. + TotalExclusions < MaxRequests`. An excluded trial leaves the state half-open and frees the slot; the + next request is admitted as the trial. A success closes the breaker; a counted failure re-opens it. - `ErrTooManyRequests` (a second concurrent request in half-open) is returned unwrapped from - `exportThroughBreaker` and becomes Transient in `classifyExportFailure`. Unchanged; not in scope. + `exportThroughBreaker` and becomes Transient in `classifyExportFailure`. Unchanged. ## Testing approach -Unit tests drive `exportThroughBreaker` directly and one boundary test drives `RunExportEnvelope` -with a stub backend, as `TestRunExportEnvelope_circuitBreakerTripsAfterRepeatedFailures` does -(`circuit_breaker_test.go:26`). +Unit tests drive `exportThroughBreaker` directly; boundary tests drive `RunExportEnvelope` with a stub +backend (as `TestRunExportEnvelope_circuitBreakerTripsAfterRepeatedFailures`, +`circuit_breaker_test.go:26`; the existing `stubBackend` returns one error for every path, so SCB-1b +needs a stub that branches on the object path), and one test drives a real Git ownership rejection +through `RunExportEnvelope` with the local-repository fixtures the Git ownership tests already use +(`internal/sink/inventory_identity_test.go`, `internal/sink/git/*owned*_test.go`; T0.1 names the +fixture). Isolation. The registry is process-wide and `ResetBreakersForTest` clears every key (`circuit_breaker.go:68-70`). Rules: (1) a key is unique per test *invocation*, not per test name, because `-count=N` reruns a test against the breakers its earlier run left behind (e.g. `t.Name()` -plus a package-level atomic counter); (2) no `t.Parallel()` test calls `ResetBreakersForTest`, so -the existing `TestResetBreakersForTest_clearsOpenBreaker` (`circuit_breaker_test.go:95-96`, parallel, -resets at `:155`) becomes non-parallel in G1. The other callers (`release_on_delete_test.go`, -`inventory_identity_test.go`, `git/owned_prune_fuzz_test.go`) are checked in G1 for the same rule and -listed in the evidence. - -Timeout seam. Half-open cases must not sleep 30 s. The only allowed seam is a constructor: -`newBreaker(key string, timeout time.Duration)`, which `breakerFor` calls with -`circuitBreakerTimeout`; a test builds a short-timeout breaker and stores it under its own unique key -with `breakerRegistry.LoadOrStore` before the first call. No mutable package variable (a write from -one parallel test is a data race with `breakerFor` in another and would leak into their breakers). -State assertions read `breakerFor(key).State()`. +plus a package-level atomic counter); (2) no `t.Parallel()` test calls `ResetBreakersForTest`; the +existing `TestResetBreakersForTest_clearsOpenBreaker` (`circuit_breaker_test.go:95-96`, parallel, +resets at `:155`) becomes non-parallel in G1; the other callers are non-parallel (round-2 review, +verified again in T0.1); (3) a non-parallel test that depends on a clean breaker resets at its start +and in `t.Cleanup`. + +Constructor seam. `newBreaker(key string, timeout time.Duration) *gobreaker.CircuitBreaker[struct{}]` +holds all settings; `breakerFor` calls it with `circuitBreakerTimeout`. G1 extracts it first as a pure +refactor (settings unchanged), so the SCB-3 tests compile on a base that behaves like `origin/main` +and fail on their target assertion. A test builds a short-timeout breaker and stores it under its own +key with `breakerRegistry.LoadOrStore` before the first call. No mutable package variable. + +Timing protocol for half-open tests: timeout 1 s; sleep 1.1 s once to reach half-open; every later +assertion runs without sleeping, well inside the next 1 s window, so a wrongly re-opened breaker +cannot drift back to half-open before `State()` is read, and a correctly re-opened one cannot either. +State is read with `breakerFor(key).State()` immediately after each call. ## Risks -- Mislabelled Terminal errors hide an outage from the breaker (see proposal *Assumptions*). -- A test that relies on Terminal errors tripping the breaker would now fail; none is known +- The marker is lost by a re-wrap without `Unwrap` between the raise site and the breaker: covered by + SCB-1e through the real Git backend. +- A shared-state rejection marked as inventory-scoped would stop shedding load for a sink-wide fault: + T0.1's per-site classification, reviewed in G1. +- A test that relies on ownership rejections tripping the breaker would now fail; none is known (`rg -n 'circuit breaker open|ErrOpenState' internal` at G1 records the search). diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index 8635d048..d4b728ff 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -1,19 +1,19 @@ # Loop — sink-breaker-terminal-isolation (spec-loop-duo run state) Repo: `/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation` · Branch: `fix/sink-breaker-terminal-isolation` · Base: `origin/main` @ `96c41b0d0` · Started: 2026-11-05 · Reviewers: two-model duo -Next: R spec-set review round 2 +Next: L — dispatch T0.1 (probe) Task prompt: local, `agent-context/loops/sink-breaker-terminal-isolation/task-prompt.md` (harness, not committed) · decisions by model alignment, logged in agent-context/decisions.md -Budget: paid review legs 1/12 (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 +Budget: paid review legs 2/12 (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 Backlog SINK-BREAKER-TENANT-01 (P2) and RACE-COUNT-BREAKER-ISOLATION (P3). Spec set: proposal.md, design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `sink-breaker-terminal` -(both legs A, pending operator validation). +(first run A; revised to B after R2 showed A's premise false, both legs B; pending operator validation). ## Stages - [x] 0 orient — new change; file:line references measured against `origin/main` @ `96c41b0d0` (2026-11-05). Open PRs touching these files: none (#523 mongo-driver v2 touches `internal/sink/mongodb/` only; #520 helm v4 CI only) - [x] P plan + tasks — written; `task spec:validate` green (22 items); spec set committed: (this commit) -- [ ] R spec-set review — round 1 on `a505029da`: reviewer-a REQUEST_CHANGES (1 CRITICAL), reviewer-b CONCERNS; all 8 findings verified and fixed (`reviews/R-spec-set/verification.md`); CRITICAL had a mechanical fix both legs named → round 2 (last) on the fix commit +- [x] R spec-set review — round 1 on `a505029da`: reviewer-a REQUEST_CHANGES (1 CRITICAL), reviewer-b CONCERNS; all 8 findings verified and fixed (`reviews/R-spec-set/verification.md`); CRITICAL had a mechanical fix both legs named → round 2 (last) on `de1e7df12`: reviewer-a REQUEST_CHANGES, reviewer-b BLOCK; new CRITICAL (Git sink-wide Terminal reaches the breaker; A's premise false) settled by decision rerun → B (unanimous), spec set rewritten for B; 7 WARNING/NOTE fixed (`reviews/R-spec-set/round-2/verification.md`). No third round (limit); premises re-verified in T0.1, rewritten set reviewed again at B - [ ] L task loop — see *Tasks* - [ ] B branch review — `reviews/B-branch/round-N/` - [ ] Hand-off — `pr-description.md` @@ -42,6 +42,12 @@ Holistic run at B: not yet run. | R1 | SCB-3a proves no state; no concurrent trial | WARNING | reviewer-a | fix | SCB-3a/3b `State()`, SCB-3c | no | | R1 | Flake mechanism overstated | WARNING | reviewer-a, reviewer-b | fix | proposal, 2.1 ten runs, 2.2 always | no | | R1 | Docs "self-closes after timeout"; Terminal trial keeps half-open | NOTE | reviewer-a, reviewer-b | fix | 3.1 wording; SCB-3 accepted consequence | no | +| R2 | Git auth/protected-branch Terminal reaches the breaker; Terminal exports repeat on events; A drops load-shedding | CRITICAL | reviewer-a, reviewer-b | fix (decision) | decide.sh rerun → B; whole spec set rewritten | validation pending | +| R2 | SCB-2d does not guard the classification boundary | WARNING | reviewer-a | fix | SCB-2d via `RunExportEnvelope`; SCB-1e real Git | no | +| R2 | Counting-interval reset omitted | WARNING | reviewer-a | fix | SCB-2, design | no | +| R2 | Half-open timing flakes | WARNING | reviewer-a, reviewer-b | fix | design timing protocol | no | +| R2 | SCB-3 reds not on target assertion | WARNING | reviewer-b | fix | 1.1 refactor base | no | +| R2 | G2 red rule; reset only in cleanup; path-branching stub | NOTE | reviewer-a, reviewer-b | fix | tasks preamble, 2.1, 2.2, design | no | ## Tasks diff --git a/openspec/changes/sink-breaker-terminal-isolation/proposal.md b/openspec/changes/sink-breaker-terminal-isolation/proposal.md index faea8112..879887da 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/proposal.md +++ b/openspec/changes/sink-breaker-terminal-isolation/proposal.md @@ -4,52 +4,55 @@ Every export to a sink runs through one process-wide circuit breaker keyed `/` (`internal/sink/export.go:200`, `internal/sink/circuit_breaker.go:26-62`, -sony/gobreaker v2.4.0). It trips after 5 consecutive failures (`circuitBreakerTripAt`), stays open -30 s (`circuitBreakerTimeout`), and while open every export to that sink, from every inventory, fails -fast with a Transient "sink circuit breaker open" error. - -The breaker counts **every** non-nil error from the export as a failure, Terminal ones included -(gobreaker's default `IsSuccessful` returns false for any non-nil error; `Settings` sets neither -`IsSuccessful` nor `IsExcluded`). Terminal errors are raised inside the export for problems that -belong to one inventory, most visibly the Git ownership/prune rejection -(`internal/sink/git/prune_owned.go:255`), raised from inside the export plan. (Identity, payload, -layout and backend-acquire errors return before the breaker, `internal/sink/export.go:86-190`, and -never reach it; only errors returned by `plan.run` at `export.go:200` do.) Five such -rejections from one misconfigured inventory open the breaker for every healthy inventory on the same -sink for 30 s (backlog SINK-BREAKER-TENANT-01, found by the #412 review 2026-10-04). - -This contradicts the documented design: ADR-0602 says the breaker is "per sink/doc backend for -`ErrTransient` storms" (`docs/adr/0602-error-taxonomy.md:38`), and the troubleshooting page says it -opens after "5 consecutive transient failures per sink key" -(`docs/operator-manual/troubleshooting.md:214`). +sony/gobreaker v2.4.0). It trips after 5 consecutive failures (`circuitBreakerTripAt`) within its +60 s closed-state counting interval, stays open 30 s (`circuitBreakerTimeout`), and while open every +export to that sink, from every inventory, fails fast with a Transient "sink circuit breaker open" +error. + +The breaker counts **every** non-nil result of the export plan as a failure (gobreaker's default +`IsSuccessful`; `Settings` sets neither `IsSuccessful` nor `IsExcluded`). Some of those results are +about **one inventory**: the Git ownership/prune rejection (`internal/sink/git/prune_owned.go:254-256`, +`invalidPrune`), raised inside the export plan. Five such rejections from one misconfigured inventory +open the breaker for every healthy inventory on the same sink for 30 s (backlog +SINK-BREAKER-TENANT-01, found by the #412 review 2026-10-04). Identity, payload, layout and +backend-acquire errors are inventory-scoped too but return before the breaker +(`internal/sink/export.go:86-190`) and never reach it. + +Other Terminal results are **sink-wide** and must keep counting. The Git backend classifies its own +errors before returning to the breaker (`internal/sink/git/export.go:128-149` → +`ClassifyExportError`, `internal/sink/git/errors.go:35-42`), so a revoked token (401/403), a protected +branch or a pre-receive rejection reaches the breaker already Terminal; BigQuery and MongoDB classify +some write errors Terminal the same way. Inventories are event-driven: a collected-resource change +triggers a reconcile and a failed export is attempted again on the next one, so a sink-wide Terminal +failure repeats across every inventory on the sink. Today the breaker sheds that load (fail fast for +30 s after 5); it should keep doing so. A related flake: `go test -race -count=3 -skip TestControllers ./internal/controller/` fails in four reconciler tests (backlog RACE-COUNT-BREAKER-ISOLATION, P3; 4 of 6 base runs failed 2026-10, no DATA RACE, not yet re-measured). Those tests share the `default/postgres-demo` breaker through `newExportFailureReconciler` (`internal/controller/kollectinventory_export_test.go:339-375`): the Terminal-failure test, the Transient-failure test and successful exports all record against one key, -so whether it opens depends on the mix and order of outcomes. Excluding Terminal results removes one -contributor; the Transient test still contributes, so this change re-measures the flake and isolates -the tests if it persists (tasks group 2). - -A Terminal error is never retried, so counting it protects no backend; it only widens one -inventory's failure to the whole sink. +so whether it opens depends on the mix and order of outcomes. The breaker change does not remove that +sharing (the Terminal there is not inventory-scoped), so this change isolates those tests. ## What Changes -- The sink circuit breaker **excludes** errors classified Terminal (`kollecterrors.IsTerminal`) at - the breaker boundary: they count neither as failures nor as successes (gobreaker `IsExcluded`). - Transient and unclassified errors keep counting, so the breaker still trips after 5 consecutive - retryable failures. -- An excluded error during half-open neither closes nor re-opens the breaker and releases the trial - slot, so the next export is admitted as the trial. -- The four reconciler tests named in RACE-COUNT-BREAKER-ISOLATION pass under `-race -count=3`; - where a test still shares a breaker key through Transient failures, it is isolated (its own sink - key or a breaker reset), not retried. +- A new error marker, `kollecterrors.ErrInventoryScoped`, says "this Terminal failure is about one + inventory, not the sink". It keeps the Terminal class and the error text. +- The ownership rejections inside the Git export plan carry the marker; T0.1 lists every Terminal + raised inside the export plan and which of them are inventory-scoped. +- The sink circuit breaker **excludes** marked results (gobreaker `IsExcluded`): they count neither as + failures nor as successes. Every other non-nil result keeps counting, sink-wide Terminal included. +- An excluded result during half-open neither closes nor re-opens the breaker and releases the trial + slot. +- The reconciler tests that share `default/postgres-demo` are isolated from each other's breaker + state, not retried. - Docs say which errors count. -Decision: option A of three (exclude all Terminal / tag inventory-scoped Terminal only / key per -inventory), chosen by two independent legs, both A (`agent-context/decisions.md`, 2026-11-05 +Decision: option B of three (exclude all Terminal / mark inventory-scoped Terminal only / key per +inventory). A first run chose A on the premise that Terminal exports never repeat; the spec-set review +showed that premise false (event-driven re-export, Git sink-wide Terminal at the breaker); the +corrected brief was re-run and both legs chose B (`agent-context/decisions.md`, 2026-11-05 `sink-breaker-terminal`, pending operator validation). ## Capabilities @@ -65,31 +68,30 @@ None. ## Impact -- Code: `internal/sink/circuit_breaker.go` (breaker settings). No change to `export.go`'s - classification after the breaker. -- Tests: `internal/sink/circuit_breaker_test.go` (new cases), `internal/controller` tests named in - RACE-COUNT-BREAKER-ISOLATION (isolation only if the fix alone does not make them pass). +- Code: `internal/errors/errors.go` (marker, constructor, predicate), `internal/sink/git/prune_owned.go` + (marked ownership rejections), `internal/sink/circuit_breaker.go` (`newBreaker`, `IsExcluded`). +- Tests: `internal/errors`, `internal/sink/circuit_breaker_test.go`, a Git-backend boundary test, and + the `internal/controller` tests named in RACE-COUNT-BREAKER-ISOLATION. - Docs: `docs/operator-manual/troubleshooting.md:214`, `docs/adr/0602-error-taxonomy.md:38` (clarifying note, not a new ADR). -- Behaviour visible to users: a misconfigured inventory no longer opens the breaker for other - inventories on its sink. A sink-wide Terminal error returned during the export (e.g. a BigQuery - authorization failure classified Terminal in `internal/sink/bigquery/errors.go`) no longer opens - the breaker either; it was never retried, and it stays visible in each inventory's status. Export - error metrics are recorded after the breaker (`export.go:203-212`) and are unchanged. A Terminal - result on the half-open trial leaves the breaker half-open, so one misconfigured inventory can keep - a recovering sink admitting one trial at a time until another export's trial succeeds or fails - retryably; bounded and stated in SCB-3. -- Release note: v0.22.x / next minor, under fixes. +- Behaviour visible to users: one inventory's ownership conflict no longer opens the breaker for the + other inventories on its sink. Sink-wide failures, Terminal or not, still open it. Export error + metrics are recorded after the breaker (`export.go:203-212`) and are unchanged. A marked result on + the half-open trial leaves the breaker half-open; the sink admits one export at a time until a trial + succeeds or fails with a counted error (stated in SCB-3). +- Release note: next release, under fixes. ## Non-goals - Changing the breaker key, trip threshold, interval or timeout. -- Reclassifying any error (e.g. making Git auth failures Terminal before the breaker). +- Reclassifying any error between Transient and Terminal. - A breaker-state metric or condition. ## Assumptions -- Terminal classification at the breaker boundary is correct for the backends: an error labelled - Terminal is not a symptom of an outage. A backend that mislabels an outage as Terminal makes the - breaker blind to it; those exports are still not requeued, so the load on the backend stays bounded - (main risk named by both decision legs). +- The marker reaches the breaker: no layer between `invalidPrune` and `exportThroughBreaker` re-wraps + the error without `Unwrap` (`ClassError` unwraps, `internal/errors/errors.go:42-48`). The boundary + test in 1.1 (SCB-1e) checks this through the real Git backend, not only at the raise site (main + risk named by a decision leg). +- A rejection is marked only when the condition is about the inventory's own paths; a rejection caused + by shared state in the sink (e.g. corrupt prune metadata) stays unmarked so it keeps counting. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/brief-reviewer-a.md new file mode 100644 index 00000000..6fda32fb --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/brief-reviewer-a.md @@ -0,0 +1,28 @@ +# Cold review: spec set `sink-breaker-terminal-isolation` (lens: spec) + +You are an independent reviewer. You have read-only access to this repository (Go Kubernetes +operator "kollect"). Review the OpenSpec change at +`openspec/changes/sink-breaker-terminal-isolation/`: `proposal.md`, `design.md`, +`specs/sink-circuit-breaker/spec.md`, `tasks.md` (commit `de1e7df12`; round 2 — round 1 findings and their dispositions are in `reviews/R-spec-set/verification.md`, check that each fix holds and look for new defects the fixes introduced). Ignore `loop.md` except as context. + +Question: **would executing these tasks as written deliver the spec, safely?** + +Check, against the actual code (read it; do not trust the spec's file:line claims): + +1. Every factual claim in proposal.md/design.md: breaker settings and key + (`internal/sink/circuit_breaker.go`, `internal/sink/export.go`), which errors reach the breaker + already Terminal vs classified after it, gobreaker v2.4.0 `IsExcluded` semantics in closed and + half-open state (module source under the Go module cache, `github.com/sony/gobreaker/v2@v2.4.0`), + and the RACE-COUNT-BREAKER-ISOLATION root cause (`internal/controller/*_test.go`). +2. Requirements SCB-1..3: are they testable, unambiguous, and complete? Missing scenarios (e.g. + concurrency in half-open, `ErrTooManyRequests`, Forbidden-class errors, a Terminal error that is + in fact sink-wide)? +3. Tasks: missing, misordered or unverifiable tasks; a test case that cannot be red on base for its + stated reason; a case whose stated failure it would not actually expose; mutations a case would + not kill; a test seam for the 30 s timeout that could change production behaviour or race. +4. Anything the change would break elsewhere (other tests or docs relying on Terminal errors + tripping the breaker; metrics; docs claims). + +Output: findings as a list, each with severity CRITICAL / WARNING / NOTE, file:line evidence, and a +concrete fix. CRITICAL only for something that makes the tasks fail to deliver the spec or ship a +defect. End with `## Verdict` and one of APPROVE / CONCERNS / REQUEST_CHANGES / BLOCK. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/leg-reviewer-a-spec.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/leg-reviewer-a-spec.md new file mode 100644 index 00000000..ea1da58f --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/leg-reviewer-a-spec.md @@ -0,0 +1,33 @@ +Reviewed commit `de1e7df12` against repository code and cached gobreaker v2.4.0 source. Round 1’s reset isolation, immutable timeout constructor, Forbidden/inferred-Terminal coverage, and half-open concurrency fixes are sound in principle. The classification-boundary fix remains factually incorrect. + +1. **CRITICAL — Git authentication failures already reach the breaker as Terminal.** + Evidence: `design.md:22–24` says they become Terminal only afterwards; `tasks.md:29–34` requires verifying that claim and stopping if false. But `internal/sink/git/export.go:128–149` calls `ClassifyExportError` before returning from the backend. `git/backend.go:65,77` and `layout_export.go:151,217` connect those returns to the export plan inside the breaker. Authentication failures become Terminal at `git/errors.go:35–36`. + + Executing task 0.1 correctly therefore stops the plan. Proceeding regardless would ship a broader behavior change than documented: Git authentication and protected-branch failures also stop tripping the breaker. + **Fix:** correct the classification inventory, proposal non-goals, and impact statement. Explicitly accept exclusion of these sink-wide Git failures and add coverage using the actual Git classification path. Retain the raw-error scenario as a separate synthetic boundary case. + +2. **WARNING — SCB-2d does not protect the real classification boundary.** + Evidence: `design.md:50–52` specifies direct breaker tests except for SCB-1b; `tasks.md:56–58` claims SCB-2d guards against moving classification before the breaker. A mutation changing `export.go:201` to return `git.ClassifyExportError(plan.run(exportCtx))` passes that direct test. SCB-1b also passes because its backend already returns Terminal. + + **Fix:** run SCB-2d through `RunExportEnvelope` with a backend returning raw `"authentication required"`. Assert five backend invocations return Terminal to callers, while the sixth is rejected as breaker-open. Require this test to kill the classification-movement mutation. + +3. **WARNING — The normative trip requirement omits the existing interval reset.** + Evidence: `specs/sink-circuit-breaker/spec.md:39–41` unconditionally requires opening after five consecutive counted failures. Production sets `Interval: 60s` at `internal/sink/circuit_breaker.go:54`; gobreaker’s `gobreaker.go:304–309,348–359` clears the closed-state generation when that interval expires. Five consecutive failures spanning that reset need not open it. + + **Fix:** qualify the requirement and streak scenarios with “within the same closed-state counting interval,” and state that interval expiration resets counts independently of exclusions. Preserve the existing 60-second behavior. + +4. **WARNING — Short-timeout state assertions can conceal reopening or introduce flakes.** + Evidence: `tasks.md:61–70` observes `State()` and immediate subsequent admission using an unspecified short timeout. Gobreaker’s `State()` advances time-dependent state (`gobreaker.go:195–201`); an incorrectly reopened breaker becomes half-open again after timeout (`:312–315`). Thus SCB-3a/3c can miss reopening if assertions run late, while SCB-3b can fail correctly implemented code after its second timeout expires. + + **Fix:** specify a positive timeout and bounded timing protocol, and capture state transitions through a per-instance callback configured at construction. Assert that a Terminal trial causes no transition to open or closed. Keep callback configuration immutable and avoid global seams. + +5. **NOTE — G2’s evidence rules contradict its intentional unconditional isolation.** + Evidence: `tasks.md:3–8` requires each group’s failing target assertion, but `:79–86` permits ten green G1 runs and still mandates G2. The proposal also still describes isolation as conditional (`proposal.md:70–71`). + + **Fix:** explicitly allow G2 to record “no reproduction; preventive isolation,” without manufacturing a red. Align the proposal with unconditional isolation and correct “six times” at `tasks.md:19` to ten. + +Runtime tests were not run: the configured Go launcher attempted to install Go 1.27.3 and failed under the read-only filesystem. + +## Verdict + +REQUEST_CHANGES diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/leg-reviewer-b-spec.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/leg-reviewer-b-spec.md new file mode 100644 index 00000000..a5a0a9dc --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/leg-reviewer-b-spec.md @@ -0,0 +1,34 @@ +## Verdict: BLOCK + +## Findings +- [CRITICAL] The design says Git auth failures become Terminal only after the breaker. The code does not support that: the Git backend classifies its own errors inside `plan.run`. — `internal/sink/git/export.go:128-149`, `internal/sink/git/backend.go:65,77` + Failure: `Backend.Export` and `Backend.ExportFiles` both return `ClassifyExportError(...)`. That call turns "authentication required", 401/403, "protected branch" and "pre-receive hook declined" into Terminal before `exportThroughBreaker` returns (`errors.go:35-42`). With `IsExcluded: IsTerminal`, a revoked Git token or a protected branch, both of which fail the whole sink, will never open the breaker. Every inventory then does a full clone and push attempt before failing. Three places describe the opposite: design *Context* (`design.md:22-24`), the SCB-2 wording "errors that become Terminal only after the breaker", and the "authentication failure" example in the SCB-1 scenario. The proposal *Impact* names BigQuery as the only sink-wide Terminal case. Task 0.1 checks exactly this claim and says a claim that fails stops the loop, so as written the loop stops at 0.1. Round 2 is the last spec-set round. + Fix: rewrite the three zones in the design and the proposal *Impact* so they say Git auth and push rejections are Terminal at the breaker and will be excluded. Get an operator decision on whether that is acceptable (option A as chosen) or whether these errors need a marker so they keep counting (option B narrowed). Change the SCB-1 scenario example to one that really is classified only after the breaker, or drop the example. Fix the 3.1 docs line to match. + Confidence: 90 +- [WARNING] SCB-3b can be flaky under `-race -count=3`: gobreaker's `State()` moves from open to half-open on its own once the timeout has passed. — `tasks.md:65-66`, `design.md:63-68` + Failure: with a short constructor timeout (for example 10 ms), a scheduling delay between the trial's Transient failure and the `State()` read or the "following call does not run" check lets the timeout pass. The next call is then admitted as a new trial and runs. The test fails intermittently in the ten-run gate that 2.x relies on. + Fix: say how far the timeout must exceed the assertion window (for example a timeout of 1 s or more, sleeping just past it once), or assert `open` straight after the failing trial and drop the "following call does not run" check. + Confidence: 70 +- [WARNING] On base, the SCB-3a/b/c reds either fail to compile or fail on a nil breaker, not on the target assertion. — `tasks.md:3-6,61-72` + Failure: the SCB-3 cases need `newBreaker(key, timeout)`, which does not exist on `origin/main`, and "red on base" is defined as `origin/main`. The permitted "not implemented (no-op)" stub returns nil, so `Execute` panics. That breaks the stated rule that a red must fail on the target assertion. + Fix: in 1.1, extract `newBreaker` as a pure refactor that keeps the current settings. Record the SCB-3 reds against that refactor, and leave only `IsExcluded` to 1.2. + Confidence: 65 +- [NOTE] With `-count=3`, the reset in 2.2's non-parallel option runs only at cleanup, not at start. — `tasks.md:85-86` + Failure: a serial test in run 2 executes after the parallel tests of run 1. If any parallel test is still on `default/postgres-demo`, the serial test inherits a breaker that run 1 tripped. + Fix: call `sink.ResetBreakersForTest()` at the start as well as in `t.Cleanup`, or require that no parallel test keeps the shared key. + Confidence: 45 +- [NOTE] Round-1 fixes that hold against the code: + - Fix #1 holds. The other callers of `ResetBreakersForTest` are non-parallel: `inventory_identity_test.go`, `release_on_delete_test.go`, the fuzz harness and `circuit_breaker_test.go:167`. Only `:95-96` is parallel, and 1.2 makes it serial. + - Fix #3 holds. `export.go:79-190` returns before the breaker call at `:200`. + - Fix #4 holds. `errors.go:86-108`: `ClassOf(nil)` is Transient, so a success is never excluded. An unwrapped NotFound is Terminal. + - Fix #6 holds. `kollectinventory_export_test.go:339-454`: both tests are parallel and share `default/postgres-demo`. + - Mutation coverage holds. Dropping `IsExcluded` is killed by SCB-1a, `IsSuccessful` by SCB-1c and SCB-3a, exclude-all by SCB-2a, and `!IsTransient` by SCB-2c. + - The SCB-2 open-breaker message "sink circuit breaker open for %q" holds (`circuit_breaker.go:40`). + - SCB-1b needs a new stub that branches on path, because `stubBackend` returns one error for every path (`export_test.go:31-35`). The task implies this but does not say it. + Confidence: 80 + +## Could not check +- gobreaker v2.4.0 source (`gobreaker.go:237-302`, `counter.go`): it is in the Go module cache, outside the repository, so I did not read it. The `IsExcluded` claims (no count, no reset, the half-open slot released through `Requests - TotalExclusions`) are unverified here. SCB-3 depends on them, and so does T0.1. +- I did not read `plan.run` and `resolveSnapshotExport`, so I have not confirmed that every Git layout goes through `Backend.Export` or `Backend.ExportFiles`. +- The other sink backends' Terminal sites (`s3/backend.go`, `mongodb/errors.go`, `bigquery/errors.go`) were not checked for further sink-wide Terminal errors at the breaker. +- I ran no tests and found no recorded sensor results for this round. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/verification.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/verification.md new file mode 100644 index 00000000..f8b40056 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/R-spec-set/round-2/verification.md @@ -0,0 +1,20 @@ +# R round 2 (last) — verification and triage + +Spec set `de1e7df12`. Reviewer A (spec): REQUEST_CHANGES. Reviewer B (spec): BLOCK. Every finding +checked against the code by the orchestrator on 2026-11-05. + +| # | Finding | Sev | Reviewers | Verified | Disposition | +|---|---|---|---|---|---| +| 1 | Git auth, protected-branch and pre-receive rejections reach the breaker already Terminal (`git/export.go:128-149` returns `ClassifyExportError`); option A would stop sink-wide failures from opening the breaker | CRITICAL | A, B | yes; also: inventories re-export on every collected-resource event, so Terminal exports do repeat — the decision brief's premise "Terminal is never retried" was false | fix by decision: brief rewritten with the corrected facts, `decide.sh` re-run, both legs chose B (mark inventory-scoped results only); proposal, design, spec and tasks rewritten for B | +| 2 | SCB-2d at the breaker only does not guard the classification boundary | WARNING | A | yes | fix: SCB-2d through `RunExportEnvelope` with the Git classifier; SCB-1e through the real Git backend | +| 3 | Requirement omits the 60 s counting-interval reset | WARNING | A | yes (`circuit_breaker.go:54`) | fix: SCB-2 text, design *Behaviour* | +| 4 | Half-open assertions timing-sensitive; `State()` advances time-dependent state | WARNING | A, B | yes | fix: design timing protocol (1 s timeout, one 1.1 s sleep, assertions without sleep) | +| 5 | SCB-3 reds on `origin/main` are compile errors or nil-breaker panics | WARNING | B | yes (`newBreaker` absent) | fix: 1.1 starts with a pure `newBreaker` extraction and marker stubs; red base redefined | +| 6 | G2 red rule contradicts preventive isolation; "six times" left over | NOTE | A | yes | fix: tasks preamble and 2.1 allow "not reproduced; preventive"; test rules updated | +| 7 | Serial test with reset only in cleanup inherits a tripped breaker under `-count` | NOTE | B | yes | fix: 2.2 resets at start and in cleanup | +| 8 | SCB-1b needs a path-branching stub | NOTE | B | yes (`export_test.go:31-35`) | fix: design *Testing approach*, 1.1 | + +Round 2 was the last spec-set round. Finding 1 was settled by the decision procedure (fact check, then +a rerun on the corrected brief, unanimous), not by a third review. Its premises are re-verified in +T0.1, which stops the loop if any does not hold, and the rewritten spec set is reviewed again as part +of the branch review at stage B. diff --git a/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md b/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md index f24b2548..a514d53f 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md +++ b/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md @@ -2,44 +2,38 @@ ## ADDED Requirements -### Requirement: SCB-1 Terminal export errors do not count toward the sink breaker +### Requirement: SCB-1 Inventory-scoped export failures do not count toward the sink breaker -The shared per-sink circuit breaker SHALL exclude an export plan result for which -`kollecterrors.ClassOf` is Terminal: it SHALL count neither as a failure nor as a success, and SHALL -NOT reset the consecutive-failure count. Classification is evaluated on the result at the breaker, -not after it. +The shared per-sink circuit breaker SHALL exclude an export plan result that carries the +inventory-scoped marker (`errors.Is(err, kollecterrors.ErrInventoryScoped)`): it SHALL count neither +as a failure nor as a success, and SHALL NOT reset the consecutive-failure count. The marker SHALL +keep the error's Terminal class and its text. -#### Scenario: One inventory's terminal rejections do not open the breaker +#### Scenario: One inventory's ownership rejections do not open the breaker -- **GIVEN** a sink with a closed breaker -- **WHEN** one inventory's export to that sink fails with a Terminal error 10 times in a row +- **GIVEN** a Git sink with a closed breaker +- **WHEN** one inventory's export to that sink is rejected for ownership 10 times in a row - **THEN** the breaker stays closed - **AND** another inventory's next export to the same sink runs against the backend and succeeds -#### Scenario: An inferred Terminal status is excluded - -- **GIVEN** a sink with a closed breaker -- **WHEN** the export plan returns an unwrapped Kubernetes NotFound status 10 times in a row -- **THEN** the breaker stays closed - -#### Scenario: An error classified Terminal only after the breaker still counts +#### Scenario: The marker survives to the breaker -- **GIVEN** a sink with a closed breaker -- **WHEN** the export plan returns 5 times an unclassified error that the post-breaker Git classifier would mark Terminal (an authentication failure) -- **THEN** the sixth export does not call the backend +- **GIVEN** a Git sink whose export plan rejects an inventory for ownership +- **WHEN** the export returns +- **THEN** the error carries the inventory-scoped marker and is Terminal -#### Scenario: An excluded error does not break a streak of retryable failures +#### Scenario: An excluded result does not break a streak of counted failures - **GIVEN** a sink with a closed breaker -- **WHEN** exports fail Transient, Terminal, Transient, Transient, Transient, Transient +- **WHEN** exports fail Transient, inventory-scoped, Transient, Transient, Transient, Transient - **THEN** the breaker opens after the fifth Transient failure -### Requirement: SCB-2 Retryable export errors still open the sink breaker +### Requirement: SCB-2 Every other export failure still counts toward the sink breaker -The breaker SHALL count every other non-nil export result as a failure (Transient, Forbidden, and -errors without a class, including errors that become Terminal only after the breaker) and SHALL open -after 5 consecutive ones. While open, an export to the sink SHALL NOT reach the backend and SHALL fail with a -Transient error naming the open breaker. +The breaker SHALL count every non-nil export plan result without the marker as a failure (Transient, +Forbidden, Terminal, and errors without a class) and SHALL open after 5 consecutive ones within its +closed-state counting interval. While open, an export to the sink SHALL NOT reach the backend and SHALL +fail with a Transient error naming the open breaker. #### Scenario: Five transient failures open the breaker @@ -48,33 +42,38 @@ Transient error naming the open breaker. - **THEN** the sixth export does not call the backend - **AND** it fails with a Transient error whose message contains "circuit breaker open" +#### Scenario: Sink-wide Terminal failures open the breaker + +- **GIVEN** a Git sink with a closed breaker +- **WHEN** 5 consecutive exports fail with an authentication error the Git backend classifies Terminal +- **THEN** the sixth export does not call the backend + #### Scenario: Unclassified and Forbidden errors count - **GIVEN** a sink with a closed breaker - **WHEN** 5 consecutive exports fail with an error that carries no class, or with a Forbidden error - **THEN** the sixth export does not call the backend -### Requirement: SCB-3 A terminal result in half-open neither closes nor wedges the breaker +### Requirement: SCB-3 An inventory-scoped result in half-open neither closes nor wedges the breaker -When the breaker is half-open, an export whose result is a Terminal error SHALL leave the breaker -half-open and SHALL release the trial slot, so that the next export is admitted as the trial. While a -trial is in flight, a concurrent export SHALL NOT reach the backend. A successful trial SHALL close the -breaker; a counted failure on the trial SHALL re-open it. Consequence (accepted): while Terminal -results keep winning the trial, the sink stays half-open and admits one export at a time. +When the breaker is half-open, an export whose result is inventory-scoped SHALL leave it half-open +and SHALL release the trial slot, so the next export is admitted as the trial. While a trial is in +flight, a concurrent export SHALL NOT reach the backend. A successful trial SHALL close the breaker; a +counted failure on the trial SHALL re-open it. -#### Scenario: Terminal trial, then a successful trial +#### Scenario: Inventory-scoped trial, then a successful trial - **GIVEN** a breaker that opened on Transient failures and whose open timeout has elapsed -- **WHEN** the trial export fails with a Terminal error +- **WHEN** the trial export fails inventory-scoped - **THEN** the breaker is half-open - **WHEN** the next export succeeds - **THEN** that export reached the backend - **AND** the breaker is closed afterwards -#### Scenario: Terminal trial, then a failing trial +#### Scenario: Inventory-scoped trial, then a failing trial - **GIVEN** a breaker that opened on Transient failures and whose open timeout has elapsed -- **WHEN** the trial export fails with a Terminal error +- **WHEN** the trial export fails inventory-scoped - **AND** the next export fails with a Transient error - **THEN** the breaker is open afterwards @@ -83,4 +82,4 @@ results keep winning the trial, the sink stays half-open and admits one export a - **GIVEN** a half-open breaker whose trial export is in flight - **WHEN** a second export starts before the trial finishes - **THEN** the second export does not reach the backend -- **AND** after the trial finishes with a Terminal error, the next export is admitted +- **AND** after the trial finishes inventory-scoped, the next export is admitted diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index cad6db80..86236432 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -2,10 +2,13 @@ Red first. Each group's test task comes before its implementation and records its red in `evidence/G.md`: the named test runs (`=== RUN `) and fails on the target assertion, not on -a compile error or a skip. "Red on base" means red on `origin/main` for G1 and on the G1 commit for -G2. A stub with a `not implemented` (no-op) body is in scope so tests compile. Each group closes in -one commit (tests and implementation), so the tree is green at every commit. A case marked *control* -is expected green on its base and is recorded as a control, not a red. +a compile error or a skip. "Red on base" means red on the group's refactor base: for G1, `origin/main` +plus the pure `newBreaker` extraction and the marker API stubs named in 1.1; for G2, the G1 commit. A +stub with the signature from `design.md` and a `not implemented` body (a no-op that returns its input +unmarked) is in scope so tests compile. Each group closes in one commit (tests and implementation), +so the tree is green at every commit. A case marked *control* is expected green on its base and is +recorded as a control, not a red. G2 may close without a red when its target is preventive (see 2.1); +it says so in its evidence. Each case reads: given / when / then · the failure it exposes. @@ -13,10 +16,10 @@ Test rules for every group: - Breaker unit tests live in `internal/sink/circuit_breaker_test.go` and use `t.Parallel()` with a key unique per test invocation (`-count` safe). No parallel test calls `ResetBreakersForTest`. The - only timeout seam is the `newBreaker(key, timeout)` constructor; no test sleeps the production 30 s - timeout and none writes a package variable. See `design.md` *Testing approach*. -- Each group records `go test -race -count=3 ./internal/sink/ ./internal/controller/` (G2 adds - `-skip TestControllers` six times, see 2.1) and `task lint`. + only timeout seam is the `newBreaker(key, timeout)` constructor; half-open tests follow the timing + protocol in `design.md` *Testing approach*; no test writes a package variable. +- Each group records `go test -race -count=3 ./internal/errors/ ./internal/sink/... ./internal/controller/` + and `task lint`. - Mutations are judged by the test command's exit status, with one no-op control that must survive; mutant trees live outside the repository. @@ -24,77 +27,91 @@ Test rules for every group: - [ ] 0.1 Verify the load-bearing claims in `proposal.md` and `design.md` on `origin/main` and record each with file:line or command output in `evidence/T0.1.md`: the breaker settings and key; that - gobreaker v2.4.0 `IsExcluded` neither counts nor resets `ConsecutiveFailures` and frees the - half-open slot; the three zones in `design.md` *Context* — for each Terminal site cited there, is - it before, at or after the breaker; that Git auth failures are classified only after the breaker; - every caller of `ResetBreakersForTest` and whether its test is parallel; RACE-COUNT-BREAKER-ISOLATION - on base (`go test -race -count=3 -skip TestControllers ./internal/controller/`, ten runs, exit - status each, failing test names). A flake that does not reproduce in ten runs is recorded as "not - reproduced", not as "mechanism disproved". A claim that does not hold stops the loop with a - decision request. + gobreaker v2.4.0 `IsExcluded` neither counts nor resets `ConsecutiveFailures`, frees the half-open + slot, and that the 60 s `Interval` resets closed-state counts; the three zones in `design.md` + *Context*; that Git auth/protected-branch errors reach the breaker Terminal; that a failed export is + attempted again on the next event-driven reconcile (cite the code path). Then the two tables + design.md *Marker* depends on: (a) every one of the 26 `invalidPrune` call sites and every other + Terminal raised inside an export plan (Git, GitLab, object-store and database backends), each + classified inventory-scoped / sink-wide / undecided with a one-line reason; (b) every caller of + `ResetBreakersForTest` and whether its test is parallel. Name the local-repository fixture that + produces a real Git ownership rejection through `RunExportEnvelope`. Finally + RACE-COUNT-BREAKER-ISOLATION on base (`go test -race -count=3 -skip TestControllers + ./internal/controller/`, ten runs, exit status each, failing test names); a flake that does not + reproduce in ten runs is recorded as "not reproduced", not as "mechanism disproved". A claim that + does not hold stops the loop with a decision request. -## 1. Breaker excludes Terminal errors (SCB-1, SCB-2, SCB-3) +## 1. Inventory-scoped results are excluded from the breaker (SCB-1, SCB-2, SCB-3) -- [ ] 1.1 Tests (red on base unless marked *control*): - - SCB-1a: given a fresh key, when `exportThroughBreaker` returns a Terminal error 10 times, then an - 11th call with a succeeding function runs it and returns nil · exposes Terminal errors counting - as failures. - - SCB-1b (boundary): given a stub backend that fails Terminal for inventory path A and succeeds for - path B on one sink, when `RunExportEnvelope` runs A 10 times, then B's export reaches the backend - and succeeds · exposes the cross-inventory blast radius at the real call site. - - SCB-1c: given a fresh key, when calls fail T, X, T, T, T (T Transient, X Terminal), then the - sixth call still runs its function (four counted failures); when that sixth call fails T, the - seventh does not run and returns a Transient error containing "circuit breaker open" · on base - the breaker opens after the fifth call (X counted), so the sixth call's assertion is the red; - after the fix it exposes an implementation that resets the streak on exclusion (`IsSuccessful` - instead of `IsExcluded`), which would let the seventh call run. +- [ ] 1.1 Refactor base first, in the same group: extract `newBreaker(key, timeout)` from `breakerFor` + with today's settings (pure refactor, existing tests green), and add `ErrInventoryScoped`, + `InventoryScoped` and `IsInventoryScoped` with no-op bodies (`InventoryScoped` returns its input). + Then the tests (red on that base unless marked *control*): + - M-1 (`internal/errors`): `InventoryScoped(e)` is Terminal, `IsInventoryScoped` true, + `errors.Is(_, e)` true, `Error()` equals `e.Error()`; `InventoryScoped(nil)` is nil; a plain + `Terminal(e)` is not inventory-scoped · exposes a marker that drops the class, the text or the + wrapped chain. + - SCB-1a: given a fresh key, when `exportThroughBreaker` returns `InventoryScoped(...)` 10 times, + then an 11th call with a succeeding function runs it and returns nil · exposes marked results + counting as failures. + - SCB-1b (boundary): given a stub backend that branches on object path, failing + `InventoryScoped(...)` for inventory A and succeeding for B on one sink, when `RunExportEnvelope` + runs A 10 times, then B's export reaches the backend and succeeds · exposes the cross-inventory + blast radius at the real call site. + - SCB-1c: given a fresh key, when calls fail T, X, T, T, T (T Transient, X marked), then the sixth + call still runs its function; when it fails T, the seventh does not run and returns a Transient + error containing "circuit breaker open" · red on base because X counts; after the fix it exposes + an implementation that resets the streak on exclusion (`IsSuccessful`). + - SCB-1e (boundary, real Git backend): with the fixture T0.1 names, an ownership rejection returned + by `RunExportEnvelope` is Terminal and inventory-scoped, and after 6 such rejections a different + inventory's export to the same sink runs · exposes a marker lost between `invalidPrune`'s + replacement and the breaker, or a site left unmarked. - SCB-2a (*control*): 5 Transient failures, then the 6th call's function does not run and the error is Transient with "circuit breaker open" · guards against over-excluding. - - SCB-2b (*control*): same with an unclassified `errors.New` error · guards against excluding - everything that is not Transient. - - SCB-2c (*control*): same with `kollecterrors.Forbidden(...)` · guards against excluding Forbidden. - - SCB-2d (*control*): same with a raw error whose text a later classifier would call a Git auth - failure (e.g. `errors.New("authentication required")`) · guards against classifying at the - breaker with the after-breaker classifier. - - SCB-1d: an unwrapped `apierrors.NewNotFound(...)` returned 10 times leaves the breaker closed (an - 11th succeeding call runs) · documents that `ClassOf`-inferred Terminal is excluded (red on base). - - SCB-3a: given a short-timeout breaker (constructor seam) opened by 5 Transient failures and its - timeout elapsed, when the trial returns Terminal, then `State()` is half-open; when the next call - succeeds, its function ran and `State()` is closed · exposes an excluded trial closing the - breaker (state after the trial) or wedging the slot (next call not run). - - SCB-3b: as 3a, but the call after the Terminal trial fails Transient, then `State()` is open and - the following call does not run · exposes an excluded trial closing the breaker. + - SCB-2b/2c/2f (*control*): same with an unclassified `errors.New`, with `kollecterrors.Forbidden`, + and with an unwrapped `apierrors.NewNotFound` (Terminal by `ClassOf`, unmarked) · guard against + excluding by class instead of by marker. + - SCB-2d (*control*, boundary): `RunExportEnvelope` with a stub backend returning + `git.ClassifyExportError(errors.New("authentication required"))` (Terminal, sink-wide): the + sixth export does not reach the backend · guards against excluding every Terminal (option A). + - SCB-3a: short-timeout breaker opened by 5 Transient failures, timeout elapsed; the trial returns + `InventoryScoped(...)`; `State()` is half-open; the next call succeeds, its function ran, and + `State()` is closed · exposes an excluded trial closing the breaker or wedging the slot. + - SCB-3b: as 3a, but the call after the marked trial fails Transient; `State()` is open and the next + call does not run · exposes an excluded trial closing the breaker. - SCB-3c: half-open breaker; a trial blocks on a channel; a second concurrent call does not run its - function and returns an error (gobreaker `ErrTooManyRequests`); release the trial with a Terminal - result; a third call runs · exposes a Terminal trial wedging the slot. *Control* for the - concurrency half on base. -- [ ] 1.2 Add `newBreaker(key, timeout)` and set `IsExcluded: kollecterrors.IsTerminal` in it; - `breakerFor` passes `circuitBreakerTimeout`. Make `TestResetBreakersForTest_clearsOpenBreaker` - non-parallel and fix any other parallel caller of `ResetBreakersForTest` found in 0.1. Mutations: - drop `IsExcluded`; replace it with `IsSuccessful`; exclude every non-nil error; exclude - `!IsTransient` (would exclude Forbidden). Each must be killed by a named 1.1 case. + function and returns an error; release the trial with a marked result; a third call runs · + exposes a marked trial wedging the slot. *Control* for the concurrency half on base. +- [ ] 1.2 Implement the marker (`design.md` *Marker*), route the inventory-scoped sites from T0.1 + table (a) through the new helper, set `IsExcluded: kollecterrors.IsInventoryScoped` in + `newBreaker`, and make `TestResetBreakersForTest_clearsOpenBreaker` non-parallel. Mutations, each + killed by a named case: drop `IsExcluded` (SCB-1a); `IsSuccessful` instead (SCB-1c, SCB-3a); + exclude `IsTerminal` (SCB-2d, SCB-2f); exclude every non-nil error (SCB-2a); marker drops `Unwrap` + (M-1); one routed site reverted to `invalidPrune` (SCB-1e). ## 2. Reconciler tests stop sharing a tripped breaker (RACE-COUNT-BREAKER-ISOLATION) - [ ] 2.1 On the G1 commit run `go test -race -count=3 -skip TestControllers ./internal/controller/` ten times and record each exit status, against T0.1's base count. Any red → name the failing tests - and the breaker key they share; that red is the group's red. All ten green is weak evidence (the - Transient-failure test still shares `default/postgres-demo`), so 2.2 runs either way. + and the breaker key they share; that red is the group's red. All ten green → record "not + reproduced; isolation is preventive" (the tests still share `default/postgres-demo`); 2.2 runs + either way. - [ ] 2.2 Isolate the reconciler tests built on `newExportFailureReconciler` and the other tests that share the `default/postgres-demo` key from each other's breaker state, without changing what they - assert (a per-invocation sink name where the assertion does not name the sink; otherwise a - non-parallel test with `t.Cleanup(sink.ResetBreakersForTest)`), never a retry. Ten green runs of - the 2.1 command afterwards. + assert: a per-invocation sink name where no assertion names the sink; otherwise a non-parallel test + that calls `sink.ResetBreakersForTest()` at its start and in `t.Cleanup`. Never a retry. Ten green + runs of the 2.1 command afterwards. ## 3. Docs -- [ ] 3.1 `docs/operator-manual/troubleshooting.md:214`: the breaker counts retryable export errors - (Transient, Forbidden, unclassified), not Terminal ones; a Terminal failure stays in that - inventory's status. Replace "self-closes after timeout": after 30 s one trial export is admitted, - success closes it, a counted failure re-opens it, a Terminal result leaves it half-open. - `docs/adr/0602-error-taxonomy.md:38`: add that Terminal errors are excluded (dated note, ADR not - re-decided). `task lint:markdown` 0 issues; `rg -n 'circuit breaker' docs` reviewed for any other - claim the change contradicts, the list recorded in `evidence/G3.md`. +- [ ] 3.1 `docs/operator-manual/troubleshooting.md:214`: the breaker opens after 5 consecutive counted + export failures per sink key (Transient, Forbidden, sink-wide Terminal such as Git auth); a failure + that is about one inventory (ownership conflict) does not count and stays in that inventory's status. + Replace "self-closes after timeout": after 30 s one trial export is admitted, success closes it, a + counted failure re-opens it, an inventory-scoped result leaves it half-open. + `docs/adr/0602-error-taxonomy.md:38`: add a dated note on what counts (ADR not re-decided). `task + lint:markdown` 0 issues; `rg -n 'circuit breaker' docs` reviewed for any other claim the change + contradicts, the list recorded in `evidence/G3.md`. ## 4. Integration From 3541629945aa279c11159dfb7cefadee33d1ba47 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 11:38:11 +0100 Subject: [PATCH 04/13] :page_facing_up: docs(spec): record breaker premise probe T0.1 Verifies the spec set's premises on base: breaker settings, gobreaker v2.4.0 exclusion semantics, the three zones, Git sink-wide Terminal at the breaker, re-export on the next reconcile. Classifies every Terminal raised inside an export plan (25 invalidPrune call sites plus backend classifiers) and every ResetBreakersForTest caller, names the SCB-1e fixture, and measures the controller race flake on base (2 of 10 runs red). --- .../evidence/T0.1.md | 366 ++++++++++++++++++ .../sink-breaker-terminal-isolation/loop.md | 5 + .../reviews/L-tasks/T0.1/brief-reviewer-a.md | 61 +++ .../L-tasks/T0.1/leg-reviewer-a-diff.md | 9 + .../sink-breaker-terminal-isolation/tasks.md | 4 +- 5 files changed, 443 insertions(+), 2 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/T0.1/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/T0.1/leg-reviewer-a-diff.md diff --git a/openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md b/openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md new file mode 100644 index 00000000..ce6b801b --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md @@ -0,0 +1,366 @@ +# T0.1 — Probe: verify the spec set's premises on base + +- Feature: `openspec/changes/sink-breaker-terminal-isolation/` · branch: `fix/sink-breaker-terminal-isolation` · + pre-task sha: `f16ec6baa` (code identical to `origin/main` @ `96c41b0d0`: `git diff --stat origin/main HEAD -- ':!openspec'` + is empty, verified 2026-11-05) · started: 2026-11-05 +- Satisfies: tasks.md 0.1 (premise gate for SCB-1, SCB-2, SCB-3 and G1/G2) · design: `design.md` *Context*, *Marker*, + *Behaviour*, *Testing approach* · constitution: none (evidence only) +- Smallest coherent change: this evidence file, the tasks.md tick and the loop.md row; no code. +- Status: CLOSED +- Next action: dispatch G1 (1.1 + 1.2), routing exactly the sites table (a) marks inventory-scoped. + +All file:line references are on base `f16ec6baa` and were read 2026-11-05 (verified) unless tagged *believed*. + +## Verification matrix + +| # | Claim / invariant | Sensor | Class | Result | +|---|---|---|---|---| +| 1 | Breaker settings and key | read `internal/sink/circuit_breaker.go:16-20,46-62`, `internal/sink/export.go:200` | independent-review | pass | +| 2 | gobreaker v2.4.0 `IsExcluded` neither counts nor resets `ConsecutiveFailures` | read `gobreaker.go:264-266`, `counter.go:32-34,106-115`; scratch probe P1/P2 | machine | pass | +| 3 | An excluded half-open trial frees the slot and leaves the state half-open | read `gobreaker.go:246`, `counter.go:36-41`; probe P4/P5 | machine | pass | +| 4 | 60 s `Interval` resets closed-state counts, independent of exclusions | read `gobreaker.go:128-136,304-309,348-359`; probe P3 | machine | pass | +| 5 | Three zones in design.md *Context* | read `internal/sink/export.go:77-216` | independent-review | pass | +| 6 | Git auth / protected-branch / pre-receive errors reach the breaker Terminal | read `git/export.go:86-150`, `git/errors.go:24-50`, `git/backend.go:59-78` | independent-review | pass | +| 7 | A failed export is attempted again on the next event-driven reconcile | read controller path (section *Re-export*) | independent-review | pass | +| 8 | "26 `invalidPrune` call sites" (design.md:21,47; tasks.md:34) | `grep -rn invalidPrune --include='*.go' . \| grep -v _test.go` | machine | fail (25 call sites + 1 definition = 26 lines; not load-bearing, see *Spec corrections*) | +| 9 | Table (a): every Terminal raised inside an export plan, classified | read every site below | independent-review | pass | +| 10 | Table (b): every `ResetBreakersForTest` caller and its parallelism | `grep -rn ResetBreakersForTest --include='*.go' .` + per-file `t.Parallel()` scan | machine | pass | +| 11 | A local-repository fixture produces a real Git ownership rejection through `RunExportEnvelope` | `go test -count=1 -v -run 'TestRunExportEnvelope_sharedPathRejectedNamesOwner' ./internal/sink/` | machine | pass | +| 12 | RACE-COUNT-BREAKER-ISOLATION on base, ten runs | `go test -race -count=3 -skip TestControllers ./internal/controller/` ×10 | machine | pass (measured: 2/10 red) | + +## Premises + +### 1. Breaker settings and key (verified) + +- Key `req.SinkNamespace + "/" + req.SinkName`: `internal/sink/export.go:200`. +- `breakerRegistry sync.Map`, lazily filled by `breakerFor`: `circuit_breaker.go:22-24,46-62`. +- `MaxRequests: 1`, `Interval: 60s`, `Timeout: 30s`, `ReadyToTrip: ConsecutiveFailures >= 5`: + `circuit_breaker.go:16-20,51-59`. No `IsSuccessful`, no `IsExcluded`, no `BucketPeriod`, so gobreaker's + defaults apply: `defaultIsSuccessful` = `err == nil`, `defaultIsExcluded` = false (`gobreaker.go:157-167,181-187`). +- Open state maps to Transient "sink circuit breaker open for %q": `circuit_breaker.go:39-41`. +- `ResetBreakersForTest` clears every key: `circuit_breaker.go:68-70`. +- `go.mod`: `github.com/sony/gobreaker/v2 v2.4.0`. + +### 2-4. gobreaker v2.4.0 (verified by reading and by a scratch probe) + +Source: `$(go env GOMODCACHE)/github.com/sony/gobreaker/v2@v2.4.0`. + +- `afterRequest` checks `isExcluded` first and returns after `onExclusion` (`gobreaker.go:264-266`); + `onExclusion` → `rollingCounts.onExclusion` → `Counts.onExclusion`, which only increments + `TotalExclusions` (`counter.go:32-34,106-115`). `ConsecutiveFailures` is neither incremented nor reset; + `readyToTrip` is not evaluated. +- Half-open admission: `cb.counts.validRequests() >= cb.maxRequests` → `ErrTooManyRequests` + (`gobreaker.go:246`); `validRequests = Requests - TotalExclusions` (`counter.go:36-41`). An excluded trial + therefore frees the slot; the state does not change (no `setState` on exclusion). +- With `Interval > 0` and no `BucketPeriod`, there is one bucket (`gobreaker.go:133-136`); entering closed + sets `expiry = now + interval` (`gobreaker.go:355-359`); `currentState` starts a new generation when it + expires (`gobreaker.go:306-308`), clearing all counts (`toNewGeneration`, `gobreaker.go:348-351`). +- Results from an earlier generation are dropped (`gobreaker.go:259-262`). +- `Counts()` returns the stored counts without calling `currentState` (`gobreaker.go:205-210`); read + `State()` first when a test asserts counts after a time step. + +Probe (scratch module outside the repository, run 2026-11-05 with +`GOFLAGS=-mod=mod GOPROXY=off GOWORK=off go run .`, repo `go.sum` copied in; same `ReadyToTrip`/`MaxRequests` +as production, `IsExcluded: errors.Is(err, errX)`, short interval/timeout for P3-P5): + +```go +// P1: closed, results T X T T T then T +for _, e := range []error{errT, errX, errT, errT, errT} { call(cb, e) } +ran, err := call(cb, errT) // 6th call +ran, err = call(cb, nil) // 7th call +// P2: 10 × X then a success +// P3: interval 200ms, 4 × T, sleep 250ms, State(), Counts(), one more T +// P4: timeout 300ms, 5 × T, sleep 350ms, trial X, trial X, trial nil +// P5: half-open, trial blocks on a channel and returns X; concurrent call; then T +``` + +Output (abridged to the asserting lines): + +```text +p1 step2 counts={Requests:2 TotalSuccesses:0 TotalFailures:1 TotalExclusions:1 ConsecutiveSuccesses:0 ConsecutiveFailures:1} state=closed +p1 step5 counts={Requests:5 TotalSuccesses:0 TotalFailures:4 TotalExclusions:1 ConsecutiveSuccesses:0 ConsecutiveFailures:4} state=closed +p1 step6 ran=true err=counted state=open +p1 step7 ran=false err=circuit breaker is open +p2 after10X counts-before-success ran=true err= state=closed +p3 before interval counts={Requests:4 ... ConsecutiveFailures:4} +p3 after interval state=closed counts={Requests:0 ... ConsecutiveFailures:0} +p4 after timeout state=half-open +p4 trial X ran=true err=excluded state=half-open counts={Requests:1 ... TotalExclusions:1 ...} +p4 trial X again ran=true err=excluded state=half-open +p4 trial ok ran=true err= state=closed +p5 concurrent ran=false err=too many requests +p5 after X trial, T: ran=true err=counted state=open +``` + +So T, X, T, T, T, T trips on the fifth T (design.md *Behaviour*, SCB-1c), ten exclusions never trip (SCB-1a), +an excluded trial keeps half-open and frees the slot (SCB-3a/3c), a counted failure after it re-opens (SCB-3b). + +### 5. Three zones (verified, `internal/sink/export.go`) + +- Before the breaker, `:78-190`: registry/spec checks `:78-84`, identity `:86-90` (`checkInventoryIdentity` + `:222-230`), acquire `:92-111`, payload decode `:115-121`, redaction re-marshal `:133-166`, spill gate + `:168-178`, layout resolve `:184-190`. All return before `:200`. design.md says `:79-190`; same range. +- At the breaker, `:200-202`: `exportThroughBreaker(key, plan.run)`. +- After the breaker, `:207-213`: `git.ClassifyExportError` then `classifyExportFailure`. +- Metrics after the breaker, `:203-212` (proposal *Impact*): matches. + +### 6. Git sink-wide Terminal reaches the breaker (verified) + +`plan.run` for Git/GitLab is `backend.Export` or `fileExporter.ExportFiles` (`layout_export.go:122,151,176,185,217`); +both end in `git.ExportFilesWithBranch` (`git/backend.go:59-78`, `git/export.go:37-50`), which returns +`ClassifyExportError(err)` / `ClassifyExportError(exportErr)` for both engines (`git/export.go:115-149`). +`ClassifyExportError` returns `Terminal("git auth failed: …")` for auth (`git/errors.go:35-37`) and +`Terminal("git push rejected: …")` for protected branch, pre-receive, and remote-rejected with 401/403 +(`git/errors.go:39-43`), and passes an already-Terminal error through unchanged (`:29-31`), so a marker on a +prune error survives it. GitLab wraps the same call and returns its error as is (`gitlab/backend.go:70-74,114-116`). + +### 7. Re-export on the next event-driven reconcile (verified) + +- A failed sink export calls `failSinkExport` (`internal/controller/kollectinventory_controller.go:439-443`), + which calls `tracker.forget(invKey, exportKey)` (`per_sink_export.go:259-270,165-170`). +- The next reconcile's `shouldSkip` finds no state and returns false (`per_sink_export.go:99-101`), so the + export runs again (`kollectinventory_controller.go:367`). +- A Terminal total failure sets `RequeueAfter = 0` (`kollectinventory_controller.go:268-270`): no timed retry; + the next attempt comes from an event. Events: the inventory itself, its sinks + (`kollectinventory_controller.go:698-712`) and the collect store (collected-resource changes, + `WatchesRawSource(newInventoryStoreSource(...))`, `:715-717`, `inventory_store_source.go:29`). +- Same pattern in the cluster-inventory controller (`kollectclusterinventory_controller.go:219-241,315,364`). + +## Table (a): Terminal results raised inside an export plan + +"Inside an export plan" = reachable from `plan.run` at `export.go:200`. The only `exportThroughBreaker` call +is `export.go:200` (verified by grep). Deletion/cleanup (`RunCleanupExport`, `cleanup.go:240,273`) does not +go through the breaker. + +Classes: **inv** = inventory-scoped (about the inventory's own paths or its ownership claim, design.md +*Marker*); **sink** = sink-wide (shared repository or prune-metadata state, endpoint, credentials); +**undecided** (G1 leaves it unmarked = today's behaviour); **n/a** = not reached inside an export plan. + +### `invalidPrune` (definition `git/prune_owned.go:254-256`), 25 call sites + +| # | Site | Message | Reached from (export plan) | Class | Reason | +|---|---|---|---|---|---| +| 1 | `git/prune_owned.go:53` | invalid prune path | `validatePrunePath`: own keep set `:272`, own claim paths `:289`, own file paths `validate.go:82`; **other owners' recorded paths** `loadPruneRecords:152` | undecided | shared helper: own path when called from `:272/:289/validate.go:82`, corrupt shared metadata when called from `:152`; the raise site cannot tell | +| 2 | `git/prune_owned.go:57` | reserved or unsafe prune path | same callers as #1 | undecided | same as #1 | +| 3 | `git/prune_owned.go:77` | prune path is a symlink | `checkPruneFile`: record file `readPruneRecord:92` (metadata); own recorded paths `prepareOwnedPrune:189`; own keep set `:282` | undecided | shared helper: metadata file (sink) vs own path (inv) | +| 4 | `git/prune_owned.go:80` | prune parent is not a directory | same callers as #3 | undecided | same as #3 | +| 5 | `git/prune_owned.go:83` | prune path is not a regular file | same callers as #3 | undecided | same as #3 | +| 6 | `git/prune_owned.go:105` | prune metadata exceeds N bytes (read) | `readPruneRecord` ← `loadPruneRecords:146` | sink | total size of all owners' records | +| 7 | `git/prune_owned.go:110` | decode prune record | `readPruneRecord` | sink | any owner's record file is corrupt; every owned export on the branch fails | +| 8 | `git/prune_owned.go:114` | prune record has trailing data | `readPruneRecord` | sink | as #7 | +| 9 | `git/prune_owned.go:117` | invalid prune record | `readPruneRecord` | sink | as #7 | +| 10 | `git/prune_owned.go:135` | prune metadata must be a real directory | `loadPruneRecords` | sink | shape of the shared `.kollect-prune` directory | +| 11 | `git/prune_owned.go:142` | prune metadata exceeds N owners (on disk) | `loadPruneRecords` | sink | shared owner count already over the cap | +| 12 | `git/prune_owned.go:156` | duplicate ownership of a path between two records | `loadPruneRecords` | sink | existing records conflict with each other; the message tells the operator to repair the branch; blocks every owned export | +| 13 | `git/prune_owned.go:174` | prune owner exceeds 4096 bytes | `prepareOwnedPrune` (own `cfg.PruneOwner`, from the inventory identity, `layout_export.go:197-201`) | inv | the inventory's own owner string | +| 14 | `git/prune_owned.go:215` | prune metadata exceeds N owners (after adding own record) | `prepareOwnedPrune` | undecided | cause is shared capacity, effect is per-inventory (only an inventory without a record is refused; existing owners still export); dispatch rule says shared metadata state → sink, so unmarked (leans sink) | +| 15 | `git/prune_owned.go:230` | prune metadata exceeds N bytes (after own record) | `prepareOwnedPrune` | undecided | as #14: shared byte budget, tipped by this inventory's record growth | +| 16 | `git/prune_owned.go:261` | prune inventory exceeds N paths | `validateOwnedPrunePaths` (own keep set) | inv | size of the inventory's own path set | +| 17 | `git/prune_owned.go:268` | written path missing from prune keep set | `validateOwnedPrunePaths` (own written vs own keep set) | inv | the inventory's own export is inconsistent (multipart union) | +| 18 | `git/prune_owned.go:310` | prune path belongs to another inventory (`foreignClaimError`) | `checkForeignClaim:306` ← `:278,:292`; `checkSingleClaims:333,:335` | inv | the inventory's own path or claim collides with another owner's record: the ownership conflict SINK-BREAKER-TENANT-01 is about | +| 19 | `git/prune_owned.go:337` | ownership records would claim a path twice (neither is self) | `checkSingleClaims` | sink | conflict between two other owners' records; *believed* unreachable after `:156` (defensive) | +| 20 | `git/release.go:46` | release without files needs an owner | `planRelease` ← `delete.go:453,834` | n/a | deletion path only, never inside `export.go:200` | +| 21 | `git/release.go:92` | prune owner exceeds 4096 bytes | `ownRecord` ← `planRelease` | n/a | deletion path only | +| 22 | `git/release.go:104` | ownership record names another inventory | `ownRecord` ← `planRelease` | n/a | deletion path only | +| 23 | `git/prune_target.go:66` | prune path belongs to another inventory on merge target branch | `checkMergeTargetClaims` ← `git/export.go:210`, `checkCLIMergeTargetClaims` ← `export_file.go:74` | inv | the inventory's own path claimed by another owner on the merge target (same conflict as #18) | +| 24 | `git/prune_target.go:93` | prune metadata exceeds N owners (merge target) | `pruneMetadataFromTree` ← `checkMergeTargetClaims:48` | sink | shared metadata on the target branch | +| 25 | `git/prune_target.go:120` | prune metadata exceeds N bytes (merge target) | `copyTreeEntry` ← `pruneMetadataFromTree:100` | sink | as #24 | + +Totals: inv 5 (#13, #16, #17, #18, #23) · sink 10 (#6-#12, #19, #24, #25) · undecided 7 (#1-#5, #14, #15) · n/a 3 +(#20-#22). Errors from `loadPruneRecords` on the merge target are wrapped with `%w` (`prune_target.go:50,54`), +which keeps class and any marker (`ClassError.Unwrap`, `internal/errors/errors.go:42-48`). + +GitLab branch-MR mode (*believed*, from `prune_target.go:22-28`, `gitlab/backend.go:99-106`): the checked-out +branch is the inventory's own feature branch, so `loadPruneRecords` on it reads metadata copied from the target. +This does not change any class above (the records are still shared state), but it is why #23 exists. + +### Other Terminal results inside an export plan + +Every `kollecterrors.Terminal(` / `ErrTerminal` / `ClassifyAPI` under `internal/sink` (non-test) was checked +(grep, 2026-11-05). Those not listed are raised before or outside the breaker: `export.go:79-186,229` +(zone 1), `registry.go:89`, `ca_source.go:42`, `git/backend.go:33`, `sinktls/sinktls.go:90,111`, +`s3/config.go:138,141`, `postgres/ca_trust.go:66,87,92,367,372,377` (via `poolConfig`, `postgres/pool.go:20-66`, +at backend construction), `mongodb/ca_trust.go:110-282` (via `clientOptions`, `mongodb/backend.go:97`, +`connection.go:33`), and `cleanup.go:169-372` (cleanup path). No sink backend calls the Kubernetes API inside +`Export` (`apierrors` imported only by `secret_refs.go`, `credentials.go`, `resolver.go`, all acquire-time). + +| # | Site | Condition | Backend | Class | Reason | +|---|---|---|---|---|---| +| 26 | `git/errors.go:36` | auth failure (401/403 text or status, `ErrAuthenticationRequired`/`ErrAuthorizationFailed`) | Git, GitLab | sink | credentials of the sink | +| 27 | `git/errors.go:42` | protected branch, pre-receive hook declined, remote rejected with 401/403 | Git, GitLab | sink | branch protection and hooks are repository settings | +| 28 | `s3/backend.go:230` (`redactedRedirectClient.Do`) and `:266` (`refuseInsecureRedirect`) | endpoint redirects to non-https with a CA configured | S3, GCS (S3-compatible inner, `gcs/backend.go:58-60`) | sink | endpoint behaviour; reaching the breaker still Terminal depends on the AWS SDK keeping the chain (*believed*) | +| 29 | `bigquery/errors.go:32` via `bigquery/backend.go:255,284,303,426` | googleapi code 400/401/403/404 or reason invalid/invalidquery/accessdenied/notfound (`errors.go:62-85`) | BigQuery | undecided | 401/403/404/accessdenied/notfound are sink-wide; 400/invalid on a MERGE with `PARSE_JSON(payload)` may be one inventory's rows; one site cannot tell | +| 30 | `bigquery/errors.go:32` via `bigquery/backend.go:146,158` | decode payload / build rows | BigQuery | n/a | input is not a `googleapi.Error`/`bigquery.Error`, so `isTerminal` is false and the result is Transient (`errors.go:62-104`) | +| 31 | `mongodb/errors.go:53` via `mongodb/backend.go:171` | duplicate key (11000) or document validation (121) on upsert | MongoDB | undecided | a collection validator or unique index is sink configuration, but the rejected document is one inventory's; one site cannot tell | + +Postgres, Kafka, NATS, local, GCS (beyond #28) and S3 (beyond #28) raise no Terminal inside `Export`. + +Not in the table's scope but relevant to SCB-1's limits: unclassified inventory-scoped errors inside the plan +also count today and keep counting under option B (marker is Terminal-only), e.g. `git/validate.go:65,72,77,86` +(no files / empty payload / object path / duplicate path), `git/export.go:94-96`, `gitlab/backend.go:88-90`, +`mongodb/backend.go:158,167`, `postgres/backend.go:116`. Recorded, not a contradiction: proposal.md marks only +Terminal results. + +## Table (b): `ResetBreakersForTest` callers + +Go runs a package's non-parallel top-level tests to completion before releasing the paused `t.Parallel()` +tests, and each `-count` iteration completes before the next, so a reset in a non-parallel test cannot clear a +parallel test's breaker mid-trip; only a reset in a parallel test can. Each test package is its own process, +so `internal/sink/git` resets do not touch the `internal/sink` or `internal/controller` registries. + +| # | Caller (file:line) | Enclosing function | Test(s) | Parallel? | +|---|---|---|---|---| +| 1 | `internal/sink/circuit_breaker_test.go:155` | `TestResetBreakersForTest_clearsOpenBreaker` (`:95`) | itself | **yes** (`t.Parallel()` at `:96`); G1 makes it non-parallel (tasks.md 1.2) | +| 2 | `internal/sink/circuit_breaker_test.go:175` | `TestResetBreakersForTest_safeAgainstConcurrentExport` (`:167`) | itself (resets from its own goroutines) | no (comment `:162-166`) | +| 3 | `internal/sink/inventory_identity_test.go:52` | `newIdentityRemote` cleanup (`:43`) | every `identityRemote` test in `inventory_identity_test.go` and `release_on_delete_test.go` | no (0 `t.Parallel()` in either file) | +| 4 | `internal/sink/inventory_identity_test.go:85` | `(*identityRemote).export` (`:83`), **before every export** | as #3 | no | +| 5 | `internal/sink/inventory_identity_test.go:312` | `runRecorded` (`:310`) | `TestRunExportEnvelope_ownerFromIdentity` etc. | no | +| 6 | `internal/sink/inventory_identity_test.go:333` | `TestRunExportEnvelope_ownerFromIdentity` cleanup | itself | no | +| 7 | `internal/sink/inventory_identity_test.go:357` | `TestRunExportEnvelope_commitContextCarriesKind` cleanup | itself | no | +| 8 | `internal/sink/inventory_identity_test.go:373` | `TestRunExportEnvelope_identityPathMismatchRejected` cleanup | itself | no | +| 9 | `internal/sink/inventory_identity_test.go:402` | `TestRunExportEnvelope_treeExportRequiresIdentity` cleanup | itself | no | +| 10 | `internal/sink/release_on_delete_test.go:33` | `(*identityRemote).cleanup` (`:31`) | release tests using `identityRemote` | no | +| 11 | `internal/sink/release_on_delete_test.go:201` | `TestRunCleanupExport_gitReleaseFailureIsRetried` cleanup | itself | no | +| 12 | `internal/sink/git/owned_prune_fuzz_test.go:63` | `FuzzOwnedPrune` `f.Cleanup` (`:48`) | `FuzzOwnedPrune` (seed corpus runs under `go test`) | no (0 `t.Parallel()` in the file) | +| 13 | `internal/sink/git/owned_prune_fuzz_test.go:542` | `(*ownedPruneHarness).deleteInventory` (`:513`) | `FuzzOwnedPrune`, `TestOwnedPrune_rejectedExportLeavesOtherOwnersBytes` (`:1145`) | no | +| 14 | `internal/sink/git/owned_prune_fuzz_test.go:657` | `(*ownedPruneHarness).export` (`:652`), before every export | as #13 | no | + +`internal/controller`: no caller (grep). Its tests share the process-wide registry with no reset; that is +RACE-COUNT-BREAKER-ISOLATION (below). + +Note for G1: the fuzz harness resets before every export because "five in a row would trip the per-sink +breaker" (`owned_prune_fuzz_test.go:655-656`), which is the ownership-rejection trip this change removes; the +reset stays valid afterwards (sink-wide sites still count). + +## Fixture for SCB-1e + +`identityRemote` (`internal/sink/inventory_identity_test.go:34-107`): a bare repository (`git init --bare`, +`:56`) behind a `file://` endpoint, per-resource layout, driven through `RunExportEnvelope` with the real Git +backend (CLI engine). `TestRunExportEnvelope_sharedPathRejectedNamesOwner` (`:230-252`) gets a Terminal +"belongs to another inventory" rejection from site #18 (`prune_owned.go:310`; the message has no "merge target", +so not #23) when `namespacedPlatform` exports a path `clusterPlatform` owns. + +```text +$ go test -count=1 -v -run 'TestRunExportEnvelope_sharedPathRejectedNamesOwner' ./internal/sink/ # 2026-11-05, exit 0 +=== RUN TestRunExportEnvelope_sharedPathRejectedNamesOwner +--- PASS: TestRunExportEnvelope_sharedPathRejectedNamesOwner (0.32s) +``` + +Constraints for G1 (verified from the fixture code): `identityRemote.export` calls `ResetBreakersForTest()` +before every export (`:85`) and uses the fixed key `default/identity-git` (`:97-98`). SCB-1e must therefore +call `RunExportEnvelope` itself (same request shape as `:94-104`) with a per-invocation sink name and no reset, +or the trip it measures is wiped; and it must not be parallel while it uses `newIdentityRemote` (cleanup resets, +`:52`). Needs the `git` CLI (`:45-47`). + +## RACE-COUNT-BREAKER-ISOLATION on base + +Command, ten runs, sequential, on `f16ec6baa`, 2026-11-05 (runner script kept outside the repository): +`go test -race -count=3 -skip TestControllers ./internal/controller/` + +| Run | Exit | Seconds | DATA RACE | Failing tests (failures across the 3 iterations) | +|---|---|---|---|---| +| 1 | 0 | 49 (incl. build) | 0 | none | +| 2 | 0 | 11 | 0 | none | +| 3 | 0 | 11 | 0 | none | +| 4 | 0 | 10 | 0 | none | +| 5 | 0 | 12 | 0 | none | +| 6 | 0 | 10 | 0 | none | +| 7 | 0 | 11 | 0 | none | +| 8 | 1 | 10 | 0 | deleteExportsEmptyAndRemovesFinalizer ×2, exportsDeploymentToSink ×2, terminalCleanupDoesNotRequeue ×1, terminalTotalExportFailureDoesNotRequeue ×2 | +| 9 | 1 | 10 | 0 | deleteExportsEmptyAndRemovesFinalizer ×2, exportsDeploymentToSink ×2, terminalCleanupDoesNotRequeue ×2, terminalTotalExportFailureDoesNotRequeue ×2 | +| 10 | 0 | 11 | 0 | none | + +Result: **2 of 10 red**, reproduced (backlog said 4 of 6, 2026-10). All four names are +`TestKollectInventoryReconciler_*`; failure lines (run 8): + +```text +kollectinventory_finalizer_test.go:180: Reconcile: export to "postgres-demo": sink circuit breaker open for "default/postgres-demo" +kollectinventory_export_test.go:312: export count = 0, want 1 +kollectinventory_export_test.go:418: RequeueAfter = 30s, want 0 (terminal export must not requeue) +kollectinventory_finalizer_test.go:269: Reconcile err = export to "postgres-demo": sink circuit breaker open for "default/postgres-demo", want nil (terminal cleanup must not requeue with backoff) +``` + +Shared key: `default/postgres-demo` (named in the failure text). Tests: `exportsDeploymentToSink` +(`kollectinventory_export_test.go:239`), `terminalTotalExportFailureDoesNotRequeue` (`:405`, via +`newExportFailureReconciler` `:339`), `deleteExportsEmptyAndRemovesFinalizer` +(`kollectinventory_finalizer_test.go:104`), `terminalCleanupDoesNotRequeue` (`:201`). `postgres-demo` also +appears in `cluster_inventory_finalizer_test.go` (G2 scope). Mechanism (open breaker on the shared key) is +consistent with the failure text; ordering was not instrumented. + +## Spec corrections (non-blocking) + +Neither changes option B, a requirement or a test; both are wording for the orchestrator. + +1. design.md:21,47 and tasks.md:34 say "26 `invalidPrune` call sites"; there are **25** call sites (grep shows 26 + lines including the definition at `prune_owned.go:254`). Table (a) lists all 25. +2. proposal.md *Impact* names only `internal/sink/git/prune_owned.go` for the marked rejections; site #23 + (`git/prune_target.go:66`) is also inventory-scoped, so G1 touches `prune_target.go` too. + +Neither is a premise that "does not hold" in the sense of tasks.md 0.1: the load-bearing premises (rows 1-7, 11, +12) hold. + +## Sensors run + +| Tier | Command | Exit | Seconds | Note | +|---|---|---|---|---| +| inner | gobreaker probe `go run .` (scratch module, `GOPROXY=off`) | 0 | ~2 | output above | +| inner | `go test -count=1 -v -run 'TestRunExportEnvelope_sharedPathRejectedNamesOwner\|TestRunExportEnvelope_circuitBreaker\|TestResetBreakersForTest' ./internal/sink/` | 0 | ~3 | 4 `=== RUN`, 4 PASS | +| milestone | `go test -race -count=3 -skip TestControllers ./internal/controller/` ×10 | see table | 10-49 each | 2/10 exit 1 (known flake, G2) | +| task | `task lint:markdown` (files staged first) | 0 | ~20 | `Summary: 0 issues in 0 files`, 421 files incl. this one | + +No code changed, so `task lint`, the unit-test fitness row and `task check` are not run for this task (diff is +evidence and state files only). + +## Iterations + +1 · grep count · SPECIFICATION (non-blocking) · 25 call sites, spec says 26 → recorded under *Spec corrections*. + +## Known red + +None added, none cleared. + +## Test changes + +None. + +## Harness gaps + +None for this probe: every row has a command or a file:line a reviewer can re-read. + +## Maintainability + +1 Locality: evidence, tasks.md tick, loop.md row only. 2 Surface: no code. 3 Duplication: none. 4 Second way: none. +5 Coupling: none. 6 Abstraction: none. 7 Debt for green: none. 8 Legibility: n/a (no code). Easier: G1 has an exact +site list; harder: nothing. + +## Independent review + +Round 1 (one leg, reviewer-a, diff lens, `reviews/L-tasks/T0.1/leg-reviewer-a-diff.md`; brief +`brief-reviewer-a.md`; no paid Claude leg: evidence-only diff, no secrets/CI/infra/manifests trigger, verdict +not BLOCK). Verbatim: + +```text +verdict: APPROVE +findings: + - [P2] evidence/T0.1.md:309: staged lint result references absent "Final checks"; the completed result exists only unstaged. +matrix_disputes: + - None affecting implementation: all 25 invalidPrune sites and 14 reset callers are accounted for; undecided sites remain unmarked. +could_not_check: + - Independent test reruns: Go shim attempts an installation blocked by the read-only filesystem; race logs corroborate the recorded 2/10 failures. +summary: Load-bearing premises hold; both spec corrections are non-blocking. No missing site or unsafe inventory-scoped classification found. +``` + +Disposition: P2 verified (the lint row was edited after staging); fixed by restaging, no second round +(evidence wording only). + +## Verdict + +CLOSED. Load-bearing premises hold; two non-blocking spec corrections for the orchestrator (*Spec corrections*). +Matrix: machine 6 / independent-review 6 / agent 0 / gaps 0 (row 8 `fail` is the count correction, not a premise). + +## Learnings + +- spec · a raise-site class is not always decidable: five `invalidPrune` sites sit in shared helpers called for + both own paths and shared metadata; G1 leaves them unmarked per design.md *Marker*. +- guidance · `identityRemote.export` resets every breaker before each export; a breaker test built on it must + call `RunExportEnvelope` directly. diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index d4b728ff..decfb75d 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -53,6 +53,7 @@ Holistic run at B: not yet run. | Task | Verdict | Review (legs, rounds, paid leg?) | Gaps / decision request | |---|---|---|---| +| T0.1 | CLOSED | reviewer-a APPROVE (1 P2 fixed), 1 round, no paid leg (`reviews/L-tasks/T0.1/`) | none; 2 non-blocking spec wording corrections (25 not 26 `invalidPrune` call sites; proposal *Impact* misses `git/prune_target.go`), see `evidence/T0.1.md` *Spec corrections*. Race on base: 2/10 red | ## Known red @@ -70,6 +71,10 @@ Tests from a test task changed by a later task (close-task `TEST` class); each o ## Lessons +- T0.1: G1 marks exactly 5 sites (`prune_owned.go:174,261,268,310`, `prune_target.go:66`); 7 sites are undecided (shared helpers `prune_owned.go:53,57,77,80,83` called for own paths and shared metadata; capacity sites `:215,:230`) and stay unmarked. +- T0.1: `identityRemote.export` (`internal/sink/inventory_identity_test.go:85`) resets every breaker before each export and uses the fixed key `default/identity-git`; SCB-1e must call `RunExportEnvelope` directly with a per-invocation sink name, non-parallel. +- T0.1: gobreaker `Counts()` does not advance state; call `State()` before reading counts after a time step. + ## Where a human should look first ## Proposed harness changes diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/T0.1/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/T0.1/brief-reviewer-a.md new file mode 100644 index 00000000..9638d743 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/T0.1/brief-reviewer-a.md @@ -0,0 +1,61 @@ +# Review brief: task 0.1 (probe, evidence only) + +You are reviewing one task, independently of whoever implemented it. Read-only: do not edit, commit or +create files in the repository. Work from the repository root +(`/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation`). + +## Inputs + +1. TASK: `openspec/changes/sink-breaker-terminal-isolation/tasks.md`, item 0.1 (read it). Spec set it checks: + `proposal.md`, `design.md`, `specs/sink-circuit-breaker/spec.md` in the same directory. +2. MATRIX and all evidence: `openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md`. +3. DIFF: base `f16ec6baa`, staged (not yet committed): `git diff --cached f16ec6baa`. The diff is evidence, + the tasks.md tick and loop.md only; no code. The code the evidence describes is the tree at `f16ec6baa`. +4. SENSORS: recorded in the evidence (*Sensors run*, race table). gobreaker source is in + `$(go env GOMODCACHE)/github.com/sony/gobreaker/v2@v2.4.0`. +5. RULES: `AGENTS.md`, `docs/development/spec-workflow.md`. Committed evidence is Markdown only. + +## What matters + +This probe is the premise gate for the implementation that follows: table (a) decides which error sites get +marked inventory-scoped (excluded from the sink circuit breaker) and table (b) decides test isolation. A wrong +row means either a sink-wide fault stops shedding load (marked wrongly) or the tenant bug stays (missed). + +Judge, in this order: + +a. INTENT: does the evidence cover every clause of task 0.1 (breaker settings and key; gobreaker `IsExcluded` + semantics incl. half-open slot and interval reset; three zones; Git auth/protected-branch Terminal at the + breaker; re-export on next event-driven reconcile; table (a) with every `invalidPrune` call site and every + other Terminal inside an export plan across Git, GitLab, object-store and database backends; table (b); + the SCB-1e fixture; ten race runs with exit status and failing names)? +b. ACCURACY: spot-check file:line citations against the code at `f16ec6baa`. Above all check table (a): + is any site missing (grep `invalidPrune`, `kollecterrors.Terminal(`, other classifiers under + `internal/sink`)? Is any classification wrong under design.md's definition (inventory-scoped = about the + inventory's own paths or its ownership claim; shared repository or prune-metadata state = sink-wide; unsure + = undecided)? Is any site wrongly placed inside or outside the export plan (`internal/sink/export.go:200` + is the only breaker call)? Check table (b) callers and their parallelism. +c. SPEC: does any finding contradict a load-bearing claim in proposal.md or design.md that the evidence fails + to flag? Are the two "non-blocking" spec corrections really non-blocking? +d. Honesty: does the evidence claim more than it ran or read (verified vs believed tags, evidence classes)? + +You may rerun anything under a minute (e.g. a focused `go test -run`); do not rerun the ten-run race loop. + +For each P0/P1 you are about to file, argue the opposite for one line; keep only what survives. + +RETURN at most 15 lines, nothing else, starting with a `## Verdict` heading: + +```text +## Verdict +verdict: APPROVE | REQUEST_CHANGES | BLOCK +findings: + - [P0|P1|P2] +matrix_disputes: + - — +could_not_check: + - +summary: +``` + +Severity: P0 (a wrong classification that would mark a sink-wide fault, a missed load-bearing contradiction, +evidence claiming a run that did not happen) → BLOCK. P1 (a missing site, a wrong file:line that changes a +row, an unflagged spec contradiction) → REQUEST_CHANGES. P2 → note only. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/T0.1/leg-reviewer-a-diff.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/T0.1/leg-reviewer-a-diff.md new file mode 100644 index 00000000..07969742 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/T0.1/leg-reviewer-a-diff.md @@ -0,0 +1,9 @@ +## Verdict +verdict: APPROVE +findings: + - [P2] openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md:309: staged lint result references absent “Final checks”; the completed result exists only unstaged. +matrix_disputes: + - None affecting implementation: all 25 invalidPrune sites and 14 reset callers are accounted for; undecided sites remain unmarked. +could_not_check: + - Independent test reruns: Go shim attempts an installation blocked by the read-only filesystem; race logs corroborate the recorded 2/10 failures. +summary: Load-bearing premises hold; both spec corrections are non-blocking. No missing site or unsafe inventory-scoped classification found. diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index 86236432..6ceb7879 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -25,7 +25,7 @@ Test rules for every group: ## 0. Probe -- [ ] 0.1 Verify the load-bearing claims in `proposal.md` and `design.md` on `origin/main` and record +- [x] 0.1 Verify the load-bearing claims in `proposal.md` and `design.md` on `origin/main` and record each with file:line or command output in `evidence/T0.1.md`: the breaker settings and key; that gobreaker v2.4.0 `IsExcluded` neither counts nor resets `ConsecutiveFailures`, frees the half-open slot, and that the 60 s `Interval` resets closed-state counts; the three zones in `design.md` @@ -39,7 +39,7 @@ Test rules for every group: RACE-COUNT-BREAKER-ISOLATION on base (`go test -race -count=3 -skip TestControllers ./internal/controller/`, ten runs, exit status each, failing test names); a flake that does not reproduce in ten runs is recorded as "not reproduced", not as "mechanism disproved". A claim that - does not hold stops the loop with a decision request. + does not hold stops the loop with a decision request. — closed 2026-11-05, evidence: evidence/T0.1.md ## 1. Inventory-scoped results are excluded from the breaker (SCB-1, SCB-2, SCB-3) From 1048a66d2dd57abcf14b441ab958441cc71bb92d Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 11:39:05 +0100 Subject: [PATCH 05/13] :page_facing_up: docs(spec): apply T0.1 probe corrections --- .../changes/sink-breaker-terminal-isolation/design.md | 4 ++-- .../changes/sink-breaker-terminal-isolation/proposal.md | 4 ++-- openspec/changes/sink-breaker-terminal-isolation/tasks.md | 8 ++++---- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/openspec/changes/sink-breaker-terminal-isolation/design.md b/openspec/changes/sink-breaker-terminal-isolation/design.md index c0608240..ecb24a1b 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/design.md +++ b/openspec/changes/sink-breaker-terminal-isolation/design.md @@ -18,7 +18,7 @@ Three zones (`internal/sink/export.go`): protected-branch and pre-receive rejections arrive Terminal (`git/errors.go:35-42`); BigQuery and MongoDB classify some write errors Terminal (`bigquery/errors.go`, `mongodb/errors.go`). The ownership/prune rejection arrives Terminal through `invalidPrune` (`git/prune_owned.go:254-256`, - 26 call sites). + 25 call sites). - **After the breaker** (`:207-212`): `git.ClassifyExportError` again and `classifyExportFailure`, for errors the plan returned unclassified. @@ -44,7 +44,7 @@ In `internal/errors`: not change). - `func IsInventoryScoped(err error) bool { return errors.Is(err, ErrInventoryScoped) }`. -At the raise sites: T0.1 classifies the 26 `invalidPrune` calls and every other Terminal raised inside +At the raise sites: T0.1 classifies the 25 `invalidPrune` calls and every other Terminal raised inside an export plan as inventory-scoped (about the inventory's own paths or its ownership claim) or sink-wide/shared (e.g. corrupt or unreadable prune metadata, repository state). G1 routes the inventory-scoped ones through a new helper (e.g. `ownershipConflict(format, args...)` returning diff --git a/openspec/changes/sink-breaker-terminal-isolation/proposal.md b/openspec/changes/sink-breaker-terminal-isolation/proposal.md index 879887da..5ad9e811 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/proposal.md +++ b/openspec/changes/sink-breaker-terminal-isolation/proposal.md @@ -68,8 +68,8 @@ None. ## Impact -- Code: `internal/errors/errors.go` (marker, constructor, predicate), `internal/sink/git/prune_owned.go` - (marked ownership rejections), `internal/sink/circuit_breaker.go` (`newBreaker`, `IsExcluded`). +- Code: `internal/errors/errors.go` (marker, constructor, predicate), `internal/sink/git/prune_owned.go` and + `prune_target.go` (marked ownership rejections, the 5 sites in `evidence/T0.1.md` table a), `internal/sink/circuit_breaker.go` (`newBreaker`, `IsExcluded`). - Tests: `internal/errors`, `internal/sink/circuit_breaker_test.go`, a Git-backend boundary test, and the `internal/controller` tests named in RACE-COUNT-BREAKER-ISOLATION. - Docs: `docs/operator-manual/troubleshooting.md:214`, `docs/adr/0602-error-taxonomy.md:38` diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index 6ceb7879..0e5664ee 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -31,7 +31,7 @@ Test rules for every group: slot, and that the 60 s `Interval` resets closed-state counts; the three zones in `design.md` *Context*; that Git auth/protected-branch errors reach the breaker Terminal; that a failed export is attempted again on the next event-driven reconcile (cite the code path). Then the two tables - design.md *Marker* depends on: (a) every one of the 26 `invalidPrune` call sites and every other + design.md *Marker* depends on: (a) every one of the 25 `invalidPrune` call sites and every other Terminal raised inside an export plan (Git, GitLab, object-store and database backends), each classified inventory-scoped / sink-wide / undecided with a one-line reason; (b) every caller of `ResetBreakersForTest` and whether its test is parallel. Name the local-repository fixture that @@ -62,7 +62,7 @@ Test rules for every group: call still runs its function; when it fails T, the seventh does not run and returns a Transient error containing "circuit breaker open" · red on base because X counts; after the fix it exposes an implementation that resets the streak on exclusion (`IsSuccessful`). - - SCB-1e (boundary, real Git backend): with the fixture T0.1 names, an ownership rejection returned + - SCB-1e (boundary, real Git backend, not parallel): with the fixture T0.1 names (`identityRemote`), calling `RunExportEnvelope` directly with a per-invocation sink name (its `export` helper resets every breaker), an ownership rejection returned by `RunExportEnvelope` is Terminal and inventory-scoped, and after 6 such rejections a different inventory's export to the same sink runs · exposes a marker lost between `invalidPrune`'s replacement and the breaker, or a site left unmarked. @@ -82,8 +82,8 @@ Test rules for every group: - SCB-3c: half-open breaker; a trial blocks on a channel; a second concurrent call does not run its function and returns an error; release the trial with a marked result; a third call runs · exposes a marked trial wedging the slot. *Control* for the concurrency half on base. -- [ ] 1.2 Implement the marker (`design.md` *Marker*), route the inventory-scoped sites from T0.1 - table (a) through the new helper, set `IsExcluded: kollecterrors.IsInventoryScoped` in +- [ ] 1.2 Implement the marker (`design.md` *Marker*), route exactly the 5 inventory-scoped sites from T0.1 + table (a) through the new helper (undecided sites stay unmarked), set `IsExcluded: kollecterrors.IsInventoryScoped` in `newBreaker`, and make `TestResetBreakersForTest_clearsOpenBreaker` non-parallel. Mutations, each killed by a named case: drop `IsExcluded` (SCB-1a); `IsSuccessful` instead (SCB-1c, SCB-3a); exclude `IsTerminal` (SCB-2d, SCB-2f); exclude every non-nil error (SCB-2a); marker drops `Unwrap` From dff6a86bb3bf3b121d96a823b8c7e18f86486abe Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 11:50:38 +0100 Subject: [PATCH 06/13] :bug: fix(sink): keep inventory-scoped failures out of the breaker An ownership conflict is about one inventory, yet it counted toward the per-sink circuit breaker, so five rejections of one inventory blocked every other inventory on the same sink for 30 s. Add an InventoryScoped marker in internal/errors (Terminal, text and chain unchanged), route the five inventory-scoped prune sites through ownershipConflict, and set IsExcluded on the breaker. Sink-wide failures (Git auth, shared prune metadata) still count. Extract newBreaker(key, timeout) so half-open tests can use a 1 s breaker. Tasks 1.1 + 1.2 of openspec change sink-breaker-terminal-isolation. --- internal/errors/errors.go | 29 ++ internal/errors/errors_test.go | 38 ++ internal/sink/circuit_breaker.go | 17 +- internal/sink/circuit_breaker_test.go | 372 +++++++++++++++++- internal/sink/git/prune_owned.go | 15 +- internal/sink/git/prune_target.go | 2 +- .../evidence/G1.md | 147 +++++++ .../sink-breaker-terminal-isolation/loop.md | 3 + .../reviews/L-tasks/G1/brief-reviewer-a.md | 63 +++ .../reviews/L-tasks/G1/leg-reviewer-a-diff.md | 10 + .../sink-breaker-terminal-isolation/tasks.md | 6 +- 11 files changed, 688 insertions(+), 14 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/evidence/G1.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G1/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G1/leg-reviewer-a-diff.md diff --git a/internal/errors/errors.go b/internal/errors/errors.go index 5ec2b00e..14a4dca3 100644 --- a/internal/errors/errors.go +++ b/internal/errors/errors.go @@ -147,3 +147,32 @@ func Format(class error, format string, args ...any) error { return &ClassError{Class: class, Err: msg} } + +// ErrInventoryScoped marks an export failure that is about one inventory (its own paths or its +// ownership claim), not about the sink. The per-sink circuit breaker does not count it. +var ErrInventoryScoped = errors.New("inventory-scoped") + +// InventoryScoped marks err as Terminal and inventory-scoped. The result keeps err's text and +// its errors.Is/errors.As chain, so status messages and logs do not change. +func InventoryScoped(err error) error { + if err == nil { + return nil + } + + return Terminal(&inventoryScopedError{err: err}) +} + +// IsInventoryScoped reports whether err carries the inventory-scoped marker. +func IsInventoryScoped(err error) bool { + return errors.Is(err, ErrInventoryScoped) +} + +type inventoryScopedError struct { + err error +} + +func (e *inventoryScopedError) Error() string { return e.err.Error() } + +func (e *inventoryScopedError) Unwrap() error { return e.err } + +func (e *inventoryScopedError) Is(target error) bool { return target == ErrInventoryScoped } diff --git a/internal/errors/errors_test.go b/internal/errors/errors_test.go index 92cbec86..238497b6 100644 --- a/internal/errors/errors_test.go +++ b/internal/errors/errors_test.go @@ -144,3 +144,41 @@ func TestClassifyAPIBranches(t *testing.T) { t.Fatal("ClassError with nil Class should default to transient") } } + +// TestInventoryScoped (M-1): the marker keeps the Terminal class, the text and the wrapped chain, +// and a plain Terminal is not inventory-scoped. +func TestInventoryScoped(t *testing.T) { + t.Parallel() + + base := errors.New("owner record") + inner := fmt.Errorf("prune path %q belongs to another inventory: %w", "a.yaml", base) + marked := InventoryScoped(inner) + + if !IsTerminal(marked) { + t.Fatalf("ClassOf(InventoryScoped(e)) = %q, want %q", ClassOf(marked), ClassTerminal) + } + if !IsInventoryScoped(marked) || !errors.Is(marked, ErrInventoryScoped) { + t.Fatalf("InventoryScoped(e) is not inventory-scoped: %v", marked) + } + if !errors.Is(marked, inner) || !errors.Is(marked, base) { + t.Fatal("InventoryScoped(e) dropped the wrapped chain") + } + if marked.Error() != inner.Error() { + t.Fatalf("InventoryScoped(e).Error() = %q, want %q", marked.Error(), inner.Error()) + } + notFound := apierrors.NewNotFound(schema.GroupResource{Resource: "secrets"}, "x") + var status *apierrors.StatusError + if !errors.As(InventoryScoped(notFound), &status) { + t.Fatal("errors.As through InventoryScoped did not reach the wrapped StatusError") + } + if InventoryScoped(nil) != nil { + t.Fatal("InventoryScoped(nil) != nil") + } + if IsInventoryScoped(Terminal(inner)) || IsInventoryScoped(nil) { + t.Fatal("a plain Terminal (or nil) reports inventory-scoped") + } + wrapped := fmt.Errorf("export to %q: %w", "sink", marked) + if !IsInventoryScoped(wrapped) || !IsTerminal(wrapped) { + t.Fatal("a %w re-wrap lost the marker or the class") + } +} diff --git a/internal/sink/circuit_breaker.go b/internal/sink/circuit_breaker.go index ca1a1197..abe1cee0 100644 --- a/internal/sink/circuit_breaker.go +++ b/internal/sink/circuit_breaker.go @@ -48,18 +48,25 @@ func breakerFor(key string) *gobreaker.CircuitBreaker[struct{}] { return existing.(*gobreaker.CircuitBreaker[struct{}]) } - br := gobreaker.NewCircuitBreaker[struct{}](gobreaker.Settings{ + actual, _ := breakerRegistry.LoadOrStore(key, newBreaker(key, circuitBreakerTimeout)) + return actual.(*gobreaker.CircuitBreaker[struct{}]) +} + +// newBreaker holds every breaker setting. The timeout is a parameter so tests can build a +// short-timeout breaker under their own key; production always passes circuitBreakerTimeout. +func newBreaker(key string, timeout time.Duration) *gobreaker.CircuitBreaker[struct{}] { + return gobreaker.NewCircuitBreaker[struct{}](gobreaker.Settings{ Name: key, MaxRequests: 1, Interval: circuitBreakerInterval, - Timeout: circuitBreakerTimeout, + Timeout: timeout, ReadyToTrip: func(counts gobreaker.Counts) bool { return counts.ConsecutiveFailures >= circuitBreakerTripAt }, + // A failure about one inventory (an ownership conflict) neither counts nor resets the + // streak, so it cannot open the breaker for every other inventory on the sink. + IsExcluded: kollecterrors.IsInventoryScoped, }) - - actual, _ := breakerRegistry.LoadOrStore(key, br) - return actual.(*gobreaker.CircuitBreaker[struct{}]) } // ResetBreakersForTest clears per-sink circuit breakers between tests. It diff --git a/internal/sink/circuit_breaker_test.go b/internal/sink/circuit_breaker_test.go index 9e6ba9bb..61eab42b 100644 --- a/internal/sink/circuit_breaker_test.go +++ b/internal/sink/circuit_breaker_test.go @@ -4,13 +4,20 @@ package sink import ( + "context" "errors" "fmt" + "strings" "sync" + "sync/atomic" "testing" + "time" + "github.com/sony/gobreaker/v2" + apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" "sigs.k8s.io/controller-runtime/pkg/client/fake" kollectdevv1alpha1 "github.com/platformrelay/kollect/api/v1alpha1" @@ -18,6 +25,7 @@ import ( kollecterrors "github.com/platformrelay/kollect/internal/errors" "github.com/platformrelay/kollect/internal/export" "github.com/platformrelay/kollect/internal/sink/cap" + "github.com/platformrelay/kollect/internal/sink/git" ) // The breaker pair drives the live RunExportEnvelope path — production calls @@ -92,8 +100,9 @@ func TestRunExportEnvelope_circuitBreakerTripsAfterRepeatedFailures(t *testing.T } } +// Not t.Parallel: ResetBreakersForTest clears every key, including the breakers of parallel tests +// mid-trip. func TestResetBreakersForTest_clearsOpenBreaker(t *testing.T) { - t.Parallel() const ( sinkNamespace = "cb-reset-ns" @@ -184,3 +193,364 @@ func TestResetBreakersForTest_safeAgainstConcurrentExport(t *testing.T) { } wg.Wait() } + +// breakerKeySeq makes breaker keys unique per test invocation: -count=N reruns a test against the +// process-wide registry its earlier run left behind. +var breakerKeySeq atomic.Uint64 + +func uniqueBreakerName(t *testing.T) string { + t.Helper() + + return fmt.Sprintf("%s-%d", strings.NewReplacer("/", "-", "_", "-").Replace(strings.ToLower(t.Name())), breakerKeySeq.Add(1)) +} + +func uniqueBreakerKey(t *testing.T) string { + t.Helper() + + return "cb-unit/" + uniqueBreakerName(t) +} + +var errInventoryConflict = errors.New("prune path \"a.yaml\" belongs to another inventory") + +// callBreaker runs one export through the breaker and reports whether fn ran. +func callBreaker(key string, result error) (bool, error) { + ran := false + err := exportThroughBreaker(key, func() error { + ran = true + + return result + }) + + return ran, err +} + +func requireOpenBreakerError(t *testing.T, err error) { + t.Helper() + if err == nil || !kollecterrors.IsTransient(err) || !strings.Contains(err.Error(), "circuit breaker open") { + t.Fatalf("err = %v (class %s), want a transient error containing %q", err, kollecterrors.ClassOf(err), "circuit breaker open") + } +} + +// pathStubBackend fails or succeeds per object path, and counts the calls that reached it. The +// backend sees the layout's object path, so both match on a substring (the inventory name). +type pathStubBackend struct { + mu sync.Mutex + fail map[string]error + calls map[string]int +} + +func (s *pathStubBackend) Type() string { return "stub" } + +func (s *pathStubBackend) Capabilities() cap.Capabilities { return cap.SnapshotStore() } + +func (s *pathStubBackend) Export(_ context.Context, _ []byte, p string) error { + s.mu.Lock() + defer s.mu.Unlock() + if s.calls == nil { + s.calls = map[string]int{} + } + s.calls[p]++ + for substr, err := range s.fail { + if strings.Contains(p, substr) { + return err + } + } + + return nil +} + +func (s *pathStubBackend) callCount(substr string) int { + s.mu.Lock() + defer s.mu.Unlock() + n := 0 + for p, c := range s.calls { + if strings.Contains(p, substr) { + n += c + } + } + + return n +} + +// stubExportRequest builds a RunExportEnvelope request for a stub sink named per invocation. +func stubExportRequest(t *testing.T, backend Backend, objectPath string) ExportEnvelopeRequest { + t.Helper() + const sinkNamespace = "cb-boundary-ns" + sinkName := uniqueBreakerName(t) + + scheme := runtime.NewScheme() + if err := kollectdevv1alpha1.AddToScheme(scheme); err != nil { + t.Fatal(err) + } + sinkObj := &kollectdevv1alpha1.KollectSnapshotSink{ + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: sinkNamespace}, + Spec: kollectdevv1alpha1.KollectSnapshotSinkSpec{ + Type: "stub", + SinkCommonFields: kollectdevv1alpha1.SinkCommonFields{Endpoint: "https://example.com/repo.git"}, + }, + } + reg := NewRegistry() + reg.Register("stub", func(_ kollectdevv1alpha1.KollectSinkSpec, _ BuildContext) (Backend, error) { + return backend, nil + }) + envelope, err := export.MarshalEnvelope([]collect.Item{{Name: "demo"}}, export.Metadata{Generation: 1}) + if err != nil { + t.Fatal(err) + } + + return ExportEnvelopeRequest{ + Ctx: t.Context(), + Client: fake.NewClientBuilder().WithScheme(scheme).WithObjects(sinkObj).Build(), + Registry: reg, + SinkNamespace: sinkNamespace, + SinkName: sinkName, + ObjectPath: objectPath, + Envelope: envelope, + SinkSpec: kollectdevv1alpha1.KollectSinkSpec{Type: "stub", Endpoint: "https://example.com/repo.git"}, + } +} + +// SCB-1a: inventory-scoped results do not count as failures. +func TestExportThroughBreaker_inventoryScopedResultsDoNotTrip(t *testing.T) { + t.Parallel() + key := uniqueBreakerKey(t) + + for i := range 10 { + if _, err := callBreaker(key, kollecterrors.InventoryScoped(errInventoryConflict)); !errors.Is(err, errInventoryConflict) { + t.Fatalf("call %d: err = %v, want the inventory-scoped result itself", i+1, err) + } + } + ran, err := callBreaker(key, nil) + if !ran || err != nil { + t.Fatalf("11th call after 10 inventory-scoped results: ran = %v, err = %v; want ran, nil", ran, err) + } +} + +// SCB-1b: one inventory's inventory-scoped rejections do not block another inventory on the same +// sink, through RunExportEnvelope. +func TestRunExportEnvelope_inventoryScopedRejectionsSpareOtherInventory(t *testing.T) { + t.Parallel() + backend := &pathStubBackend{fail: map[string]error{"inv-a": kollecterrors.InventoryScoped(errInventoryConflict)}} + reqA := stubExportRequest(t, backend, "cb-boundary-ns/inv-a.json") + reqB := reqA + reqB.ObjectPath = "cb-boundary-ns/inv-b.json" + + var errA error + for i := range 10 { + if _, errA = RunExportEnvelope(reqA); errA == nil { + t.Fatalf("export A #%d succeeded, want a rejection", i+1) + } + } + if _, err := RunExportEnvelope(reqB); err != nil { + t.Fatalf("export B after 10 inventory-scoped rejections of A: %v", err) + } + if !kollecterrors.IsTerminal(errA) || !kollecterrors.IsInventoryScoped(errA) || !errors.Is(errA, errInventoryConflict) { + t.Fatalf("export A's 10th error = %v (class %s), want the terminal inventory-scoped rejection", errA, kollecterrors.ClassOf(errA)) + } + if got := backend.callCount("inv-a"); got != 10 { + t.Fatalf("export A reached the backend %d times, want 10", got) + } + if got := backend.callCount("inv-b"); got != 1 { + t.Fatalf("export B reached the backend %d times, want 1", got) + } +} + +// SCB-1c: an excluded result neither counts nor resets the consecutive-failure streak. +func TestExportThroughBreaker_inventoryScopedKeepsFailureStreak(t *testing.T) { + t.Parallel() + key := uniqueBreakerKey(t) + transient := kollecterrors.Transient(errors.New("network down")) + + for i, result := range []error{transient, kollecterrors.InventoryScoped(errInventoryConflict), transient, transient, transient} { + if ran, _ := callBreaker(key, result); !ran { + t.Fatalf("call %d did not run", i+1) + } + } + ran, _ := callBreaker(key, transient) + if !ran { + t.Fatal("sixth call (fourth counted failure) did not run: the inventory-scoped result counted") + } + ran, err := callBreaker(key, nil) + if ran { + t.Fatal("seventh call ran: the breaker did not open on the fifth counted failure") + } + requireOpenBreakerError(t, err) +} + +// SCB-2a/2b/2c/2f (controls): every unmarked failure counts, whatever its class. +func TestExportThroughBreaker_unmarkedFailuresTrip(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + name string + err error + }{ + {"transient", kollecterrors.Transient(errors.New("network down"))}, + {"unclassified", errors.New("boom")}, + {"forbidden", kollecterrors.Forbidden(errors.New("denied"))}, + {"unwrapped-notfound", apierrors.NewNotFound(schema.GroupResource{Resource: "secrets"}, "x")}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + key := uniqueBreakerKey(t) + for i := range circuitBreakerTripAt { + if ran, _ := callBreaker(key, tc.err); !ran { + t.Fatalf("call %d did not run before the trip", i+1) + } + } + ran, err := callBreaker(key, nil) + if ran { + t.Fatalf("sixth call ran after %d %s failures", circuitBreakerTripAt, tc.name) + } + requireOpenBreakerError(t, err) + }) + } +} + +// SCB-2d (control): a sink-wide Terminal from the Git classifier still opens the breaker. +func TestRunExportEnvelope_sinkWideTerminalTrips(t *testing.T) { + t.Parallel() + authErr := git.ClassifyExportError(errors.New("authentication required")) + if !kollecterrors.IsTerminal(authErr) { + t.Fatalf("precondition: git auth error class = %s, want terminal", kollecterrors.ClassOf(authErr)) + } + backend := &pathStubBackend{fail: map[string]error{"inv": authErr}} + req := stubExportRequest(t, backend, "cb-boundary-ns/inv.json") + + for range circuitBreakerTripAt { + _, _ = RunExportEnvelope(req) + } + _, err := RunExportEnvelope(req) + if got := backend.callCount("inv"); got != int(circuitBreakerTripAt) { + t.Fatalf("backend reached %d times, want %d (sixth export must not reach it)", got, circuitBreakerTripAt) + } + requireOpenBreakerError(t, err) +} + +// halfOpenBreaker stores a 1 s breaker under a fresh key, opens it with Transient failures and +// sleeps once past its timeout (design.md timing protocol). Later steps must not sleep. +func halfOpenBreaker(t *testing.T) string { + t.Helper() + key := uniqueBreakerKey(t) + if _, loaded := breakerRegistry.LoadOrStore(key, newBreaker(key, time.Second)); loaded { + t.Fatalf("breaker key %q already in use", key) + } + for range circuitBreakerTripAt { + _, _ = callBreaker(key, kollecterrors.Transient(errors.New("network down"))) + } + if st := breakerFor(key).State(); st != gobreaker.StateOpen { + t.Fatalf("state after %d transient failures = %s, want open", circuitBreakerTripAt, st) + } + time.Sleep(1100 * time.Millisecond) + + return key +} + +// SCB-3a: an inventory-scoped trial leaves the breaker half-open; the next success closes it. +func TestExportThroughBreaker_halfOpenInventoryScopedThenSuccess(t *testing.T) { + t.Parallel() + key := halfOpenBreaker(t) + + if ran, _ := callBreaker(key, kollecterrors.InventoryScoped(errInventoryConflict)); !ran { + t.Fatal("trial did not run after the timeout") + } + if st := breakerFor(key).State(); st != gobreaker.StateHalfOpen { + t.Fatalf("state after an inventory-scoped trial = %s, want half-open", st) + } + ran, err := callBreaker(key, nil) + if !ran || err != nil { + t.Fatalf("next trial: ran = %v, err = %v; want ran, nil", ran, err) + } + if st := breakerFor(key).State(); st != gobreaker.StateClosed { + t.Fatalf("state after a successful trial = %s, want closed", st) + } +} + +// SCB-3b: an inventory-scoped trial, then a counted failure, re-opens the breaker. +func TestExportThroughBreaker_halfOpenInventoryScopedThenFailure(t *testing.T) { + t.Parallel() + key := halfOpenBreaker(t) + + if ran, _ := callBreaker(key, kollecterrors.InventoryScoped(errInventoryConflict)); !ran { + t.Fatal("trial did not run after the timeout") + } + ran, _ := callBreaker(key, kollecterrors.Transient(errors.New("network down"))) + if !ran { + t.Fatal("the export after an inventory-scoped trial was not admitted as the trial") + } + if st := breakerFor(key).State(); st != gobreaker.StateOpen { + t.Fatalf("state after a failing trial = %s, want open", st) + } + if ran, err := callBreaker(key, nil); ran { + t.Fatal("a call ran on the re-opened breaker") + } else { + requireOpenBreakerError(t, err) + } +} + +// SCB-3c: a concurrent export does not run while a trial is in flight; an inventory-scoped trial +// frees the slot for the next export. +func TestExportThroughBreaker_halfOpenConcurrentTrial(t *testing.T) { + t.Parallel() + key := halfOpenBreaker(t) + + started, release := make(chan struct{}), make(chan struct{}) + done := make(chan error, 1) + go func() { + done <- exportThroughBreaker(key, func() error { + close(started) + <-release + + return kollecterrors.InventoryScoped(errInventoryConflict) + }) + }() + <-started + ran, err := callBreaker(key, nil) + if ran || err == nil { + t.Fatalf("concurrent call during the trial: ran = %v, err = %v; want not run, an error", ran, err) + } + close(release) + if trialErr := <-done; !errors.Is(trialErr, errInventoryConflict) { + t.Fatalf("trial err = %v, want the inventory-scoped result", trialErr) + } + if ran, err := callBreaker(key, nil); !ran || err != nil { + t.Fatalf("export after an inventory-scoped trial: ran = %v, err = %v; want ran, nil", ran, err) + } +} + +// SCB-1e: a real Git ownership rejection reaches the breaker inventory-scoped, and does not block a +// different inventory on the same sink. Not parallel: newIdentityRemote resets every breaker. +func TestRunExportEnvelope_gitOwnershipRejectionSparesOtherInventory(t *testing.T) { + r := newIdentityRemote(t) + sinkName := uniqueBreakerName(t) + exportTo := func(id InventoryIdentity, objectPath string, items []collect.Item) error { + t.Helper() + envelope, err := export.MarshalEnvelope(items, export.Metadata{Generation: 1, ExportedAt: time.Now().UTC()}) + if err != nil { + t.Fatalf("marshal envelope: %v", err) + } + _, err = RunExportEnvelope(ExportEnvelopeRequest{ + Ctx: t.Context(), Registry: NewRegistry(), SinkNamespace: "default", SinkName: sinkName, + ObjectPath: objectPath, Inventory: id, Envelope: envelope, SinkSpec: r.spec, + }) + + return err + } + + if err := exportTo(clusterPlatform, platformObjectPath, []collect.Item{identityItem("team-a", "Deployment", "api", "L")}); err != nil { + t.Fatalf("owner export: %v", err) + } + for i := range circuitBreakerTripAt + 1 { + err := exportTo(namespacedPlatform, platformObjectPath, []collect.Item{identityItem("team-a", "Deployment", "api", "N")}) + if !kollecterrors.IsTerminal(err) || !kollecterrors.IsInventoryScoped(err) || !strings.Contains(err.Error(), "belongs to another inventory") { + t.Fatalf("ownership rejection #%d: err = %v (class %s, inventory-scoped %v), want a terminal inventory-scoped rejection", + i+1, err, kollecterrors.ClassOf(err), kollecterrors.IsInventoryScoped(err)) + } + } + other := InventoryIdentity{Kind: InventoryKindNamespaced, Namespace: "team-b", Name: "other"} + if err := exportTo(other, "inventory/team-b/other.json", []collect.Item{identityItem("team-b", "ConfigMap", "settings", "B")}); err != nil { + t.Fatalf("other inventory's export after %d ownership rejections: %v", circuitBreakerTripAt+1, err) + } + if _, ok := r.files()["default/team-b/configmap/settings.yaml"]; !ok { + t.Fatalf("other inventory's file missing; files %v", dataPaths(r.files())) + } +} diff --git a/internal/sink/git/prune_owned.go b/internal/sink/git/prune_owned.go index efbb26f8..ae6ddfc2 100644 --- a/internal/sink/git/prune_owned.go +++ b/internal/sink/git/prune_owned.go @@ -171,7 +171,7 @@ func prepareOwnedPrune(fs billy.Filesystem, cfg Config, written []string) (*owne return nil, nil } if len(cfg.PruneOwner) > 4096 { - return nil, invalidPrune("prune owner exceeds 4096 bytes") + return nil, ownershipConflict("prune owner exceeds 4096 bytes") } owners, records, total, err := loadPruneRecords(fs) if err != nil { @@ -255,17 +255,24 @@ func invalidPrune(format string, args ...any) error { return kollecterrors.Terminal(fmt.Errorf(format, args...)) } +// ownershipConflict is invalidPrune for a rejection about the exporting inventory's own paths or +// ownership claim. It is marked inventory-scoped so the per-sink circuit breaker does not count it; +// a rejection caused by shared prune metadata stays on invalidPrune and still counts. +func ownershipConflict(format string, args ...any) error { + return kollecterrors.InventoryScoped(fmt.Errorf(format, args...)) +} + func validateOwnedPrunePaths(fs billy.Filesystem, cfg Config, written []string, owners map[string]string) (map[string]struct{}, error) { keep := pruneKeepSet(cfg, written) if len(keep) > maxPrunePaths { - return nil, invalidPrune("prune inventory exceeds %d paths", maxPrunePaths) + return nil, ownershipConflict("prune inventory exceeds %d paths", maxPrunePaths) } current := pathSet(keep) // Written paths must be present in an explicit union; otherwise the ownership // record would silently omit data just written by this export. for _, p := range written { if _, ok := current[p]; !ok { - return nil, invalidPrune("written path %q missing from prune keep set", p) + return nil, ownershipConflict("written path %q missing from prune keep set", p) } } for p := range current { @@ -307,7 +314,7 @@ func checkForeignClaim(cfg Config, owners map[string]string, p string) error { } func foreignClaimError(p, owner string) error { - return invalidPrune("prune path %q belongs to another inventory: %s, ownership record %s", + return ownershipConflict("prune path %q belongs to another inventory: %s, ownership record %s", p, describePruneOwner(owner), pruneRecordPath(owner)) } diff --git a/internal/sink/git/prune_target.go b/internal/sink/git/prune_target.go index 86a7508a..71e88bea 100644 --- a/internal/sink/git/prune_target.go +++ b/internal/sink/git/prune_target.go @@ -63,7 +63,7 @@ func checkMergeTargetClaims(repo *git.Repository, targetRef plumbing.ReferenceNa continue } - return invalidPrune("prune path %q belongs to another inventory on merge target branch %q: %s, ownership record %s", + return ownershipConflict("prune path %q belongs to another inventory on merge target branch %q: %s, ownership record %s", p, targetBranch, describePruneOwner(owner), pruneRecordPath(owner)) } diff --git a/openspec/changes/sink-breaker-terminal-isolation/evidence/G1.md b/openspec/changes/sink-breaker-terminal-isolation/evidence/G1.md new file mode 100644 index 00000000..c7676ab6 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/evidence/G1.md @@ -0,0 +1,147 @@ +# G1 — inventory-scoped results are excluded from the sink breaker (tasks 1.1 + 1.2) + +- Change: `openspec/changes/sink-breaker-terminal-isolation/` · branch: `fix/sink-breaker-terminal-isolation` · pre-task sha: `7f8c88804` · started: 2026-11-05 +- Satisfies: SCB-1, SCB-2, SCB-3 (`specs/sink-circuit-breaker/spec.md`) · design: `design.md` *Marker*, *Behaviour*, *Testing approach* · decision `agent-context/decisions.md` 2026-11-05 `sink-breaker-terminal` (option B) +- Smallest coherent change: an `InventoryScoped` marker in `internal/errors`, the 5 inventory-scoped sites from T0.1 table (a) routed through `ownershipConflict`, and `IsExcluded: kollecterrors.IsInventoryScoped` on the breaker. +- Files in scope: `internal/errors/errors.go`, `internal/errors/errors_test.go`, `internal/sink/circuit_breaker.go`, `internal/sink/circuit_breaker_test.go`, `internal/sink/git/prune_owned.go`, `internal/sink/git/prune_target.go` +- Status: CLOSED +- Next action: dispatch G2 (2.1 + 2.2). + +## Verification matrix + +All rows verified 2026-11-05 on the final tree unless marked otherwise. + +| # | Claim / invariant | Sensor | Class | Result | +|---|---|---|---|---| +| 1 | Refactor base: `newBreaker(key, timeout)` holds today's settings; existing tests green on it | `go test -count=1 ./internal/errors/ ./internal/sink/` on the base (exit 0) | machine | pass | +| 2 | M-1: `InventoryScoped(e)` is Terminal, inventory-scoped, keeps `errors.Is`/`errors.As` chain and text; nil → nil; plain Terminal not marked; survives a `%w` re-wrap | `TestInventoryScoped` (`internal/errors`) | machine | pass | +| 3 | SCB-1a: 10 marked results, then an 11th call runs and returns nil | `TestExportThroughBreaker_inventoryScopedResultsDoNotTrip` | machine | pass | +| 4 | SCB-1b: inventory A rejected 10× inventory-scoped; B's export on the same sink reaches the backend and succeeds (`RunExportEnvelope`, path-branching stub) | `TestRunExportEnvelope_inventoryScopedRejectionsSpareOtherInventory` | machine | pass | +| 5 | SCB-1c: T, X, T, T, T → 6th runs; fails T → 7th does not run, Transient "circuit breaker open" | `TestExportThroughBreaker_inventoryScopedKeepsFailureStreak` | machine | pass | +| 6 | SCB-1e: real Git ownership rejection via `RunExportEnvelope` is Terminal + inventory-scoped 6×; another inventory's export on the same sink then runs and writes its file | `TestRunExportEnvelope_gitOwnershipRejectionSparesOtherInventory` (non-parallel, `identityRemote`, per-invocation sink name) | machine | pass | +| 7 | SCB-2a/2b/2c/2f (controls): 5 Transient / unclassified / Forbidden / unwrapped NotFound failures open the breaker | `TestExportThroughBreaker_unmarkedFailuresTrip/{transient,unclassified,forbidden,unwrapped-notfound}` | machine | pass | +| 8 | SCB-2d (control): Git auth Terminal (`git.ClassifyExportError`) via `RunExportEnvelope`: the 6th export does not reach the backend | `TestRunExportEnvelope_sinkWideTerminalTrips` | machine | pass | +| 9 | SCB-3a: marked trial → half-open; next success ran → closed | `TestExportThroughBreaker_halfOpenInventoryScopedThenSuccess` | machine | pass | +| 10 | SCB-3b: marked trial, next call admitted and fails T → open; next call does not run | `TestExportThroughBreaker_halfOpenInventoryScopedThenFailure` | machine | pass | +| 11 | SCB-3c: concurrent call during a trial does not run; after a marked trial the next call runs | `TestExportThroughBreaker_halfOpenConcurrentTrial` | machine | pass | +| 12 | Exactly the 5 inventory-scoped sites are routed (`prune_owned.go:174,261,268,310`, `prune_target.go:66`); the 7 undecided and 10 sink-wide sites stay on `invalidPrune` | `git diff 7f8c88804 -- internal/sink/git` (5 `-invalidPrune`/`+ownershipConflict` lines, no others); M6 for site 310 | independent-review (sites 174/261/268/66), machine (site 310) | pass | +| 13 | Isolation: every new test's key is unique per invocation (`uniqueBreakerKey`/`uniqueBreakerName`: test name plus a package-level atomic counter); no parallel test calls `ResetBreakersForTest`; `TestResetBreakersForTest_clearsOpenBreaker` non-parallel. The pre-existing parallel `TestRunExportEnvelope_circuitBreakerTripsAfterRepeatedFailures` keeps its fixed key (out of G1's scope; it only asserts a trip, which a leftover open breaker also satisfies) | uniqueness: structural, diff review (the `-count=3` race run is consistent with it but does not prove it); reset callers: `rg -n 'ResetBreakersForTest' internal/sink/circuit_breaker_test.go` (callers `:164` in the test at `:105` and `:184` in the test at `:176`, both non-parallel; SCB-1e at `:522` is non-parallel and resets only through `newIdentityRemote`'s cleanup) | independent-review | pass | +| 14 | Marker text unchanged: existing message assertions (e.g. `TestRunExportEnvelope_sharedPathRejectedNamesOwner`) still pass | `go test ./internal/sink/...` | machine | pass | +| 15 | No test relied on ownership rejections tripping the breaker | `rg -n 'circuit breaker open\|ErrOpenState' internal` (base 2026-11-05: only `circuit_breaker_test.go:88`, `circuit_breaker.go:39-40`); `go test ./internal/sink/...` green | machine | pass | +| 16 | Production timeout unchanged (30 s); no mutable package variable | `breakerFor` passes `circuitBreakerTimeout`; diff review | independent-review | pass | + +## Red on the refactor base + +Base = `7f8c88804` + `newBreaker` extraction + stubs (`InventoryScoped` returns its input, +`IsInventoryScoped` returns false). Command (2026-11-05): +`go test -count=1 -v -run 'TestExportThroughBreaker_|TestRunExportEnvelope_inventoryScoped|TestRunExportEnvelope_sinkWide|TestRunExportEnvelope_gitOwnership' ./internal/sink/` and +`go test -count=1 -v -run TestInventoryScoped ./internal/errors/`. + +| Case | Result on base | Failure line | +|---|---|---| +| M-1 | RED | `errors_test.go:158: ClassOf(InventoryScoped(e)) = "transient", want "terminal"` | +| SCB-1a | RED | `call 6: err = sink circuit breaker open for "cb-unit/…", want the inventory-scoped result itself` | +| SCB-1b | RED | `export B after 10 inventory-scoped rejections of A: … sink circuit breaker open for "cb-boundary-ns/…"` | +| SCB-1c | RED | `sixth call (fourth counted failure) did not run: the inventory-scoped result counted` | +| SCB-1e | RED | `ownership rejection #1: err = export to "…": prune path "default/team-a/deployment/api.yaml" belongs to another inventory: … (class terminal, inventory-scoped false)` | +| SCB-3a | RED | `state after an inventory-scoped trial = open, want half-open` | +| SCB-3b | RED | `the export after an inventory-scoped trial was not admitted as the trial` | +| SCB-3c | RED (concurrency half green = control) | `export after an inventory-scoped trial: ran = false, err = sink circuit breaker open …` | +| SCB-2a/2b/2c/2f | control, PASS | — | +| SCB-2d | control, PASS | — | + +## Mutations (1.2) + +Runner kept outside the repository (scratch copy of the worktree via rsync, each file restored from +the first saved copy between mutants). Judged by the test command's exit status; the kill line is +the target assertion in every case (no build failure). + +| Mutant | Command `-run` | Exit | Kill line | +|---|---|---|---| +| M0 no-op control (rename local `br` → `cb` in `exportThroughBreaker`) | all 9 new sink tests (13 `=== RUN`) | 0 (survives) | — | +| M1 drop `IsExcluded` | `TestExportThroughBreaker_inventoryScopedResultsDoNotTrip$` (SCB-1a) | 1 | `call 6: err = sink circuit breaker open …` | +| M2 `IsSuccessful` instead | SCB-1c / SCB-3a | 1 / 1 | `seventh call ran: the breaker did not open on the fifth counted failure` / `state after an inventory-scoped trial = closed, want half-open` | +| M3 `IsExcluded: kollecterrors.IsTerminal` | SCB-2d / SCB-2f | 1 / 1 | `backend reached 6 times, want 5` / `sixth call ran after 5 unwrapped-notfound failures` | +| M4 exclude every non-nil error | SCB-2a (`…/^transient$`) | 1 | `sixth call ran after 5 transient failures` | +| M5 marker drops `Unwrap` | `TestInventoryScoped$` (M-1) | 1 | `InventoryScoped(e) dropped the wrapped chain` | +| M6 site `prune_owned.go:310` reverted to `invalidPrune` | SCB-1e | 1 | `ownership rejection #1: … inventory-scoped false` | + +## Sensors run + +| Tier | Command | sha / tree | Exit | Seconds | Note | +|---|---|---|---|---|---| +| inner | `go build ./...`, `gofmt -l internal` | final working tree | 0 | <5 | no output | +| inner | `go test -count=1 ./internal/errors/ ./internal/sink/...` | final working tree | 0 | ~30 | all ok | +| task (fitness: race) | `go test -race -count=3 ./internal/errors/ ./internal/sink/...` | final working tree | 0 | 54 | 20 packages ok | +| task (fitness: race) | `go test -race -count=3 -skip TestControllers ./internal/controller/` | final working tree | 0 | 11 | ok; the four reconciler flakes (G2's target) did not reproduce in this run, 1 run only | +| task (fitness: arch) | `task lint` (golangci-lint v2 + go-arch-lint) | final working tree | 0 | 38 | `0 issues.`, arch `OK - No warnings found` | +| task (fitness: unit, FINAL) | `go test -count=1 -v ./internal/sink/ ./internal/controller/ -skip TestControllers` | final code tree (after review; only evidence changed since) | 0 | 13 | both packages ok, 618 `=== RUN` | +| milestone | mutations M0-M6 above | scratch copy of final tree | as listed | | | + +Race row note: the dispatch command includes `./internal/controller/` without `-skip TestControllers`; +there the Ginkgo suite refuses `-count=3` ("Ginkgo does not support using go test -count … Only +-count=1 is allowed") and the package reports FAIL in 1.3 s with no test failure. Run separately with +`-skip TestControllers` as T0.1 did (verified 2026-11-05). + +## Iterations + +1 · test SCB-1b/SCB-2d · TEST · stub matched `req.ObjectPath`, but the backend sees `objectstore.ObjectPath(spec, ns, name, gen)` (`export.go:179-181`), so it never failed; stub changed to match on the inventory-name substring → reds/controls as recorded. +2 · test SCB-1b · TEST · first base red was on the marker check (Transient class), not the cross-inventory assertion; reordered so B's export is asserted first and the marker on A's last error after it → red on target (`export B after 10 … circuit breaker open`). +3 · race row with controller · HARNESS · Ginkgo refuses `-count=3`; split as above (not a product red). + +## Known red + +None added; none cleared. + +## Test changes + +`TestResetBreakersForTest_clearsOpenBreaker` (pre-existing, not from a test task): `t.Parallel()` +removed, as tasks.md 1.2 and design.md *Isolation* rule (2) require. Assertions unchanged. + +## Harness gaps + +None on a MUST row. Sites 174/261/268 and prune_target.go:66 have no dedicated kill test (M6 covers +site 310 only, as tasks.md names); their routing is diff-reviewed (row 12). + +## Maintainability + +1. Locality: yes, only the six files in scope. +2. Change surface: ~40 product lines for the marker, helper, setting and 5 sites; tests ~370 lines, proportional to 11 cases. +3. Duplication: `stubExportRequest` repeats the request setup of the two existing breaker tests; not refactored to keep the existing tests unchanged. +4. Second way: `ownershipConflict` beside `invalidPrune` is the design's named helper; same shape, different class. +5. Coupling: `internal/sink` already imported `internal/errors`; no new edge. +6. Abstraction: marker built on `ClassError` (`Terminal(...)`), no bypass. +7. Debt for green: none; no special case in product code. +8. Legibility: yes, `newBreaker` holds all settings in one place. + +Easier: adding another inventory-scoped site is one call to `ownershipConflict`. Harder: none known; a re-wrap without `Unwrap` would drop the marker (covered by SCB-1e for the Git path). + +## Independent review + +Round 1, one leg (reviewer-a, read-only, `reviews/L-tasks/G1/leg-reviewer-a-diff.md`), no paid leg +(no trigger: no secrets, CI, infra or manifests; verdict not BLOCK). Brief: `reviews/L-tasks/G1/brief-reviewer-a.md`. + +```text +verdict: APPROVE +findings: + - [P2] evidence/G1.md:28: Qualify unique keys as applying to new tests; the existing parallel trip test retains its fixed key. +matrix_disputes: + - Row 13 — passing `-count=3` does not prove key uniqueness; the new tests establish it structurally. +could_not_check: + - Independent test and mutation reruns: Go build cache not writable in the read-only sandbox; reds and mutations reviewed statically. + - Unit fitness remains pending as specified. +summary: Both tasks are implemented correctly; exactly five sites are marked, sink-wide failures still count, and new half-open tests follow the prescribed protocol. +``` + +Disposition: P2 and the row-13 dispute verified (the existing test at `circuit_breaker_test.go:34` uses the +fixed key `cb-test-ns/cb-test-sink`) and fixed in the evidence wording (row 13 reworded, class +independent-review). Evidence-only, so no second round. + +## Verdict + +CLOSED. All rows machine or independent-review, pass; reviewer-a APPROVE. + +## Learnings + +- tooling: the controller package's Ginkgo suite rejects `-count>1`; every race/-count fitness command that names `./internal/controller/` must carry `-skip TestControllers` (loop.md fitness row already does for G2's 2.1 command, the dispatch race row did not). +- guidance: the snapshot stub backend sees `objectstore.ObjectPath(...)`, not `req.ObjectPath`; a path-branching stub matches on the inventory name. diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index decfb75d..5babfaa3 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -54,6 +54,7 @@ Holistic run at B: not yet run. | Task | Verdict | Review (legs, rounds, paid leg?) | Gaps / decision request | |---|---|---|---| | T0.1 | CLOSED | reviewer-a APPROVE (1 P2 fixed), 1 round, no paid leg (`reviews/L-tasks/T0.1/`) | none; 2 non-blocking spec wording corrections (25 not 26 `invalidPrune` call sites; proposal *Impact* misses `git/prune_target.go`), see `evidence/T0.1.md` *Spec corrections*. Race on base: 2/10 red | +| G1 (1.1 + 1.2) | CLOSED | reviewer-a APPROVE (1 P2 evidence wording fixed), 1 round, no paid leg (`reviews/L-tasks/G1/`) | none; race row split: Ginkgo `TestControllers` refuses `-count=3`, run with `-skip TestControllers` | ## Known red @@ -74,6 +75,8 @@ Tests from a test task changed by a later task (close-task `TEST` class); each o - T0.1: G1 marks exactly 5 sites (`prune_owned.go:174,261,268,310`, `prune_target.go:66`); 7 sites are undecided (shared helpers `prune_owned.go:53,57,77,80,83` called for own paths and shared metadata; capacity sites `:215,:230`) and stay unmarked. - T0.1: `identityRemote.export` (`internal/sink/inventory_identity_test.go:85`) resets every breaker before each export and uses the fixed key `default/identity-git`; SCB-1e must call `RunExportEnvelope` directly with a per-invocation sink name, non-parallel. - T0.1: gobreaker `Counts()` does not advance state; call `State()` before reading counts after a time step. +- G1: the controller package's Ginkgo suite refuses `-count>1` (package FAIL, no test failure); every `-count=3` command naming `./internal/controller/` needs `-skip TestControllers`. +- G1: a stub snapshot backend receives `objectstore.ObjectPath(...)`, not `req.ObjectPath`; branch a stub on the inventory name. Breaker test helpers `uniqueBreakerKey`/`uniqueBreakerName`/`stubExportRequest`/`halfOpenBreaker` live in `internal/sink/circuit_breaker_test.go` for G2 to reuse. ## Where a human should look first diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G1/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G1/brief-reviewer-a.md new file mode 100644 index 00000000..7d0d11ac --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G1/brief-reviewer-a.md @@ -0,0 +1,63 @@ +# Review brief: group G1 (tasks 1.1 + 1.2) + +You are reviewing one task group, independently of whoever implemented it. Read-only: do not edit, commit or +create files in the repository. Work from the repository root +(`/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation`). You may run +`go test` commands under a minute (they write only to the Go build cache); any scratch copy for a mutation +goes under `$TMPDIR`, never in the repository. + +## Inputs + +1. TASK: `openspec/changes/sink-breaker-terminal-isolation/tasks.md`, items 1.1 and 1.2 and the preamble + (read them). Spec: `specs/sink-circuit-breaker/spec.md` (SCB-1, SCB-2, SCB-3). Design: + `design.md` (*Marker*, *Behaviour*, *Testing approach*). The site classification it relies on: + `evidence/T0.1.md` table (a) and table (b). +2. MATRIX and all evidence: `openspec/changes/sink-breaker-terminal-isolation/evidence/G1.md`. +3. DIFF: base `7f8c88804`, staged (not yet committed): `git diff --cached 7f8c88804`. Product code: + `internal/errors/errors.go`, `internal/sink/circuit_breaker.go`, `internal/sink/git/prune_owned.go`, + `internal/sink/git/prune_target.go`; tests: `internal/errors/errors_test.go`, + `internal/sink/circuit_breaker_test.go`; plus evidence, tasks.md tick and loop.md. +4. SENSORS: recorded in the evidence (*Red on the refactor base*, *Mutations*, *Sensors run*). gobreaker + source: `$(go env GOMODCACHE)/github.com/sony/gobreaker/v2@v2.4.0`. The unit fitness row runs once + after review (pending, not missing). +5. RULES: `AGENTS.md`, `docs/development/spec-workflow.md`. The breaker registry is process-wide: a parallel + test must not call `ResetBreakersForTest`, and keys must be unique per test invocation (`-count` safe). + +## What matters + +The per-sink circuit breaker must stop counting failures that are about one inventory (ownership +conflicts) while still counting every sink-wide failure. Getting it wrong either keeps the cross-tenant +outage (marker lost or a site missed) or stops load-shedding for a dead sink (over-exclusion). + +Judge, in this order: + +a. INTENT: does the diff do all of 1.1 and 1.2 and nothing else? Exactly the 5 sites T0.1 table (a) marks + inventory-scoped are routed (`prune_owned.go:174,261,268,310`, `prune_target.go:66` on the base); no + undecided or sink-wide site is. The marker meets design.md *Marker* (Terminal, `errors.Is` to the + sentinel and to everything wrapped, text unchanged, nil for nil). `IsExcluded` is set in `newBreaker`; + production timeout unchanged; no mutable package variable. +b. MATRIX: each row's sensor exists and would fail if the claim were false. Check that each recorded base + red is on the target assertion. Mutation-verify any clause you doubt in a scratch copy. +c. RULES: test isolation (parallel tests vs `ResetBreakersForTest`, unique keys, half-open timing protocol + in design.md); committed evidence is Markdown only. +d. AI-code failure modes: tautological tests, weakened assertions, a dropped half of the task, prose that + claims more than the code does, flaky timing. + +For each P0/P1 you are about to file, argue the opposite for one line; keep only what survives. + +RETURN at most 15 lines, nothing else, under a heading `## Verdict`: + +```text +verdict: APPROVE | REQUEST_CHANGES | BLOCK +findings: + - [P0|P1|P2] +matrix_disputes: + - — +could_not_check: + - +summary: +``` + +Severity: P0 (security, data loss, invariant broken, vacuous gate presented as proof) → BLOCK. P1 +(correctness gap, untested new behaviour, dishonest evidence class, missing half of the task) → +REQUEST_CHANGES. P2/P3 → note only, APPROVE allowed. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G1/leg-reviewer-a-diff.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G1/leg-reviewer-a-diff.md new file mode 100644 index 00000000..ecc2a409 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G1/leg-reviewer-a-diff.md @@ -0,0 +1,10 @@ +## Verdict +verdict: APPROVE +findings: + - [P2] openspec/changes/sink-breaker-terminal-isolation/evidence/G1.md:28: Qualify unique keys as applying to new tests; the existing parallel trip test retains its fixed key. +matrix_disputes: + - Row 13 — passing `-count=3` does not prove key uniqueness; the new tests establish it structurally. +could_not_check: + - Independent test and mutation reruns: installed Go cannot create `/tmp/go-build…` under the read-only sandbox; recorded reds and mutation assertions were reviewed statically. + - Unit fitness remains pending as specified. +summary: Both tasks are implemented correctly; exactly five sites are marked, sink-wide failures still count, and new half-open tests follow the prescribed protocol. diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index 0e5664ee..727d7d34 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -43,7 +43,7 @@ Test rules for every group: ## 1. Inventory-scoped results are excluded from the breaker (SCB-1, SCB-2, SCB-3) -- [ ] 1.1 Refactor base first, in the same group: extract `newBreaker(key, timeout)` from `breakerFor` +- [x] 1.1 Refactor base first, in the same group: extract `newBreaker(key, timeout)` from `breakerFor` with today's settings (pure refactor, existing tests green), and add `ErrInventoryScoped`, `InventoryScoped` and `IsInventoryScoped` with no-op bodies (`InventoryScoped` returns its input). Then the tests (red on that base unless marked *control*): @@ -82,12 +82,12 @@ Test rules for every group: - SCB-3c: half-open breaker; a trial blocks on a channel; a second concurrent call does not run its function and returns an error; release the trial with a marked result; a third call runs · exposes a marked trial wedging the slot. *Control* for the concurrency half on base. -- [ ] 1.2 Implement the marker (`design.md` *Marker*), route exactly the 5 inventory-scoped sites from T0.1 +- [x] 1.2 Implement the marker (`design.md` *Marker*), route exactly the 5 inventory-scoped sites from T0.1 table (a) through the new helper (undecided sites stay unmarked), set `IsExcluded: kollecterrors.IsInventoryScoped` in `newBreaker`, and make `TestResetBreakersForTest_clearsOpenBreaker` non-parallel. Mutations, each killed by a named case: drop `IsExcluded` (SCB-1a); `IsSuccessful` instead (SCB-1c, SCB-3a); exclude `IsTerminal` (SCB-2d, SCB-2f); exclude every non-nil error (SCB-2a); marker drops `Unwrap` - (M-1); one routed site reverted to `invalidPrune` (SCB-1e). + (M-1); one routed site reverted to `invalidPrune` (SCB-1e). — 1.1 and 1.2 closed 2026-11-05, evidence: evidence/G1.md ## 2. Reconciler tests stop sharing a tripped breaker (RACE-COUNT-BREAKER-ISOLATION) From 13b5a54d106b9017d1ea21633bf31dade682efad Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 12:02:12 +0100 Subject: [PATCH 07/13] :white_check_mark: test(controller): isolate sink breaker per test invocation Controller tests shared the fixed sink name postgres-demo, so parallel siblings and -count reruns fed one process-wide circuit breaker key and could open it for a test that expects an export (3 of 10 race runs red on the previous commit). Each test now derives a per-invocation sink name from uniqueSinkName and builds its expected status name from it. After the change the race command shows no breaker failure in ten runs; one run failed on the unrelated debounce-metric delta test, so task 2.2 stays open on its ten-green clause pending a decision. --- .../cluster_inventory_finalizer_test.go | 15 +- .../kollectinventory_export_test.go | 27 ++- .../kollectinventory_finalizer_test.go | 20 +- .../evidence/G2.md | 184 ++++++++++++++++++ .../sink-breaker-terminal-isolation/loop.md | 3 + .../reviews/L-tasks/G2/brief-reviewer-a.md | 59 ++++++ .../reviews/L-tasks/G2/leg-reviewer-a-diff.md | 11 ++ .../sink-breaker-terminal-isolation/tasks.md | 4 +- 8 files changed, 302 insertions(+), 21 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/leg-reviewer-a-diff.md diff --git a/internal/controller/cluster_inventory_finalizer_test.go b/internal/controller/cluster_inventory_finalizer_test.go index 45fef997..c1200c23 100644 --- a/internal/controller/cluster_inventory_finalizer_test.go +++ b/internal/controller/cluster_inventory_finalizer_test.go @@ -27,6 +27,7 @@ import ( func TestKollectClusterInventoryReconciler_deleteExportsEmptyAndRemovesFinalizer(t *testing.T) { t.Parallel() + sinkName := uniqueSinkName(t, "postgres-demo") scheme := runtime.NewScheme() if err := kollectdevv1alpha1.AddToScheme(scheme); err != nil { @@ -37,7 +38,7 @@ func TestKollectClusterInventoryReconciler_deleteExportsEmptyAndRemovesFinalizer } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "kollect-system"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "kollect-system"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.DatabaseSinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -55,7 +56,7 @@ func TestKollectClusterInventoryReconciler_deleteExportsEmptyAndRemovesFinalizer DeletionTimestamp: &now, }, Spec: kollectdevv1alpha1.KollectClusterInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } @@ -113,6 +114,7 @@ func TestKollectClusterInventoryReconciler_deleteExportsEmptyAndRemovesFinalizer // 12-hour informer resync. func TestKollectClusterInventoryReconciler_terminalCleanupDoesNotRequeue(t *testing.T) { t.Parallel() + sinkName := uniqueSinkName(t, "postgres-demo") scheme := runtime.NewScheme() if err := kollectdevv1alpha1.AddToScheme(scheme); err != nil { @@ -123,7 +125,7 @@ func TestKollectClusterInventoryReconciler_terminalCleanupDoesNotRequeue(t *test } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "kollect-system"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "kollect-system"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.DatabaseSinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -141,7 +143,7 @@ func TestKollectClusterInventoryReconciler_terminalCleanupDoesNotRequeue(t *test DeletionTimestamp: &now, }, Spec: kollectdevv1alpha1.KollectClusterInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } @@ -277,6 +279,7 @@ func TestKollectClusterInventoryReconciler_deleteWithMissingSinkRemovesFinalizer // EC-P1-03 counterpart: transient cleanup failures keep the error-driven retry. func TestKollectClusterInventoryReconciler_transientCleanupStillRetries(t *testing.T) { t.Parallel() + sinkName := uniqueSinkName(t, "postgres-demo") scheme := runtime.NewScheme() if err := kollectdevv1alpha1.AddToScheme(scheme); err != nil { @@ -287,7 +290,7 @@ func TestKollectClusterInventoryReconciler_transientCleanupStillRetries(t *testi } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "kollect-system"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "kollect-system"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.DatabaseSinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -305,7 +308,7 @@ func TestKollectClusterInventoryReconciler_transientCleanupStillRetries(t *testi DeletionTimestamp: &now, }, Spec: kollectdevv1alpha1.KollectClusterInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } diff --git a/internal/controller/kollectinventory_export_test.go b/internal/controller/kollectinventory_export_test.go index b6d9f4fc..b2b47f7b 100644 --- a/internal/controller/kollectinventory_export_test.go +++ b/internal/controller/kollectinventory_export_test.go @@ -9,6 +9,7 @@ import ( "fmt" "strings" "sync" + "sync/atomic" "testing" corev1 "k8s.io/api/core/v1" @@ -238,6 +239,7 @@ func TestKollectInventoryReconciler_recordsLastExportPaths(t *testing.T) { func TestKollectInventoryReconciler_exportsDeploymentToSink(t *testing.T) { t.Parallel() + sinkName := uniqueSinkName(t, "postgres-demo") store := collect.NewStore() store.Upsert(collect.Item{ @@ -260,7 +262,7 @@ func TestKollectInventoryReconciler_exportsDeploymentToSink(t *testing.T) { } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "default"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "default"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.DatabaseSinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -273,7 +275,7 @@ func TestKollectInventoryReconciler_exportsDeploymentToSink(t *testing.T) { inv := &kollectdevv1alpha1.KollectInventory{ ObjectMeta: metav1.ObjectMeta{Name: "team-inventory", Namespace: "default"}, Spec: kollectdevv1alpha1.KollectInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } @@ -325,7 +327,7 @@ func TestKollectInventoryReconciler_exportsDeploymentToSink(t *testing.T) { if len(got.Status.SinkExports) != 1 { t.Fatalf("SinkExports = %d, want 1", len(got.Status.SinkExports)) } - wantName := string(kollectdevv1alpha1.SinkFamilyDatabase) + "/postgres-demo" + wantName := string(kollectdevv1alpha1.SinkFamilyDatabase) + "/" + sinkName if got.Status.SinkExports[0].Name != wantName { t.Fatalf("sink export name = %q, want %s", got.Status.SinkExports[0].Name, wantName) } @@ -334,10 +336,25 @@ func TestKollectInventoryReconciler_exportsDeploymentToSink(t *testing.T) { } } +// sinkNameSeq makes sink names unique per test invocation. The sink circuit +// breaker is process-wide and keyed by sink namespace/name, so tests that +// share a sink name share one breaker: parallel siblings, and -count=N reruns, +// add their failures to it and can open it for a test that expects an export. +var sinkNameSeq atomic.Uint64 + +// uniqueSinkName returns base with a per-invocation suffix, for a sink whose +// breaker no other test (or earlier run of this test) touches. +func uniqueSinkName(t *testing.T, base string) string { + t.Helper() + + return fmt.Sprintf("%s-%d", base, sinkNameSeq.Add(1)) +} + // newExportFailureReconciler builds a reconciler whose single postgres sink // always fails export with exportErr. func newExportFailureReconciler(t *testing.T, exportErr error) (*KollectInventoryReconciler, client.Client) { t.Helper() + sinkName := uniqueSinkName(t, "postgres-demo") store := collect.NewStore() store.Upsert(collect.Item{ @@ -359,7 +376,7 @@ func newExportFailureReconciler(t *testing.T, exportErr error) (*KollectInventor } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "default"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "default"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.DatabaseSinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -372,7 +389,7 @@ func newExportFailureReconciler(t *testing.T, exportErr error) (*KollectInventor inv := &kollectdevv1alpha1.KollectInventory{ ObjectMeta: metav1.ObjectMeta{Name: "team-inventory", Namespace: "default"}, Spec: kollectdevv1alpha1.KollectInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } diff --git a/internal/controller/kollectinventory_finalizer_test.go b/internal/controller/kollectinventory_finalizer_test.go index 61e9cd5e..d6b12ea1 100644 --- a/internal/controller/kollectinventory_finalizer_test.go +++ b/internal/controller/kollectinventory_finalizer_test.go @@ -103,6 +103,7 @@ func TestKollectInventoryReconciler_addsCleanupFinalizer(t *testing.T) { func TestKollectInventoryReconciler_deleteExportsEmptyAndRemovesFinalizer(t *testing.T) { t.Parallel() + sinkName := uniqueSinkName(t, "postgres-demo") store := collect.NewStore() store.Upsert(collect.Item{ @@ -124,7 +125,7 @@ func TestKollectInventoryReconciler_deleteExportsEmptyAndRemovesFinalizer(t *tes } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "default"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "default"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.SinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -143,7 +144,7 @@ func TestKollectInventoryReconciler_deleteExportsEmptyAndRemovesFinalizer(t *tes DeletionTimestamp: &now, }, Spec: kollectdevv1alpha1.KollectInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } @@ -200,6 +201,7 @@ func TestKollectInventoryReconciler_deleteExportsEmptyAndRemovesFinalizer(t *tes // result) and must keep the finalizer in place for manual intervention. func TestKollectInventoryReconciler_terminalCleanupDoesNotRequeue(t *testing.T) { t.Parallel() + sinkName := uniqueSinkName(t, "postgres-demo") scheme := runtime.NewScheme() if err := kollectdevv1alpha1.AddToScheme(scheme); err != nil { @@ -210,7 +212,7 @@ func TestKollectInventoryReconciler_terminalCleanupDoesNotRequeue(t *testing.T) } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "default"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "default"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.SinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -229,7 +231,7 @@ func TestKollectInventoryReconciler_terminalCleanupDoesNotRequeue(t *testing.T) DeletionTimestamp: &now, }, Spec: kollectdevv1alpha1.KollectInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } @@ -371,6 +373,7 @@ func TestKollectInventoryReconciler_deleteWithMissingSinkRemovesFinalizer(t *tes // EC-P1-03 counterpart: transient cleanup failures keep the error-driven retry. func TestKollectInventoryReconciler_transientCleanupStillRetries(t *testing.T) { t.Parallel() + sinkName := uniqueSinkName(t, "postgres-demo") scheme := runtime.NewScheme() if err := kollectdevv1alpha1.AddToScheme(scheme); err != nil { @@ -381,7 +384,7 @@ func TestKollectInventoryReconciler_transientCleanupStillRetries(t *testing.T) { } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "default"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "default"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.SinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -400,7 +403,7 @@ func TestKollectInventoryReconciler_transientCleanupStillRetries(t *testing.T) { DeletionTimestamp: &now, }, Spec: kollectdevv1alpha1.KollectInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } @@ -1053,6 +1056,7 @@ func TestKollectInventoryReconciler_terminalCleanupIncrementsMetric(t *testing.T // K-30 escape hatch: force-cleanup drops the finalizer without backend contact. func TestKollectInventoryReconciler_forceCleanupAnnotationDropsFinalizer(t *testing.T) { t.Parallel() + sinkName := uniqueSinkName(t, "postgres-demo") scheme := runtime.NewScheme() if err := kollectdevv1alpha1.AddToScheme(scheme); err != nil { @@ -1063,7 +1067,7 @@ func TestKollectInventoryReconciler_forceCleanupAnnotationDropsFinalizer(t *test } sinkObj := &kollectdevv1alpha1.KollectDatabaseSink{ - ObjectMeta: metav1.ObjectMeta{Name: "postgres-demo", Namespace: "default"}, + ObjectMeta: metav1.ObjectMeta{Name: sinkName, Namespace: "default"}, Spec: kollectdevv1alpha1.KollectDatabaseSinkSpec{ Type: kollectdevv1alpha1.SinkTypePostgres, Postgres: &kollectdevv1alpha1.PostgresSpec{ @@ -1084,7 +1088,7 @@ func TestKollectInventoryReconciler_forceCleanupAnnotationDropsFinalizer(t *test }, }, Spec: kollectdevv1alpha1.KollectInventorySpec{ - DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList("postgres-demo"), + DatabaseSinkRefs: kollectdevv1alpha1.NewSinkRefList(sinkName), }, } pgSecret := &corev1.Secret{ diff --git a/openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md b/openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md new file mode 100644 index 00000000..5e77f3d6 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md @@ -0,0 +1,184 @@ +# G2 — reconciler tests stop sharing a tripped breaker (tasks 2.1 + 2.2) + +- Change: `openspec/changes/sink-breaker-terminal-isolation/` · branch: `fix/sink-breaker-terminal-isolation` · pre-task sha: `0e7897dd1` (the G1 commit) · started: 2026-11-05 +- Satisfies: backlog RACE-COUNT-BREAKER-ISOLATION; tasks.md 2.1, 2.2 · design: `design.md` *Testing approach* (breaker registry is process-wide; keys unique per invocation) +- Smallest coherent change: a test helper `uniqueSinkName(t, base)` (package-level atomic counter) and every + controller test that used the fixed sink name `postgres-demo` builds its sink, its inventory ref and (where + asserted) its expected status name from that per-invocation name. Test-only; no production code. +- Files in scope: `internal/controller/kollectinventory_export_test.go`, `internal/controller/kollectinventory_finalizer_test.go`, `internal/controller/cluster_inventory_finalizer_test.go` +- Status: BLOCKED on 2.2's "ten green runs" clause (decision request below); 2.1 closed; 2.2's isolation committed +- Next action: orchestrator decides the *Decision request*; G3 (3.1) does not depend on it. + +## Verification matrix + +All rows verified 2026-11-05 unless marked otherwise. + +| # | Claim / invariant | Sensor | Class | Result | +|---|---|---|---|---| +| 1 | 2.1: on the G1 commit the race command is still red through the shared key; failing tests and key named | ten runs of `go test -race -count=3 -skip TestControllers ./internal/controller/` on `0e7897dd1` (*2.1* below) | machine | pass (3/10 red, key `default/postgres-demo`) | +| 2 | 2.2: no controller test uses the fixed sink name `postgres-demo` any more (default and `kollect-system` namespaces) | `rg -n '"postgres-demo"' internal/controller/` → only the 9 `uniqueSinkName(t, "postgres-demo")` lines | machine | pass | +| 3 | Per-invocation names are distinct across parallel siblings and `-count` reruns | `uniqueSinkName` = base + `sinkNameSeq.Add(1)` (process-wide atomic); M2 (constant suffix) killed | machine | pass | +| 4 | Isolation removes the shared-breaker red, deterministically | focused command (*Deterministic sensor*) green on the tree; red with the helper reverted to `base` (M1, = base behaviour) | machine | pass | +| 5 | Assertions unchanged: the one assertion that names the sink (`exportsDeploymentToSink`, `wantName`) is derived from the per-invocation name, not dropped; no other assertion or test body changed | `git diff 0e7897dd1 -- internal/controller` (only name lines, 9 declarations, `wantName`, helper, import) | independent-review | pass | +| 6 | Never a retry, never a sleep; no `ResetBreakersForTest` added | diff review; added lines of `git diff 0e7897dd1 -- internal/controller` matching ResetBreakersForTest, time.Sleep, Eventually or retry: 0 | independent-review | pass | +| 7 | 2.2: ten green runs of the 2.1 command afterwards | ten runs on the G2 tree (*2.2* below) | machine | **fail on the literal clause**: 9/10 green; run 9 red on `TestClusterInventory_DebouncedExportIncrementsMetric` (metric-delta flake, not the breaker). Zero breaker reds in 10 runs. See *Harness gaps* | +| 8 | Lint / architecture | `task lint` exit 0, `0 issues.`, go-arch-lint `OK - No warnings found` | machine | pass | +| 9 | Race fitness, other packages | `go test -race -count=3 ./internal/errors/ ./internal/sink/...`: every package `ok`, no FAIL or race line | machine | pass | +| 10 | Unit fitness (per task) | FINAL: `go test -v ./internal/sink/ ./internal/controller/ -skip TestControllers` on the final code tree | machine | pass (exit 0, 11 s, both packages `ok`, 618 `=== RUN`) | + +## 2.1 — base measurement on the G1 commit + +Command, ten runs, sequential, foreground, on `0e7897dd1`, 2026-11-05 (runner script outside the repository): +`go test -race -count=3 -skip TestControllers ./internal/controller/` + +| Run | Exit | Seconds | Failing tests (failures across the 3 iterations) | +|---|---|---|---| +| 1 | 0 | 9 | none | +| 2 | 0 | 11 | none | +| 3 | 0 | 10 | none | +| 4 | 1 | 10 | deleteExportsEmptyAndRemovesFinalizer ×1, exportsDeploymentToSink ×1, terminalCleanupDoesNotRequeue ×1, terminalTotalExportFailureDoesNotRequeue ×1 | +| 5 | 0 | 11 | none | +| 6 | 1 | 10 | the same four, ×2 each | +| 7 | 0 | 10 | none | +| 8 | 0 | 11 | none | +| 9 | 1 | 10 | deleteExportsEmptyAndRemovesFinalizer ×1, exportsDeploymentToSink ×1, **TestClusterInventory_DebouncedExportIncrementsMetric ×1** | +| 10 | 0 | 11 | none | + +Result: **3 of 10 red** (T0.1 on `origin/main`: 2 of 10). This is the group's red. The four named tests are +`TestKollectInventoryReconciler_*`; failure lines (runs 4, 6, 9): + +```text +kollectinventory_finalizer_test.go:180: Reconcile: export to "postgres-demo": sink circuit breaker open for "default/postgres-demo" +kollectinventory_export_test.go:312: export count = 0, want 1 +kollectinventory_export_test.go:418: RequeueAfter = 30s, want 0 (terminal export must not requeue) +kollectinventory_finalizer_test.go:269: Reconcile err = export to "postgres-demo": sink circuit breaker open for "default/postgres-demo", want nil (terminal cleanup must not requeue with backoff) +kollectclusterinventory_debounce_checksum_test.go:279: ExportDebouncedTotal{controller=KollectClusterInventory} delta = 2, want 1 +``` + +Shared key: `default/postgres-demo`. Counted failures on it come from `terminalTotalExportFailureDoesNotRequeue`, +`transientTotalExportFailureRequeues` (both via `newExportFailureReconciler`), `terminalCleanupDoesNotRequeue` and +`transientCleanupStillRetries` (backend returns unmarked Terminal or Transient, which G1 keeps counting by design). +`forceCleanupAnnotationDropsFinalizer` also used the key but never contacts the backend (it asserts zero backend +constructions), so it adds no failure; it is isolated anyway so that it never reads another test's breaker. The debounce-metric line is a different mechanism (a parallel +test reading the process-global `ExportDebouncedTotal` counter as a delta); not a breaker failure, out of G2's scope. + +## Deterministic sensor for the isolation + +The probabilistic ten-run loop cannot kill a mutant reliably, so the isolation has a focused, deterministic +command: the four default-namespace tests whose backend fails (plus `forceCleanupAnnotationDropsFinalizer`, which +adds no failure), three iterations, no success in between to reset the streak. On a shared key iteration 1 adds +4 counted failures and iteration 2's first one is the 5th, which opens the breaker; the later iteration-2 tests +then get the Transient "circuit breaker open" error. + +`go test -count=3 -skip TestControllers -run 'TestKollectInventoryReconciler_(terminalTotalExportFailureDoesNotRequeue|transientTotalExportFailureRequeues|terminalCleanupDoesNotRequeue|transientCleanupStillRetries|forceCleanupAnnotationDropsFinalizer)$' ./internal/controller/` + +- G2 tree: exit 0, 15 `--- PASS` lines (5 tests × 3) with `-v`. + +## Mutations + +Scratch copy of the worktree outside the repository (rsync, no `.git`), file restored from the first saved +copy between mutants; judged by the deterministic command's exit status. + +| Mutant | Exit | Kill line | +|---|---|---| +| none (control run) | 0 | — | +| M1 `uniqueSinkName` returns `base` (= base behaviour) | 1 | `kollectinventory_finalizer_test.go:271: Reconcile err = export to "postgres-demo": sink circuit breaker open for "default/postgres-demo", want nil` | +| M2 suffix from `sinkNameSeq.Load()` (constant per process) | 1 | `kollectinventory_export_test.go:435: RequeueAfter = 30s, want 0 (terminal export must not requeue)` | +| C0 no-op (comment edit), must survive | 0 | — | + +## 2.2 — ten runs after the change + +Same command as 2.1, on the G2 working tree (diff above `0e7897dd1`), 2026-11-05, sequential, foreground. + +| Run | Exit | Seconds | Failing tests | +|---|---|---|---| +| 1–8 | 0 each | 10–11 | none | +| 9 | 1 | 10 | TestClusterInventory_DebouncedExportIncrementsMetric ×1 (`delta = 2, want 1`) | +| 10 | 0 | 10 | none | + +Result: 9 of 10 green; **0 breaker failures in 10 runs** (base 3 of 10). The one red is the debounce-metric +flake also seen on the G1 base (run 9 of 2.1). + +## Sensors + +| Sensor | Exit | Seconds | +|---|---|---| +| `go vet ./internal/controller/` | 0 | ~5 | +| focused `-race -count=1 -run 'TestKollectInventoryReconciler_\|TestKollectClusterInventoryReconciler_'` | 0 | 2 | +| deterministic sensor (above) | 0 | ~5 | +| 2.1 ten runs (G1 commit) | 3 of 10 exit 1 | ~10 each | +| 2.2 ten runs (G2 tree) | 1 of 10 exit 1 (non-breaker) | ~10 each | +| `task lint` | 0 | 25 | +| `go test -race -count=3 ./internal/errors/ ./internal/sink/...` | all `ok` | — | +| FINAL `go test -v ./internal/sink/ ./internal/controller/ -skip TestControllers` | 0 | 11 | + +## Scope note + +`cluster_inventory_finalizer_test.go` (key `kollect-system/postgres-demo`, three parallel tests including a +Terminal and a Transient failure) is included: T0.1 named it as G2 scope, and it has the same shared-key defect. + +## Maintainability self-check + +1. New abstraction without a second caller? No: 9 callers. +2. Duplicated logic? The sink package has `uniqueBreakerName`; not importable (test-only, other package), so a 3-line controller-side helper. +3. Hidden global state? One package-level atomic counter, documented; it only grows. +4. Behaviour change outside the task? No production code changed. +5. Weakened assertion? No; `wantName` derives from the same name the sink was created with. +6. Naming unclear? `uniqueSinkName` states what it returns. +7. Dead code? None. +8. Test depends on order or timing? Less than before; the remaining flake is the metric delta (gap). + +## Harness gaps + +- G-1: `TestClusterInventory_DebouncedExportIncrementsMetric` (`kollectclusterinventory_debounce_checksum_test.go:210`) + is parallel and asserts a delta on the process-global `ExportDebouncedTotal{controller=KollectClusterInventory}`; + a concurrent debounced cluster export adds to it (`delta = 2, want 1`). Seen once in 10 runs on the G1 commit and + once in 10 after G2. Not the breaker; outside 2.2's text. Deferred: new backlog item (proposed name + RACE-COUNT-DEBOUNCE-METRIC; likely fix: make the test non-parallel, as `terminalCleanupIncrementsMetric` already + is). Because of it, 2.2's "ten green runs" holds for the breaker (0/10) but not for the whole command (9/10). + +## Iterations + +1 · 2.1 ten runs · RED (not a failure class) · 3/10 exit 1, `sink circuit breaker open for "default/postgres-demo"` +2 · 2.2 ten runs · ENVIRONMENT/other-test flake (not the breaker) · run 9 `delta = 2, want 1`; classified as a separate pre-existing flake (seen on base), recorded as G-1, not chased + +## Review + +One leg, reviewer-a (diff lens), round 1, on the staged diff above `0e7897dd1`; no paid leg (test-only diff, +no secrets/CI/infra/manifests, no BLOCK). Records: `reviews/L-tasks/G2/` (brief, leg file). +Verdict: **REQUEST_CHANGES**. + +| Finding | Verified | Disposition | +|---|---|---| +| P1 `tasks.md:103`: 9/10 green does not satisfy 2.2's explicit "ten green runs"; keep 2.2 open until satisfied or the clause is revised | yes (row 7) | accepted: 2.2 left unticked; *Decision request* | +| P2 `evidence/G2.md`: `forceCleanupAnnotationDropsFinalizer` adds no failure (asserts zero backend contact); breaker opens in iteration 2, not 1 | yes (`kollectinventory_finalizer_test.go:1135`) | fixed (wording in *2.1* and *Deterministic sensor*); no new round (evidence only) | + +Reviewer confirmed: references and assertions preserved; focused sensor passes; non-breaker classification of +the debounce red supported. + +## Verdict + +**BLOCKED** (2.2 clause), with 2.1 closed and the isolation committed. 2.1: CLOSED (row 1). 2.2: every row +`machine`/`independent-review` pass except row 7, which fails only on a different test's flake (G-1); fixing that +test is outside 2.2's named scope, so it is a decision, not a correction. + +## Decision request + +2.2 says "Ten green runs of the 2.1 command afterwards". After the isolation: 0 breaker reds in 10 runs (base 3/10), +1 red from `TestClusterInventory_DebouncedExportIncrementsMetric` (parallel test asserting a delta on a +process-global counter; seen on the G1 base too). Options: + +- A (recommended): widen 2.2 by one line: make `TestClusterInventory_DebouncedExportIncrementsMetric` non-parallel + (precedent: `TestKollectInventoryReconciler_terminalCleanupIncrementsMetric`), then rerun the ten runs and tick 2.2. + Same class of defect (shared process-global test state in the same race command), test-only, cheap. +- B: revise 2.2's clause to "ten runs without a breaker failure", tick 2.2 now, and file the metric flake as a new + backlog item (RACE-COUNT-DEBOUNCE-METRIC). + +## Learnings + +Class: sensor. + +- A ten-run race loop is probabilistic; pair it with a deterministic `-count=3 -run` over only the failing-backend + tests (no success in between) so mutants of the isolation are killed by exit status. +- The 2.1 race command has a second, unrelated flake (debounce-metric delta, ~1/10); "ten green runs" clauses on + that command should name the failure mechanism they target. diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index 5babfaa3..180e75a5 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -55,6 +55,7 @@ Holistic run at B: not yet run. |---|---|---|---| | T0.1 | CLOSED | reviewer-a APPROVE (1 P2 fixed), 1 round, no paid leg (`reviews/L-tasks/T0.1/`) | none; 2 non-blocking spec wording corrections (25 not 26 `invalidPrune` call sites; proposal *Impact* misses `git/prune_target.go`), see `evidence/T0.1.md` *Spec corrections*. Race on base: 2/10 red | | G1 (1.1 + 1.2) | CLOSED | reviewer-a APPROVE (1 P2 evidence wording fixed), 1 round, no paid leg (`reviews/L-tasks/G1/`) | none; race row split: Ginkgo `TestControllers` refuses `-count=3`, run with `-skip TestControllers` | +| G2 (2.1 + 2.2) | 2.1 CLOSED; 2.2 BLOCKED (decision) | reviewer-a REQUEST_CHANGES (P1: 9/10 green misses 2.2's "ten green runs"; P2 evidence wording fixed), 1 round, no paid leg (`reviews/L-tasks/G2/`) | isolation committed: 0 breaker reds in 10 runs (base 3/10); 1 red from `TestClusterInventory_DebouncedExportIncrementsMetric` (metric-delta flake, not the breaker). Decision: A widen 2.2 to make that test non-parallel (recommended) or B revise the clause and file a backlog item; `evidence/G2.md` *Decision request* | ## Known red @@ -77,6 +78,8 @@ Tests from a test task changed by a later task (close-task `TEST` class); each o - T0.1: gobreaker `Counts()` does not advance state; call `State()` before reading counts after a time step. - G1: the controller package's Ginkgo suite refuses `-count>1` (package FAIL, no test failure); every `-count=3` command naming `./internal/controller/` needs `-skip TestControllers`. - G1: a stub snapshot backend receives `objectstore.ObjectPath(...)`, not `req.ObjectPath`; branch a stub on the inventory name. Breaker test helpers `uniqueBreakerKey`/`uniqueBreakerName`/`stubExportRequest`/`halfOpenBreaker` live in `internal/sink/circuit_breaker_test.go` for G2 to reuse. +- G2: the ten-run race loop is probabilistic and cannot kill a mutant; a deterministic sensor for breaker sharing is `-count=3 -run` over only the tests whose backend fails (no success between iterations to reset the streak): 5 counted failures in iteration 1 open a shared breaker. +- G2: `TestClusterInventory_DebouncedExportIncrementsMetric` is a separate parallel metric-delta flake in the same race command (~1/10); a red there is not the breaker. ## Where a human should look first diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/brief-reviewer-a.md new file mode 100644 index 00000000..f555423d --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/brief-reviewer-a.md @@ -0,0 +1,59 @@ +# Review brief: group G2 (tasks 2.1 + 2.2) + +You are reviewing one task group, independently of whoever implemented it. Read-only: do not edit, commit or +create files in the repository. Work from the repository root +(`/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation`). You may run +`go test` commands under a minute (they write only to the Go build cache); any scratch copy for a mutation +goes under `$TMPDIR`, never in the repository. The Ginkgo suite in `internal/controller` refuses `-count>1`: +add `-skip TestControllers` to any `-count=N` command there. + +## Inputs + +1. TASK: `openspec/changes/sink-breaker-terminal-isolation/tasks.md`, items 2.1 and 2.2 and the preamble + (read them). Design: `design.md` *Testing approach*. Base measurement: `evidence/T0.1.md` + *RACE-COUNT-BREAKER-ISOLATION on base*. +2. MATRIX and all evidence: `openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md`. +3. DIFF: base `0e7897dd1`, staged (not yet committed): `git diff --cached 0e7897dd1`. Code: test files only, + `internal/controller/kollectinventory_export_test.go`, `internal/controller/kollectinventory_finalizer_test.go`, + `internal/controller/cluster_inventory_finalizer_test.go`; plus evidence, tasks.md tick and loop.md. +4. SENSORS: recorded in the evidence (*2.1*, *Deterministic sensor*, *Mutations*, *2.2*, *Sensors*). The unit + fitness row runs once after review (pending, not missing). +5. RULES: `AGENTS.md`, `docs/development/spec-workflow.md`. The breaker registry + (`internal/sink/circuit_breaker.go`) is process-wide, keyed by sink namespace/name + (`internal/sink/export.go:200`). No parallel test may call `ResetBreakersForTest`; never a retry or sleep. + +## What matters + +Controller tests must stop sharing one breaker key, without changing what they assert. Check in particular: +every test that used the fixed sink `postgres-demo` now uses a per-invocation name for the sink object AND +the inventory's ref (a mismatch would turn the test into a "sink not found" test that may still pass); the +one assertion naming the sink is derived, not dropped; nothing else in the test bodies changed; the +deterministic sensor and its mutants are real. Judge whether the debounce-metric red in 2.2's ten runs is +honestly classified (not the breaker) and whether deferring it is acceptable against 2.2's "ten green runs" +clause, or whether it makes the group's verdict wrong. + +## Judge, in this order + +a. INTENT: does the diff do what 2.1 and 2.2 ask, all of it, and nothing they did not ask for? +b. MATRIX: per row, is the sensor real, does it exercise the claim, is the class honest, would it fail if the + claim were false? Mutation-verify where cheap. +c. RULES: any rule the diff touches or violates. +d. AI-code failure modes: tautological tests, weakened assertions, a dropped half of the task, prose claiming + more than the code does. + +For each P0/P1 you are about to file, argue the opposite for one line; keep only what survives. + +RETURN at most 15 lines, nothing else, under a `## Verdict` heading: + +```text +verdict: APPROVE | REQUEST_CHANGES | BLOCK +findings: + - [P0|P1|P2] +matrix_disputes: + - — +could_not_check: + - +summary: +``` + +P0 → BLOCK, P1 → REQUEST_CHANGES, P2 → note only, APPROVE allowed. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/leg-reviewer-a-diff.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/leg-reviewer-a-diff.md new file mode 100644 index 00000000..5d59c606 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/leg-reviewer-a-diff.md @@ -0,0 +1,11 @@ +## Verdict +verdict: REQUEST_CHANGES +findings: + - [P1] openspec/changes/sink-breaker-terminal-isolation/tasks.md:103 — Counterargument: the unrelated, pre-existing metric flake justifies separate remediation; nevertheless, 9/10 green does not satisfy the explicit completion condition. Keep 2.2 open until satisfied or explicitly revised. + - [P2] openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md:68 — Only four selected tests produce failures: forceCleanupAnnotationDropsFinalizer asserts zero backend contact. The breaker opens during iteration 2, not iteration 1; correct the explanation. +matrix_disputes: + - Row 7 — “partial” honestly reports the result, but cannot support closing 2.2; the non-breaker classification is supported. +could_not_check: + - Independently replaying mutations was prevented by the read-only filesystem; historical ten-run results and lint were inspected as recorded evidence. + - Unit fitness remains pending as scheduled. +summary: Isolation code preserves references and assertions; the focused sensor passes, but the completion claim exceeds the evidence. diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index 727d7d34..022720bd 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -91,7 +91,7 @@ Test rules for every group: ## 2. Reconciler tests stop sharing a tripped breaker (RACE-COUNT-BREAKER-ISOLATION) -- [ ] 2.1 On the G1 commit run `go test -race -count=3 -skip TestControllers ./internal/controller/` +- [x] 2.1 On the G1 commit run `go test -race -count=3 -skip TestControllers ./internal/controller/` ten times and record each exit status, against T0.1's base count. Any red → name the failing tests and the breaker key they share; that red is the group's red. All ten green → record "not reproduced; isolation is preventive" (the tests still share `default/postgres-demo`); 2.2 runs @@ -100,7 +100,7 @@ Test rules for every group: share the `default/postgres-demo` key from each other's breaker state, without changing what they assert: a per-invocation sink name where no assertion names the sink; otherwise a non-parallel test that calls `sink.ResetBreakersForTest()` at its start and in `t.Cleanup`. Never a retry. Ten green - runs of the 2.1 command afterwards. + runs of the 2.1 command afterwards. — 2.1 closed 2026-11-05; 2.2 isolation landed, open on its ten-green clause (decision request), evidence: evidence/G2.md ## 3. Docs From b3cd2f50ac850d64430acb28f72a7e22b3c84cd8 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 12:02:53 +0100 Subject: [PATCH 08/13] :page_facing_up: docs(spec): extend 2.2 to the debounce metric flake --- openspec/changes/sink-breaker-terminal-isolation/loop.md | 1 + openspec/changes/sink-breaker-terminal-isolation/tasks.md | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index 180e75a5..f2b4d2e4 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -48,6 +48,7 @@ Holistic run at B: not yet run. | R2 | Half-open timing flakes | WARNING | reviewer-a, reviewer-b | fix | design timing protocol | no | | R2 | SCB-3 reds not on target assertion | WARNING | reviewer-b | fix | 1.1 refactor base | no | | R2 | G2 red rule; reset only in cleanup; path-branching stub | NOTE | reviewer-a, reviewer-b | fix | tasks preamble, 2.1, 2.2, design | no | +| L G2 | 2.1 race command also flakes on `TestClusterInventory_DebouncedExportIncrementsMetric` (parallel test reading a global counter delta), 1 in 10 | WARNING | reviewer-a (P1), task agent | fix (option A, by fact check: one line with in-repo precedent `kollectinventory_finalizer_test.go:974`; B weakens the criterion and leaves a known flake) | tasks.md 2.2 extended | no | ## Tasks diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index 022720bd..432c0008 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -99,8 +99,10 @@ Test rules for every group: - [ ] 2.2 Isolate the reconciler tests built on `newExportFailureReconciler` and the other tests that share the `default/postgres-demo` key from each other's breaker state, without changing what they assert: a per-invocation sink name where no assertion names the sink; otherwise a non-parallel test - that calls `sink.ResetBreakersForTest()` at its start and in `t.Cleanup`. Never a retry. Ten green - runs of the 2.1 command afterwards. — 2.1 closed 2026-11-05; 2.2 isolation landed, open on its ten-green clause (decision request), evidence: evidence/G2.md + that calls `sink.ResetBreakersForTest()` at its start and in `t.Cleanup`. Never a retry. Also make + `TestClusterInventory_DebouncedExportIncrementsMetric` non-parallel with a comment (it reads a + process-global counter as a delta; precedent `terminalCleanupIncrementsMetric`; added after G2 + round 1, decision in loop.md *Triage*). Ten green runs of the 2.1 command afterwards. — 2.1 closed 2026-11-05; 2.2 isolation landed, open on its ten-green clause (decision request), evidence: evidence/G2.md ## 3. Docs From 228d0c9e28939627dd3513e7054a9686b2fd9542 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 12:07:22 +0100 Subject: [PATCH 09/13] :white_check_mark: test(controller): serialize debounce metric delta test TestClusterInventory_DebouncedExportIncrementsMetric read the process-global kollect_export_debounced_total counter as a delta while running in parallel, so a concurrent debounced cluster export made the delta 2 (about 1 in 10 race runs). Run it non-parallel, as terminalCleanupIncrementsMetric already does. With the per-invocation sink names, ten race -count=3 runs of the controller package are green. --- ...clusterinventory_debounce_checksum_test.go | 5 +- .../evidence/G2.md | 41 +++++++++++----- .../sink-breaker-terminal-isolation/loop.md | 2 +- .../L-tasks/G2/round-2/brief-reviewer-a.md | 48 +++++++++++++++++++ .../L-tasks/G2/round-2/leg-reviewer-a-diff.md | 7 +++ .../sink-breaker-terminal-isolation/tasks.md | 4 +- 6 files changed, 91 insertions(+), 16 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/round-2/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/round-2/leg-reviewer-a-diff.md diff --git a/internal/controller/kollectclusterinventory_debounce_checksum_test.go b/internal/controller/kollectclusterinventory_debounce_checksum_test.go index 739c8036..17f7c015 100644 --- a/internal/controller/kollectclusterinventory_debounce_checksum_test.go +++ b/internal/controller/kollectclusterinventory_debounce_checksum_test.go @@ -208,7 +208,10 @@ func TestClusterInventory_CeilingChangeWithinDebounceWindowReExports(t *testing. // export must increment kollect_export_debounced_total for KollectClusterInventory, so cluster // debounce decisions are visible to dashboards/alerts exactly like the namespaced path. func TestClusterInventory_DebouncedExportIncrementsMetric(t *testing.T) { - t.Parallel() + // Not parallel: the assertion reads the process-global + // kollect_export_debounced_total counter as a delta, so no other test may + // increment the same {controller=KollectClusterInventory} label + // concurrently (a parallel debounced cluster export made the delta 2). const ( sinkNS = sink.DefaultSecretNamespace diff --git a/openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md b/openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md index 5e77f3d6..5b930657 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md +++ b/openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md @@ -5,9 +5,9 @@ - Smallest coherent change: a test helper `uniqueSinkName(t, base)` (package-level atomic counter) and every controller test that used the fixed sink name `postgres-demo` builds its sink, its inventory ref and (where asserted) its expected status name from that per-invocation name. Test-only; no production code. -- Files in scope: `internal/controller/kollectinventory_export_test.go`, `internal/controller/kollectinventory_finalizer_test.go`, `internal/controller/cluster_inventory_finalizer_test.go` -- Status: BLOCKED on 2.2's "ten green runs" clause (decision request below); 2.1 closed; 2.2's isolation committed -- Next action: orchestrator decides the *Decision request*; G3 (3.1) does not depend on it. +- Files in scope: `internal/controller/kollectinventory_export_test.go`, `internal/controller/kollectinventory_finalizer_test.go`, `internal/controller/cluster_inventory_finalizer_test.go`; after decision A: `internal/controller/kollectclusterinventory_debounce_checksum_test.go` +- Status: CLOSED (2.1 and 2.2); decision A applied after round 1 +- Next action: dispatch G3 (3.1). ## Verification matrix @@ -21,7 +21,8 @@ All rows verified 2026-11-05 unless marked otherwise. | 4 | Isolation removes the shared-breaker red, deterministically | focused command (*Deterministic sensor*) green on the tree; red with the helper reverted to `base` (M1, = base behaviour) | machine | pass | | 5 | Assertions unchanged: the one assertion that names the sink (`exportsDeploymentToSink`, `wantName`) is derived from the per-invocation name, not dropped; no other assertion or test body changed | `git diff 0e7897dd1 -- internal/controller` (only name lines, 9 declarations, `wantName`, helper, import) | independent-review | pass | | 6 | Never a retry, never a sleep; no `ResetBreakersForTest` added | diff review; added lines of `git diff 0e7897dd1 -- internal/controller` matching ResetBreakersForTest, time.Sleep, Eventually or retry: 0 | independent-review | pass | -| 7 | 2.2: ten green runs of the 2.1 command afterwards | ten runs on the G2 tree (*2.2* below) | machine | **fail on the literal clause**: 9/10 green; run 9 red on `TestClusterInventory_DebouncedExportIncrementsMetric` (metric-delta flake, not the breaker). Zero breaker reds in 10 runs. See *Harness gaps* | +| 7 | 2.2: ten green runs of the 2.1 command afterwards | ten runs on the final tree (*2.2 after decision A* below) | machine | pass (10/10 exit 0, 2026-11-05). First attempt before decision A: 9/10, run 9 on the debounce-metric flake (G-1) | +| 11 | Decision A: `TestClusterInventory_DebouncedExportIncrementsMetric` non-parallel with a comment in the precedent's style (`kollectinventory_finalizer_test.go:974`) | diff review; the ten runs above (probabilistic: no deterministic mutant for a scheduling race) | independent-review | pass | | 8 | Lint / architecture | `task lint` exit 0, `0 issues.`, go-arch-lint `OK - No warnings found` | machine | pass | | 9 | Race fitness, other packages | `go test -race -count=3 ./internal/errors/ ./internal/sink/...`: every package `ok`, no FAIL or race line | machine | pass | | 10 | Unit fitness (per task) | FINAL: `go test -v ./internal/sink/ ./internal/controller/ -skip TestControllers` on the final code tree | machine | pass (exit 0, 11 s, both packages `ok`, 618 `=== RUN`) | @@ -99,6 +100,17 @@ Same command as 2.1, on the G2 working tree (diff above `0e7897dd1`), 2026-11-05 Result: 9 of 10 green; **0 breaker failures in 10 runs** (base 3 of 10). The one red is the debounce-metric flake also seen on the G1 base (run 9 of 2.1). +## 2.2 after decision A — ten runs on the final tree + +`TestClusterInventory_DebouncedExportIncrementsMetric` made non-parallel (decision A, loop.md *Triage* row "L G2", +tasks.md 2.2 extended in `8c4f6f875`). Same command as 2.1, on the working tree above `8c4f6f875`, 2026-11-05, +sequential, foreground: runs 1–10 all exit 0, 10–13 s each, every run's output `ok github.com/platformrelay/kollect/internal/controller`, +no FAIL line. **10 of 10 green.** + +Also on that tree: the deterministic sensor exit 0 (15 `--- PASS` with `-v`); the target test alone +(`-run 'TestClusterInventory_DebouncedExportIncrementsMetric$' -v`) `=== RUN` and `--- PASS`; `go vet` exit 0; +`task lint` exit 0, `0 issues.`, go-arch-lint `OK - No warnings found`. + ## Sensors | Sensor | Exit | Seconds | @@ -107,7 +119,8 @@ flake also seen on the G1 base (run 9 of 2.1). | focused `-race -count=1 -run 'TestKollectInventoryReconciler_\|TestKollectClusterInventoryReconciler_'` | 0 | 2 | | deterministic sensor (above) | 0 | ~5 | | 2.1 ten runs (G1 commit) | 3 of 10 exit 1 | ~10 each | -| 2.2 ten runs (G2 tree) | 1 of 10 exit 1 (non-breaker) | ~10 each | +| 2.2 ten runs (G2 tree, before decision A) | 1 of 10 exit 1 (non-breaker) | ~10 each | +| 2.2 ten runs (final tree, after decision A) | 10 of 10 exit 0 | 10–13 each | | `task lint` | 0 | 25 | | `go test -race -count=3 ./internal/errors/ ./internal/sink/...` | all `ok` | — | | FINAL `go test -v ./internal/sink/ ./internal/controller/ -skip TestControllers` | 0 | 11 | @@ -133,14 +146,14 @@ Terminal and a Transient failure) is included: T0.1 named it as G2 scope, and it - G-1: `TestClusterInventory_DebouncedExportIncrementsMetric` (`kollectclusterinventory_debounce_checksum_test.go:210`) is parallel and asserts a delta on the process-global `ExportDebouncedTotal{controller=KollectClusterInventory}`; a concurrent debounced cluster export adds to it (`delta = 2, want 1`). Seen once in 10 runs on the G1 commit and - once in 10 after G2. Not the breaker; outside 2.2's text. Deferred: new backlog item (proposed name - RACE-COUNT-DEBOUNCE-METRIC; likely fix: make the test non-parallel, as `terminalCleanupIncrementsMetric` already - is). Because of it, 2.2's "ten green runs" holds for the breaker (0/10) but not for the whole command (9/10). + once in 10 after G2's isolation. Not the breaker. **Resolved** by decision A (2.2 extended): the test is now + non-parallel; ten runs afterwards 10/10 green. ## Iterations 1 · 2.1 ten runs · RED (not a failure class) · 3/10 exit 1, `sink circuit breaker open for "default/postgres-demo"` 2 · 2.2 ten runs · ENVIRONMENT/other-test flake (not the breaker) · run 9 `delta = 2, want 1`; classified as a separate pre-existing flake (seen on base), recorded as G-1, not chased +3 · decision A · test made non-parallel · ten runs 10/10 exit 0 ## Review @@ -156,13 +169,17 @@ Verdict: **REQUEST_CHANGES**. Reviewer confirmed: references and assertions preserved; focused sensor passes; non-breaker classification of the debounce red supported. +Round 2 (last; round 1 found a verified defect against 2.2's clause): one leg, reviewer-a (diff lens), on the +staged diff above `8c4f6f875` only (decision A). No paid leg. Records: `reviews/L-tasks/G2/round-2/`. +Verdict: **APPROVE**, no findings, no matrix disputes. The reviewer ran its own race `-count=3` run (green) and +confirmed that neither paralleltest nor tparallel is enabled. + ## Verdict -**BLOCKED** (2.2 clause), with 2.1 closed and the isolation committed. 2.1: CLOSED (row 1). 2.2: every row -`machine`/`independent-review` pass except row 7, which fails only on a different test's flake (G-1); fixing that -test is outside 2.2's named scope, so it is a decision, not a correction. +**CLOSED** (2.1 and 2.2). Every matrix row `machine` or `independent-review`, result pass. Round 1 left 2.2 +blocked on its ten-green clause; decision A resolved it (see *Decision request*, kept for the record). -## Decision request +## Decision request (resolved: A, by the coordinator, 2026-11-05) 2.2 says "Ten green runs of the 2.1 command afterwards". After the isolation: 0 breaker reds in 10 runs (base 3/10), 1 red from `TestClusterInventory_DebouncedExportIncrementsMetric` (parallel test asserting a delta on a diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index f2b4d2e4..b82766d9 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -56,7 +56,7 @@ Holistic run at B: not yet run. |---|---|---|---| | T0.1 | CLOSED | reviewer-a APPROVE (1 P2 fixed), 1 round, no paid leg (`reviews/L-tasks/T0.1/`) | none; 2 non-blocking spec wording corrections (25 not 26 `invalidPrune` call sites; proposal *Impact* misses `git/prune_target.go`), see `evidence/T0.1.md` *Spec corrections*. Race on base: 2/10 red | | G1 (1.1 + 1.2) | CLOSED | reviewer-a APPROVE (1 P2 evidence wording fixed), 1 round, no paid leg (`reviews/L-tasks/G1/`) | none; race row split: Ginkgo `TestControllers` refuses `-count=3`, run with `-skip TestControllers` | -| G2 (2.1 + 2.2) | 2.1 CLOSED; 2.2 BLOCKED (decision) | reviewer-a REQUEST_CHANGES (P1: 9/10 green misses 2.2's "ten green runs"; P2 evidence wording fixed), 1 round, no paid leg (`reviews/L-tasks/G2/`) | isolation committed: 0 breaker reds in 10 runs (base 3/10); 1 red from `TestClusterInventory_DebouncedExportIncrementsMetric` (metric-delta flake, not the breaker). Decision: A widen 2.2 to make that test non-parallel (recommended) or B revise the clause and file a backlog item; `evidence/G2.md` *Decision request* | +| G2 (2.1 + 2.2) | CLOSED | round 1 reviewer-a REQUEST_CHANGES (P1 9/10 green missed 2.2's clause → decision A; P2 evidence wording fixed); round 2 reviewer-a APPROVE; no paid leg (`reviews/L-tasks/G2/`, `round-2/`) | none. Base 3/10 red (shared `default/postgres-demo`); after per-invocation sink names and the debounce-metric test made non-parallel: 10/10 green | ## Known red diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/round-2/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/round-2/brief-reviewer-a.md new file mode 100644 index 00000000..31b65857 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/round-2/brief-reviewer-a.md @@ -0,0 +1,48 @@ +# Review brief: group G2 round 2 (task 2.2, decision A) + +You are reviewing one change, independently of whoever implemented it. Read-only: do not edit, commit or create +files in the repository. Work from the repository root +(`/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation`). You may run `go test` +commands under a minute; the Ginkgo suite in `internal/controller` refuses `-count>1`, so add +`-skip TestControllers` to any `-count=N` command there. + +## Inputs + +1. TASK: `openspec/changes/sink-breaker-terminal-isolation/tasks.md` item 2.2 as extended in `8c4f6f875` + (make `TestClusterInventory_DebouncedExportIncrementsMetric` non-parallel with a comment; ten green runs of + the 2.1 command). Decision: `loop.md` *Triage* row "L G2". Round 1 review: `reviews/L-tasks/G2/leg-reviewer-a-diff.md`. +2. MATRIX and evidence: `openspec/changes/sink-breaker-terminal-isolation/evidence/G2.md` (rows 7 and 11, + *2.2 after decision A*, *Harness gaps*, *Verdict*). +3. DIFF (this round only): staged, `git diff --cached 8c4f6f875`. Code: one test file, + `internal/controller/kollectclusterinventory_debounce_checksum_test.go`. +4. SENSORS: recorded in the evidence. Round 1 already reviewed the sink-name isolation (`8ada865fd`); do not + re-review it. +5. RULES: `AGENTS.md`, `docs/development/spec-workflow.md`. Precedent for the change: + `internal/controller/kollectinventory_finalizer_test.go:974` (`terminalCleanupIncrementsMetric`). + +## What matters + +Does removing `t.Parallel()` actually stop concurrent increments of the counter label the test reads (Go runs +non-parallel top-level tests before releasing parallel ones)? Does the assertion stay unchanged? Does any linter +rule (e.g. paralleltest/tparallel) or other test now conflict? Is the evidence honest that the ten-run result is +probabilistic and that the verdict CLOSED is supported? + +## Judge, in this order + +a. INTENT, b. MATRIX (rows 7, 11), c. RULES, d. AI-code failure modes. For each P0/P1 argue the opposite for one +line; keep only what survives. + +RETURN at most 15 lines, nothing else, under a `## Verdict` heading: + +```text +verdict: APPROVE | REQUEST_CHANGES | BLOCK +findings: + - [P0|P1|P2] +matrix_disputes: + - — +could_not_check: + - +summary: +``` + +P0 → BLOCK, P1 → REQUEST_CHANGES, P2 → note only, APPROVE allowed. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/round-2/leg-reviewer-a-diff.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/round-2/leg-reviewer-a-diff.md new file mode 100644 index 00000000..a06e6649 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G2/round-2/leg-reviewer-a-diff.md @@ -0,0 +1,7 @@ +## Verdict +verdict: APPROVE +findings: [] +matrix_disputes: [] +could_not_check: + - Historical ten-run results and lint were inspected as recorded evidence, not independently replayed. +summary: Removing t.Parallel isolates the counter delta from parallel tests; assertions remain unchanged, and neither paralleltest nor tparallel is enabled. Independent race/count=3 run passed. Rows 7 and 11 acknowledge probabilistic evidence and support CLOSED. diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index 432c0008..f01cb4ed 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -96,13 +96,13 @@ Test rules for every group: and the breaker key they share; that red is the group's red. All ten green → record "not reproduced; isolation is preventive" (the tests still share `default/postgres-demo`); 2.2 runs either way. -- [ ] 2.2 Isolate the reconciler tests built on `newExportFailureReconciler` and the other tests that +- [x] 2.2 Isolate the reconciler tests built on `newExportFailureReconciler` and the other tests that share the `default/postgres-demo` key from each other's breaker state, without changing what they assert: a per-invocation sink name where no assertion names the sink; otherwise a non-parallel test that calls `sink.ResetBreakersForTest()` at its start and in `t.Cleanup`. Never a retry. Also make `TestClusterInventory_DebouncedExportIncrementsMetric` non-parallel with a comment (it reads a process-global counter as a delta; precedent `terminalCleanupIncrementsMetric`; added after G2 - round 1, decision in loop.md *Triage*). Ten green runs of the 2.1 command afterwards. — 2.1 closed 2026-11-05; 2.2 isolation landed, open on its ten-green clause (decision request), evidence: evidence/G2.md + round 1, decision in loop.md *Triage*). Ten green runs of the 2.1 command afterwards. — 2.1 and 2.2 closed 2026-11-05, evidence: evidence/G2.md ## 3. Docs From 4e50a9a2f26132429c70902e284d1f1480c827d7 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 12:15:03 +0100 Subject: [PATCH 10/13] :page_facing_up: docs(sink): document what the sink breaker counts The troubleshooting row said the breaker opens on 5 transient failures and self-closes after the timeout. It counts every backend write failure except an inventory-scoped one (a Git ownership conflict), and after 30 s admits one trial that closes, re-opens or stays half-open. ADR-0602 gets a dated note; the decision is unchanged. --- docs/adr/0602-error-taxonomy.md | 6 + docs/operator-manual/troubleshooting.md | 4 +- .../evidence/G3.md | 116 ++++++++++++++++++ .../sink-breaker-terminal-isolation/loop.md | 2 + .../reviews/L-tasks/G3/brief-reviewer-a.md | 51 ++++++++ .../reviews/L-tasks/G3/leg-reviewer-a-diff.md | 11 ++ .../sink-breaker-terminal-isolation/tasks.md | 4 +- 7 files changed, 190 insertions(+), 4 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/evidence/G3.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G3/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G3/leg-reviewer-a-diff.md diff --git a/docs/adr/0602-error-taxonomy.md b/docs/adr/0602-error-taxonomy.md index 98c9b81b..0d26e803 100644 --- a/docs/adr/0602-error-taxonomy.md +++ b/docs/adr/0602-error-taxonomy.md @@ -36,6 +36,12 @@ Additional rules: - No `panic` in reconcilers; entrypoint `recover` → requeue + Event. - Context deadlines on all external calls; never log secrets or full payloads. - Circuit breaker (`gobreaker`) per sink/doc backend for `ErrTransient` storms. + *Note 2026-11-05:* the breaker counts more than `ErrTransient`. Every failure the sink + backend returns while writing an export counts (transient, forbidden, unclassified, and + sink-wide terminal errors such as Git auth or protected-branch rejections), except a result + marked inventory-scoped (today a Git ownership conflict on one inventory's own paths), which + counts neither as a failure nor as a success. The decision above is unchanged; see the + troubleshooting guide for thresholds and recovery. Conditions follow Kubernetes conventions: `Ready`, `Synced`, `Degraded` with `observedGeneration`. diff --git a/docs/operator-manual/troubleshooting.md b/docs/operator-manual/troubleshooting.md index 222436de..89884756 100644 --- a/docs/operator-manual/troubleshooting.md +++ b/docs/operator-manual/troubleshooting.md @@ -211,7 +211,7 @@ Family CRDs: **`KollectSnapshotSink`** (git/gitlab/s3/gcs), **`KollectDatabaseSi | Postgres connection failures | DSN, network policy, TLS, pool timeout | Database sink conditions; `transient` sink errors; connection test metric | Verify Secret keys, egress NetworkPolicy, server reachable | DBA for server-side limits | | Postgres rows stale while status looks healthy | Upsert-only drift, or an export error on that ref | Compare `lastExportTime` on the database entry in `status.sinkExports[]` | Follow [Postgres state store — Troubleshooting](../examples/postgres-state-store.md#troubleshooting) | DBA for server-side limits | | S3/GCS 403 | IAM, wrong bucket, signature | Export logs; `terminal`/`forbidden` | Fix credentials and bucket policy | Cloud IAM review | -| `sink circuit breaker open` in logs | 5 consecutive transient failures per sink key | `transient` errors then silence ~30s | Fix backend; breaker self-closes after timeout | Backend SLA breach | +| `sink circuit breaker open` in logs | 5 consecutive counted backend write failures to one sink (breaker key `/`) within its 60 s counting window. Counted: transient, forbidden and unclassified errors, and sink-wide terminal errors such as a Git auth or protected-branch rejection. A failure about one inventory (an ownership conflict on its own paths) does not count; it still fails that inventory's export (status, Warning event, logs) | Sink errors, then `sink circuit breaker open for "/"` (transient) for ~30 s; exports to that sink do not reach the backend while it is open | Fix the backend. After 30 s one trial export is admitted: success closes the breaker, a counted failure re-opens it for another 30 s, an inventory-scoped result (ownership conflict) leaves it half-open so the next export is the trial | Backend SLA breach | | `SinkReachable=False`, `SinkNotFound` | Wrong sink name or cross-namespace ref in `*SinkRefs` | Inventory message; `kubectl get kollect*sink -n ` | Fix ref name; create sink in inventory namespace | — | | `SinkReachable=False`, `SinkUnreachable` | Backend down despite CR present; bad DSN or TLS failure | Sink `ConnectionVerified`; probe annotation | Fix network/credentials first | — | | Inventory / namespace stuck `Terminating` | Sink cleanup failed terminally during deletion; finalizer retained | Events reason `CleanupTerminal` on the inventory; `increase(kollect_cleanup_terminal_total[1h]) > 0`; Degraded reason `CleanupTerminal` | Fix the sink backend and let the next reconcile finish cleanup — the terminal wedge re-checks every 5 minutes, so a Secret-only credential fix (no watch fires on Secret edits) clears within minutes; or accept the exported objects and set `kollect.dev/force-cleanup: "true"` on the deleting object to drop the finalizer | — | @@ -325,7 +325,7 @@ Structured controller logs (`logr`). Grep operator pod logs (namespace typically | `export failed` | Sink export path | | `export payload exceeds spill warn threshold` | Approaching 1 MiB — shard soon | | `debounced` | Export skipped by interval/coalesce | -| `sink circuit breaker open` | Repeated transient sink failures | +| `sink circuit breaker open` | Repeated sink failures (transient, forbidden or sink-wide terminal); see *Export — sink backends* | | `reconcile panic recovered` | Panic converted to requeue (EC-P2-01) | | `git push` / `git auth failed` | Snapshot sink transport | diff --git a/openspec/changes/sink-breaker-terminal-isolation/evidence/G3.md b/openspec/changes/sink-breaker-terminal-isolation/evidence/G3.md new file mode 100644 index 00000000..30b3b586 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/evidence/G3.md @@ -0,0 +1,116 @@ +# G3 — docs: what the sink breaker counts (task 3.1) + +- Change: `openspec/changes/sink-breaker-terminal-isolation/` · branch: `fix/sink-breaker-terminal-isolation` · pre-task sha: `b8296d025` · started: 2026-11-05 +- Satisfies: tasks.md 3.1 (operator docs for SCB-1, SCB-2, SCB-3) +- Smallest coherent change: rewrite the troubleshooting row for `sink circuit breaker open`, fix the log-pattern + row that still says "transient", add a dated note under ADR-0602's circuit-breaker rule. Docs only. +- Files in scope: `docs/operator-manual/troubleshooting.md`, `docs/adr/0602-error-taxonomy.md` +- Status: CLOSED +- Next action: 4.1 (integrator). + +## Source of the documented behaviour + +Verified 2026-11-05 by reading the code at `b8296d025`: + +- `internal/sink/circuit_breaker.go`: `circuitBreakerInterval = 60s`, `circuitBreakerTimeout = 30s`, + `circuitBreakerTripAt = 5`; `MaxRequests: 1`; `ReadyToTrip` on `ConsecutiveFailures >= 5`; + `IsExcluded: kollecterrors.IsInventoryScoped`; open state returns + `Transient("sink circuit breaker open for %q")`. +- `internal/sink/export.go:200`: key `req.SinkNamespace + "/" + req.SinkName`. +- `internal/sink/git/prune_owned.go:258-263`: `ownershipConflict` marks a rejection about the exporting + inventory's own paths or ownership claim; rejections caused by shared prune metadata stay unmarked and count. +- Half-open behaviour (success closes, counted failure re-opens, excluded result leaves half-open and frees the + slot): SCB-3a/3b/3c, closed in G1 (`evidence/G1.md`). + +## Verification matrix + +All rows verified 2026-11-05 on the final tree unless marked otherwise. + +| # | Claim / invariant | Sensor | Class | Result | +|---|---|---|---|---| +| 1 | troubleshooting row: opens after 5 consecutive counted failures per sink key; counted = transient, forbidden, sink-wide terminal (Git auth) | diff vs `circuit_breaker.go` / `export.go:200` above | independent-review | pass | +| 2 | troubleshooting row: a failure about one inventory (ownership conflict) does not count and still fails that inventory's export (wording after review: not "only in that inventory's status") | diff vs `IsExcluded` and `ownershipConflict` above | independent-review | pass | +| 3 | "self-closes after timeout" replaced: after 30 s one trial; success closes; counted failure re-opens; inventory-scoped result leaves it half-open | `rg -n 'self-closes' docs` → 0 matches; diff review | independent-review | pass | +| 4 | ADR-0602:38 gets a dated note on what counts (failures the backend returns while writing; pre-breaker failures in `export.go` before line 200 are outside the breaker); decision not re-decided | diff review (6 added lines, `*Note 2026-11-05:*`, says the decision is unchanged) | independent-review | pass | +| 5 | `rg -n 'circuit breaker' docs` reviewed for other contradicted claims; list recorded | *Other mentions* below | independent-review | pass | +| 6 | Markdown lint | `task lint:markdown` exit 0, `Summary: 0 issues in 0 files` (423 files linted) | machine | pass | +| 7 | Site builds strict with the edits | `mkdocs build --strict` (pinned `docs/requirements-docs.txt` in a scratch venv) exit 0, `Documentation built`; `python3 -m unittest discover -s test/docs -p 'test_*.py'` 37 tests OK | machine | pass | +| 8 | Doc guards that name the edited files | `hack/test/docs_adr_kollectsink_retcon_test.sh`, `docs_launch_truth_test.sh`, `docs_map_contract_test.sh` each exit 0 | machine | pass | +| 9 | Public-docs hygiene: no internal ids, no agent-context references, no private strings | `task scrub` and `gitleaks protect --staged` at commit (*Sensors*); diff review | machine + independent-review | pass | + +## Other mentions (`rg -n -i 'circuit breaker|breaker' docs`, 2026-11-05) + +| Location | Claim | Contradicted? | +|---|---|---| +| `operator-manual/troubleshooting.md:158` | Transient class includes "circuit breaker open" | no: the open error is Transient | +| `operator-manual/troubleshooting.md:162` | circuit-breaker rules in ADR-0602 | no (note added there) | +| `operator-manual/troubleshooting.md:214` | the row this task rewrites | rewritten | +| `operator-manual/troubleshooting.md:328` | log pattern: "Repeated transient sink failures" | yes: changed to transient, forbidden or sink-wide terminal, pointing to the sink-backends row | +| `operator-manual/troubleshooting.md:336` | `rg` log filter | no | +| `adr/0602-error-taxonomy.md:38` | breaker "for `ErrTransient` storms" | yes: dated note added (task clause) | +| `REQUIREMENTS.md:89` | each sink has its own breaker | no | +| `ROADMAP.md:23` | per-sink breakers exist | no | +| `adr/0402-sink-backends-database-kafka.md:108` | breaker per ADR-0602 | no | +| `adr/0406-sink-registry.md:68` | long exports rely on deadlines and the breaker | no | +| `adr/0420-bigquery-database-sink.md:230,292` | throttling (transient) absorbed by the breaker | no: transient still counts | +| `operator-manual/lab-scenario-matrix.md:42` | outage → breaker closes on recovery | no | +| `development/guidelines.md:41,83` | per-sink `gobreaker`; transient requeue, terminal stops requeue | no: about requeue, not counting | + +## Sensors + +| Sensor | Sha / tree | Exit | Seconds | Result | +|---|---|---|---|---| +| `task lint:markdown` | working tree on `b8296d025` | 0 | ~10 | 0 issues (rerun after the review wording fix: 0 issues) | +| `rg -n 'self-closes' docs` | same | 1 | <1 | no match (old wording gone) | +| `task docs:verify` | same | 201 | ~120 | not-run (ENVIRONMENT): `demo_04_samples_kustomize_test.sh` fails, mise has no `kubectl` version for the shim; unrelated to the diff. Substituted by rows 7 and 8 | +| `mkdocs build --strict` (scratch venv, pinned requirements) | same | 0 | ~15 | built (rerun after the wording fix: exit 0) | +| `python3 -m unittest discover -s test/docs` | same | 0 | <1 | 37 OK (rerun after the fix: OK) | +| 3 doc guard scripts (row 8) | same | 0 each | <5 | pass (rerun after the fix: 0 each) | +| `task scrub` | staged diff (docs + evidence) | 0 | <5 | `scrub: ok` | +| `MISE_GITLEAKS_VERSION=8.30.1 gitleaks protect --staged --config .github/gitleaks.toml` | staged diff | 0 | <1 | `no leaks found` (rerun at commit) | +| Go tiers | n/a | | | not run: docs-only diff, no code touched (test budget: fast check only) | + +## Maintainability self-check + +1. New abstraction? no. 2. Duplication? The row restates breaker settings already in code; the ADR note points +to the row for numbers instead of repeating them. 3. Dead text? no. 4. Hidden coupling? the 30 s / 60 s / 5 +numbers must change with `circuit_breaker.go`; same coupling the old row had. 5. Naming? matches the log text. +6. Scope creep? line 328 edit is inside 3.1's "any other claim the change contradicts". 7. Tests weakened? none. +8. Instruction files touched? no. + +## Harness gaps + +- No machine check ties the documented numbers (5, 30 s, 60 s) to the constants; drift is caught only by review. + Pre-existing; not in this task's scope. + +## Iterations + +1 · `task docs:verify` · ENVIRONMENT · mise `kubectl` shim has no version (demo_04 kustomize guard); substituted +by strict mkdocs build, the docs unit contracts and the guards that name the edited files. + +## Review + +One leg, reviewer-a (diff lens), round 1, on the staged diff above `b8296d025`. No paid leg: docs-only diff, no +secrets, CI, infra or manifests, no BLOCK. Records: `reviews/L-tasks/G3/` (brief, leg file). +Verdict: **REQUEST_CHANGES**. + +| Finding | Verified | Disposition | +|---|---|---| +| P1 `troubleshooting.md:214`: "reported only in that inventory's status" is false; ownership failures also log, emit a Warning event and a sink-error metric | yes (`export.go:210` `classifyExportFailure`; controller logs/events) | fixed: "it still fails that inventory's export (status, Warning event, logs)" | +| P1 `0602-error-taxonomy.md:39`: "Every failed export to a sink counts" over-claims; backend acquisition, payload validation and layout resolution fail before `exportThroughBreaker` | yes (`export.go` returns before line 200) | fixed: "Every failure the sink backend returns while writing an export counts"; row said "backend write failures" | + +Both findings are wording, not behaviour: fixed, lint and strict build rerun, no second round (task prompt rule). +Reviewer confirmed: scope, five-column format, other breaker mentions, the fixed 60 s counting window and the +half-open transitions. + +## Verdict + +**CLOSED**. Every matrix row `machine` or `independent-review`, result pass. `task docs:verify` not-run +(ENVIRONMENT, host `kubectl` shim) and substituted by its strict-build, docs-contract and guard components that +cover the edited files. + +## Learnings + +Class: tooling. `task docs:verify` stops on a missing mise `kubectl` version before `mkdocs build --strict`, and +the main checkout's `.mkdocs-venv` has a foreign interpreter path and lacks the redirects plugin; a scratch venv +from `docs/requirements-docs.txt` (`pip install --require-hashes --only-binary=:all:`) builds strict in ~15 s. diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index b82766d9..b9189378 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -57,6 +57,7 @@ Holistic run at B: not yet run. | T0.1 | CLOSED | reviewer-a APPROVE (1 P2 fixed), 1 round, no paid leg (`reviews/L-tasks/T0.1/`) | none; 2 non-blocking spec wording corrections (25 not 26 `invalidPrune` call sites; proposal *Impact* misses `git/prune_target.go`), see `evidence/T0.1.md` *Spec corrections*. Race on base: 2/10 red | | G1 (1.1 + 1.2) | CLOSED | reviewer-a APPROVE (1 P2 evidence wording fixed), 1 round, no paid leg (`reviews/L-tasks/G1/`) | none; race row split: Ginkgo `TestControllers` refuses `-count=3`, run with `-skip TestControllers` | | G2 (2.1 + 2.2) | CLOSED | round 1 reviewer-a REQUEST_CHANGES (P1 9/10 green missed 2.2's clause → decision A; P2 evidence wording fixed); round 2 reviewer-a APPROVE; no paid leg (`reviews/L-tasks/G2/`, `round-2/`) | none. Base 3/10 red (shared `default/postgres-demo`); after per-invocation sink names and the debounce-metric test made non-parallel: 10/10 green | +| G3 (3.1) | CLOSED | reviewer-a REQUEST_CHANGES (2 P1 operator-wording over-claims, verified and fixed, no new round); no paid leg (`reviews/L-tasks/G3/`) | none. `task docs:verify` not-run (host `kubectl` shim); strict mkdocs build, docs unit contracts and doc guards green instead | ## Known red @@ -81,6 +82,7 @@ Tests from a test task changed by a later task (close-task `TEST` class); each o - G1: a stub snapshot backend receives `objectstore.ObjectPath(...)`, not `req.ObjectPath`; branch a stub on the inventory name. Breaker test helpers `uniqueBreakerKey`/`uniqueBreakerName`/`stubExportRequest`/`halfOpenBreaker` live in `internal/sink/circuit_breaker_test.go` for G2 to reuse. - G2: the ten-run race loop is probabilistic and cannot kill a mutant; a deterministic sensor for breaker sharing is `-count=3 -run` over only the tests whose backend fails (no success between iterations to reset the streak): 5 counted failures in iteration 1 open a shared breaker. - G2: `TestClusterInventory_DebouncedExportIncrementsMetric` is a separate parallel metric-delta flake in the same race command (~1/10); a red there is not the breaker. +- G3: `task docs:verify` can fail on a missing mise `kubectl` version before its mkdocs step; build strict from a scratch venv of `docs/requirements-docs.txt` and run `test/docs` unit contracts plus the guards naming the edited files. ## Where a human should look first diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G3/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G3/brief-reviewer-a.md new file mode 100644 index 00000000..f4f36b38 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G3/brief-reviewer-a.md @@ -0,0 +1,51 @@ +# Review brief: group G3 (task 3.1, docs) + +You are reviewing one task, independently of whoever implemented it. Read-only: do not edit, commit or create +files in the repository. Work from the repository root +(`/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation`). + +## Inputs + +1. TASK: `openspec/changes/sink-breaker-terminal-isolation/tasks.md`, item 3.1 (read it). Spec: + `openspec/changes/sink-breaker-terminal-isolation/specs/*/spec.md` (SCB-1, SCB-2, SCB-3). +2. MATRIX and evidence: `openspec/changes/sink-breaker-terminal-isolation/evidence/G3.md`. +3. DIFF: base `b8296d025`, staged (not yet committed): `git diff --cached b8296d025`. Product changes: + `docs/operator-manual/troubleshooting.md` (two table rows) and `docs/adr/0602-error-taxonomy.md` (a dated + note); plus the evidence file and the tasks.md tick. +4. SENSORS: recorded in the evidence (*Sensors*). `task docs:verify` could not run here (a host tool missing, + recorded as not-run); a strict mkdocs build, the docs unit contracts and three doc guards ran instead. +5. RULES: `AGENTS.md`, `docs/development/spec-workflow.md`. These docs are public: written for operators, no + internal backlog ids, no references to internal planning files, no private strings. The troubleshooting table + keeps its five-column row format. The ADR gets a short dated note, not a rewrite. + +## What matters + +Check every claim the new text makes against the code, not against the evidence's summary: +`internal/sink/circuit_breaker.go` (thresholds, interval, timeout, `MaxRequests`, `IsExcluded`, the open-state +error text), `internal/sink/export.go` (the breaker key), `internal/errors/errors.go` (the inventory-scoped +marker) and `internal/sink/git/prune_owned.go` (which Git rejections are marked). gobreaker is +`github.com/sony/gobreaker/v2` (module cache). In particular: is "within its 60 s counting window" accurate for +gobreaker's closed-state `Interval`; does a counted failure on the half-open trial re-open it; does an +inventory-scoped result really leave the breaker half-open; is the claim that an ownership conflict "is reported +only in that inventory's status" true; and is every Git ownership rejection excluded, or only some (the docs +must not over-claim). Also: does 3.1's requirement to review `rg -n 'circuit breaker' docs` hold, i.e. is any +other docs claim left contradicted. + +Then judge, in this order: (a) INTENT, all of 3.1 and nothing else; (b) MATRIX rows honest; (c) RULES; (d) +prose that claims more than the code does. + +RETURN at most 15 lines, nothing else: + +```text +verdict: APPROVE | REQUEST_CHANGES | BLOCK +findings: + - [P0|P1|P2] +matrix_disputes: + - — +could_not_check: + - +summary: +``` + +P0 → BLOCK; P1 (a false or over-claiming operator statement, a missed clause of 3.1) → REQUEST_CHANGES; +P2/P3 → note, APPROVE allowed. Write the result as `## Verdict` followed by those lines. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G3/leg-reviewer-a-diff.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G3/leg-reviewer-a-diff.md new file mode 100644 index 00000000..c66d566a --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/G3/leg-reviewer-a-diff.md @@ -0,0 +1,11 @@ +## Verdict +verdict: REQUEST_CHANGES +findings: + - [P1] docs/operator-manual/troubleshooting.md:214: “reported only in that inventory's status” is false: ownership failures also produce logs (internal/controller/kollectinventory_controller.go:440), warning Events (:277), and sink-error metrics (internal/sink/export.go:210). Remove “only”. + - [P1] docs/adr/0602-error-taxonomy.md:39: “Every failed export to a sink counts” overclaims: backend acquisition, payload validation and layout resolution can fail before exportThroughBreaker (internal/sink/export.go:92–200). Limit this statement to failures returned by the export plan inside the breaker. +matrix_disputes: + - Row 2 — IsExcluded and ownershipConflict establish exclusion, not exclusive status reporting; the controller and metrics paths contradict that wording. + - Row 4 — A short dated addition preserves ADR structure but does not establish the accuracy of “Every failed export”. +could_not_check: + - Recorded lint/build/unit/guard results were not rerun; full docs:verify remains unverified because of the recorded missing host tool. +summary: Scope, five-column format, other breaker mentions, fixed 60 s counting window and half-open transitions check out; correct the two operator-facing overclaims. diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index f01cb4ed..b6781a3a 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -106,14 +106,14 @@ Test rules for every group: ## 3. Docs -- [ ] 3.1 `docs/operator-manual/troubleshooting.md:214`: the breaker opens after 5 consecutive counted +- [x] 3.1 `docs/operator-manual/troubleshooting.md:214`: the breaker opens after 5 consecutive counted export failures per sink key (Transient, Forbidden, sink-wide Terminal such as Git auth); a failure that is about one inventory (ownership conflict) does not count and stays in that inventory's status. Replace "self-closes after timeout": after 30 s one trial export is admitted, success closes it, a counted failure re-opens it, an inventory-scoped result leaves it half-open. `docs/adr/0602-error-taxonomy.md:38`: add a dated note on what counts (ADR not re-decided). `task lint:markdown` 0 issues; `rg -n 'circuit breaker' docs` reviewed for any other claim the change - contradicts, the list recorded in `evidence/G3.md`. + contradicts, the list recorded in `evidence/G3.md`. — closed 2026-11-05, evidence: evidence/G3.md ## 4. Integration From 7a26657c052b15344f2b6f432319ae6f7a990a23 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 13:11:10 +0100 Subject: [PATCH 11/13] :page_facing_up: docs(spec): triage branch review round 1 --- .../sink-breaker-terminal-isolation/design.md | 8 ++- .../sink-breaker-terminal-isolation/loop.md | 16 ++++-- .../B-branch/round-1/brief-reviewer-a.md | 42 +++++++++++++++ .../round-1/leg-reviewer-a-adversarial.md | 52 +++++++++++++++++++ .../B-branch/round-1/leg-reviewer-b-spec.md | 49 +++++++++++++++++ .../reviews/B-branch/round-1/verification.md | 23 ++++++++ .../specs/sink-circuit-breaker/spec.md | 2 +- .../sink-breaker-terminal-isolation/tasks.md | 23 ++++++-- 8 files changed, 204 insertions(+), 11 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/leg-reviewer-a-adversarial.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/leg-reviewer-b-spec.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/verification.md diff --git a/openspec/changes/sink-breaker-terminal-isolation/design.md b/openspec/changes/sink-breaker-terminal-isolation/design.md index ecb24a1b..34cf3c58 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/design.md +++ b/openspec/changes/sink-breaker-terminal-isolation/design.md @@ -65,7 +65,10 @@ Breaker: `IsExcluded: kollecterrors.IsInventoryScoped`. TotalExclusions < MaxRequests`. An excluded trial leaves the state half-open and frees the slot; the next request is admitted as the trial. A success closes the breaker; a counted failure re-opens it. - `ErrTooManyRequests` (a second concurrent request in half-open) is returned unwrapped from - `exportThroughBreaker` and becomes Transient in `classifyExportFailure`. Unchanged. + `exportThroughBreaker`, passes `git.ClassifyExportError` (for every sink type, `export.go:208`), + where it is wrapped Transient as "git transport: too many requests", then `classifyExportFailure`. + Pre-existing and out of scope (misleading message only; class is Transient); deferred to the + backlog. ## Testing approach @@ -100,7 +103,8 @@ State is read with `breakerFor(key).State()` immediately after each call. ## Risks - The marker is lost by a re-wrap without `Unwrap` between the raise site and the breaker: covered by - SCB-1e through the real Git backend. + SCB-1e through the real Git backend (CLI engine) and B1.1 (per-site table and the go-git + return path). - A shared-state rejection marked as inventory-scoped would stop shedding load for a sink-wide fault: T0.1's per-site classification, reviewed in G1. - A test that relies on ownership rejections tripping the breaker would now fail; none is known diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index b9189378..1d75f87a 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -1,9 +1,9 @@ # Loop — sink-breaker-terminal-isolation (spec-loop-duo run state) Repo: `/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation` · Branch: `fix/sink-breaker-terminal-isolation` · Base: `origin/main` @ `96c41b0d0` · Started: 2026-11-05 · Reviewers: two-model duo -Next: L — dispatch T0.1 (probe) +Next: L — dispatch B1.1 (branch review fixes) Task prompt: local, `agent-context/loops/sink-breaker-terminal-isolation/task-prompt.md` (harness, not committed) · decisions by model alignment, logged in agent-context/decisions.md -Budget: paid review legs 2/12 (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 +Budget: paid review legs 3/12 (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 Backlog SINK-BREAKER-TENANT-01 (P2) and RACE-COUNT-BREAKER-ISOLATION (P3). Spec set: proposal.md, design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `sink-breaker-terminal` @@ -14,8 +14,8 @@ design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `si - [x] 0 orient — new change; file:line references measured against `origin/main` @ `96c41b0d0` (2026-11-05). Open PRs touching these files: none (#523 mongo-driver v2 touches `internal/sink/mongodb/` only; #520 helm v4 CI only) - [x] P plan + tasks — written; `task spec:validate` green (22 items); spec set committed: (this commit) - [x] R spec-set review — round 1 on `a505029da`: reviewer-a REQUEST_CHANGES (1 CRITICAL), reviewer-b CONCERNS; all 8 findings verified and fixed (`reviews/R-spec-set/verification.md`); CRITICAL had a mechanical fix both legs named → round 2 (last) on `de1e7df12`: reviewer-a REQUEST_CHANGES, reviewer-b BLOCK; new CRITICAL (Git sink-wide Terminal reaches the breaker; A's premise false) settled by decision rerun → B (unanimous), spec set rewritten for B; 7 WARNING/NOTE fixed (`reviews/R-spec-set/round-2/verification.md`). No third round (limit); premises re-verified in T0.1, rewritten set reviewed again at B -- [ ] L task loop — see *Tasks* -- [ ] B branch review — `reviews/B-branch/round-N/` +- [x] L task loop — see *Tasks* (T0.1, G1, G2, G3 closed; B1.1 from stage B) +- [ ] B branch review — round 1 on `4e50a9a2f` (rebased on main after #520/#523): reviewer-a CONCERNS, reviewer-b CONCERNS, no CRITICAL; SCB-1..3 hold; 6 fix (B1.1 + spec wording), 1 defer, 2 reject (`reviews/B-branch/round-1/verification.md`). Next: B1.1, then round 2 + per-commit gate - [ ] Hand-off — `pr-description.md` ## Fitness functions @@ -29,7 +29,7 @@ design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `si | OpenSpec strict | `task spec:validate` | triggered | green | per spec-set edit | | Full local gate | `task check` | holistic | green on base, minus known environment reds | at B | -Holistic run at B: not yet run. +Holistic run at B: `task check` on `4e50a9a2f` exit 1; four gates red, none from this branch (git-ignored mkdocs cache hit, `kubectl` shim, two `operator-sdk` guards; the last three red on clean `origin/main` too). Details `reviews/B-branch/round-1/verification.md`. ## Triage @@ -49,6 +49,12 @@ Holistic run at B: not yet run. | R2 | SCB-3 reds not on target assertion | WARNING | reviewer-b | fix | 1.1 refactor base | no | | R2 | G2 red rule; reset only in cleanup; path-branching stub | NOTE | reviewer-a, reviewer-b | fix | tasks preamble, 2.1, 2.2, design | no | | L G2 | 2.1 race command also flakes on `TestClusterInventory_DebouncedExportIncrementsMetric` (parallel test reading a global counter delta), 1 in 10 | WARNING | reviewer-a (P1), task agent | fix (option A, by fact check: one line with in-repo precedent `kollectinventory_finalizer_test.go:974`; B weakens the criterion and leaves a known flake) | tasks.md 2.2 extended | no | +| B1 | Only one routed raise site pinned by a test | WARNING | reviewer-a, reviewer-b | fix | B1.1 | no | +| B1 | go-git engine return path untested | WARNING | reviewer-b | fix | B1.1, design *Risks* | no | +| B1 | Race command in test rules fails on Ginkgo | WARNING | reviewer-a | fix | tasks.md test rules | no | +| B1 | Green-at-every-commit claim; T0.1 cleanup wording; marker always Terminal | NOTE | reviewer-a, reviewer-b | fix | tasks.md, B1.1, SCB-1 text | no | +| B1 | `ErrTooManyRequests` logged as "git transport" for every sink | NOTE | reviewer-b | defer | backlog SINK-BREAKER-TOOMANY-MSG | no | +| B1 | Own-path rejections at undecided sites still count; marked trial keeps half-open | NOTE | reviewer-a, reviewer-b | reject | spec fallback; accepted consequence (proposal *Impact*) | no | ## Tasks diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/brief-reviewer-a.md new file mode 100644 index 00000000..f7b2c5f9 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/brief-reviewer-a.md @@ -0,0 +1,42 @@ +# Cold review: branch `fix/sink-breaker-terminal-isolation` (lens: adversarial, whole branch) + +You are an independent reviewer with read-only access to this repository (Go Kubernetes operator +"kollect"). Review the whole branch: `git diff $(git merge-base HEAD origin/main)..HEAD` (10 commits). +The spec is the OpenSpec change `openspec/changes/sink-breaker-terminal-isolation/` (`proposal.md`, +`design.md`, `specs/sink-circuit-breaker/spec.md`, `tasks.md`); per-task evidence is in its +`evidence/` and per-task reviews in `reviews/L-tasks/`. Per-task reviews saw one diff each; you see +the feature. Note: the spec set was rewritten after its last spec-set review round (decision changed +from "exclude all Terminal errors" to "exclude only results marked inventory-scoped"), so it has not +been reviewed in its final form — review it too. + +Goal of the change: one inventory's Git ownership rejections must no longer open the shared per-sink +circuit breaker for every other inventory on that sink, while sink-wide failures (Transient, +Forbidden, unclassified, and sink-wide Terminal such as Git auth / protected-branch rejections) keep +opening it. Plus: reconciler tests that shared a breaker key are isolated. + +Attack it: + +1. Compliance per requirement SCB-1, SCB-2, SCB-3: holds / partial / contradicted / absent / + stronger than spec; and what the code does that no requirement mentions. +2. The marker (`internal/errors`): can it be lost between the raise site and the breaker (any wrap + without `Unwrap`, `%v`, re-classification, `ClassError` nesting where the outer class wins, Git + `ClassifyExportError` running on an already-marked error, multipart/layout export paths that call + a different prune path)? Can it be attached to a result that is actually sink-wide (shared prune + metadata, repository state, capacity limits)? Check every marked site against T0.1's table in + `evidence/T0.1.md` and the code. +3. Ownership rejections from paths other than the 5 marked sites (GitLab MR backend, object-store + layouts, cleanup/delete paths): do any still trip the breaker for other inventories? +4. Half-open behaviour under load: can one inventory's repeated marked results starve recovery + (serialise a sink indefinitely)? Is that what the spec accepts? +5. Tests: could any new test pass vacuously (no `=== RUN`, keys that collide across `-count`, timing + windows, a stub that never reaches the breaker)? Were any existing assertions weakened in G2's + isolation (sink names changed — did every assertion that names the sink follow)? +6. Docs (`docs/operator-manual/troubleshooting.md`, `docs/adr/0602-error-taxonomy.md`): accurate + against the code; nothing private (no internal ids, agent-context references). +7. Commit-by-commit: does each commit build on its own (the change claims the tree is green at every + commit)? Conventional gitmoji titles; no AI/co-author trailers; a `fix` that changes user-visible + behaviour has a title that will read well in a changelog. + +Output: findings, each with severity CRITICAL / WARNING / NOTE, file:line evidence, and a concrete +fix; CRITICAL only for a defect that ships wrong behaviour or makes a requirement fail. End with +`## Verdict` and one of APPROVE / CONCERNS / REQUEST_CHANGES / BLOCK. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/leg-reviewer-a-adversarial.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/leg-reviewer-a-adversarial.md new file mode 100644 index 00000000..c09accfc --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/leg-reviewer-a-adversarial.md @@ -0,0 +1,52 @@ +No CRITICAL defect found. SCB-1, SCB-2 and SCB-3 hold in the final implementation. The remaining concerns are regression coverage and inaccurate validation claims. + +1. **WARNING — The merge-target ownership exclusion lacks a regression assertion.** + `internal/sink/git/prune_target.go:66` is a separate marked raise site, but the new boundary test at `internal/sink/circuit_breaker_test.go:522` exercises the ordinary ownership conflict. Existing merge-target tests assert Terminal class and message, without asserting the marker (`internal/sink/git/prune_merge_target_test.go:139`, `internal/sink/gitlab/backend_branchmr_claims_test.go:165`). Reverting just this site to `invalidPrune` would preserve those assertions while restoring cross-inventory breaker trips for this path. G1’s recorded site mutation covers only `foreignClaimError` (`evidence/G1.md:67`). + + **Fix:** Assert `IsInventoryScoped` in the warm-mirror merge-target test for both engines and in the GitLab boundary test. Add marker assertions for the other three newly marked validation sites, and verify each site-specific reversion fails. + +2. **WARNING — The final task contract still prescribes a known-failing validation command.** + `openspec/changes/sink-breaker-terminal-isolation/tasks.md:21` requires `-count=3` across the controller package without excluding its Ginkgo suite. `evidence/G1.md:81` explicitly records that this command fails because Ginkgo rejects that repetition count. The implementation evidence uses a sensible substitute, but the final contract remains inconsistent. + + **Fix:** Specify separate commands: repeated race tests for errors/sinks; repeated controller tests with `-skip TestControllers`; and a separate single-run Ginkgo gate. + +3. **NOTE — T0.1 overstates which cleanup paths bypass the breaker.** + `evidence/T0.1.md:132` says deletion/cleanup does not go through it. Git ownership release does bypass it, but relational cleanup calls `RunExportEnvelope` at `internal/sink/cleanup.go:379`, thereby using the shared breaker. + + **Fix:** Narrow the statement to Git/GitLab ownership release and snapshot deletion; explicitly document the relational exception. + +4. **NOTE — “Green at every commit” is stronger than the evidence supports.** + `tasks.md:8` claims every commit is green. G2 records a failing debounce test before its follow-up fix (`evidence/G2.md:97`), and earlier commits retain the breaker-isolation flake being repaired. This does not demonstrate a compilation failure, but it contradicts an unconditional green-tree claim. + + **Fix:** Say that tests and implementation land together without deliberate red-test commits, while identifying inherited flakes and the commit that resolves each. + +**Compliance and attack results** + +| Requirement | Assessment | Evidence | +|---|---|---| +| SCB-1 | **Holds** | Marker preserves text and wrapping; `IsExcluded` excludes marked results without resetting failures (`internal/errors/errors.go:157`, `internal/sink/circuit_breaker.go:69`). | +| SCB-2 | **Holds** | Unmarked failures retain gobreaker’s failure semantics. Git classification preserves already-Terminal marked errors and leaves auth/protected-branch errors unmarked. | +| SCB-3 | **Holds** | Exclusions release the half-open slot without changing state; success closes and counted failure reopens. Concurrent-trial tests passed. | + +- All five marked sites match T0.1’s classification. Shared metadata corruption, duplicate records and aggregate capacity limits remain unmarked. +- I found no marker loss through the Git CLI, go-git, GitLab, multipart or layout return paths. Relevant wrappers return errors directly or use `%w`; no reachable outer reclassification overrides the marked Terminal class. +- GitLab ownership checks use the same marked helpers. Object-store exports have no corresponding Git ownership rejection. Git cleanup’s unmarked ownership errors cannot trip this breaker. +- Repeated marked trials **can indefinitely retain half-open operation and deny competing exports while a trial runs**. The final design explicitly accepts this; SCB-3 provides slot release, not fairness or bounded recovery. +- Behavior outside the three requirements includes the three marked validation failures, controller-test isolation and debounce-test serialization. These are described in the design/tasks. +- G2 preserves existing assertions, including the sink-qualified status name. New keys are invocation-unique. The new tests visibly ran and reached their intended boundaries. +- The two public documentation changes accurately describe counting and recovery; I found no private identifiers or agent-context references there. +- All ten commit titles follow the gitmoji/conventional pattern. Commit messages contain no prohibited attribution. The `fix(sink)` title is suitable for a changelog. + +**Validation and limits** + +Passed independently: + +- New marker/breaker tests with verbose execution. +- `go test -race -count=3 ./internal/errors/ ./internal/sink/...` +- `go test -race -count=3 -skip TestControllers ./internal/controller/` + +I could not independently build every historical commit or rerun mutations under read-only access. `go build ./...` was blocked creating its temporary build directory. Historical compilation therefore remains unverified; static inspection found no cross-commit symbol dependency. + +## Verdict + +CONCERNS diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/leg-reviewer-b-spec.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/leg-reviewer-b-spec.md new file mode 100644 index 00000000..3cd26a19 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/leg-reviewer-b-spec.md @@ -0,0 +1,49 @@ +## Verdict: CONCERNS + +## Findings +- [WARNING] Five of the six converted raise sites have no test that checks the marker. Turning any of them back into `invalidPrune` leaves every test green. — `internal/sink/git/prune_target.go:66`, `internal/sink/git/prune_owned.go:174,268,275` + Failure: only `foreignClaimError` (`prune_owned.go:317`) is exercised end to end, by SCB-1e (`circuit_breaker_test.go:522-556`). A grep for `IsInventoryScoped` finds no test in `internal/sink/git`. If someone changes the merge-target claim rejection (GitLab MR / branch export) back to `invalidPrune`, it starts counting again and opens the breaker for every inventory, and no test fails. + Fix: add a table test in `internal/sink/git` that asserts `IsInventoryScoped && IsTerminal` for each `ownershipConflict` site. Calling `checkMergeTargetClaims` (as `prune_merge_target_test.go:318` already does) and `prepareOwnedPrune` is enough. + Confidence: 90 +- [WARNING] The design says SCB-1e covers "marker lost by a re-wrap … through the real Git backend". It only covers the CLI engine. — `internal/sink/inventory_identity_test.go:62`, `internal/sink/git/export.go:115-131` vs `:134-149,186-215` + Failure: the fixture's `file://` endpoint makes `isFileRemote` route to `exportViaCLI`. The go-git engine (`exportRemoteInWorkdir` → `checkMergeTargetClaims` / `writeBillyExportFiles`), which serves production https/ssh sinks, is never driven through the breaker. Reading the code, the chain does survive: `ClassifyExportError` returns early on Terminal (`git/errors.go:29`), and `classifyExportFailure` wraps with `%w`. But that is not tested. + Fix: narrow the claim in the design.md Risks section, or add a go-git-engine case alongside the unit table above. + Confidence: 85 +- [NOTE] SCB-1 says the marker "SHALL keep the error's Terminal class". `InventoryScoped` sets Terminal unconditionally (stronger than spec). — `internal/errors/errors.go:156-162` + Failure: `InventoryScoped(Transient(x))` comes out Terminal, so a transient cause would stop being requeued. Every current caller passes a fresh unclassified `fmt.Errorf`, so nothing breaks today. + Fix: document "always Terminal" in the spec and the doc comment, or preserve an existing class. + Confidence: 60 +- [NOTE] design.md:67-68 says `ErrTooManyRequests` "becomes Transient in classifyExportFailure". It actually goes through `git.ClassifyExportError` first, for every sink type, and comes out as `git transport: too many requests`. — `internal/sink/export.go:208`, `internal/sink/git/errors.go:45,72` + Failure: during a half-open trial, a concurrent export on an S3 or Postgres sink logs a misleading "git transport" error. The troubleshooting row also does not mention this message. The behaviour is pre-existing. + Fix: map `gobreaker.ErrTooManyRequests` in `exportThroughBreaker` the same way as `ErrOpenState`, or correct design.md. + Confidence: 75 +- [NOTE] Some own-path rejections still count toward the breaker. The prune-path checks at `prune_owned.go:53,57` are listed "undecided" in T0.1 (#1–2). The empty export at `git/export.go:95` is the same case (my reading; T0.1 does not list it). — `internal/sink/git/export.go:95,100` + Failure: all of these fail inside `plan.run` without touching the backend, and still count. The spec allows this fallback, and I could not find a reachable input that triggers them. + Fix: if a reachable input turns up, mark the error at the `validateExportFiles` call site, where the scope is known. + Confidence: 40 + +**Compliance** +- **SCB-1: holds.** + - `IsExcluded: kollecterrors.IsInventoryScoped` at `circuit_breaker.go:68`. + - Tests for the streak case (`:359-378`) and the other-inventory case (`:331-356`). + - Marker survival holds on the CLI engine (SCB-1e). The go-git engine is unverified, as in the second finding. +- **SCB-2: holds.** + - `circuit_breaker_test.go:381-427` covers transient, unclassified, forbidden, NotFound and Git-auth Terminal. + - The open state maps to Transient "circuit breaker open" at `circuit_breaker.go:39-41`. + - Gaps: the 60 s interval is untested, and the Git-auth case uses a stub with a pre-classified error, not a real backend. +- **SCB-3: holds.** + - `:449-518` covers half-open followed by success, by failure, and a concurrent export during the trial. + - The concurrent case asserts only `err != nil`, not the error's class, which matches the spec. + +**Behaviour no requirement mentions** +- Two limits now skip the breaker: the 4096-byte owner limit (unreachable for k8s names) and `maxPrunePaths`. +- `SinkErrorsTotal` still counts excluded failures. +- An inventory that keeps failing ownership checks can keep the breaker half-open indefinitely. It does not lock out other inventories, because each excluded trial frees the slot. + +**Docs**: the troubleshooting rows (`troubleshooting.md:214,328`) and the ADR note match the code: key format, 60 s/30 s, counted classes, half-open semantics. + +## Could not check +- Did not run any tests or `task check`. The ones behind gobreaker v2.4.0 behaviour (that `onExclusion` frees the half-open slot) are `circuit_breaker_test.go:449-518`, and I read only the diff and the design's citation of the library. +- Did not read the controller test diffs (`internal/controller/*_test.go`) in depth, or `evidence/G1-G3.md`, `loop.md`, `tasks.md`, `proposal.md`, or the committed review records. I checked the rewritten spec only against the code, not against those records. +- Did not check whether other backends (BigQuery, MongoDB) raise inventory-scoped Terminal errors that should also be marked. +- Did not check what `reviews/B-branch/` contains (untracked). diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/verification.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/verification.md new file mode 100644 index 00000000..5852c263 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-1/verification.md @@ -0,0 +1,23 @@ +# B round 1 — verification and triage + +Branch tip `4e50a9a2f` (rebased on `origin/main` after the Helm v4 and mongo-driver v2 merges). +Reviewer A (adversarial): CONCERNS. Reviewer B (spec): CONCERNS. Both: SCB-1, SCB-2, SCB-3 hold. No +CRITICAL. Each finding below was checked against the code by the orchestrator on 2026-11-05. + +| # | Finding | Sev | Reviewers | Verified | Disposition | +|---|---|---|---|---|---| +| 1 | Only one of the routed raise sites (`foreignClaimError`) is pinned by a test; reverting any other to `invalidPrune` keeps every test green | WARNING | A, B | yes: no `IsInventoryScoped` assertion in `internal/sink/git` tests; G1's site mutation covers one site | fix: task B1.1 (per-site table + per-site mutations) | +| 2 | SCB-1e drives only the CLI engine (`file://` remote); the go-git engine's return path is untested | WARNING | B | yes: `isFileRemote` routes the fixture to `exportViaCLI` | fix: B1.1 go-git case; design *Risks* narrowed | +| 3 | tasks.md test-rule race command fails on Ginkgo `-count>1` | WARNING | A | yes (`evidence/G1.md`) | fix: tasks.md test rules | +| 4 | "Green at every commit" stronger than evidence (inherited flakes until G2) | NOTE | A | yes | fix: tasks.md preamble | +| 5 | T0.1 says cleanup bypasses the breaker; relational cleanup calls `RunExportEnvelope` (`cleanup.go:379`) | NOTE | A | yes | fix: B1.1 evidence wording | +| 6 | `InventoryScoped` always yields Terminal, stronger than "keeps the class" | NOTE | B | yes (`errors.go:156-162`) | fix: SCB-1 text "make the error Terminal"; doc comment in B1.1 | +| 7 | `ErrTooManyRequests` passes the Git classifier for every sink type and logs "git transport: too many requests" | NOTE | B | yes (`export.go:208`, `git/errors.go:45`) | defer: pre-existing, class unaffected; backlog SINK-BREAKER-TOOMANY-MSG; design.md corrected | +| 8 | Some own-path rejections (`prune_owned.go:53,57`, `git/export.go:95`) still count | NOTE | B | yes; no reachable input found | reject: the spec's fallback (undecided sites stay unmarked, T0.1) | +| 9 | A Terminal-marked trial can keep the breaker half-open indefinitely | NOTE | A, B | yes | reject: accepted consequence, recorded in proposal *Impact*; each excluded trial frees the slot, so no lock-out | + +Holistic `task check` on `4e50a9a2f`: exit 1, four gates red, none from this branch — `gitleaks-detect` +(one hit in the git-ignored mkdocs cache `.cache/plugin/social/manifest.json` left by a local strict +docs build; removed), `demo_04_samples_kustomize_test` (mise `kubectl` shim without a version), +`dist_olm_bundle_test` and `dist_operatorhub_pr_test` (no local `operator-sdk`). The last three fail +identically on a clean `origin/main` worktree (checked 2026-11-05). All other gates green. diff --git a/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md b/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md index a514d53f..a1b40a5b 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md +++ b/openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md @@ -7,7 +7,7 @@ The shared per-sink circuit breaker SHALL exclude an export plan result that carries the inventory-scoped marker (`errors.Is(err, kollecterrors.ErrInventoryScoped)`): it SHALL count neither as a failure nor as a success, and SHALL NOT reset the consecutive-failure count. The marker SHALL -keep the error's Terminal class and its text. +make the error Terminal and keep its text. #### Scenario: One inventory's ownership rejections do not open the breaker diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index b6781a3a..7c453ddd 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -5,8 +5,9 @@ Red first. Each group's test task comes before its implementation and records it a compile error or a skip. "Red on base" means red on the group's refactor base: for G1, `origin/main` plus the pure `newBreaker` extraction and the marker API stubs named in 1.1; for G2, the G1 commit. A stub with the signature from `design.md` and a `not implemented` body (a no-op that returns its input -unmarked) is in scope so tests compile. Each group closes in one commit (tests and implementation), -so the tree is green at every commit. A case marked *control* is expected green on its base and is +unmarked) is in scope so tests compile. Each group closes in one commit (tests and implementation), so no +commit deliberately carries a red test; flakes inherited from the base stay until the group that +fixes them (G2 for the breaker-sharing and debounce-metric flakes). A case marked *control* is expected green on its base and is recorded as a control, not a red. G2 may close without a red when its target is preventive (see 2.1); it says so in its evidence. @@ -18,7 +19,8 @@ Test rules for every group: key unique per test invocation (`-count` safe). No parallel test calls `ResetBreakersForTest`. The only timeout seam is the `newBreaker(key, timeout)` constructor; half-open tests follow the timing protocol in `design.md` *Testing approach*; no test writes a package variable. -- Each group records `go test -race -count=3 ./internal/errors/ ./internal/sink/... ./internal/controller/` +- Each group records `go test -race -count=3 ./internal/errors/ ./internal/sink/...`, + `go test -race -count=3 -skip TestControllers ./internal/controller/` (Ginkgo refuses `-count>1`), and `task lint`. - Mutations are judged by the test command's exit status, with one no-op control that must survive; mutant trees live outside the repository. @@ -115,6 +117,21 @@ Test rules for every group: lint:markdown` 0 issues; `rg -n 'circuit breaker' docs` reviewed for any other claim the change contradicts, the list recorded in `evidence/G3.md`. — closed 2026-11-05, evidence: evidence/G3.md +## B1. Branch review round 1 fixes + +- [ ] B1.1 Marker regression per site (branch review round 1, both reviewers): a table test in + `internal/sink/git` asserting `kollecterrors.IsInventoryScoped(err) && kollecterrors.IsTerminal(err)` + for every site routed through the inventory-scoped helper (`prune_owned.go` and `prune_target.go`; + list them from the code at HEAD), driven through the functions that raise them (e.g. + `checkMergeTargetClaims`, `prepareOwnedPrune`), plus one boundary case through the go-git engine + (non-`file://` path, `exportRemoteInWorkdir`) next to SCB-1e's CLI-engine case, or, if the go-git + engine cannot be driven without a network remote, a unit case on the go-git return path that shows + the marker survives `ClassifyExportError` and say so. Mutation: revert each site to `invalidPrune` + one at a time; each must be killed by the new table. Also: the doc comment on `InventoryScoped` + says it always yields Terminal (whatever the input's class); `evidence/T0.1.md`'s cleanup statement + is narrowed (Git/GitLab ownership release and snapshot deletion bypass the breaker; relational + cleanup goes through `RunExportEnvelope`, `internal/sink/cleanup.go:379`). + ## 4. Integration - [ ] 4.1 Integrator (orchestrator): branch review (stage B), PR with the release-note line, merge on From d243fd5f4bf5d2da70f25234d095d5e6f371efa5 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 13:21:08 +0100 Subject: [PATCH 12/13] :white_check_mark: test(sink): pin the inventory-scoped marker per site Branch review round 1 found that only one ownershipConflict raise site was pinned by a test. A table in internal/sink/git now drives every routed site through the function that raises it, plus the go-git engine through ClassifyExportError; each site reverted to invalidPrune fails its row. The InventoryScoped doc comment now says it always yields Terminal, pinned by a test, and T0.1's cleanup statement is narrowed: relational cleanup goes through RunExportEnvelope. --- internal/errors/errors.go | 6 +- internal/errors/errors_test.go | 20 ++ internal/sink/git/prune_scoped_marker_test.go | 182 ++++++++++++++++++ .../evidence/B1.md | 149 ++++++++++++++ .../evidence/T0.1.md | 6 +- .../sink-breaker-terminal-isolation/loop.md | 3 + .../reviews/L-tasks/B1/brief-reviewer-a.md | 76 ++++++++ .../reviews/L-tasks/B1/leg-reviewer-a-diff.md | 6 + .../sink-breaker-terminal-isolation/tasks.md | 4 +- 9 files changed, 446 insertions(+), 6 deletions(-) create mode 100644 internal/sink/git/prune_scoped_marker_test.go create mode 100644 openspec/changes/sink-breaker-terminal-isolation/evidence/B1.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/B1/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/B1/leg-reviewer-a-diff.md diff --git a/internal/errors/errors.go b/internal/errors/errors.go index 14a4dca3..6f386ecd 100644 --- a/internal/errors/errors.go +++ b/internal/errors/errors.go @@ -152,8 +152,10 @@ func Format(class error, format string, args ...any) error { // ownership claim), not about the sink. The per-sink circuit breaker does not count it. var ErrInventoryScoped = errors.New("inventory-scoped") -// InventoryScoped marks err as Terminal and inventory-scoped. The result keeps err's text and -// its errors.Is/errors.As chain, so status messages and logs do not change. +// InventoryScoped marks err as inventory-scoped and always makes it Terminal, whatever class err +// carried before: ClassOf and IsTerminal report Terminal for a Transient or Forbidden input too (the +// outer class wins; the inner class sentinel stays reachable through errors.Is). The result keeps +// err's text and its errors.Is/errors.As chain, so status messages and logs do not change. func InventoryScoped(err error) error { if err == nil { return nil diff --git a/internal/errors/errors_test.go b/internal/errors/errors_test.go index 238497b6..61e6f6d7 100644 --- a/internal/errors/errors_test.go +++ b/internal/errors/errors_test.go @@ -182,3 +182,23 @@ func TestInventoryScoped(t *testing.T) { t.Fatal("a %w re-wrap lost the marker or the class") } } + +// TestInventoryScoped_alwaysTerminal (B1.1, the InventoryScoped doc comment): the marker makes any +// input Terminal; the input's own class sentinel stays reachable through errors.Is. +func TestInventoryScoped_alwaysTerminal(t *testing.T) { + t.Parallel() + + for name, in := range map[string]error{ + "transient": Transient(errors.New("dial")), + "forbidden": Forbidden(errors.New("denied")), + "plain": errors.New("plain"), + } { + marked := InventoryScoped(in) + if ClassOf(marked) != ClassTerminal || !IsTerminal(marked) || IsTransient(marked) || IsForbidden(marked) { + t.Fatalf("%s: ClassOf(InventoryScoped(e)) = %q, want %q", name, ClassOf(marked), ClassTerminal) + } + } + if !errors.Is(InventoryScoped(Transient(errors.New("dial"))), ErrTransient) { + t.Fatal("InventoryScoped hid the input's class sentinel from errors.Is") + } +} diff --git a/internal/sink/git/prune_scoped_marker_test.go b/internal/sink/git/prune_scoped_marker_test.go new file mode 100644 index 00000000..cec82362 --- /dev/null +++ b/internal/sink/git/prune_scoped_marker_test.go @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 Konrad Heimel + +package git + +import ( + "fmt" + "path/filepath" + "strings" + "testing" + "time" + + billy "github.com/go-git/go-billy/v5" + "github.com/go-git/go-billy/v5/memfs" + "github.com/go-git/go-git/v5" + "github.com/go-git/go-git/v5/plumbing/object" + "github.com/go-git/go-git/v5/storage/memory" + + kollecterrors "github.com/platformrelay/kollect/internal/errors" +) + +// assertInventoryScopedTerminal fails unless err is Terminal and carries the inventory-scoped marker +// (SCB-1: the per-sink circuit breaker must not count it) and its text contains want. +func assertInventoryScopedTerminal(t *testing.T, err error, want string) { + t.Helper() + if !kollecterrors.IsInventoryScoped(err) || !kollecterrors.IsTerminal(err) { + t.Fatalf("err = %v (class %s, inventory-scoped %v), want terminal and inventory-scoped", + err, kollecterrors.ClassOf(err), kollecterrors.IsInventoryScoped(err)) + } + if !strings.Contains(err.Error(), want) { + t.Fatalf("err = %v, want text containing %q", err, want) + } +} + +// TestOwnershipConflict_everyRoutedSiteIsInventoryScoped (B1.1): each raise site routed through +// ownershipConflict is driven through the function that raises it and must come back Terminal and +// inventory-scoped. Reverting any one site to invalidPrune fails its row. The last row is the +// control: a rejection about shared prune metadata stays unmarked, so the breaker still counts it. +func TestOwnershipConflict_everyRoutedSiteIsInventoryScoped(t *testing.T) { + t.Parallel() + const shared = "prod/team-a/Deployment/api.yaml" + tooManyPaths := make([]string, maxPrunePaths+1) + for i := range tooManyPaths { + tooManyPaths[i] = fmt.Sprintf("p/%d.yaml", i) + } + for name, tc := range map[string]struct { + site string // the raise site at the time of writing, for the reader + setup func(t *testing.T, fs billy.Filesystem) + cfg Config + written []string + want string // substring of the rejection + scoped bool + }{ + "owner too long": { + site: "prune_owned.go:174", cfg: Config{Prune: true, PruneOwner: strings.Repeat("o", 4097)}, + written: []string{"own.yaml"}, want: "prune owner exceeds 4096 bytes", scoped: true, + }, + "keep set too large": { + site: "prune_owned.go:268", cfg: Config{Prune: true, PruneOwner: "b", PruneKeepPaths: tooManyPaths}, + written: []string{"p/0.yaml"}, want: "prune inventory exceeds", scoped: true, + }, + "written path missing from keep set": { + site: "prune_owned.go:275", cfg: Config{Prune: true, PruneOwner: "b", PruneKeepPaths: []string{"keep.yaml"}}, + written: []string{"other.yaml"}, want: "missing from prune keep set", scoped: true, + }, + "foreign claim on a written path (no prune)": { + site: "prune_owned.go:317 via checkForeignClaim", + setup: func(t *testing.T, fs billy.Filesystem) { commitOwnedPaths(t, fs, "a", shared) }, + cfg: Config{PruneOwner: "b"}, written: []string{shared}, + want: "belongs to another inventory", scoped: true, + }, + "foreign claim on a later part's claim path": { + site: "prune_owned.go:317 via checkForeignClaim (PruneClaimPaths)", + setup: func(t *testing.T, fs billy.Filesystem) { commitOwnedPaths(t, fs, "a", shared) }, + cfg: Config{PruneOwner: "b", PruneClaimPaths: []string{shared}}, written: []string{"own.yaml"}, + want: "belongs to another inventory", scoped: true, + }, + "record-advancing claim, other owner sorts first": { + site: "prune_owned.go:340 (checkSingleClaims)", + setup: func(t *testing.T, fs billy.Filesystem) { commitOwnedPaths(t, fs, "a", shared) }, + cfg: Config{Prune: true, PruneOwner: "b"}, written: []string{shared}, + want: "belongs to another inventory", scoped: true, + }, + "record-advancing claim, exporter sorts first": { + site: "prune_owned.go:342 (checkSingleClaims)", + setup: func(t *testing.T, fs billy.Filesystem) { commitOwnedPaths(t, fs, "b", shared) }, + cfg: Config{Prune: true, PruneOwner: "a"}, written: []string{shared}, + want: "belongs to another inventory", scoped: true, + }, + "control: shared metadata is not a directory": { + site: "prune_owned.go:135 (invalidPrune, sink-wide)", + setup: func(t *testing.T, fs billy.Filesystem) { mustWriteBillyFile(t, fs, pruneMetadataDir, "x") }, + cfg: Config{Prune: true, PruneOwner: "b"}, written: []string{"own.yaml"}, + want: "must be a real directory", + }, + } { + t.Run(name, func(t *testing.T) { + t.Parallel() + fs := memfs.New() + if tc.setup != nil { + tc.setup(t, fs) + } + _, err := prepareOwnedPrune(fs, tc.cfg, tc.written) + if tc.scoped { + assertInventoryScopedTerminal(t, err, tc.want) + return + } + if !kollecterrors.IsTerminal(err) || kollecterrors.IsInventoryScoped(err) || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("%s: err = %v (inventory-scoped %v), want an unmarked terminal %q", + tc.site, err, kollecterrors.IsInventoryScoped(err), tc.want) + } + }) + } + + t.Run("merge target claim", func(t *testing.T) { // prune_target.go:66 (checkMergeTargetClaims) + t.Parallel() + fs := memfs.New() + repo, err := git.Init(memory.NewStorage(), fs) + if err != nil { + t.Fatal(err) + } + commitOwnedPaths(t, fs, "a", shared) + wt, err := repo.Worktree() + if err != nil { + t.Fatal(err) + } + if addErr := wt.AddGlob("."); addErr != nil { + t.Fatal(addErr) + } + sig := &object.Signature{Name: "Kollect Tests", Email: "kollect-tests@example.com", When: time.Now()} + if _, commitErr := wt.Commit("target", &git.CommitOptions{Author: sig}); commitErr != nil { + t.Fatal(commitErr) + } + head, err := repo.Head() + if err != nil { + t.Fatal(err) + } + err = checkMergeTargetClaims(repo, head.Name(), "main", Config{Prune: true, PruneOwner: "b"}, []string{shared}) + assertInventoryScopedTerminal(t, err, "belongs to another inventory on merge target branch") + }) +} + +// TestOwnershipConflict_goGitEngineKeepsMarker (B1.1): the go-git engine (exportRemoteInWorkdir, the +// engine for every non-file:// remote) returns an ownership rejection that is still Terminal and +// inventory-scoped after ClassifyExportError, the classifier ExportFilesWithBranch and +// RunExportEnvelope apply. SCB-1e covers the CLI engine (a file:// remote routes there); this drives +// the go-git engine directly against a local bare repository, offline, as the warm-mirror tests in +// prune_merge_target_test.go do. +func TestOwnershipConflict_goGitEngineKeepsMarker(t *testing.T) { + const ( + shared = "prod/team-a/Deployment/api.yaml" + ownPath = "prod/team-b/Deployment/worker.yaml" + ) + for name, tc := range map[string]struct { + branch *BranchSpec + want string + }{ + "same branch (prepareOwnedPrune)": {want: "belongs to another inventory"}, + "merge target (checkMergeTargetClaims)": {branch: &BranchSpec{PushBranch: "kollect/team-b/apps", CloneBranch: "main"}, want: `on merge target branch "main"`}, + } { + t.Run(name, func(t *testing.T) { + skipWithoutGit(t) + remote := createBareRemoteWithMainCommit(t) + cloneURL := "file://" + remote + a, _ := InventoryPruneOwner("KollectInventory", "default", "team-a", "apps") + b, _ := InventoryPruneOwner("KollectInventory", "default", "team-b", "apps") + aCfg := Config{Endpoint: cloneURL, Prune: true, PruneOwner: a}.withDefaults() + if err := ExportFilesWithBranch(t.Context(), aCfg, Auth{}, []FileEntry{{Path: shared, Data: []byte("a")}}, nil, CommitContext{}); err != nil { + t.Fatalf("A's export: %v", err) + } + cfg := Config{Endpoint: cloneURL, Prune: true, PruneOwner: b}.withDefaults() + files := []FileEntry{{Path: ownPath, Data: []byte("b")}, {Path: shared, Data: []byte("b")}} + req, validated, err := validateExportFiles(cfg, files, tc.branch) + if err != nil { + t.Fatal(err) + } + workdir := filepath.Join(t.TempDir(), "mirror") + err = ClassifyExportError(exportRemoteInWorkdir(t.Context(), cfg, nil, req, validated, CommitContext{}, workdir)) + assertInventoryScopedTerminal(t, err, tc.want) + }) + } +} diff --git a/openspec/changes/sink-breaker-terminal-isolation/evidence/B1.md b/openspec/changes/sink-breaker-terminal-isolation/evidence/B1.md new file mode 100644 index 00000000..ff51ec13 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/evidence/B1.md @@ -0,0 +1,149 @@ +# B1 — branch review round 1 fixes (task B1.1) + +- Change: `openspec/changes/sink-breaker-terminal-isolation/` · branch: `fix/sink-breaker-terminal-isolation` · pre-task sha: `7a26657c0` · started: 2026-11-05 +- Origin: `reviews/B-branch/round-1/verification.md` findings 1, 2, 5, 6. +- Satisfies: SCB-1 (`specs/sink-circuit-breaker/spec.md`): every routed ownership rejection is Terminal and inventory-scoped, so the breaker does not count it; on both Git engines. +- Smallest coherent change: one table test over the routed raise sites, one go-git engine case, one test pinning the `InventoryScoped` doc claim, the doc comment, and the T0.1 cleanup sentence. No production behaviour change. +- Files: `internal/sink/git/prune_scoped_marker_test.go` (new), `internal/errors/errors_test.go` (one new test), `internal/errors/errors.go` (doc comment only), `evidence/T0.1.md` (one statement narrowed). +- Status: see *Verdict*. + +## Framing: no red on base for the table + +The code already behaves correctly (G1 routed the sites), so the table has no red on the base. +Its proof is the mutation run: each routed site reverted to `invalidPrune`, one at a time, must +turn the table red on that site's row; a no-op control must survive. Same for the new +`TestInventoryScoped_alwaysTerminal` (pins the doc comment) and for the go-git case. + +## Routed sites at HEAD (`7a26657c0`) + +Listed from the code, `rg -n 'ownershipConflict|foreignClaimError' internal/sink/git/*.go` minus tests (verified 2026-11-05): + +| Site | Raised in | Reached through | Table row | +|---|---|---|---| +| `prune_owned.go:174` owner > 4096 bytes | `prepareOwnedPrune` | `prepareOwnedPrune` | owner too long | +| `prune_owned.go:268` keep set > `maxPrunePaths` | `validateOwnedPrunePaths` | `prepareOwnedPrune` | keep set too large | +| `prune_owned.go:275` written path missing from keep set | `validateOwnedPrunePaths` | `prepareOwnedPrune` | written path missing from keep set | +| `prune_owned.go:317` `foreignClaimError` body | `checkForeignClaim` (written path, no prune; claim path) | `prepareOwnedPrune` | foreign claim on a written path (no prune); foreign claim on a later part's claim path | +| `prune_owned.go:340` call `foreignClaimError(p, prev)` | `checkSingleClaims` | `prepareOwnedPrune` (prune on) | record-advancing claim, other owner sorts first | +| `prune_owned.go:342` call `foreignClaimError(p, owner)` | `checkSingleClaims` | `prepareOwnedPrune` (prune on) | record-advancing claim, exporter sorts first | +| `prune_target.go:66` merge-target claim | `checkMergeTargetClaims` | `checkMergeTargetClaims` on an in-memory go-git repo | merge target claim | + +`ownershipConflict` itself is `prune_owned.go:261-263`. Control row: `prune_owned.go:135` (metadata +directory is a file, `invalidPrune`, sink-wide) must stay Terminal and unmarked. + +## go-git engine case + +Existing offline driver found: `prune_merge_target_test.go:108,185` call `exportRemoteInWorkdir` +directly on a local bare repository (`file://` URL, go-git's file transport, no network; the +`isFileRemote` routing in `ExportFilesWithBranch` is bypassed). The new +`TestOwnershipConflict_goGitEngineKeepsMarker` does the same and passes the engine's return value +through `ClassifyExportError` (what `export.go:149` and `internal/sink/export.go:208` apply), asserting +Terminal + inventory-scoped. Two subtests: same branch (go-git `prepareOwnedPrune` at `export.go:582`, +raise site `:340`) and merge-request branch (`checkMergeTargetClaims` at `export.go:210`, raise site +`prune_target.go:66`). So the go-git engine is driven, not only a unit case on the return path. It +does not go through `RunExportEnvelope` and the breaker (that needs a non-`file://` remote, available +only in the integration build's loopback HTTP server); the breaker side is SCB-1e on the CLI engine +plus G1's unit cases. (verified 2026-11-05) + +## Verification matrix + +All rows verified 2026-11-05 on the working tree after the lint fix (no code changed since). + +| # | Claim / invariant | Sensor | Class | Result | +|---|---|---|---|---| +| 1 | Every routed site (7 above) returns Terminal + inventory-scoped through the function that raises it | `TestOwnershipConflict_everyRoutedSiteIsInventoryScoped` (9 subtests) | machine | pass | +| 2 | Reverting any one routed site to `invalidPrune` turns its row red | mutants M1-M7 (below), each exit 1 on the site's row | machine | pass | +| 3 | A shared-metadata rejection stays unmarked (no over-marking) | control row of the table | machine | pass | +| 4 | go-git engine: ownership rejection keeps the marker through `ClassifyExportError` (same branch and merge-request branch) | `TestOwnershipConflict_goGitEngineKeepsMarker` (2 subtests, git present, not skipped: `--- PASS` lines) | machine | pass | +| 5 | Doc comment: `InventoryScoped` always yields Terminal whatever the input's class; inner class sentinel still reachable via `errors.Is` | `TestInventoryScoped_alwaysTerminal`; mutant M8 | machine | pass | +| 6 | T0.1 cleanup statement narrowed: Git/GitLab release (`cleanup.go:273`) and snapshot deletion (`cleanup.go:240`) bypass the breaker; relational cleanup calls `RunExportEnvelope` (`cleanup.go:379`) | `rg -n 'RunExportEnvelope\(\|DeleteExport\(\|ReleaseExport\(' internal/sink/cleanup.go` → `240`, `273`, `379` | independent-review | pass | +| 7 | No production behaviour change | `git diff 7a26657c0 -- ':!*_test.go' ':!openspec'` shows only the `errors.go` comment | independent-review | pass | +| 8 | New tests isolated: no breaker key used, no `ResetBreakersForTest`; the go-git test is not parallel (uses temp dirs only) | diff review | independent-review | pass | + +## Mutations + +Runner: a Python script kept in the session scratch directory (outside the repository), applied to an +rsync copy of the worktree; each file restored from the first saved copy after each mutant; the copy +was `diff -r`-identical to the worktree after the run. Judged by `go test` exit status. Commands: +`go test -count=1 -v -run TestOwnershipConflict_ ./internal/sink/git/` (13 `=== RUN`) and +`go test -count=1 -v -run TestInventoryScoped ./internal/errors/` (2 `=== RUN`). + +| Mutant | Exit | Kill line (first) | +|---|---|---| +| M0 control: rename `ownershipConflict`'s `format` parameter | 0 (survives) | — | +| M1 `:174` → `invalidPrune` | 1 | `owner_too_long`: `err = prune owner exceeds 4096 bytes (class terminal, inventory-scoped false), want terminal and inventory-scoped` | +| M2 `:268` → `invalidPrune` | 1 | `keep_set_too_large`: `err = prune inventory exceeds 100000 paths (class terminal, inventory-scoped false)` | +| M3 `:275` → `invalidPrune` | 1 | `written_path_missing_from_keep_set`: `… missing from prune keep set (class terminal, inventory-scoped false)` | +| M4 `:317` body → `invalidPrune` | 1 | four `checkForeignClaim`/`checkSingleClaims` rows and the go-git same-branch subtest | +| M5 `:340` call → `invalidPrune` | 1 | `record-advancing_claim,_other_owner_sorts_first` and go-git `same_branch_(prepareOwnedPrune)` | +| M6 `:342` call → `invalidPrune` | 1 | `record-advancing_claim,_exporter_sorts_first` only | +| M7 `prune_target.go:66` → `invalidPrune` | 1 | `merge_target_claim` and go-git `merge_target_(checkMergeTargetClaims)` | +| M8 `InventoryScoped` returns an explicitly classed input unwrapped by `Terminal` | 1 | `errors_test.go:198: transient: ClassOf(InventoryScoped(e)) = "transient", want "terminal"` (only the new test; `TestInventoryScoped` survives it) | + +## Sensors run + +| Tier | Command | sha / tree | Exit | Seconds | Note | +|---|---|---|---|---|---| +| inner | `go build ./...`, `gofmt -l internal`, `go vet ./internal/errors/ ./internal/sink/git/`, `go vet -tags=integration ./internal/sink/git/` | working tree on `7a26657c0` | 0 | 9 | no output | +| inner | `go test -count=1 ./internal/errors/ ./internal/sink/...` | working tree | 0 | 13 | all ok | +| task (fitness: arch) | `task lint` | working tree after the shadow fix | 0 | 18 | `0 issues.`, arch `OK - No warnings found`; first run had 2 `govet shadow` in the new test (fixed) | +| task (fitness: race) | `go test -race -count=3 ./internal/errors/ ./internal/sink/...` | working tree | 0 | 42 | all ok | +| task (fitness: race) | `go test -race -count=3 -skip TestControllers ./internal/controller/` | working tree | 0 | 13 | ok (N = 3) | +| milestone | mutants M0-M8 | scratch copy of the final test tree | as listed | ~60 | | +| task (fitness: unit, FINAL) | `go test -count=1 -v ./internal/sink/ ./internal/controller/ -skip TestControllers` | final code tree (after review; only evidence and task/loop files changed since) | 0 | 10 | both packages ok, 618 `=== RUN` | +| task (fitness: markdown) | `task lint:markdown` | final tree, files staged | 0 | ~20 | `Linting: 425 files`, `Summary: 0 issues in 0 files` (B1.md included; `reviews/**` excluded by the glob) | + +## Iterations + +1 · `task lint` · IMPLEMENTATION · `govet shadow` on `err` in the merge-target subtest (`AddGlob`, `Commit`); renamed to `addErr`/`commitErr`; lint 0 issues; mutation sweep rerun on the fixed tree, same results. + +## Known red + +None added; none cleared. + +## Test changes + +None. Two new test functions and one new test file; no earlier test edited. + +## Harness gaps + +None. + +## Maintainability + +1. Locality: yes, tests in the package whose sites they pin; one doc comment. +2. Change surface: ~190 test lines for 7 sites, one control and 2 engine cases. +3. Duplication: the go-git case repeats the setup pattern of `prune_merge_target_test.go` (helpers reused: `createBareRemoteWithMainCommit`, `validateExportFiles`, `commitOwnedPaths`). +4. Second way: no. +5. Coupling: no new import edge in production code; test imports go-git `storage/memory` (already a module dependency). +6. Abstraction: tests call the raising functions, not `ownershipConflict` directly. +7. Debt for green: none. +8. Legibility: each row names its site in a `site` field (line numbers as of `7a26657c0`). + +## Independent review + +Round 1, one leg (reviewer-a, read-only, `reviews/L-tasks/B1/leg-reviewer-a-diff.md`), brief +`reviews/L-tasks/B1/brief-reviewer-a.md`. No paid leg: no trigger (no secrets, CI, infra or +manifests; verdict not BLOCK). The reviewer received the working-tree diff against `7a26657c0` (new +files intent-to-add). + +```text +verdict: APPROVE +findings: [] +matrix_disputes: [] +could_not_check: + - Did not independently rerun M0-M8, full build/vet/lint, or race suites; mutation writes are outside this session's read-only permissions. +summary: B1.1 is satisfied: all routed rejection paths are covered, all nine table cases and both go-git cases passed without skips, marker tests passed, cleanup wording matches code, and production changes are comment-only. +``` + +No findings, so no second round. + +## Verdict + +CLOSED (2026-11-05). Rows 1-5 machine, rows 6-8 independent-review, all pass; reviewer-a APPROVE; +FINAL unit fitness run green. + +## Learnings + +- tooling: `exportRemoteInWorkdir` can be driven offline on a local bare repository (`file://`, go-git's file transport) by calling it directly; only `ExportFilesWithBranch`'s `isFileRemote` routing sends `file://` to the CLI engine. A go-git engine test does not need the integration build's HTTP server unless it must pass through `RunExportEnvelope`. +- guidance: a test-strengthening task has no base red; record the mutation table as its proof and add one mutant per routed call (here `foreignClaimError`'s body and its two `checkSingleClaims` call sites are separate mutants with separate rows). diff --git a/openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md b/openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md index ce6b801b..86ac6eea 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md +++ b/openspec/changes/sink-breaker-terminal-isolation/evidence/T0.1.md @@ -129,8 +129,10 @@ prune error survives it. GitLab wraps the same call and returns its error as is ## Table (a): Terminal results raised inside an export plan "Inside an export plan" = reachable from `plan.run` at `export.go:200`. The only `exportThroughBreaker` call -is `export.go:200` (verified by grep). Deletion/cleanup (`RunCleanupExport`, `cleanup.go:240,273`) does not -go through the breaker. +is `export.go:200` (verified by grep). Deletion/cleanup (`RunCleanupExport`): the Git/GitLab ownership +release (`ReleaseExport`, `cleanup.go:273`) and snapshot deletion (`DeleteExport`, `cleanup.go:240`) do not go +through the breaker; relational cleanup (`pruneRelationalExport`) calls `RunExportEnvelope` (`cleanup.go:379`) +and so does (narrowed in B1.1, branch review round 1 finding 5, verified 2026-11-05). Classes: **inv** = inventory-scoped (about the inventory's own paths or its ownership claim, design.md *Marker*); **sink** = sink-wide (shared repository or prune-metadata state, endpoint, credentials); diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index 1d75f87a..9f5fc636 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -64,6 +64,7 @@ Holistic run at B: `task check` on `4e50a9a2f` exit 1; four gates red, none from | G1 (1.1 + 1.2) | CLOSED | reviewer-a APPROVE (1 P2 evidence wording fixed), 1 round, no paid leg (`reviews/L-tasks/G1/`) | none; race row split: Ginkgo `TestControllers` refuses `-count=3`, run with `-skip TestControllers` | | G2 (2.1 + 2.2) | CLOSED | round 1 reviewer-a REQUEST_CHANGES (P1 9/10 green missed 2.2's clause → decision A; P2 evidence wording fixed); round 2 reviewer-a APPROVE; no paid leg (`reviews/L-tasks/G2/`, `round-2/`) | none. Base 3/10 red (shared `default/postgres-demo`); after per-invocation sink names and the debounce-metric test made non-parallel: 10/10 green | | G3 (3.1) | CLOSED | reviewer-a REQUEST_CHANGES (2 P1 operator-wording over-claims, verified and fixed, no new round); no paid leg (`reviews/L-tasks/G3/`) | none. `task docs:verify` not-run (host `kubectl` shim); strict mkdocs build, docs unit contracts and doc guards green instead | +| B1 (B1.1) | CLOSED | reviewer-a APPROVE, no findings, 1 round, no paid leg (`reviews/L-tasks/B1/`) | none. No base red by design (test-strengthening); proof is mutants M1-M7 (each routed site) + M8 (doc claim), control M0 survives | ## Known red @@ -89,6 +90,8 @@ Tests from a test task changed by a later task (close-task `TEST` class); each o - G2: the ten-run race loop is probabilistic and cannot kill a mutant; a deterministic sensor for breaker sharing is `-count=3 -run` over only the tests whose backend fails (no success between iterations to reset the streak): 5 counted failures in iteration 1 open a shared breaker. - G2: `TestClusterInventory_DebouncedExportIncrementsMetric` is a separate parallel metric-delta flake in the same race command (~1/10); a red there is not the breaker. - G3: `task docs:verify` can fail on a missing mise `kubectl` version before its mkdocs step; build strict from a scratch venv of `docs/requirements-docs.txt` and run `test/docs` unit contracts plus the guards naming the edited files. +- B1: go-git engine tests run offline by calling `exportRemoteInWorkdir` directly on a local bare repo (`file://`); only `ExportFilesWithBranch` routes `file://` to the CLI engine. +- B1: per-site marker table `TestOwnershipConflict_everyRoutedSiteIsInventoryScoped` (`internal/sink/git/prune_scoped_marker_test.go`) lists routed sites with their line as of `7a26657c0`; a new `ownershipConflict` site needs a row there. ## Where a human should look first diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/B1/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/B1/brief-reviewer-a.md new file mode 100644 index 00000000..1a1048fc --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/B1/brief-reviewer-a.md @@ -0,0 +1,76 @@ +# Review brief — task B1.1 (diff-only, independent) + +You are reviewing one task, independently of whoever implemented it. Read-only: do not modify any +file in the repository. You may copy the tree to a temp directory outside it for mutation checks. + +Repository: current directory (git worktree, branch `fix/sink-breaker-terminal-isolation`). +Base: `7a26657c0`. The change is the working tree against the base (new files are intent-to-add): +run `git diff 7a26657c0` to see it. Judge only that diff. + +## 1. TASK + +`openspec/changes/sink-breaker-terminal-isolation/tasks.md`, line B1.1 (read it there). In short: +a table test in `internal/sink/git` asserting `kollecterrors.IsInventoryScoped(err) && +kollecterrors.IsTerminal(err)` for every site routed through the inventory-scoped helper +(`prune_owned.go`, `prune_target.go`, listed from the code at HEAD), driven through the functions +that raise them; one boundary case through the go-git engine (`exportRemoteInWorkdir`), or, if that +cannot run without a network remote, a unit case on the go-git return path through +`ClassifyExportError`; mutation: revert each site to `invalidPrune` one at a time, each killed by the +table; the `InventoryScoped` doc comment says it always yields Terminal whatever the input's class; +`evidence/T0.1.md`'s cleanup statement is narrowed (Git/GitLab ownership release and snapshot +deletion bypass the breaker; relational cleanup goes through `RunExportEnvelope`, +`internal/sink/cleanup.go:379`). Spec: `openspec/changes/sink-breaker-terminal-isolation/specs/sink-circuit-breaker/spec.md` (SCB-1). +Constraint: no production behaviour change beyond the doc comment. + +This is a test-strengthening task on code that already behaves correctly: there is no red on the +base for the table; its proof is the mutation run. + +## 2. MATRIX + +`openspec/changes/sink-breaker-terminal-isolation/evidence/B1.md`, sections *Routed sites*, +*go-git engine case*, *Verification matrix*, *Mutations*. + +## 3. DIFF + +`git diff 7a26657c0`. + +## 4. SENSORS + +`evidence/B1.md` *Sensors run* (build/vet/gofmt, unit tests, `task lint`, two race commands, +mutants M0-M8 all as recorded). The full unit fitness run is `pending: runs once in FINAL`, not missing. + +## 5. RULES + +Repo rules: tests must not use a shared circuit-breaker key or call `ResetBreakersForTest` from a +parallel test; committed evidence is Markdown only; no production behaviour change in this task. + +## Judge, in order + +a. INTENT — does the diff do all of B1.1 and nothing more? Check the site list against + `rg -n 'ownershipConflict|foreignClaimError' internal/sink/git/` at the base: is any routed site + missing from the table? +b. MATRIX — is each sensor real, would it fail if the claim were false, is the class honest? Rerun + anything under a minute that a disputed row depends on (e.g. one mutant, in a scratch copy). +c. RULES — any rule above the diff violates. +d. AI-code failure modes — tautological tests, weakened assertions, a dropped half of the task, + prose (doc comment, evidence) claiming more than the code does. + +For each P0/P1 you are about to file, argue the opposite for one line; keep only what survives. +APPROVE with no findings is valid when substantiated. + +RETURN ≤15 lines, nothing else: + +```text +verdict: APPROVE | REQUEST_CHANGES | BLOCK +findings: + - [P0|P1|P2] +matrix_disputes: + - — +could_not_check: + - +summary: +``` + +Severity: P0 (invariant broken, vacuous gate presented as proof) → BLOCK; P1 (correctness gap, +untested claim, dishonest evidence class, missing half of the task) → REQUEST_CHANGES; P2 → note, +APPROVE allowed. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/B1/leg-reviewer-a-diff.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/B1/leg-reviewer-a-diff.md new file mode 100644 index 00000000..2b198b92 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/L-tasks/B1/leg-reviewer-a-diff.md @@ -0,0 +1,6 @@ +verdict: APPROVE +findings: [] +matrix_disputes: [] +could_not_check: + - Did not independently rerun M0–M8, full build/vet/lint, or race suites; mutation writes are outside this session’s read-only permissions. +summary: B1.1 is satisfied: all routed rejection paths are covered, all nine table cases and both go-git cases passed without skips, marker tests passed, cleanup wording matches code, and production changes are comment-only. diff --git a/openspec/changes/sink-breaker-terminal-isolation/tasks.md b/openspec/changes/sink-breaker-terminal-isolation/tasks.md index 7c453ddd..b2bea429 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/tasks.md +++ b/openspec/changes/sink-breaker-terminal-isolation/tasks.md @@ -119,7 +119,7 @@ Test rules for every group: ## B1. Branch review round 1 fixes -- [ ] B1.1 Marker regression per site (branch review round 1, both reviewers): a table test in +- [x] B1.1 Marker regression per site (branch review round 1, both reviewers): a table test in `internal/sink/git` asserting `kollecterrors.IsInventoryScoped(err) && kollecterrors.IsTerminal(err)` for every site routed through the inventory-scoped helper (`prune_owned.go` and `prune_target.go`; list them from the code at HEAD), driven through the functions that raise them (e.g. @@ -130,7 +130,7 @@ Test rules for every group: one at a time; each must be killed by the new table. Also: the doc comment on `InventoryScoped` says it always yields Terminal (whatever the input's class); `evidence/T0.1.md`'s cleanup statement is narrowed (Git/GitLab ownership release and snapshot deletion bypass the breaker; relational - cleanup goes through `RunExportEnvelope`, `internal/sink/cleanup.go:379`). + cleanup goes through `RunExportEnvelope`, `internal/sink/cleanup.go:379`). — closed 2026-11-05, evidence: evidence/B1.md ## 4. Integration From 6cc25f4b9c3628c7523ae27afd7eed21eaaa1a14 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Thu, 5 Nov 2026 13:27:21 +0100 Subject: [PATCH 13/13] :page_facing_up: docs(spec): record branch review round 2 and hand-off Round 2 approve/clean, per-commit gate green on all commits, and the troubleshooting wording for the breaker counting interval. --- docs/operator-manual/troubleshooting.md | 2 +- .../sink-breaker-terminal-isolation/loop.md | 17 +++- .../pr-description.md | 85 +++++++++++++++++++ .../B-branch/round-2/brief-reviewer-a.md | 46 ++++++++++ .../round-2/leg-reviewer-a-adversarial.md | 36 ++++++++ .../B-branch/round-2/leg-reviewer-b-spec.md | 31 +++++++ .../reviews/B-branch/round-2/verification.md | 34 ++++++++ 7 files changed, 246 insertions(+), 5 deletions(-) create mode 100644 openspec/changes/sink-breaker-terminal-isolation/pr-description.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/brief-reviewer-a.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/leg-reviewer-a-adversarial.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/leg-reviewer-b-spec.md create mode 100644 openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/verification.md diff --git a/docs/operator-manual/troubleshooting.md b/docs/operator-manual/troubleshooting.md index 89884756..71ea589e 100644 --- a/docs/operator-manual/troubleshooting.md +++ b/docs/operator-manual/troubleshooting.md @@ -211,7 +211,7 @@ Family CRDs: **`KollectSnapshotSink`** (git/gitlab/s3/gcs), **`KollectDatabaseSi | Postgres connection failures | DSN, network policy, TLS, pool timeout | Database sink conditions; `transient` sink errors; connection test metric | Verify Secret keys, egress NetworkPolicy, server reachable | DBA for server-side limits | | Postgres rows stale while status looks healthy | Upsert-only drift, or an export error on that ref | Compare `lastExportTime` on the database entry in `status.sinkExports[]` | Follow [Postgres state store — Troubleshooting](../examples/postgres-state-store.md#troubleshooting) | DBA for server-side limits | | S3/GCS 403 | IAM, wrong bucket, signature | Export logs; `terminal`/`forbidden` | Fix credentials and bucket policy | Cloud IAM review | -| `sink circuit breaker open` in logs | 5 consecutive counted backend write failures to one sink (breaker key `/`) within its 60 s counting window. Counted: transient, forbidden and unclassified errors, and sink-wide terminal errors such as a Git auth or protected-branch rejection. A failure about one inventory (an ownership conflict on its own paths) does not count; it still fails that inventory's export (status, Warning event, logs) | Sink errors, then `sink circuit breaker open for "/"` (transient) for ~30 s; exports to that sink do not reach the backend while it is open | Fix the backend. After 30 s one trial export is admitted: success closes the breaker, a counted failure re-opens it for another 30 s, an inventory-scoped result (ownership conflict) leaves it half-open so the next export is the trial | Backend SLA breach | +| `sink circuit breaker open` in logs | 5 consecutive counted backend write failures to one sink (breaker key `/`) within one 60 s counting interval (the counts reset every 60 s while the breaker is closed). Counted: transient, forbidden and unclassified errors, and sink-wide terminal errors such as a Git auth or protected-branch rejection. A failure about one inventory (an ownership conflict on its own paths) does not count; it still fails that inventory's export (status, Warning event, logs) | Sink errors, then `sink circuit breaker open for "/"` (transient) for ~30 s; exports to that sink do not reach the backend while it is open | Fix the backend. After 30 s one trial export is admitted: success closes the breaker, a counted failure re-opens it for another 30 s, an inventory-scoped result (ownership conflict) leaves it half-open so the next export is the trial | Backend SLA breach | | `SinkReachable=False`, `SinkNotFound` | Wrong sink name or cross-namespace ref in `*SinkRefs` | Inventory message; `kubectl get kollect*sink -n ` | Fix ref name; create sink in inventory namespace | — | | `SinkReachable=False`, `SinkUnreachable` | Backend down despite CR present; bad DSN or TLS failure | Sink `ConnectionVerified`; probe annotation | Fix network/credentials first | — | | Inventory / namespace stuck `Terminating` | Sink cleanup failed terminally during deletion; finalizer retained | Events reason `CleanupTerminal` on the inventory; `increase(kollect_cleanup_terminal_total[1h]) > 0`; Degraded reason `CleanupTerminal` | Fix the sink backend and let the next reconcile finish cleanup — the terminal wedge re-checks every 5 minutes, so a Secret-only credential fix (no watch fires on Secret edits) clears within minutes; or accept the exported objects and set `kollect.dev/force-cleanup: "true"` on the deleting object to drop the finalizer | — | diff --git a/openspec/changes/sink-breaker-terminal-isolation/loop.md b/openspec/changes/sink-breaker-terminal-isolation/loop.md index 9f5fc636..9214b8a0 100644 --- a/openspec/changes/sink-breaker-terminal-isolation/loop.md +++ b/openspec/changes/sink-breaker-terminal-isolation/loop.md @@ -1,9 +1,9 @@ # Loop — sink-breaker-terminal-isolation (spec-loop-duo run state) Repo: `/home/koni/Projects/PlatformRelay/worktrees/kollect/sink-breaker-terminal-isolation` · Branch: `fix/sink-breaker-terminal-isolation` · Base: `origin/main` @ `96c41b0d0` · Started: 2026-11-05 · Reviewers: two-model duo -Next: L — dispatch B1.1 (branch review fixes) +Next: 4.1 integrator — push, PR, merge on green CI, post-merge evidence, archive Task prompt: local, `agent-context/loops/sink-breaker-terminal-isolation/task-prompt.md` (harness, not committed) · decisions by model alignment, logged in agent-context/decisions.md -Budget: paid review legs 3/12 (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 +Budget: paid review legs 4/12 (counted from legs.log: R1, R2, B1, B2 spec legs; no per-task paid leg) (B reserve 3; default) · active hours 0/8 (sessions: 2026-11-05) · per task estimate 0 legs (no extra-leg trigger expected: no secrets, CI, infra or manifests), R 1, B 2 → planned 3 Backlog SINK-BREAKER-TENANT-01 (P2) and RACE-COUNT-BREAKER-ISOLATION (P3). Spec set: proposal.md, design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `sink-breaker-terminal` @@ -15,8 +15,8 @@ design.md, specs/sink-circuit-breaker/spec.md, tasks.md. Decision 2026-11-05 `si - [x] P plan + tasks — written; `task spec:validate` green (22 items); spec set committed: (this commit) - [x] R spec-set review — round 1 on `a505029da`: reviewer-a REQUEST_CHANGES (1 CRITICAL), reviewer-b CONCERNS; all 8 findings verified and fixed (`reviews/R-spec-set/verification.md`); CRITICAL had a mechanical fix both legs named → round 2 (last) on `de1e7df12`: reviewer-a REQUEST_CHANGES, reviewer-b BLOCK; new CRITICAL (Git sink-wide Terminal reaches the breaker; A's premise false) settled by decision rerun → B (unanimous), spec set rewritten for B; 7 WARNING/NOTE fixed (`reviews/R-spec-set/round-2/verification.md`). No third round (limit); premises re-verified in T0.1, rewritten set reviewed again at B - [x] L task loop — see *Tasks* (T0.1, G1, G2, G3 closed; B1.1 from stage B) -- [ ] B branch review — round 1 on `4e50a9a2f` (rebased on main after #520/#523): reviewer-a CONCERNS, reviewer-b CONCERNS, no CRITICAL; SCB-1..3 hold; 6 fix (B1.1 + spec wording), 1 defer, 2 reject (`reviews/B-branch/round-1/verification.md`). Next: B1.1, then round 2 + per-commit gate -- [ ] Hand-off — `pr-description.md` +- [x] B branch review — round 1 on `4e50a9a2f` (rebased on main after #520/#523): reviewer-a CONCERNS, reviewer-b CONCERNS, no CRITICAL; SCB-1..3 hold; 6 fix (B1.1 + spec wording), 1 defer, 2 reject (`reviews/B-branch/round-1/verification.md`). B1.1 closed. Round 2 (last) on `d243fd5f4`: reviewer-a APPROVE, reviewer-b CLEAN (one NOTE, wording fixed after round 2, not reviewed again); per-commit gate green on all 12 commits; final-tree gates green (`reviews/B-branch/round-2/verification.md`) +- [x] Hand-off — 2026-11-05 · `pr-description.md` · PR: opened by the integrator (4.1) ## Fitness functions @@ -95,10 +95,19 @@ Tests from a test task changed by a later task (close-task `TEST` class); each o ## Where a human should look first +- `internal/sink/circuit_breaker.go` · `IsExcluded: kollecterrors.IsInventoryScoped` — the failure-policy change +- `internal/errors/errors.go` · `InventoryScoped` always yields Terminal; text and chain kept +- `internal/sink/git/prune_owned.go`, `prune_target.go` · which rejections are marked (7 routed sites); undecided sites stay counted +- `internal/controller/*_test.go` · test changes: per-invocation sink names (9 tests), debounce metric test now serial +- `docs/operator-manual/troubleshooting.md:214`, `docs/adr/0602-error-taxonomy.md` · operator-facing description of what counts + ## Proposed harness changes | Lesson | Seen in | Proposed change (AGENTS.md line / sensor / close-task.yml) | |---|---|---| +| Ginkgo in `internal/controller` refuses `-count>1` | G1, G2, B1 | AGENTS.local.md gotcha line: every `-count=N` on `./internal/controller/` needs `-skip TestControllers`, Ginkgo suite run separately | +| A fresh worktree has no `bin/k8s`, so envtest `BeforeSuite` fails there | per-commit gate (B2) | AGENTS.local.md: set `KUBEBUILDER_ASSETS` to the main checkout's `bin/k8s/` in throwaway worktrees | +| A ten-run probabilistic loop cannot kill a mutant | G2 | spec-loop test-task guidance: pair any N-run flake gate with a deterministic `-count=3 -run` sensor | ## Owner tasks (skipped by the loop) diff --git a/openspec/changes/sink-breaker-terminal-isolation/pr-description.md b/openspec/changes/sink-breaker-terminal-isolation/pr-description.md new file mode 100644 index 00000000..fcc234e5 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/pr-description.md @@ -0,0 +1,85 @@ +# PR description — sink-breaker-terminal-isolation + +- **Source:** `fix/sink-breaker-terminal-isolation` +- **Target:** `main` +- **Title:** `:bug: fix(sink): keep one inventory's ownership conflicts out of the shared breaker` +- **Opens as:** **ready**, rebase merge once every check on the head is green (maintainer branch, + reviewed through the spec loop). + +Paste everything below the line. + +--- + +## TL;DR + +- **What:** a Git ownership conflict that belongs to one inventory no longer counts toward the + shared per-sink circuit breaker. Every other export failure still counts, including sink-wide + Terminal failures such as a revoked Git token or a protected branch. +- **Why:** five ownership rejections from one misconfigured inventory opened the breaker for every + healthy inventory on the same sink for 30 s. The same shared breaker also made four reconciler + tests flaky under `-race -count=3`. +- **Risk:** low. The change is a narrow exclusion (a marker on 7 raise sites). The breaker still + sheds load for every failure that is not marked, and a site nobody marked keeps today's + behaviour. +- **Reviewer action:** read `internal/sink/circuit_breaker.go` and the marked sites in + `internal/sink/git/prune_owned.go` and `prune_target.go`. + +## What changes + +- **Marker.** `kollecterrors.InventoryScoped(err)` makes an error Terminal, keeps its text and + chain, and marks it with `ErrInventoryScoped`. `IsInventoryScoped` tests for it. +- **Marked sites.** The ownership-conflict rejections in the Git prune and merge-target claim + checks (7 sites) go through a new `ownershipConflict` helper. Rejections caused by shared state + (prune metadata, repository contents) stay unmarked and keep counting. +- **Breaker.** `IsExcluded: kollecterrors.IsInventoryScoped`: a marked result counts neither as a + failure nor as a success, does not reset the failure streak, and on a half-open trial frees the + slot without closing or re-opening the breaker. Settings, key, threshold and timeout are + unchanged; `newBreaker(key, timeout)` is extracted for tests. +- **Tests.** Breaker unit and boundary tests (incl. half-open and a concurrent trial), a + real-Git-backend boundary test, a per-site table that fails if any marked site loses its + marker, a go-git engine case. Reconciler tests that shared the `postgres-demo` breaker now use + a per-invocation sink name; one parallel test that read a global metric delta is now serial. +- **Docs.** Troubleshooting row and an ADR-0602 note describe what counts and how half-open works. + +## Risk + +- A rejection marked inventory-scoped that is really sink-wide would stop the breaker from + shedding load for that fault. Each of the 7 sites was classified in the probe and checked by + two reviewers; ambiguous sites were left unmarked. +- An inventory that keeps failing its ownership check can keep a recovering sink half-open (one + trial at a time). Each excluded trial frees the slot, so other inventories are not locked out. + +## Files to check first + +1. `internal/sink/circuit_breaker.go` +2. `internal/errors/errors.go` (`InventoryScoped`) +3. `internal/sink/git/prune_owned.go`, `internal/sink/git/prune_target.go` +4. `internal/sink/circuit_breaker_test.go`, `internal/sink/git/prune_scoped_marker_test.go` + +Review commit by commit; each commit builds and passes the packages it touches. + +## Testing + +- `go test -race -count=3 ./internal/errors/ ./internal/sink/...` and + `go test -race -count=3 -skip TestControllers ./internal/controller/` (ten consecutive green runs + of the latter; 3 of 10 failed on the base before the isolation). +- Mutations: dropping the exclusion, swapping in `IsSuccessful`, excluding every Terminal, + excluding every error, and reverting each marked site one at a time are each caught by a named + test. +- `task verify`, `task lint`, `task spec:validate`, scrub and gitleaks green on the head. + +Release note (fixes): a single inventory's Git ownership conflict no longer opens the sink circuit +breaker for the other inventories on that sink. + +--- + +
Optional: how this was decided + +The first design excluded every Terminal error. The spec-set review showed that Git auth and +protected-branch rejections reach the breaker already Terminal, and that inventories re-export on +every resource event, so that design would have stopped the breaker shedding load for a revoked +token. It was re-decided to mark only inventory-scoped results. OpenSpec change +`openspec/changes/sink-breaker-terminal-isolation/` holds the spec, the per-task evidence and the +review records. + +
diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/brief-reviewer-a.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/brief-reviewer-a.md new file mode 100644 index 00000000..0129d1f4 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/brief-reviewer-a.md @@ -0,0 +1,46 @@ +# Cold review: branch `fix/sink-breaker-terminal-isolation` (lens: adversarial, whole branch) + +You are an independent reviewer with read-only access to this repository (Go Kubernetes operator +"kollect"). Review the whole branch: `git diff $(git merge-base HEAD origin/main)..HEAD` (10 commits). +The spec is the OpenSpec change `openspec/changes/sink-breaker-terminal-isolation/` (`proposal.md`, +`design.md`, `specs/sink-circuit-breaker/spec.md`, `tasks.md`); per-task evidence is in its +`evidence/` and per-task reviews in `reviews/L-tasks/`. Per-task reviews saw one diff each; you see +the feature. Note: the spec set was rewritten after its last spec-set review round (decision changed +from "exclude all Terminal errors" to "exclude only results marked inventory-scoped"), so it has not +been reviewed in its final form — review it too. + +Goal of the change: one inventory's Git ownership rejections must no longer open the shared per-sink +circuit breaker for every other inventory on that sink, while sink-wide failures (Transient, +Forbidden, unclassified, and sink-wide Terminal such as Git auth / protected-branch rejections) keep +opening it. Plus: reconciler tests that shared a breaker key are isolated. + +Attack it: + +1. Compliance per requirement SCB-1, SCB-2, SCB-3: holds / partial / contradicted / absent / + stronger than spec; and what the code does that no requirement mentions. +2. The marker (`internal/errors`): can it be lost between the raise site and the breaker (any wrap + without `Unwrap`, `%v`, re-classification, `ClassError` nesting where the outer class wins, Git + `ClassifyExportError` running on an already-marked error, multipart/layout export paths that call + a different prune path)? Can it be attached to a result that is actually sink-wide (shared prune + metadata, repository state, capacity limits)? Check every marked site against T0.1's table in + `evidence/T0.1.md` and the code. +3. Ownership rejections from paths other than the 5 marked sites (GitLab MR backend, object-store + layouts, cleanup/delete paths): do any still trip the breaker for other inventories? +4. Half-open behaviour under load: can one inventory's repeated marked results starve recovery + (serialise a sink indefinitely)? Is that what the spec accepts? +5. Tests: could any new test pass vacuously (no `=== RUN`, keys that collide across `-count`, timing + windows, a stub that never reaches the breaker)? Were any existing assertions weakened in G2's + isolation (sink names changed — did every assertion that names the sink follow)? +6. Docs (`docs/operator-manual/troubleshooting.md`, `docs/adr/0602-error-taxonomy.md`): accurate + against the code; nothing private (no internal ids, agent-context references). +7. Commit-by-commit: does each commit build on its own (the change claims the tree is green at every + commit)? Conventional gitmoji titles; no AI/co-author trailers; a `fix` that changes user-visible + behaviour has a title that will read well in a changelog. + +Output: findings, each with severity CRITICAL / WARNING / NOTE, file:line evidence, and a concrete +fix; CRITICAL only for a defect that ships wrong behaviour or makes a requirement fail. End with +`## Verdict` and one of APPROVE / CONCERNS / REQUEST_CHANGES / BLOCK. + +## Round 2 (last) + +Head is now `d243fd5f4` (12 commits). Round-1 findings and their dispositions are in `reviews/B-branch/round-1/verification.md`; the fixes are commit `7a26657c0` (spec wording) and `d243fd5f4` (task B1.1: per-site marker table, go-git engine case, doc comment, evidence wording). Check each round-1 fix holds and look for defects the fixes introduced; do not re-raise dispositions already rejected or deferred there unless you have new evidence. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/leg-reviewer-a-adversarial.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/leg-reviewer-a-adversarial.md new file mode 100644 index 00000000..e66a4004 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/leg-reviewer-a-adversarial.md @@ -0,0 +1,36 @@ +No new CRITICAL or WARNING findings at `d243fd5f4`. The round-1 fixes hold; I found no regression introduced by either follow-up commit. + +| Requirement | Assessment | Evidence | +|---|---|---| +| SCB-1 | **Holds** | `internal/sink/circuit_breaker.go:68` excludes by marker. All five marked sites match T0.1’s inventory-scoped classification; shared metadata and capacity limits remain unmarked. | +| SCB-2 | **Holds** | Unmarked failures retain the existing counting policy. Tests verify Transient, Forbidden, unclassified and sink-wide Terminal failures, including backend-call counts. | +| SCB-3 | **Holds** | Exclusion releases the half-open slot without changing state. Tests at `internal/sink/circuit_breaker_test.go:449`, `:469`, and `:492` exercise success, failure and concurrent admission. | + +The round-1 dispositions check out: + +- `internal/sink/git/prune_scoped_marker_test.go:39` drives every marked site, including both `checkSingleClaims` branches, with a shared-metadata negative control. +- `internal/sink/git/prune_scoped_marker_test.go:149` genuinely calls the go-git engine directly. Both engine cases ran without skips. +- `internal/errors/errors.go:155` accurately documents the outer Terminal classification and preserved inner chain. +- The revised race command, commit-green wording and cleanup evidence match the implementation. + +Marker tracing found no lossy wrapper between the marked sites and the breaker. Git classification preserves already-Terminal errors; GitLab returns the Git error intact. Multipart claim paths reach the same ownership checks. Object-store layouts do not use Git ownership pruning. Git/GitLab release and snapshot deletion bypass this breaker; relational cleanup enters it through `RunExportEnvelope`. + +Repeated marked trials can keep recovery serialized indefinitely. This is explicitly accepted in the final spec/proposal; I found no new evidence warranting reopening that disposition. + +G2 preserves the existing assertions, including the expected sink status name at `internal/controller/kollectinventory_export_test.go:330`. Its atomic suffix isolates repeated invocations. The two public documentation changes match the behavior and contain no private IDs or coordination references. + +All 12 commit titles follow the gitmoji Conventional Commit form. Commit messages contain no prohibited attribution trailers. The behavior-changing `fix(sink)` title is suitable for release notes. Static inspection found no cross-commit compile dependency. + +The adversarial checks covered exclusion under concurrent recovery, classification and wrapper maintenance hazards, accidental suppression of shared/authentication failures, and unnecessary production abstraction. No actionable defect survived those checks. Additional helper behavior—nil handling and preservation of `errors.Is`/`errors.As`—is documented and tested beyond the normative requirements. + +Validation passed: + +- Focused marker, per-site, engine and breaker tests, with explicit `=== RUN` and no skips. +- `go test -race -count=3 ./internal/errors/ ./internal/sink/...` +- `go test -race -count=3 -skip TestControllers ./internal/controller/` + +**What I could not check:** I did not independently rerun mutations, full CI, integration tests, or builds at every historical commit. The attempted HEAD `go build ./...` failed before compilation because creating `/tmp/go-build…` was denied by the read-only filesystem. Thus, historical buildability is statically reviewed, not machine-proven here. + +## Verdict + +APPROVE diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/leg-reviewer-b-spec.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/leg-reviewer-b-spec.md new file mode 100644 index 00000000..8ef586e9 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/leg-reviewer-b-spec.md @@ -0,0 +1,31 @@ +## Verdict: CLEAN + +## Findings +- [NOTE] The troubleshooting row calls the breaker's counting period a "60 s counting window", but gobreaker's `Interval` is a fixed cycle — `docs/operator-manual/troubleshooting.md:214`, `internal/sink/circuit_breaker.go:61` + Failure: gobreaker clears the closed-state counts when each 60 s generation ends. Five counted failures can straddle that boundary, for example three at t=58 s and two at t=61 s. The breaker then stays closed, while an operator reading "window" expects a rolling 60 s window and expects it to open. The breaker itself behaves as SCB-2 says ("within its closed-state counting interval"); only the operator wording is loose. + Fix: write "within one 60 s counting interval (counts reset every 60 s while closed)". + Confidence: 55 + +Checked, and holds: +- **Round-1 #1 (per-site pinning):** `internal/sink/git/prune_scoped_marker_test.go:979-1081` has one row for each routed site: `prune_owned.go:174,268,275,317`, the two `checkSingleClaims` calls at `:340` and `:342`, and `prune_target.go:66`. Each row calls the function that raises the error. The control row (`:135`, `invalidPrune`) stays unmarked. I traced the inputs of each row through `prepareOwnedPrune`, `validateOwnedPrunePaths`, `pruneKeepSet` (`git/export.go:250`) and `checkSingleClaims`; each one reaches the line it names. For example, the sort order in the two `checkSingleClaims` rows selects `:340` and `:342` separately. Mutants M1–M8 are recorded in `evidence/B1.md:71-81`. +- **Round-1 #2 (go-git engine):** `prune_scoped_marker_test.go:1089` drives `exportRemoteInWorkdir` and then `ClassifyExportError`. What the test skips does not affect the marker: `exportRemote` only picks auth and the workdir (`git/export.go:152-181`), `withRepoExportLock` returns `fn()` unchanged (`export_lock.go:28`), and `ClassifyExportError` returns a Terminal error unchanged (`git/errors.go:29`). The design *Risks* text was narrowed to match. +- **Round-1 #6:** the SCB-1 text now reads "make the error Terminal". That matches `errors.go:354-360`, the doc comment, and `TestInventoryScoped_alwaysTerminal`. +- **Round-1 #7/#8/#9:** these were deferred or rejected. I found no new evidence on them. +- **SCB-1:** holds. + - `IsExcluded: kollecterrors.IsInventoryScoped` is at `circuit_breaker.go:68`. The breaker sees the raw `plan.run` error before the reclassification and wrap at `export.go:200-212`. + - The wraps keep `%w`, so `errors.Is` still finds the marker. + - The text is kept: `errors.go:371`. + - Scenarios: `circuit_breaker_test.go:643,660,688,851`. +- **SCB-2:** holds. Every non-excluded error counts (`circuit_breaker.go:63-65`), and an open breaker returns a Transient "circuit breaker open" error (`:39-40`). Tests: `circuit_breaker_test.go:710,739`. +- **SCB-3:** holds through gobreaker's half-open exclusion together with `MaxRequests: 1` (`:59`). Tests: `circuit_breaker_test.go:778,798,821`. +- **Test isolation:** `halfOpenBreaker` puts a breaker in the shared registry under a fresh key, so a `ResetBreakersForTest` that ran during these tests would replace it and break them. I looked for one that could. Every caller in package `sink` runs serially: `circuit_breaker_test.go:164,184`, `release_on_delete_test.go`, `inventory_identity_test.go`, and none of them calls `t.Parallel`. Go runs serial tests before the paused parallel ones, so no reset overlaps a half-open test. The `git`-package fuzz tests reset a separate process's registry. + +Beyond what the spec asks: the branch makes controller-test sink names unique (`kollectinventory_export_test.go:193-201`), runs the debounce metric test serially, adds an ADR 0602 note and updates the troubleshooting rows. Marking the "owner > 4096 bytes", "keep set > maxPrunePaths" and "written path missing from keep set" sites inventory-scoped matches the T0.1 classification, since each is about the exporter's own configuration. + +## Could not check +- I ran no sensors. The pass results above come from `evidence/B1.md`. +- No holistic `task check` is recorded at head `d243fd5f4`; the last one ran on `4e50a9a2f`. That means `verify`, `scrub`, `gitleaks` and `spec:validate` have not been re-run on the B1.1 tree. B1.1 recorded lint, markdown, unit and race runs. +- I did not re-run mutants M0–M8. +- I did not check CI at the head, and did not confirm that `skipWithoutGit` does not skip in CI. +- I did not check the gobreaker v2 source for its `IsExcluded` half-open semantics; I relied on design.md and the SCB-3 tests. +- I did not read the full evidence files (G1–G3, T0.1), the earlier review rounds, or the untracked `reviews/B-branch/round-2/` contents. diff --git a/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/verification.md b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/verification.md new file mode 100644 index 00000000..295e6283 --- /dev/null +++ b/openspec/changes/sink-breaker-terminal-isolation/reviews/B-branch/round-2/verification.md @@ -0,0 +1,34 @@ +# B round 2 (last) — verification + +Head `d243fd5f4`. Reviewer A (adversarial): APPROVE, no findings. Reviewer B (spec): CLEAN, one NOTE. +Both confirm every round-1 fix holds and SCB-1, SCB-2, SCB-3 hold. + +| # | Finding | Sev | Reviewers | Verified | Disposition | +|---|---|---|---|---|---| +| 1 | Troubleshooting row says "60 s counting window"; gobreaker's `Interval` is a fixed cycle, not a rolling window | NOTE | B | yes (`circuit_breaker.go:61`, gobreaker generation reset) | fix after round 2, wording only, not reviewed again: "within one 60 s counting interval (the counts reset every 60 s while the breaker is closed)" | + +Gates on the final tree (after the wording fix), 2026-11-05: `task verify` exit 0, `task scrub` exit 0, +`task spec:validate` 22/22, `gitleaks detect --no-git` exit 0, `task lint` no warnings, +`task lint:markdown` 0 issues. B1.1 recorded unit, race (`-count=3`) and mutation runs on `d243fd5f4`. + +## Per-commit gate + +Each commit checked out in its own detached worktree; `go build ./...`, then `go test` on the packages +the commit's Go files touch (`-skip TestControllers` for `internal/controller`, then the Ginkgo suite +separately with `KUBEBUILDER_ASSETS` pointing at the main checkout's `bin/k8s/1.36.0-linux-amd64`; a +fresh worktree has no `bin/`, so the first pass failed in `BeforeSuite` for that reason only). + +| Commit | Gates | Exit | +|---|---|---| +| `42a13dfb6` docs(spec): propose | build | 0 | +| `dbfdaef52` docs(spec): address R1 | build | 0 | +| `34b39b11a` docs(spec): option B | build | 0 | +| `354162994` docs(spec): T0.1 probe | build | 0 | +| `1048a66d2` docs(spec): T0.1 corrections | build | 0 | +| `dff6a86bb` fix(sink): keep inventory-scoped failures out | build; test errors, sink, sink/git | 0 | +| `13b5a54d1` test(controller): isolate breaker per invocation | build; test controller (unit + envtest) | 0 | +| `b3cd2f50a` docs(spec): extend 2.2 | build | 0 | +| `228d0c9e2` test(controller): serialize debounce metric test | build; test controller (unit + envtest) | 0 | +| `4e50a9a2f` docs(sink): breaker docs | build | 0 | +| `7a26657c0` docs(spec): triage B1 | build | 0 | +| `d243fd5f4` test(sink): pin marker per site | build; test errors, sink/git | 0 |