perf(meta): overlap bounded failover proposals across groups

[thweetkomputer]

Summary

Closes #143.

Both automatic Begin detection and the ordinary failover executor currently await one proposal before planning another Group. A blocked proposal therefore serializes unrelated Groups even when their transitions are ready.

• Add a worker-owned window of four proposal waits per reconciler, with one occupied slot per Group. Skip occupied Groups before allocating action identities; completion wakes planning immediately.
• Preserve automatic uncertain retry identities/admissions, typed CAS, fresh committed-state reconciliation and per-Group rejection backoff.
• Own immutable commands/results in proposal tasks and join every dispatched task on cancellation/shutdown before releasing the leader context. No additional threads or durable/wire format changes.
• Update the control-plane architecture and add bounded overlap, timeout/late-commit, cancellation/reelection and two-Group process tests.

The limit bounds local waits; coordinator reservations for uncertain Raft appends keep their existing lifetime. The two reconcilers have separate windows. Raft log/apply order and throughput limits remain.

Validation

• Regression fails on main b89310f: six ready Groups dispatch only one Begin while the proposal executor is held. Candidate dispatches four, respects the bound, then completes all six.
• GCC13 Debug Meta build; 88/88 focused automatic/failover/planner/validation tests pass (27.223s).
• Existing real-process gates pass: automatic owner-kill, leader-proposal and leader-commit; controlled live-leader-demotion and prepared-leader-resume.
• New two-Group owner-loss gate passes: both fence observations at1.307s; serving at5.144s/5.073s; old data and new writes verified. These are status-poll observations, not a before/after speedup benchmark.
• clang-format23.1.1, ruff0.13.0, Python syntax, CMake registration and diff checks pass.

Outstanding test observation

The first new process-gate attempt failed before fault injection, during cluster creation: group-2 replica initialization returned replication peer closed connection, with replica session supersession and source lease-expiry/cleanup logs. A compiler was running concurrently. An unchanged isolated rerun passed. Cause and reproduction on unmodified main remain unproven; this is not claimed fixed by this PR. Original failure and passing logs are retained in lavik-stability/incidents/failover-progress-143-20260930/ alongside the full report and fixture archive.

Summary by CodeRabbit

• New Features
  • Automatic failover can now make progress on multiple groups at the same time, while avoiding overlapping failover attempts for the same group.
  • Failover attempts that are rejected or delayed can be retried, and completed changes are reconciled before another attempt is made for that group.
• Tests
  • Added coverage for simultaneous failover across multiple groups, including checks that recovered replicas retain data and accept writes.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 145d4d33-c42a-4895-95a3-d63c968ca8ab

📥 Commits

Reviewing files that changed from the base of the PR and between b89310f and d2ae4c1.

📒 Files selected for processing (9)

• CMakeLists.txt
• docs/architecture/08-meta-control-plane.md
• include/lavik/meta/failover_reconciler.h
• src/meta/automatic_failover_reconciler.cpp
• src/meta/failover_reconciler.cpp
• src/meta/group_proposal_window.h
• tests/meta_automatic_failover_reconciler_test.cpp
• tests/meta_failover_reconciler_test.cpp
• tests/meta_integration/gate_automatic_failover.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

Walkthrough

Failover reconciler proposals now run concurrently for distinct Groups, with a limit of four in-flight proposals. The changes add result handling, per-Group retry and occupancy checks, shutdown draining, and tests for multi-group failover.

Changes

Failover proposal processing

Layer / File(s)	Summary
Proposal window src/meta/group_proposal_window.h	Adds a four-entry window that tracks each Group’s command and result, prevents duplicate in-flight proposals per Group, and supports waiting for and draining completions.
Controlled failover reconciliation include/lavik/meta/failover_reconciler.h, src/meta/failover_reconciler.cpp, tests/meta_failover_reconciler_test.cpp, docs/architecture/08-meta-control-plane.md	The planner skips Groups with an in-flight proposal or retry delay. The reconciler dispatches through the proposal window, refreshes its view after completion, and drains proposals on exit. Tests cover occupied Groups and ID allocation.
Automatic failover reconciliation and validation src/meta/automatic_failover_reconciler.cpp, tests/meta_automatic_failover_reconciler_test.cpp, tests/meta_integration/gate_automatic_failover.py, CMakeLists.txt, docs/architecture/08-meta-control-plane.md	The detector retains pending identities while proposals are active, handles completed results, and drains active proposals before cleanup. Tests cover concurrency limits, timeouts, shutdown, and controlled failover; the integration case exercises two Groups failing over together.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Reconciler as MetaAutomaticFailoverReconciler
  participant Window as GroupProposalWindow
  participant Leader as MetaLeaderContext
  Reconciler->>Window: Start proposal for a Group
  Window->>Leader: Propose command
  Leader-->>Window: Proposal result
  Reconciler->>Window: Take completed results
  Reconciler->>Reconciler: Reconcile result and refresh committed view

Loading

Suggested reviewers: liunyl

Merge Risk: ⚪ Minimal · up to d2ae4

Failover proposals for different Groups can now overlap, limited to four at a time and one per Group. Pending proposals are drained on shutdown or demotion. Review found no concrete defect, and tests cover the concurrency bound, retries, and shutdown behavior. The change appears ready to merge.

Security Architecture Review

Security architecture risk: 🔵 Low · up to d2ae4

Concurrent failover work changes coordination and shutdown behavior. The reviewed paths retain leadership checks, state-version checks and cleanup protections, with no introduced security flaw identified. Remaining uncertainty is limited by incomplete coverage beyond those paths.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The two independent windows permit up to eight simultaneous local failover proposal waits across the leader's two reconcilers, potentially affecting multiple Groups. This is not a bound on unresolved Raft appends after timeouts; those retain the coordinator's existing reservations.

Security Findings and Attack Paths

• inferred — No introduced authority-bypass path was identified in the inspected change. Request-selected Group and operation identities still enter through durable intent, while leader-local planning constructs the privileged transition command and deterministic apply revalidates its ownership anchors. This conclusion does not establish complete security coverage.

Trust Boundaries and Controls

• observed — The remote handler receives an AuthenticatedPrincipal. Leader proposals forward the leader actor and expected term; coordinator admission rejects a mismatched term, runs validation hooks before append, injects the trusted actor, and verifies the returned command tag.

Resilience and Maintainability Implications

• observed — Timeout and cancellation are uncertain outcomes rather than proof of non-append. Automatic retries preserve identity, and local draining keeps proposal ownership alive until tasks return. The next leadership epoch reconstructs committed transitions instead of inheriting the old local window.

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description provides useful context, implementation details, design constraints, validation results, and an outstanding test observation. However, it does not follow the required template headings…	Rewrite the description using all required template sections. Include the exact test commands and results, documentation confirmation, risks, rollback steps, reviewer guidance, and follow-up work. Preserve the existing implementation summar…
Linked Issues check	⚠️ Warning	Issue #143 requires bounded per-Group in-flight state, a concurrency limit, fair scheduling, preserved retry and admission behavior, cancellation/join handling, and regression coverage. The PR summary…	Add automated performance coverage that records Begin-fence commit and recovery-completion latency distributions during simultaneous multi-Group loss. Include slow-proposal and uncertain-timeout cases. Report the resulting measurements in t…
Docstring Coverage	⚠️ Warning	Docstring coverage is 12.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 25 functions across 7 files. (2 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the primary change: bounded overlap of failover proposals across Groups.
Out of Scope Changes check	✅ Passed	The changes stay within Issue #143. They add bounded proposal windows, per-Group occupancy checks, retry handling, committed-state reconciliation, cancellation joins, focused reconciler tests, and a m…

Full details: Description check

Explanation

The description provides useful context, implementation details, design constraints, validation results, and an outstanding test observation. However, it does not follow the required template headings and omits or incompletely covers Context, Behavior before and after, Implementation, Design decisions and alternatives, Documentation and comments, Test plan with exact commands, Risk assessment, Rollback plan, Reviewer guide, and Follow-up work.

Resolution

Rewrite the description using all required template sections. Include the exact test commands and results, documentation confirmation, risks, rollback steps, reviewer guidance, and follow-up work. Preserve the existing implementation summary, validation results, and outstanding test observation.

Full details: Linked Issues check

Explanation

Issue #143 requires bounded per-Group in-flight state, a concurrency limit, fair scheduling, preserved retry and admission behavior, cancellation/join handling, and regression coverage. The PR summary reports the four-slot windows, one proposal per Group, identity and admission preservation, committed-state reconciliation, shutdown joins, and automatic/controlled failover tests. However, Issue #143 also requires latency-distribution measurement for simultaneous multi-Group loss, including slow proposals and uncertain timeouts. The PR reports status-poll observations only and does not provide that measurement.

Resolution

Add automated performance coverage that records Begin-fence commit and recovery-completion latency distributions during simultaneous multi-Group loss. Include slow-proposal and uncertain-timeout cases. Report the resulting measurements in the linked implementation evidence.

Full details: Docstring Coverage

Explanation

Docstring coverage is 12.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 25 functions across 7 files. (2 skipped: 2 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{I’m a rabbit watching proposals hop,
Four at a time, then pause or stop.
Each Group keeps its own command in view,
Completed hops bring fresh results through.
I nibble clover as tests pass by.}

_{Comment @coderabbitai help to get the list of available commands.}