fix(meta): isolate cluster creation progress across groups

[thweetkomputer]

问题

Closes #142.

当前 planner 遇到第一个未完成的 Group child 就返回，即使它只是在等待 Data receipt、lease 或会话投影。新增回归在 main 31a1e13 上确认：Group A 的 replica receipt 被延迟时，具备启动条件的 Group B 得不到任何命令。

修改

• 从已提交 child 状态恢复并发预算，最多同时保留 4 个未完成 Group；扫描并推进其他可运行 child，遇到等待继续扫描。
• 优先推进已开始的任务；空出的名额允许后续 Group 启动。未启动 Group 的 primary 暂时不可用时，也继续检查后续 Group。
• 确定性失败优先于普通进度。一个 child abort 后，先通过已有 failure phase 撤销其他未完成 child 的 directives，再 fence/abort，最后 abort root。保留原始失败原因，已完成 Group 不回滚。
• 每次仍只提交一条已有 Meta command；不增加线程、并行 proposal task 或新的持久化格式。
• 同步更新架构说明。

验证

• 新增慢 replica 测试在原 main 上失败；修复后 29/29 reconciler 单测通过。每个 accepted command 后序列化并恢复 MetaStores，覆盖 6 Group / 4 并发名额、晚到结果、四个活跃阶段的 sibling 收尾、无 runtime 的恢复、失败优先级和已完成 Group 保留。
• 真实 Meta/Data 进程，两个 Group、worker 数 1/2/3/2：同时拦住两个 replica rebuild，只放行 Group 2。Group 1 仍被拦住时，Group 2 replica 已 population-current；本次控制阶段到就绪耗时 9.047s。之后两组的 snapshot、增量复制、路由及 READY 校验全部通过，进程正常退出。
• 原有副本重启导致 provisioning failure、result-committed 处 snapshot + SIGKILL 恢复、wire-result 处 WAL + SIGTERM 恢复场景全部通过。
• clang-format 23.1.1、ruff 0.13.0、Python 编译和 git diff --check 通过。

这是独立 Group 创建进度隔离的验证，没有将上述时间作为普遍的吞吐或总创建时长 benchmark。初始全体 Data projection barrier 保持原有语义；4 个名额全被等待占用时，后续 Group 仍需等待释放名额。本地验证使用 Debug、kernel/io_uring 私有 fixture，未修改线上服务。

Summary by CodeRabbit

• New Features
  • Cluster creation can now make progress on up to four independent Groups at once, so a blocked Group does not prevent others from completing.
  • New Groups are admitted as capacity becomes available; cluster creation completes after all Groups finish.
• Bug Fixes
  • When a Group fails, unfinished sibling work is stopped and cleaned up before the overall cluster creation is aborted.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: eab82ad9-1c6a-44e5-ad95-6a73c734600c

📥 Commits

Reviewing files that changed from the base of the PR and between 31a1e13 and aacdff1.

📒 Files selected for processing (5)

• docs/architecture/08-meta-control-plane.md
• include/lavik/meta/cluster_create_reconciler.h
• src/meta/cluster_create_reconciler.cpp
• tests/meta_cluster_create_reconciler_test.cpp
• tests/meta_integration/gate_cluster_create.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

Walkthrough

Cluster creation now advances independent Group children concurrently, with a limit of four unfinished children. Deterministic child failures take precedence over new work and trigger sibling cleanup before the root is aborted.

Changes

Independent Group creation

Layer / File(s)	Summary
Bounded Group scheduling docs/architecture/08-meta-control-plane.md, include/lavik/meta/cluster_create_reconciler.h, src/meta/cluster_create_reconciler.cpp, tests/meta_cluster_create_reconciler_test.cpp, tests/meta_integration/gate_cluster_create.py	The planner scans active Group children and admits eligible missing children up to a limit of four. Unit and integration tests cover independent progress, capacity reuse, and rebuild completion while another Group remains blocked.
Failure preemption and cleanup docs/architecture/08-meta-control-plane.md, src/meta/cluster_create_reconciler.cpp, tests/meta_cluster_create_reconciler_test.cpp	The planner handles a deterministic child failure before issuing new work. It fences and aborts unfinished siblings with the original cause before aborting the root; completed Groups remain unchanged.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~40 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant Planner as PlanV1ClusterCreateStep
  participant Children as Group child operations
  participant Projection as Primary projection
  participant Root as Root operation
  Planner->>Children: Inspect retained child operation states
  Planner->>Projection: Check eligibility and projection match
  Projection-->>Planner: Return projection state
  Planner->>Children: Submit eligible missing child within the four-child limit
  Children-->>Planner: Return child completion or failure state
  Planner->>Root: Complete after all children finish, or abort after failure cleanup

Loading

Suggested reviewers: liunyl

Merge Risk: ⚪ Minimal · up to aacdf

No merge-blocking issue is established. The change supports bounded independent Group progress and failure cleanup and is mergeable subject to normal checks.

Security Architecture Review

Security architecture risk: 🔵 Low · up to aacdf

Independent Groups can now progress together within a fixed limit. Existing identity and authority checks remain in the reviewed flow, and unfinished work is cleaned up before creation aborts. No introduced security defect was identified, but authorization and execution outside this change were not fully verified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The changed flow can overlap work and failure cleanup across up to four unfinished Groups within the creation manifest. Detected failure can cancel unfinished siblings, while completed Groups retain their state. The reviewed scheduling change does not establish new cross-cluster or cross-tenant authority.

Trust Boundaries and Controls

• observed — Runtime status and terminal receipts influence child progress through the existing Group planner. Root-derived ownership, directive and attempt identities, boot/history checks, projection matching, authority, and lease checks remain in that path. This assessment does not establish upstream caller authentication or independent downstream enforcement.

Resilience and Maintainability Implications

• observed — Directive retirement and the retained failure reason support cleanup after interruption or loss of runtime evidence. Recovery tests exercise rejection of a late result after retirement, fencing before root abort, and cleanup across four active phases.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR implements the main coding objectives in #142. The reconciler admits up to four unfinished Groups, continues past blocked children, restores capacity from committed state, preserves existing co…	Add a deterministic automated slow or retrying-Group scenario that records and asserts the later Group start and completion latency while an earlier Group is blocked. Document the concurrency, fairness, failure, and recovery acceptance crit…
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 24 functions across 4 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: isolating cluster creation progress across independent Groups.
Description check	✅ Passed	The description provides detailed context, behavior changes, implementation details, documentation updates, and validation results. It does not explicitly include the template sections for risk assess…
Out of Scope Changes check	✅ Passed	The changed reconciler code directly implements #142. The unit tests and integration scenario verify independent progress, bounded admission, failure propagation, and recovery. The architecture and he…

Full details: Linked Issues check

Explanation

The PR implements the main coding objectives in #142. The reconciler admits up to four unfinished Groups, continues past blocked children, restores capacity from committed state, preserves existing commands and dependencies, and handles failure cleanup before root abort. Unit and integration tests cover independent progress, capacity reuse, failure cleanup, and recovery cases. However, #142 explicitly requires a multi-Group slow or retrying-Group scenario that measures later Group start and completion latency. The PR reports a real-process scenario, but it states that the timing is not a general benchmark. The requested latency measurement and acceptance criteria are therefore not established.

Resolution

Add a deterministic automated slow or retrying-Group scenario that records and asserts the later Group start and completion latency while an earlier Group is blocked. Document the concurrency, fairness, failure, and recovery acceptance criteria that the test verifies.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 24 functions across 4 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit watched four Groups take flight,
While others waited just out of sight.
One found its path past a rebuild slow,
Another finished as new plans flow.
A failure came; the siblings were cleared,
Then the root closed, its outcome made clear.}

_{Comment @coderabbitai help to get the list of available commands.}