Summary
A log entry can be counted toward Raft's commit quorum — and reported to a client as successfully committed — before it has durably reached physical storage. This happens at two points on the same path:
- An entry can count toward quorum while it exists only in an in-memory index, before being handed to the storage engine at all.
- An entry can count toward quorum after reaching the storage engine, but before it has been synced to physical disk.
Both are the same underlying gap at different distances from disk: quorum accounting is derived from in-memory/write-ahead progress, not from confirmed durable persistence.
Impact
- An ordinary process crash or restart — no power loss, no disk failure, just the process exiting — can permanently lose an entry already reported to a client as committed, if the crash happens before point 1 above. This requires no correlated failure and is a routine operational event, not an edge case.
- A power-loss event affecting the specific nodes that made up a commit's quorum can permanently lose an entry already reported as committed, even after it reached the storage engine, if it hadn't yet synced to physical disk (point 2). This requires a correlated fault across those specific nodes.
- Both convert a claimed "committed" response into a write that never existed after recovery.
Background
This is a known, deliberate performance tradeoff, not an oversight: gating quorum on durable persistence has a real, previously measured cost, under an IO architecture that has since been replaced. That cost has not been re-measured under the current architecture — the actual price of closing this gap today is unknown.
This is fundamentally a contest between two goals pulling in opposite directions: write latency/throughput, versus the protocol-level guarantee that a "committed" response means the data cannot be lost. Resolving it means deciding, with current measurements, where that line sits, and applying the decision consistently across both points above — they are the same gap, not two separate ones.
Status
Root cause identified for both points. Point 1 reproduced deterministically via a regression test that blocks the storage engine's write call. Fix approach and its performance cost under the current architecture not yet decided.
Summary
A log entry can be counted toward Raft's commit quorum — and reported to a client as successfully committed — before it has durably reached physical storage. This happens at two points on the same path:
Both are the same underlying gap at different distances from disk: quorum accounting is derived from in-memory/write-ahead progress, not from confirmed durable persistence.
Impact
Background
This is a known, deliberate performance tradeoff, not an oversight: gating quorum on durable persistence has a real, previously measured cost, under an IO architecture that has since been replaced. That cost has not been re-measured under the current architecture — the actual price of closing this gap today is unknown.
This is fundamentally a contest between two goals pulling in opposite directions: write latency/throughput, versus the protocol-level guarantee that a "committed" response means the data cannot be lost. Resolving it means deciding, with current measurements, where that line sits, and applying the decision consistently across both points above — they are the same gap, not two separate ones.
Status
Root cause identified for both points. Point 1 reproduced deterministically via a regression test that blocks the storage engine's write call. Fix approach and its performance cost under the current architecture not yet decided.