Skip to content

Add KV unit test reproducing stale pending tx (#8293) - #8295

Open
Amaury Chamayou (achamayou) wants to merge 5 commits into
mainfrom
achamayou-systematic-concurrency-investigation
Open

Add KV unit test reproducing stale pending tx (#8293)#8295
Amaury Chamayou (achamayou) wants to merge 5 commits into
mainfrom
achamayou-systematic-concurrency-investigation

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Sep 5, 2026

Copy link
Copy Markdown
Member

Reject stale-view transactions in Store::commit() regardless of node role, under the existing version lock and before updating last_committable or inserting into pending_txs. This prevents rolled-back writes from being replicated after a later election and blocking subsequent replication.

Closes #8293.

  • Keep a deterministic KV regression covering primary, backup and candidate roles, with and without sequence-number reuse after rollback. Check stale-write rejection, absence of rolled-back data and continued replication using entry counts.
  • Leave rollback and follower replay unchanged, with no new locks or allocations on the normal commit path.
  • Add the fix to the 7.0.15 release notes and align the Python package version.

Validation: kv_test and history_test pass via tests.sh; all 14 scripts/ci-checks.sh -f groups pass (WSL, Clang 21.1.8, Debug, USE_SNMALLOC=OFF).

A transaction which takes its version before a view change, but reaches
Store::commit() after it, is not caught by the stale-view check, because
that check is also gated on the node still being primary. Its entry is
parked in pending_txs behind the hole the rollback left, survives the
next election because a rollback which discards nothing returns early
without clearing pending_txs, and is then replicated by the first
transaction of the new term.

The new case reproduces this against a real ccf::kv::Store and the
existing stub consensus, using a write set observer to drive the view
change at the point another thread would be parked mid-commit. Its
assertions record the current behaviour, marked BUG where incorrect.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 5, 2026 21:55
@achamayou
Amaury Chamayou (achamayou) requested a review from a team as a code owner September 5, 2026 21:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A couple of the new test’s messages/assertions are misleading/ambiguous in ways that reduce the test’s value as a clear reproduction and future regression indicator.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new KV-only unit test in kv_test to deterministically reproduce issue #8293 (stale pending transactions surviving a rollback/election boundary and being replicated in a later term), using StubConsensus state transitions and a write-set observer to trigger the view change mid-commit.

Changes:

  • Add a new test case covering the “version allocated before view change, Store::commit() reached after view change” interleaving.
  • Assert today’s observed (buggy) outcomes to document the failure mode and make future fixes explicitly update expectations.

Custom instructions used:

  • .github/copilot-instructions.md
  • .github/instructions/reviewing.instructions.md
File summaries
File Description
src/kv/test/kv_test.cpp Adds a new KV unit test that reproduces #8293 by simulating rollback/election ordering and observing stale pending-tx replication effects.
Review details

Suppressed comments (1)

src/kv/test/kv_test.cpp:3167

  • The final assertion only checks replicated_to() == 4, which is already true immediately before this write in the buggy path. Capturing the pre-write value and asserting it does not change makes it explicit that this commit was not handed to consensus (and makes the intent of the comment unambiguous).
    REQUIRE(write("lost", "4") == ccf::kv::CommitResult::SUCCESS);
    CHECK(read("lost") == "4");

    // BUG (#8293): reported as committed, but never handed to consensus
    CHECK(replicated_to() == 4);
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/kv/test/kv_test.cpp Outdated
The test previously recorded the current behaviour and passed. Flip the
expectations to what the store should do, so it is a failing repro of
 #8293 until the defect is fixed.

Also count the entries handed to consensus, rather than only the seqno
reached. The lost follow-up transaction is invisible to the seqno alone,
because the entry already sitting at that seqno is the discarded write.

Five divergences are now reported:
 - the stale write returns SUCCESS rather than FAIL_NO_REPLICATE
 - the new term's first write replicates two entries rather than one
 - it reaches seqno 4 rather than 3
 - the store version trails what has been replicated
 - the write after it is never handed to consensus

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reject transactions whose view no longer matches the store before they enter the pending replication queue, regardless of leadership state.

Extend the regression to cover all leadership states and sequence-number reuse after rollback.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve both sides of the release-note conflict, move the pending fix to 7.0.15, and synchronize the Python package version. Name the regression's retained boundary rollback_seqno rather than implying StubConsensus has committed those entries.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Include the public-header helper tests added to main while the PR update was being prepared.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rolled-back follower writes can enter a later leader's replication batch

2 participants