Skip to content

[Fix] Prevent silent subscription data loss on WAL replay gaps - #18741

Merged
jt2594838 merged 5 commits into
apache:masterfrom
Caideyipi:fix/subscription-wal-gap-loss
Sep 30, 2026
Merged

jt2594838 merged 5 commits into
apache:masterfrom
Caideyipi:fix/subscription-wal-gap-loss

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Stop WAL replay from advancing past an unavailable search-index range.
  • On the first gap, force a WAL roll and recreate the iterator from the missing search index so newly visible entries can be replayed.
  • If the same gap remains unavailable, fail closed: keep the replay cursor at the gap, mark progress as stalled, block realtime admission, and return a critical subscription error instead of silently reporting caught-up status.
  • Reset the replay-failure state during initialization, seek, deactivation, and cleanup.

Reproduction

In the subscription regression, the writer and source-side count both reported 3,999 rows, while the consumer received only 3,942 rows. The server log showed WAL replay advancing from expected searchIndex 57 to actual searchIndex 113, silently skipping the unavailable range [57, 113). The consumer reported no error and the region was still considered caught up.

Tests

  • Full reactor test compilation, English locale
  • Full reactor test compilation with with-zh-locale
  • ConsensusPrefetchingQueueTest#testWalReplayRetriesGapWithoutSkippingEntries
  • ConsensusPrefetchingQueueTest#testWalReplayFailsCriticallyWhenGapRemainsUnavailable

The targeted tests were rerun after rebasing onto the latest origin/master: 2 tests, 0 failures, 0 errors.

@jt2594838
jt2594838 merged commit 0c9fb83 into apache:master Sep 30, 2026
35 of 39 checks passed
@jt2594838
jt2594838 deleted the fix/subscription-wal-gap-loss branch September 30, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants