Skip to content

PLT-7829: graph-node BoundedQueue::clear can drop a write pushed while a failed writer clears its queue - #5

Merged
bashalex merged 1 commit into
zerion/v0.43.0from
PLT-7829-boundedqueue-clear-race
Nov 18, 2026
Merged

bashalex merged 1 commit into
zerion/v0.43.0from
PLT-7829-boundedqueue-clear-race

Conversation

@bashalex

@bashalex bashalex commented Nov 18, 2026 •

Copy link
Copy Markdown

PLT-7829: graph-node BoundedQueue::clear can drop a write pushed while a failed writer clears its queue

Changes

  • BoundedQueue::clear (graph/src/util/bounded_queue.rs) no longer loops over try_pop. That loop released a push permit for each item, so a push blocked on a full queue could insert its item mid-loop, and the next try_pop dropped it.
  • New clear, per the ticket's interface contract:
    1. acquire_all_pop_permits reads pop_semaphore.available_permits(), takes exactly that many with try_acquire_many and forgets them. If a concurrent try_pop/pop takes one first (NoPermits), it retries with the new count.
    2. It drains exactly count items from the front while holding the queue lock once.
    3. After releasing the lock, it calls push_semaphore.add_permits(count).
    4. It drops the drained items last.
  • The signature stays pub fn clear(&self). Queue::push, Queue::record_err and the writer loop are unchanged. The sole caller is still Queue::record_err in store/postgres/src/writable.rs.
  • The upstream 2022 pop race (224196b8) is not reintroduced: clear removes only as many items as the pop permits it holds. An item that a push has inserted but not yet given a pop permit stays in the queue.

Acceptance Criteria

  • 1. clear never removes an item pushed while it runs. clear_keeps_item_pushed_during_clear fills a capacity-1 queue with an item whose Drop polls a pending push(B) future once with a noop waker. After clear it asserts the push completed, len() == 1 and try_pop() == Some(B). It passes on the branch. With clear reverted to while self.try_pop().is_some() {}, it failed on all 11 runs I made (assertion left == right failed, left: 0, right: 1 on len()). I restored the file afterwards with git checkout.
  • 2. clear_restores_permits covers capacity 1..=4 and pre-fill 0..=capacity. After clear it asserts try_pop() == None and len() == 0, and that capacity consecutive push futures each complete on their first now_or_never. It passes.
  • 3. clear_skips_item_without_pop_permit pushes one real item, then appends one with queue.lock().push_back(2) and no pop permit. After clear it asserts len() == 1 and that item 2 remains. It passes. Mutation check: with the drain changed to drain(..) (all items), this test fails. Restored afterwards.
  • 4. cargo check -p graph -p graph-store-postgres --tests exits 0. cargo test -p graph --lib: 519 passed and 3 failed: components::link_resolver::ipfs::tests::{ipfs_map_file_size, max_file_size, read_json_stream}. All three fail with Connection refused against 127.0.0.1:5001 because there is no local IPFS daemon. They fail the same way with the base bounded_queue.rs from a1a67f93 (516 passed, 3 failed), so there are no new failures.
  • 5. At head b29e0ab0, the check-runs API shows Subgraph Runner integration tests (with its step Run writer restart tests), Run unit tests, Check rustfmt style, Clippy linting and Build in release mode all with conclusion success. The first run of Run unit tests failed on an unrelated graph-core polling_monitor flake; I reran only the failed jobs of run 44345811374, with no code change. git diff origin/zerion/v0.43.0 -- tests/tests/writer_restart.rs is empty.

Departures

  • AC 3's test also pushes one real, permitted item before the unpermitted one. That checks that clear drains the permitted item and keeps only the unpermitted one. This adds to the AC's len() == 1 assertion and changes nothing it requires.

Testing

  • cargo test -p graph --lib util::bounded_queue: 3 passed.
  • New tests: graph/src/util/bounded_queue.rs (#[cfg(test)] mod tests).
  • The writer-restart integration test was not run locally; CI runs it (AC 5).

Resolves PLT-7829

🤖 Generated with Claude Code

Produced by a hive session (PLT-7829)

clear now takes all pop permits at once, drains exactly that many items
under one lock, adds the push permits back, and drops the drained items
last. A push that was blocked on a full queue and inserts while clear
runs is no longer dropped, and an item whose pop permit is not yet added
is never removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zerion-hive-worker[bot] <318250561+zerion-hive-worker[bot]@users.noreply.github.com>
@bashalex

Copy link
Copy Markdown
Author

REVIEW REQUEST @ b29e0ab v=1 pr=#5 base=zerion/v0.43.0 patch-id=9b48ce42392818cd98538ed09271db099a69cd78 commits=1 since=none ac=PLT-7829

@linear-code

linear-code Bot commented Nov 18, 2026

Copy link
Copy Markdown

PLT-7829

@bashalex

Copy link
Copy Markdown
Author

Reviewer: reviewer-PLT-7829 @ b29e0ab
REVIEW STARTED @ b29e0ab v=1 pr=#5 by=reviewer-PLT-7829

@bashalex

Copy link
Copy Markdown
Author

Reviewer: reviewer-PLT-7829 @ b29e0ab
VERDICT CHANGES @ b29e0ab v=1 pr=#5 attempt=1/3 blockers=1 class=external-blocker repeat=no history=read

Review: PLT-7829

Summary: I found no problems in the code. The only blocker is CI: Run unit tests failed at the head on a known-flaky graph-core polling_monitor test that this diff does not touch. Rerun the failed job. If the rerun at this same SHA is green, I will publish APPROVE for patch-id 9b48ce42392818cd98538ed09271db099a69cd78 without another code pass.

Acceptance Criteria

  • ✅ 1. clear_keeps_item_pushed_during_clear is deterministic and passes at head. I reverted clear to while self.try_pop().is_some() {} in my scratch worktree, and the test failed (len() left 0, right 1). It also catches the wrong drop order: when I moved drop(drained) before push_semaphore.add_permits(count), the test failed.

  • ✅ 2. clear_restores_permits loops over capacity 1..=4 and prefill 0..=capacity, and asserts try_pop()==None, len()==0 and that capacity pushes each complete on their first poll. It passes.

  • ✅ 3. clear_skips_item_without_pop_permit passes. When I changed drain(..count) to drain(..), it failed, so it guards the 2022 bulk-clear race from 224196b8.

  • ✅ 4. cargo check -p graph -p graph-store-postgres --tests finished OK at head. I ran cargo test -p graph --lib myself on both sides:

    • head: 519 passed, 3 failed
    • base a1a67f93, with a forced rebuild: 516 passed, 3 failed

    The 3 failures are the same on both: link_resolver::ipfs::tests::{ipfs_map_file_size,max_file_size,read_json_stream} (Connection refused, no local IPFS). So the diff adds no failures. In CI the graph lib suite passed in full: 522 passed.

  • ❌ 5. At b29e0ab0, Check rustfmt style, Clippy linting, Subgraph Runner integration tests and Build in release mode are success, and Run unit tests is failure. git diff origin/zerion/v0.43.0 -- tests/tests/writer_restart.rs is empty.

CI failure analysis (Run unit tests, run 44345811374)

  • The failing test is graph_core::polling_monitor::tests::polling_monitor_failed_push_to_back at core/src/polling_monitor/mod.rs:342. It got Some(("req-1","res-0")) where it expected ("req-0","res-0"), which is a response-ordering race.
  • BoundedQueue is used only in graph/src/util/bounded_queue.rs and store/postgres/src/writable.rs. polling_monitor does not touch it.
  • I ran the same test 15 times at head and it passed 15 of 15.
  • The same module flaked on an unrelated branch: run 43849350933 (PLT-7750) failed on polling_monitor_unordered, and a rerun passed.
  • So this is a flake, not something the diff causes. No commit to this branch can fix it; a rerun of the failed job clears it.

Code Quality / concurrency

  • Permit accounting. acquire_all_pop_permits reads available_permits(), then try_acquire_many(n) + forget, and retries on NoPermits.
    • Every pop permit belongs to an item that is already in the deque, because push inserts before it calls add_permits.
    • A concurrent pop/try_pop that holds its own permit removes at most one more item.
    • So count + concurrent pops <= len. drain(..count) cannot panic, and it never reaches an item inserted after the permits were taken.
  • Concurrent push.
    • When the queue is full, no push permit is released until after the drain, so a blocked pusher can only insert after the lock is released.
    • When the queue is not full, a push appends at the back, behind every item that is counted.
    • Either way the new item survives, and its pop permit stays available.
  • Waking pushers. A single push_semaphore.add_permits(count) comes after the lock guard is dropped (the temporary guard drops at the end of the let drained statement). That wakes the same waiters the per-item loop did.
  • Drop order. Items are dropped last, outside the lock, so a Drop that re-enters the queue cannot deadlock. The test pins this order.
  • Caller. Queue::record_err (store/postgres/src/writable.rs) runs on the writer task itself, so no concurrent writer peek/pop can hold a permit during clear. Queue::push, record_err and the writer loop are unchanged, as the ticket requires.
  • Efficiency. One lock and one add_permits instead of n.

Interface Compliance

  • The signature stays pub fn clear(&self), synchronous, with no return value. It follows contract steps 1 to 4.

Notes (non-blocking)

  • drop: the doc comment says an entry whose pop permit has not been added yet stays in the queue. That holds for the count, but not for which item stays.
    • Example: two pushes are in flight at once, and the earlier one has not added its permit yet. clear drains that earlier item and leaves the later, permitted one.
    • The counts stay consistent, and the runner is the only pusher, so this never happens in practice.

Verdict: CHANGES REQUESTED (external blocker only: CI flake)

Scope

Scoped to b29e0ab (patch-id 9b48ce42392818cd98538ed09271db099a69cd78 from hive-engine patch-id).

Required Changes

  1. Rerun the failed Run unit tests job of run 44345811374 at this SHA (gh run rerun 44345811374 --failed -R zeriontech/graph-node), then re-DM me. No code change is needed.

@bashalex

Copy link
Copy Markdown
Author

Reviewer: reviewer-PLT-7829 @ b29e0ab
VERDICT APPROVE @ b29e0ab v=1 pr=#5 patch-id=9b48ce42392818cd98538ed09271db099a69cd78 ci=5/github-actions notes=1

Review: PLT-7829

This replaces my CHANGES verdict at this same head. That verdict's only blocker was the external CI flake, and the rerun has cleared it. The head and patch-id are unchanged, and the code review in my previous comment stands without changes.

Acceptance Criteria

  • ✅ 1. clear_keeps_item_pushed_during_clear passes. With the old try_pop loop restored, it failed when I ran it. It also failed when I moved the drop before add_permits.

  • ✅ 2. clear_restores_permits covers capacity 1..=4 and prefill 0..=capacity. It passes.

  • ✅ 3. clear_skips_item_without_pop_permit passes. It failed when I changed the drain to drain(..).

  • ✅ 4. cargo check -p graph -p graph-store-postgres --tests passes. In cargo test -p graph --lib, head and base a1a67f93 fail the same 3 IPFS tests (Connection refused, no local daemon), so the change adds no failures.

  • ✅ 5. Per-check CI at b29e0ab0 (run 44345811374, attempt 2) is all success:

    • Check rustfmt style
    • Clippy linting
    • Run unit tests
    • Subgraph Runner integration tests, including its Run writer restart tests step
    • Build in release mode

    git diff origin/zerion/v0.43.0 -- tests/tests/writer_restart.rs is empty. In attempt 1, Run unit tests failed on graph-core polling_monitor_failed_push_to_back. That was a flake in code this diff does not touch: it passed 15 of 15 runs locally, and the same module flaked in run 43849350933.

Code Quality

  • Pop permits are counted correctly: clear takes them all with one try_acquire_many and retries if another popper takes one first.
  • clear drains no more items than it holds permits for, so an item pushed while it runs is never drained.
  • Push permits are added in one add_permits(count) call, after the lock is released.
  • Drained items are dropped last, outside the lock.
  • The only caller is Queue::record_err, which runs on the writer task itself.

Interface Compliance

  • pub fn clear(&self) keeps its signature and follows contract steps 1 to 4. Queue::push, record_err and the writer loop are unchanged.

Notes (non-blocking)

  • drop: the doc comment says an entry without a pop permit stays in the queue. That is true of the count, not of which item stays. With two pushes in flight at once, clear can drain the earlier, unpermitted item and keep the later one. The runner is the only pusher, so this does not happen in practice.

Verdict: APPROVE

Scope

Scoped to b29e0ab (patch-id 9b48ce42392818cd98538ed09271db099a69cd78 from hive-engine patch-id).
The approval carries over automatically to any later head with an unchanged patch-id, including a clean merge origin/zerion/v0.43.0, and to nothing else.

@bashalex
bashalex merged commit e2a459e into zerion/v0.43.0 Nov 18, 2026
9 of 10 checks passed
@bashalex
bashalex deleted the PLT-7829-boundedqueue-clear-race branch November 18, 2026 16:55
@bashalex

Copy link
Copy Markdown
Author

Deployed: v0.43.0-zerion3 (tag at 1e572ab, ECR sha256:aa584d57e13222aa7abf6d49f8828c0b4e754e80d98da63b0fda6244f04dbdce) · image.tag k8s-cluster eksctl 6c8e1bfa5885062b86b69bb934de18a88b7ad99f · production datai, base-bootstrap-graph-node only (user decision "Base-bootstrap only"; fleet stage 2 separate). Ships #5 and #6.
rollout: statefulset/datai-indexer-v1-base-bootstrap-graph-node rolling update complete, ArgoCD datai-indexer-v1-production Synced/Healthy at 6c8e1bfa58 · health: pod Ready on the zerion3 digest, 0 restarts, 0 ERRO/panic/poisoned/store-error lines, sgd5613 committed 282 batches (59162908 → 59164090) over 17:29:22Z–17:33:27Z UTC 2026-11-18; no other datai graph-node restarted. Writer restart covered by CI runner tests at 1e572ab, not exercised live.
Linear: https://linear.app/zerion/issue/PLT-7829 · https://linear.app/zerion/issue/PLT-7830

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant