Repository navigation
PLT-7829: graph-node BoundedQueue::clear can drop a write pushed while a failed writer clears its queue - #5
Conversation
clear now takes all pop permits at once, drains exactly that many items under one lock, adds the push permits back, and drops the drained items last. A push that was blocked on a full queue and inserts while clear runs is no longer dropped, and an item whose pop permit is not yet added is never removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zerion-hive-worker[bot] <318250561+zerion-hive-worker[bot]@users.noreply.github.com>
|
Reviewer: reviewer-PLT-7829 @ b29e0ab Review: PLT-7829Summary: I found no problems in the code. The only blocker is CI: Acceptance Criteria
CI failure analysis (
|
|
Reviewer: reviewer-PLT-7829 @ b29e0ab Review: PLT-7829This replaces my CHANGES verdict at this same head. That verdict's only blocker was the external CI flake, and the rerun has cleared it. The head and patch-id are unchanged, and the code review in my previous comment stands without changes. Acceptance Criteria
Code Quality
Interface Compliance
Notes (non-blocking)
Verdict: APPROVEScopeScoped to b29e0ab (patch-id |
|
Deployed: v0.43.0-zerion3 (tag at 1e572ab, ECR sha256:aa584d57e13222aa7abf6d49f8828c0b4e754e80d98da63b0fda6244f04dbdce) · image.tag k8s-cluster eksctl 6c8e1bfa5885062b86b69bb934de18a88b7ad99f · production |
PLT-7829: graph-node BoundedQueue::clear can drop a write pushed while a failed writer clears its queue
Changes
BoundedQueue::clear(graph/src/util/bounded_queue.rs) no longer loops overtry_pop. That loop released a push permit for each item, so apushblocked on a full queue could insert its item mid-loop, and the nexttry_popdropped it.clear, per the ticket's interface contract:acquire_all_pop_permitsreadspop_semaphore.available_permits(), takes exactly that many withtry_acquire_manyandforgets them. If a concurrenttry_pop/poptakes one first (NoPermits), it retries with the new count.countitems from the front while holding the queue lock once.push_semaphore.add_permits(count).pub fn clear(&self).Queue::push,Queue::record_errand the writer loop are unchanged. The sole caller is stillQueue::record_errinstore/postgres/src/writable.rs.224196b8) is not reintroduced:clearremoves only as many items as the pop permits it holds. An item that apushhas inserted but not yet given a pop permit stays in the queue.Acceptance Criteria
clearnever removes an item pushed while it runs.clear_keeps_item_pushed_during_clearfills a capacity-1 queue with an item whoseDroppolls a pendingpush(B)future once with a noop waker. Afterclearit asserts the push completed,len() == 1andtry_pop() == Some(B). It passes on the branch. Withclearreverted towhile self.try_pop().is_some() {}, it failed on all 11 runs I made (assertion left == right failed, left: 0, right: 1onlen()). I restored the file afterwards withgit checkout.clear_restores_permitscovers capacity 1..=4 and pre-fill 0..=capacity. Afterclearit assertstry_pop() == Noneandlen() == 0, and thatcapacityconsecutivepushfutures each complete on their firstnow_or_never. It passes.clear_skips_item_without_pop_permitpushes one real item, then appends one withqueue.lock().push_back(2)and no pop permit. Afterclearit assertslen() == 1and that item 2 remains. It passes. Mutation check: with the drain changed todrain(..)(all items), this test fails. Restored afterwards.cargo check -p graph -p graph-store-postgres --testsexits 0.cargo test -p graph --lib: 519 passed and 3 failed:components::link_resolver::ipfs::tests::{ipfs_map_file_size, max_file_size, read_json_stream}. All three fail withConnection refusedagainst127.0.0.1:5001because there is no local IPFS daemon. They fail the same way with the basebounded_queue.rsfroma1a67f93(516 passed, 3 failed), so there are no new failures.b29e0ab0, the check-runs API showsSubgraph Runner integration tests(with its stepRun writer restart tests),Run unit tests,Check rustfmt style,Clippy lintingandBuild in release modeall with conclusion success. The first run ofRun unit testsfailed on an unrelated graph-corepolling_monitorflake; I reran only the failed jobs of run 44345811374, with no code change.git diff origin/zerion/v0.43.0 -- tests/tests/writer_restart.rsis empty.Departures
cleardrains the permitted item and keeps only the unpermitted one. This adds to the AC'slen() == 1assertion and changes nothing it requires.Testing
cargo test -p graph --lib util::bounded_queue: 3 passed.graph/src/util/bounded_queue.rs(#[cfg(test)] mod tests).Resolves PLT-7829
🤖 Generated with Claude Code
Produced by a hive session (PLT-7829)