Skip to content

store: Fix stale loadRelated results while blocks are queued (<> all and exclusion of unrelated queued writes) - #6727

Open
madumas wants to merge 2 commits into
graphprotocol:masterfrom
ellipfra:fix-get-derived-queued
Open

madumas wants to merge 2 commits into
graphprotocol:masterfrom
ellipfra:fix-get-derived-queued

Conversation

@madumas

@madumas madumas commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #6726.

Queue::get_derived combines queued changes with a database read at the block before the queue, and hides the queued keys from that read through excluded_keys. Two defects let older database versions through, so loadRelated could return children that a queued block had removed or moved to another parent. Whether that happened depended on writer timing.

Two commits, which can be reviewed separately:

  1. store: Exclude every listed key in FindDerivedQuery: id != any($excluded) → id <> all($excluded). With two or more keys, the old predicate excluded nothing.
  2. store: Exclude queued keys no longer related in get_derived: in effective_ops, a queued write that is no longer related to the parent now excludes its key instead of being dropped. A newer queued version always supersedes the database row, whether or not it is still related.

Tests: four store tests in store/test-store/tests/postgres/writable.rs (get_derived_pending_*). They hold the writer, call get_derived while a block is queued, then flush. Without the fixes:

  • two_removals and removal_and_creation fail without commit 1;
  • parent_change fails without commit 2;
  • one_removal is a control that passed before.

The existing get_derived_batch and get_derived_nobatch still pass.

Notes:

  • The empty array is still guarded (if !self.excluded_keys.is_empty()).
  • <> all(array) has the same planner characteristics as != any(array).
  • Commit 2 excludes every queued key of the queried entity type, so the bind array is somewhat larger on big batches.
  • Behaviour change: for subgraphs that call loadRelated on children moved or removed while blocks are queued, results and therefore PoIs can differ from released versions. The current behaviour is timing-dependent, so this probably deserves a release note.

Checks run locally on this branch: cargo fmt --all -- --check, cargo clippy --all-targets (no warnings), cargo check --release, cargo test -p graph-store-postgres --lib, cargo test -p test-store --test postgres, and the workspace unit tests.

`id != any($excluded)` holds as soon as the id differs from one element
of the array, so with two or more excluded keys no row was excluded.
`Queue::get_derived` reads the database at the block before the queue
and relies on this list to hide children that queued blocks remove or
rewrite, so `loadRelated` could return entities that a queued block had
already removed. Use `id <> all($excluded)`.

Add store tests that hold the writer and call `get_derived` while the
block is queued: one removal (control), two removals, and a removal plus
a creation.
In `Queue::get_derived`, a queued write whose `@derivedFrom` field no
longer points to the parent was dropped instead of excluding its key.
The database, read at the block before the queue, then still returned
the previous version of that child under the old parent, so
`loadRelated` could return a child that a queued block had moved to
another parent. A newer queued version always supersedes the database
row, related or not: exclude its key.

Add a store test that moves a child to another parent in a queued block.

@dimitrovmaksim dimitrovmaksim left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The root-cause analysis checks out, and the fix is correct for mutable entities. One skipDuplicates case is inline.

A question on cost: with commit 78e60f0 the exclusion list grows from the parent's related queued writes to every queued key of the type, and <> all has to check each candidate row against that list. 4d2479c moved this list to a single array bind to avoid potentially hitting the Postgres bind-parameter limit (65,535), so lists that large were already anticipated with the smaller set

Queue::get_many settles queue-vs-database precedence in Rust, and FindDerivedQuery has no LIMIT or ORDER BY. If get_derived dropped database rows whose id is in entities_in_queue in the Rust code instead, would the result differ? Would excluded_keys (and the != any / <> all question) still need to exist?

Comment on lines +1347 to 1355
.map(|op| match op {
EntityOp::Write { key, entity } if is_related(derived_query, entity) => {
Some((key.clone(), Some(entity.clone())))
(key.clone(), Some(entity.clone()))
}
EntityOp::Write { .. } => None,
EntityOp::Remove { key } => Some((key.clone(), None)),
// A newer queued version supersedes the database row even
// when it is no longer related: exclude its key
EntityOp::Write { key, .. } => (key.clone(), None),
EntityOp::Remove { key } => (key.clone(), None),
})

@dimitrovmaksim dimitrovmaksim Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#6458 introduced skipDuplicates for immutable entities: a duplicate insert doesn't fail the subgraph; when it reaches the table, ON CONFLICT DO NOTHING drops it, so the stored version stays.

With that in mind, does "a newer queued version supersedes the database row" hold for a queued duplicate of a skipDuplicates entity? For example, with a stored t { parent: p0 } and a queued duplicate t { parent: p1 }, what does loadRelated(p0) return while the block is queued, and after the flush?

Not sure how often skipDuplicates is used, but it is a real path.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I traced this through and it's real, and slightly wider than get_derived.

With a stored t { parent: p0 } and a queued duplicate t { parent: p1 } in a later block (so RowGroup::append_row doesn't see the original and the duplicate is queued; the insert's ON CONFLICT DO NOTHING discards it at write time):

Read while queued master this PR after flush
get(t) p1 p1 p0
loadRelated(p0) t nothing t
loadRelated(p1) t t nothing

So get and loadRelated(p1) already return the duplicate on master, and this PR adds loadRelated(p0). There's also a cache-side variant: EntityCache::as_modifications inserts the duplicate's data into the entity LFU cache, so later blocks can read it until it's evicted.

One way to fix all read paths at once is to drop these inserts before they're queued, since they're no-ops once written:

// in WritableStore::transact_block_operations, before Batch::new  
let keys: BTreeSet<EntityKey> = mods  
    .iter()  
    .filter(|m| matches!(m, EntityModification::Insert { .. })  
        && m.key().entity_type.skip_duplicates())  
    .map(|m| m.key().clone())  
    .collect();  
if !keys.is_empty() {  
    let existing = self.writer.get_many(keys).await?;  
    mods.retain(|m| !(matches!(m, EntityModification::Insert { .. })  
        && m.key().entity_type.skip_duplicates()  
        && existing.contains_key(m.key())));  
}  

On top of that, as_modifications should not cache keys of skipDuplicates types it produced a modification for, so later blocks read them from the store. This costs one get_many per block that inserts skipDuplicates entities. Other types are unaffected.

A store test for the scenario above: write t { parent: p0 } and flush, pause the writer, queue t { parent: p1 }, then check get(t), loadRelated(p0) and loadRelated(p1) before and after the flush. It fails on master at get(t) and passes with the change. Happy to open this as a separate PR if you'd prefer to keep this one to get_derived.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @chriswessels, it probably makes sense as a separate PR, so feel free to open one and I'll review it.

@chriswessels

Copy link
Copy Markdown

On moving queue-vs-database precedence into Rust: since FindDerivedQuery has no LIMIT or ORDER BY, querying without excluded_keys and then dropping database rows whose key is in entities_in_queue gives the same result as the SQL exclusion. It avoids the larger bind array, at the cost of the database returning rows for queued keys that are then discarded. Either shape fixes the two defects here. The skipDuplicates case above is independent of this choice and needs handling either way.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

loadRelated can return stale children while blocks are queued for writing (!= any in FindDerivedQuery; unrelated queued writes not excluded)

3 participants