Skip to content

Per-key removal notification for the client-side cache - #3350

Merged
mgravell merged 2 commits into
marc/v4-cache-result-reusefrom
marc/v4-cache-removal-notify
Nov 12, 2026
Merged

mgravell merged 2 commits into
marc/v4-cache-result-reusefrom
marc/v4-cache-removal-notify

Conversation

@mgravell

Copy link
Copy Markdown
Member

Stacked on #3349 (marc/v4-cache-result-reuse); only the last commit is new here.

What

A caching layer built on top of the client-side cache (StackExchange.Redis.HybridCache's tag-timestamp memo is the concrete consumer; a FusionCache backplane adapter is a prospective second) keeps derived structures coherent with cached keys, and needs to be told when a key stops being current. This adds that: a per-context removal observer.

var ctx = db.AsRespContext().WithRemovalHandler(static (key, reason) => ...);
  • CacheKeyRemovedHandler — void (ReadOnlySpan<byte> key, CacheKeyRemovalReason reason); the span is the key exactly as stored (prefixes included), valid only for the callback.
  • CacheKeyRemovalReason — Invalidated (server push), LocalWrite (our own write), Replaced (a fresh fill displaced a live value, e.g. refresh-ahead), Expired (aged out, fired at physical reclamation), Evicted (budget), Flushed (flush/reconnect/dispose).

Enrollment model: one-shot, armed by the read

You hear about the keys you read, nothing else. Each read through the context arms one notification for that key; the first removal at-or-after the read consumes it; reading again re-arms. That makes exactly-once structural (Interlocked.Exchange take), needs no unenrollment API, and bounds dangling handlers — a consumer that stops reading a key stops hearing about it after at most one trailing event, and dropping a notification on the floor is always safe.

Observers hang on the key-table node, not the entry: invalidation touches the node while entry release is lazy, nodes are immortal and carried by reference across growth, and Node.Key is already an immortal byte[] — so slab compaction is invisible (not a death, enrollment survives) and dispatch needs no key copy.

The contract holds under the races:

  • arm-then-recheck on the hit path: either the invalidation's take sees the new observer, or the second look sees the invalidation and fires here.
  • a fill that raced an invalidation enrolls after the store, then rechecks: the consumer gets the value and an immediate Invalidated.
  • a stale-grace serve (value handed out already dead) likewise fires immediately — an already-wrong value is announced, never smuggled.
  • only a still-valid entry fires at release/replace: an invalidated corpse already announced itself, and firing again would steal next-epoch enrollments.

Dispatch

Never on the killing thread. NotifyRemoved takes the observer set and queues one ThreadPool.QueueUserWorkItem — the same model as pub/sub (CompleteAsWorker) — so user code never runs on the socket reader, under the key table's all-stripe locks (flush queues under them, fires after), or on the evictor. Handler exceptions are swallowed per handler. Cost when unused: one null field per key-table node and a null check per death.

Scope cuts, documented in code: a fill whose CLIENT CACHING YES prefix was dropped enrolls nothing (no invalidation will ever come), error replies likewise, and the shared-fill fallback send has no dependencies to enroll. Under default admission the doorkeeper's first-miss read isn't a fill, so coverage starts when caching does.

Tests

14 new tests in RespCacheRemovalTests cover every contract point: each reason, one-shot/re-arm, the fill-race (an executor that invalidates mid-send), stale-grace announcement, two consumers, a throwing consumer not silencing another, prefix fidelity, compaction-is-not-death (real slab promotion churn), and residency at fire time. Full cache/tracking sweep: 284/284 across 17 classes; all TFMs build clean.

A context enrolled via RespDatabaseContext.WithRemovalHandler is told - with
the key and a reason - when a key it read stops being servable from the
client-side cache, so layers above (HybridCache tag memos, backplane
adapters) can keep derived state coherent without polling.

Observers hang on the key-table NODE, the one object every death already
touches: invalidation fires promptly while the corpse is still resident,
flush walks the nodes it already walks, and slab compaction is not a
removal. Enrollment is one-shot - armed by a serve, consumed by the first
removal at or after it, re-armed by reading again - which makes exactly-once
structural (an Interlocked take, as the result memo does) and bounds
dangling handlers. Arm-then-recheck closes the serve/death race, and a value
handed out already dead (a fill that raced an invalidation, a stale-grace
serve) is announced immediately rather than silently.

Dispatch is the library's existing model: never on the caller's thread -
the socket reader, the table's locks, the evictor - but queued to the
thread pool like pub/sub and connection events, with exceptions swallowed
per handler. No key copy: Node.Key is already immortal.

Reasons separate value-changed (Invalidated, LocalWrite, Replaced) from
cache hygiene (Evicted, Expired) and the bulk case (Flushed); a
correctness-only consumer treats them all as drop. A fill whose tracking
prefix was dropped enrolls nothing: no invalidation is coming for it to
hear.
WithMaxCacheAge and WithRemovalHandler are gone; the context carries a
whole CachePolicy (WithCachePolicy, on both public context types), and
CachePolicy - now a record, so callers derive with 'policy with {...}' -
gains RemovalHandler. The surface stops growing contexts-times-knobs:
a new read-time knob is one init property, and the first ship had
already dropped a cell of that matrix (no WithRemovalHandler on
RespServerContext).

A per-read policy replaces the default wholesale and narrows, never
widens: the default TimeToLive stays a ceiling, reclamation follows the
default policy alone, and a per-read grace window only functions when
the default policy enabled stamping. RefreshAfter is now genuinely
per-caller. A RemovalHandler on the DefaultPolicy is connection-wide
enrollment - every read arms it, including background refresh
completions - which is what a backplane-style consumer wants, and has a
new test.

The policy's tick conversions are memoised at construction (the old
MaxCacheAgeService did this; now the default-policy hit path gets it
too), which surfaced a static-ordering trap: Default must be declared
after the field its construction reads.
@mgravell

Copy link
Copy Markdown
Member Author

Reworked per review discussion: the per-knob withers are gone. WithCachePolicy(CachePolicy) is the one extension point (both context types), CachePolicy is now a record carrying RemovalHandler, and a handler on the DefaultPolicy gives connection-wide enrollment (new test). A per-read policy replaces the default wholesale and narrows-never-widens on lifetime; RefreshAfter/grace are now genuinely per-caller (grace still gated on the default policy's stamping). Docs updated.

@mgravell
mgravell merged commit 99ae40b into marc/v4-cache-result-reuse Nov 12, 2026
2 of 3 checks passed
@mgravell
mgravell deleted the marc/v4-cache-removal-notify branch November 12, 2026 13:50
mgravell added a commit that referenced this pull request Nov 12, 2026
…ypes (#3349)

* spike: cache hits reuse the deserialized result for provably-shareable types

An Entry memoizes ONE parsed result with the handler that produced it (first writer wins,
handler-identity keyed, carried across slab compaction); the hit funnel serves the memo
before touching the reader. Core auto-enrolls strings and no-reference value types
(IsReferenceOrContainsReferences, conservative down-level); richer rules opt in through
the new public IRespResultReuse<TResult> on the handler - the HybridCache integration seam.

* A memoized result charges the cache budget: IRespResultReuse gains SizeOf, and the core sizes what it enrolls

The charge is paired with the Memo object itself - set wins charge, whoever takes the memo
(removal, replacement, slab move via ownership transfer, with a tombstone closing the slot)
credits exactly what was charged - so MaxBytes cannot drift under the discard/late-set race.
Strings are header+UTF-16, boxed no-ref structs are header+Unsafe.SizeOf; handler-opted
results volunteer an estimate, 0 meaning uncounted.

* Per-key removal notification for the client-side cache (#3350)

* Per-key removal notification for the client-side cache

A context enrolled via RespDatabaseContext.WithRemovalHandler is told - with
the key and a reason - when a key it read stops being servable from the
client-side cache, so layers above (HybridCache tag memos, backplane
adapters) can keep derived state coherent without polling.

Observers hang on the key-table NODE, the one object every death already
touches: invalidation fires promptly while the corpse is still resident,
flush walks the nodes it already walks, and slab compaction is not a
removal. Enrollment is one-shot - armed by a serve, consumed by the first
removal at or after it, re-armed by reading again - which makes exactly-once
structural (an Interlocked take, as the result memo does) and bounds
dangling handlers. Arm-then-recheck closes the serve/death race, and a value
handed out already dead (a fill that raced an invalidation, a stale-grace
serve) is announced immediately rather than silently.

Dispatch is the library's existing model: never on the caller's thread -
the socket reader, the table's locks, the evictor - but queued to the
thread pool like pub/sub and connection events, with exceptions swallowed
per handler. No key copy: Node.Key is already immortal.

Reasons separate value-changed (Invalidated, LocalWrite, Replaced) from
cache hygiene (Evicted, Expired) and the bulk case (Flushed); a
correctness-only consumer treats them all as drop. A fill whose tracking
prefix was dropped enrolls nothing: no invalidation is coming for it to
hear.

* Per-caller cache behaviour is one policy, not a wither per knob

WithMaxCacheAge and WithRemovalHandler are gone; the context carries a
whole CachePolicy (WithCachePolicy, on both public context types), and
CachePolicy - now a record, so callers derive with 'policy with {...}' -
gains RemovalHandler. The surface stops growing contexts-times-knobs:
a new read-time knob is one init property, and the first ship had
already dropped a cell of that matrix (no WithRemovalHandler on
RespServerContext).

A per-read policy replaces the default wholesale and narrows, never
widens: the default TimeToLive stays a ceiling, reclamation follows the
default policy alone, and a per-read grace window only functions when
the default policy enabled stamping. RefreshAfter is now genuinely
per-caller. A RemovalHandler on the DefaultPolicy is connection-wide
enrollment - every read arms it, including background refresh
completions - which is what a backplane-style consumer wants, and has a
new test.

The policy's tick conversions are memoised at construction (the old
MaxCacheAgeService did this; now the default-policy hit path gets it
too), which surfaced a static-ordering trap: Default must be declared
after the field its construction reads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant