Skip to content

v4: cache hits reuse the deserialized result for provably-shareable types - #3349

Merged
mgravell merged 3 commits into
v4from
marc/v4-cache-result-reuse
Nov 12, 2026
Merged

mgravell merged 3 commits into
v4from
marc/v4-cache-result-reuse

Conversation

@mgravell

Copy link
Copy Markdown
Member

Cache hits currently re-parse the retained RESP payload on every serve. For HybridCache-style consumers (and plain string/scalar traffic), the deserialized result is often provably safe to hand back as the same instance — this PR memoizes it on the cache entry.

Design

  • Each cache entry gains one memo slot: an immutable (parser, value, bytes) atom, CAS'd in on a first-wins basis. There is no map and no lookup: a fetch is a single ReferenceEquals on the parser reference, so a different parser instance for the same entry simply re-parses (never a wrong result), and the slot never churns.

  • The decision happens post-parse in the send funnel — no flag threads through RespReader or the context.

  • The core auto-enrolls only provably-shareable results: string, and (where the runtime can detect it, #if NET) structs with no interior references via RuntimeHelpers.IsReferenceOrContainsReferences<T>(); down-level falls back to primitives and enums only.

  • Richer rules are opt-in via a new public capability side-interface alongside IRespHandler<TResult>:

    public interface IRespResultReuse<TResult>
    {
        bool MayReuse(in TResult value); // per-value veto
        long SizeOf(in TResult value);   // estimate in bytes; 0 = uncounted
    }

    This is how an external layer (e.g. HybridCache) teaches its deserializing handler to share results under its own reuse rules.

Budget accounting

A memoized value charges MaxBytes when its size is known: the core sizes its own enrollments (24 + 2·len for strings, 16 + sizeof(T) for boxed no-ref structs), and opted-in handlers volunteer via SizeOf (0 = uncounted). The charge is paired with the memo atom itself: whoever takes the memo (eviction, replacement, slab compaction — which transfers ownership rather than copying) credits back exactly what was debited, and taking leaves a tombstone so a late setter racing a discard loses the CAS and charges nothing. The budget cannot drift, lock-free.

Scope cuts (documented in code)

  • The fill-path parse is not memoized; the memo is created on the first hit, so one extra parse per entry lifetime.
  • One memo slot per entry, first parser wins; later parsers for the same entry re-parse.

Tests

11 new tests in RespCacheResultReuseTests cover instance-reuse for strings and no-ref structs, non-enrollment of ref-bearing structs (RedisValue), handler opt-in/refusal per value, parser-instance guarding, memo death with the entry on local writes, and exact charge/credit of both core-sized and volunteered sizes. The 270-test cache/tracking sweep and all-TFM library build are clean.

…e types

An Entry memoizes ONE parsed result with the handler that produced it (first writer wins,
handler-identity keyed, carried across slab compaction); the hit funnel serves the memo
before touching the reader. Core auto-enrolls strings and no-reference value types
(IsReferenceOrContainsReferences, conservative down-level); richer rules opt in through
the new public IRespResultReuse<TResult> on the handler - the HybridCache integration seam.
…zeOf, and the core sizes what it enrolls

The charge is paired with the Memo object itself - set wins charge, whoever takes the memo
(removal, replacement, slab move via ownership transfer, with a tombstone closing the slot)
credits exactly what was charged - so MaxBytes cannot drift under the discard/late-set race.
Strings are header+UTF-16, boxed no-ref structs are header+Unsafe.SizeOf; handler-opted
results volunteer an estimate, 0 meaning uncounted.
* Per-key removal notification for the client-side cache

A context enrolled via RespDatabaseContext.WithRemovalHandler is told - with
the key and a reason - when a key it read stops being servable from the
client-side cache, so layers above (HybridCache tag memos, backplane
adapters) can keep derived state coherent without polling.

Observers hang on the key-table NODE, the one object every death already
touches: invalidation fires promptly while the corpse is still resident,
flush walks the nodes it already walks, and slab compaction is not a
removal. Enrollment is one-shot - armed by a serve, consumed by the first
removal at or after it, re-armed by reading again - which makes exactly-once
structural (an Interlocked take, as the result memo does) and bounds
dangling handlers. Arm-then-recheck closes the serve/death race, and a value
handed out already dead (a fill that raced an invalidation, a stale-grace
serve) is announced immediately rather than silently.

Dispatch is the library's existing model: never on the caller's thread -
the socket reader, the table's locks, the evictor - but queued to the
thread pool like pub/sub and connection events, with exceptions swallowed
per handler. No key copy: Node.Key is already immortal.

Reasons separate value-changed (Invalidated, LocalWrite, Replaced) from
cache hygiene (Evicted, Expired) and the bulk case (Flushed); a
correctness-only consumer treats them all as drop. A fill whose tracking
prefix was dropped enrolls nothing: no invalidation is coming for it to
hear.

* Per-caller cache behaviour is one policy, not a wither per knob

WithMaxCacheAge and WithRemovalHandler are gone; the context carries a
whole CachePolicy (WithCachePolicy, on both public context types), and
CachePolicy - now a record, so callers derive with 'policy with {...}' -
gains RemovalHandler. The surface stops growing contexts-times-knobs:
a new read-time knob is one init property, and the first ship had
already dropped a cell of that matrix (no WithRemovalHandler on
RespServerContext).

A per-read policy replaces the default wholesale and narrows, never
widens: the default TimeToLive stays a ceiling, reclamation follows the
default policy alone, and a per-read grace window only functions when
the default policy enabled stamping. RefreshAfter is now genuinely
per-caller. A RemovalHandler on the DefaultPolicy is connection-wide
enrollment - every read arms it, including background refresh
completions - which is what a backplane-style consumer wants, and has a
new test.

The policy's tick conversions are memoised at construction (the old
MaxCacheAgeService did this; now the default-policy hit path gets it
too), which surfaced a static-ordering trap: Default must be declared
after the field its construction reads.
@mgravell
mgravell merged commit fcc8a64 into v4 Nov 12, 2026
2 of 3 checks passed
@mgravell
mgravell deleted the marc/v4-cache-result-reuse branch November 12, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant