feat(benchmark): UUID cache-coherency validation on the router-replay path - #2
Open
4D0R wants to merge 8 commits into
Open
feat(benchmark): UUID cache-coherency validation on the router-replay path#24D0R wants to merge 8 commits into
4D0R wants to merge 8 commits into
Conversation
Presence + cross-conversation contamination checks on the --from-dataset replay path, modeled on the cache-coherency test. Reusable primitives in benchmark/replay_uuid.go (UUID gen, marker injection, recite instruction, validateReplayResponse) and FindLeakedUUIDs moved into the benchmark package. To be re-targeted to the router-replay path next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # cli/benchmark_options.go
…play Re-targets the per-turn UUID coherency check from the dataset-replay path (--from-dataset) to the tree-aware router-replay path (--router-replay-file), composing it with the existing forced-output feature (ignore_eos + continue-generating instruction). Design (Option C — boundary injection, with tail fallback): one deterministic UUID is injected per SESSION at the boundary between its cross-session-shared leading blocks (system/tools) and its per-session content. This lands the marker in a region cached WITHIN a session (later requests repeat it, giving a genuine KV-coherency signal) while leaving the cross-session shared prefix byte-identical, preserving cache-hit reproduction against the original capture. A session with no shared leading block falls back to tail injection. New benchmark/replay_router_uuid.go: computeBlockSessionCounts (streams a replay-v3 file once, counting distinct-session references per block hash), sharedPrefixBlockCount, buildSessionUUIDs, replayReciteFromContextInstruction (a recite ask that never embeds the UUID itself, so presence reflects genuine recall), and a max_tokens recite floor (with a one-time warning) so a tiny captured output budget can't truncate the recite line into a false PRESENCE_MISS. benchmark/replay_router_wire.go: buildAnthropicMessagesBody / buildOpenAIChatCompletionsBody take a new *uuidInjection param (nil = unchanged); the marker splices into the system prefix at the boundary, or falls back to the message tail, preserving Anthropic's strict user/ assistant role alternation. benchmark/replay_router_post.go / replay_router.go: replayPoster carries the per-session UUID/marker state and builds the injection per request; after a successful response, validates it via the existing validateReplayResponse/FindLeakedUUIDs primitives (unchanged, shared with the eval CLI). benchmark/auto.go precomputes the per-session UUID array and block-hash counts once before per-model goroutines spawn, sized from the header + --replay-series/--replay-series-indices (the same formula the router stream itself uses), so no lazy-growth is needed. Drops the dataset-path UUID wiring (replay.go, replay_uuid.go) that --replay-inject-uuids used before: buildReplayUUIDSets, computeInScopeAtEachGptTurn, replayTurnInjectable, replayReciteInstruction, capRecitedUUIDs, and the --replay-uuid-mode / --replay-uuids-per-turn flags are all removed. --replay-inject-uuids now requires --router-replay-file and is rejected with --from-dataset (flip of the previous gate). --replay-recite-every-turn is renamed --replay-recite-every-request. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…est mechanics Changes the --replay-inject-uuids feature to mirror cache_coherency.go's --shared-prefix-per-series mode instead of its own single-marker scheme: - N bare UUIDs per session (was 1 wrapped "[ref-id: ...]" marker), sized via computeStampsPerSeries off each session's per-session cached-region bytes (the root request's prefix blocks at/after sharedPrefixBlockCount) — see the new computePerSessionCachedChars/requestPrefixBytes helpers. - Injection is spliced as one system block/message of N bare, space-separated UUIDs at the boundary (or tail-fallback), mirroring buildCoherencySharedSeriesPrompt's tail, byte-identical across a session's own requests. - Recite instruction now asks for the exact ordered UUID list on the FIRST line, then lets the model continue normally (replayReciteFirstLineInstruction), replacing the old "find and echo the marker" ask. - Validation adds first-line output conformity (firstLineConformity, reusing matchesExpectedUUIDList) alongside the existing per-UUID presence and cross-session leak checks; RequestMetrics/requestDataRecord gain ExactMatch/ uuid_exact_match, and the auto-summary prints both UUID correctness and output conformity, matching the coherency CLI's two-test layout. - The recite max_tokens floor now scales with N via computeMaxOutputTokens instead of a fixed 64-token constant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildInjection returns a non-nil *uuidInjection on EVERY request once --replay-inject-uuids is on -- injection happens every turn so the stamp stays warm in KV, but only inj.Recite (reciteEveryRequest || isLastRequest) says the model was actually asked to recite it. With --replay-recite-every-request=false, non-final requests were being scored anyway, guaranteeing a false PRESENCE_MISS/conformity failure on every turn the model was never asked to recite. Add inj.Recite to the scoring gate in do() while leaving injection itself ungated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/messages markerAtBoundary required SharedPrefixLen >= len(effectiveSystemBlocks), i.e. "the shared run covers at least all system blocks" -- which stays true even when the cross-session-shared run extends PAST the system blocks into shared tools or a shared leading message. Splicing the per-session UUID marker as a system block in that case puts it ahead of those shared tools/messages on the wire, making them diverge per session and losing their cross-session prefix-cache hit even though their content stays byte-identical. Flip the comparison to SharedPrefixLen <= len(effectiveSystemBlocks): boundary injection is now eligible only when the shared run ends at or within the system blocks, falling back to tail injection whenever it reaches into tools/messages. Applied to both the Anthropic and OpenAI body builders; behavior is unchanged for the common case where the shared run is contained within the system blocks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…owed recite
Replace the "N UUID stamps per session, clustered at the shared->per-session
boundary, recite ALL on line 1" scheme with one deterministic UUID injected
inline per qualifying user turn (role==user, has a text block, hash used by
exactly one session), spread through the conversation, and a bounded recite
window per request: first turn + up to 3 most-recent turns, excluding the
current turn, capped at 4. This keeps recite cost/response-budget constant
regardless of session length while spreading coverage across the whole
conversation over time; every visible turn still gets its own inline marker
stamped (keeping it warm in KV) even when it falls outside the recite window.
- replay_router_uuid.go: computeSessionTurnHashes/isQualifyingUserTurn
replace computePerSessionCachedChars/countSessionsWithUsableBoundary/
sharedPrefixBlockCount/bareUUIDBlock; buildSessionTurnUUIDs replaces
buildSessionUUIDs and also returns a uuid->owning-session reverse map;
uuidInjection is now {StampByHash, Recite, ReciteLabels, ReciteUUIDs};
replayReciteWindowInstruction replaces replayReciteFirstLineInstruction.
- replay_router_wire.go: buildMessageContent/buildMessages/
buildOpenAIMessages take a stampByHash param and append the inline
"[turn-N id: <uuid>]" marker to the stamped message's own synthesized
text, keyed by its hash for byte-identical replay; boundary/tail-splice
logic removed from both body builders.
- replay_router_post.go: replayPoster gains turnHashes/hashToTurn/owner
(drops blockCounts); buildInjection now walks a request's visible turns
to build StampByHash and derive the recite window; do()'s validation
scores against inj.ReciteUUIDs and calls the new
findLeakedUUIDsByOwner scanner instead of validateReplayResponse.
- replay_uuid.go: add uuidRe + findLeakedUUIDsByOwner, an O(response)
contamination scan via the reverse owner map (FindLeakedUUIDs, whose
O(population) scan the dataset path and the coherency CLI still use,
is left untouched).
- replay_router.go: wire the new poster fields in
runRouterReplayInstance; drop the now-dead requestPrefixBytes helper
(its only caller, computePerSessionCachedChars, is removed).
- auto.go: precompute replaySessionTurnHashes/replayUUIDSets/
replayUUIDOwner once per run; startup diagnostic reports turn counts
instead of the retired boundary-usability stats.
- cli/benchmark_options.go: update --replay-inject-uuids' description.
- Tests rewritten for the new turn/window/contamination model (turn
identification across instances, window selection at turns 1-6,
determinism/fidelity, reverse-map contamination scanning).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the router-replay UUID branch up to date with main (multi-endpoint per-request routing + failover, coherency cold/warm cycle distribution, .so bumps). Resolved 3 conflicts and reconciled the UUID feature with main's shared-poster routing: - replay_router.go / replay_router_post.go / auto.go: main's endpoint picker builds one replayPoster per endpoint, shared across concurrent sessions. Made all UUID poster state global/read-only (allUUIDSets, sessionTurnHashes, owner, reciteEveryRequest, uuidEnabled) and derive the per-session sessionIdx per request from seriesNum, so a shared poster safely serves any session -- --replay-inject-uuids is safe with multi-endpoint specs, no per-session mutation on a shared object. - cli/eval_commands.go: main's cold/warm cycle-distribution summary + findLeakedUUIDs moved to the benchmark package. - cli/command_misc_cache_coherency_test.go: main's new eval tests + dataset-path test prune. Build, vet, and benchmark/cli tests green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
--replay-inject-uuids, a cache-coherency check for the--router-replay-filebenchmark path. It brings the mechanics of the existing
eval cache-coherency-*suite to realistic replayed traffic instead of synthetic prompts, so a replay run
can also verify that the served prefix cache stays coherent under the replayed
workload.
How it works
Ndeterministic UUID "stamps"(
NviacomputeStampsPerSeries, sized to the session's per-session cachedregion) and injects them as bare, space-separated tokens at the boundary
between the session's cross-session-shared leading blocks (system/tools) and
its per-session content — mirroring
cache_coherency.go's shared-prefix mode.This keeps the shared prefix byte-identical across sessions (so router-replay's
prefix-cache-hit reproduction is preserved) while the stamps live in a region
cached within the session, so later turns read them back from cache. A session
with no shared leading block falls back to tail injection.
then continue normally, so the check composes with forced-output (
ignore_eos)without constraining output volume.
first-line list conformity (
matchesExpectedUUIDList) — plus cross-sessioncontamination (
FindLeakedUUIDs), surfaced in the auto-summary and theper-request JSONL.
Flags
--replay-inject-uuids(requires--router-replay-file),--replay-uuid-seed,--replay-recite-every-request.--replay-natural-output/--replay-output-ratio.Notes
FindLeakedUUIDsis consolidated into thebenchmarkpackage (shared by theeval CLI and replay). Reuses
newUUIDGenerator,computeStampsPerSeries,matchesExpectedUUIDList, andcomputeMaxOutputTokens.invariant, first-line conformity, and cross-session contamination.