Stop leftover covering as LoCoMo 180 benchmax - #170
Merged
Merged
Conversation
Isolated leftover covering on the skip-ingest 180 is saturating at 137/180. Industry on the same tenant is still 62/180; full product /recall is still 11.4%. Do not add another query-shape detector from that remaining ledger (no P54 peaceful-moments covering). Co-authored-by: aria <aria@engramhq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Isolated leftover covering on the skip-ingest LoCoMo 180 is benchmaxxing-adjacent and must stop. Product hybrid
/recallon tenantdiag-mh-135is 137/180; the industry lane on the same tenant is still 62/180; full product/recallis still 175/1540 = 11.4%. 137/180 is not 90%, not n=1540, and not a Mem0 same-pin.Each covering detector is generic English, but the next detector was queued from this 180’s remaining ledger. That is Goodhart.
internal/memory/recall.gois now ~50% leftover covering (24 query-shape detectors, 58 covering-line helpers). P28→P53 was +24, almost all one-item recoveries. Cycle-closeout already called covering saturating; we kept going anyway.No P54. Do not add a peaceful/nature/miss-about covering for
conv-44-q62. Remaining 43 misses are WRITE, count dumps, incomplete lists, OD hypotheticals, relative dates, and steal-slots. Optimistic covering ceiling on this 180 is ~141/180, still 21 short of 162.Beating public benchmarks still means: compiler WRITE (S1), entity-scoped enumerate (S2), industry lane top-k 200 (S5), fair Mem0 same-pin after quota 2026-09-01, n=1540 only at S6.
Existing leftover covering stays. This PR does not rip it out and does not change recall behavior beyond a stop comment.
Changes
AGENTS.md,sota-execution-plan.md,public-bench-ladder.md, competitive/research READMEs, benchmarks README honesty-stop row.leftoverCoveringRareForQueryso the next agent does not add another query-shape detector from this 180.Test plan
gofmt -l internal/memory/recall.goclean (comment only).go vet ./.../go test ./...not required for docs + comment; run if CI wants them.Linear / tracking
Risk / follow-ups
looks*Query.mainwith this unless explicitly asked. Stagingdevshould pick it up so later agents see the stop.