Skip to content

Stop leftover covering as LoCoMo 180 benchmax - #170

Merged
cursor[bot] merged 1 commit into
devfrom
pr/benchmax-audit-1e9e
Aug 25, 2026
Merged

cursor[bot] merged 1 commit into
devfrom
pr/benchmax-audit-1e9e

Conversation

@sdntsng

@sdntsng sdntsng commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Isolated leftover covering on the skip-ingest LoCoMo 180 is benchmaxxing-adjacent and must stop. Product hybrid /recall on tenant diag-mh-135 is 137/180; the industry lane on the same tenant is still 62/180; full product /recall is still 175/1540 = 11.4%. 137/180 is not 90%, not n=1540, and not a Mem0 same-pin.

Each covering detector is generic English, but the next detector was queued from this 180’s remaining ledger. That is Goodhart. internal/memory/recall.go is now ~50% leftover covering (24 query-shape detectors, 58 covering-line helpers). P28→P53 was +24, almost all one-item recoveries. Cycle-closeout already called covering saturating; we kept going anyway.

No P54. Do not add a peaceful/nature/miss-about covering for conv-44-q62. Remaining 43 misses are WRITE, count dumps, incomplete lists, OD hypotheticals, relative dates, and steal-slots. Optimistic covering ceiling on this 180 is ~141/180, still 21 short of 162.

Beating public benchmarks still means: compiler WRITE (S1), entity-scoped enumerate (S2), industry lane top-k 200 (S5), fair Mem0 same-pin after quota 2026-09-01, n=1540 only at S6.

Existing leftover covering stays. This PR does not rip it out and does not change recall behavior beyond a stop comment.

Changes

  • Add docs/research/competitive/benchmax-audit-2026-08-25.md (verdict, pin mixing table, remaining-43 classes, stop/continue).
  • Cycle-closeout 2026-08-25 honesty stop with required competitor compare (no new Mem0 score; do not refresh lead/trail from 137 vs 11).
  • Kill-list updates: AGENTS.md, sota-execution-plan.md, public-bench-ladder.md, competitive/research READMEs, benchmarks README honesty-stop row.
  • Comment on leftoverCoveringRareForQuery so the next agent does not add another query-shape detector from this 180.

Test plan

  • P53 ledger still 43 misses (RETRIEVAL 17 / PROOF 14 / READER 7 / WRITE 4 / HARNESS 1) — no new 180 run (no product recall change).
  • gofmt -l internal/memory/recall.go clean (comment only).
  • go vet ./... / go test ./... not required for docs + comment; run if CI wants them.
  • Confirm README 11.4% / 70% 1×30 were not replaced by 137/180.

Linear / tracking

  • ENG-168 (conversational long-memory epic)
  • ENG-176 (MH) — leftover covering was not the MH lever; MH still 18/33 on this 180, 7.4% on n=1540.

Risk / follow-ups

Open in Web Open in Cursor 

Isolated leftover covering on the skip-ingest 180 is saturating at 137/180.
Industry on the same tenant is still 62/180; full product /recall is still 11.4%.
Do not add another query-shape detector from that remaining ledger (no P54 peaceful-moments covering).

Co-authored-by: aria <aria@engramhq.com>
@sdntsng
sdntsng marked this pull request as ready for review August 25, 2026 21:54
@cursor
cursor Bot merged commit 0c33eb8 into dev Aug 25, 2026
2 checks passed
@sdntsng
sdntsng deployed to dev - brainy-worker-staging August 25, 2026 21:54 — with Render Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant