Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ Ladder: [research/public-bench-ladder.md](../research/public-bench-ladder.md).
| P58 historical typed-set lists S0 this-VM (2026-08-26) | Same store, hybrid on. S2 product: scan past current-state for item/skill/place-set/`activities`+`done` questions; singular `location` stays a point fact; leftover how/why does not widen hop scan. **141/180** (MH **21/33**, OD **4/11**, SH **85/98**, temporal **31/38**). [pin](./artifacts/locomo-s0-diag-mh-135-p58-20260826.md). John outdoor hiking+mountaineering. Unique losses none vs P54. Collars/tricks still miss. **Not** leftover covering. **Not** n=1540, **not** a Mem0 same-pin. Not 90% (162/180 on this sample; n=1540 for public LoCoMo). |
| P59 dest-class lists S0 this-VM (2026-08-26) | Same store, hybrid on. S2 product: dest-being skills without requiring `trick`; transfer-cue item objects (`buy`/`for`); possessed-class dest names (`dog named`, not generic `named`). **143/180** (MH **23/33**, OD **4/11**, SH **85/98**, temporal **31/38**). [pin](./artifacts/locomo-s0-diag-mh-135-p59-20260826.md). Audrey collars/tags/toys/beds; James swim/frisbee/skateboard. Unique losses none vs P58. p59/p59b 142/180 are not pins. **Not** leftover covering. **Not** n=1540, **not** a Mem0 same-pin. Not 90% (162/180 on this sample; n=1540 for public LoCoMo). |
| P61 beneficiary org sets S0 this-VM (2026-08-26) | Same store, hybrid on. S2 product: who/which beneficiary questions recover raise/for-cue affiliation objects (shelter, homeless, hospital) instead of leftover tournament slogans. **144/180** (MH **24/33**, OD **4/11**, SH **85/98**, temporal **31/38**). [pin](./artifacts/locomo-s0-diag-mh-135-p61-20260826.md). Unique losses none vs P59. p60/p60b 143/180 food-set 180s are not pins. **Not** leftover covering. **Not** n=1540, **not** a Mem0 same-pin. Not 90% (162/180 on this sample; n=1540 for public LoCoMo). |
| P62 community participation sets S0 this-VM (2026-08-26) | Same store, hybrid on. S2 product: in-what-ways community questions recover joined/organizing/host/article-participating-in objects (activist group, art show, mentorship, pride event) instead of leftover courage slogans; attended compiler junk is ranked after. **145/180** (MH **25/33**, OD **4/11**, SH **85/98**, temporal **31/38**). [pin](./artifacts/locomo-s0-diag-mh-135-p62-20260826.md). Unique losses none vs P61. **Not** leftover covering. **Not** n=1540, **not** a Mem0 same-pin. Not 90% (162/180 on this sample; n=1540 for public LoCoMo). |
| P42 how-did-start leftover covering S0 this-VM (2026-08-25) | Same store, hybrid on. How-did-start leftover covering admits duration-matched inception leftover on `how did … start … years ago`; covering prefers a multi-stem inception pair over a walking-only duration fact or a gym transformation/journey restatement; lexical search drops start/journey wrappers only on that query shape. **126/180** (MH **18/33**, OD **4/11**, SH **74/98**, temporal **30/38**). [pin](./artifacts/locomo-s0-diag-mh-135-p42-20260825.md). Diet+walking pair leftover recovered. Does not add a diet/walking or gym dictionary. **Not** n=1540, **not** a Mem0 same-pin. |
| BEAM 100K this cycle | [beam-100k-fresh-20260815.md](./artifacts/beam-100k-fresh-20260815.md) |
| BEAM 100K historical | [beam-100k-c0-async.md](./artifacts/beam-100k-c0-async.md) |
Expand Down

Large diffs are not rendered by default.

55 changes: 55 additions & 0 deletions docs/benchmarks/artifacts/locomo-s0-diag-mh-135-p62-20260826.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# LoCoMo S0 product `/recall` — P62 community participation sets — 2026-08-26

Same frozen store as [locomo-s0-diag-mh-135-20260822.md](./locomo-s0-diag-mh-135-20260822.md): tenant `diag-mh-135` + conv-30, dataset SHA `79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4`, stratified 180 seed 1, `--fail-closed --skip-ingest`. Product SHA `c02d70a` (joined/organizing/host/article-participating-in recover onto the activity hop; attended compiler junk is ranked after). Hybrid **on** (`BRAINY_RECALL_LLM=1`). Go default for `BRAINY_RECALL_LLM` stays **off**.

P61 pair: [locomo-s0-diag-mh-135-p61-20260826.md](./locomo-s0-diag-mh-135-p61-20260826.md) (`ee2baa6`, **144/180**). Honesty stop: [benchmax-audit-2026-08-25.md](../../research/competitive/benchmax-audit-2026-08-25.md). This is **S2 enumerate**, not leftover covering.

**Not** n=1540. **Not** a Mem0 same-pin. **Not** SOTA. Does not replace integrity 32/180. Does not replace the reader-off 19/180 no-LLM pin. Does not replace README 11.4% / 70% 1×30. 90% on this 180 is **162/180**. 90% on public LoCoMo is n=1540.

## Scores vs prior pins on this store

| Lane | Overall | multi-hop | open-domain | single-hop | temporal |
| --- | ---: | ---: | ---: | ---: | ---: |
| product reader **off** (`453a929`) | **19/180 (0.106)** | **12/33** | 0/11 | 5/98 | 2/38 |
| product hybrid **on** P28 (`454fbb3`) | **113/180 (0.628)** | **18/33** | **4/11** | **61/98** | **30/38** |
| product hybrid **on** P53 (`ae15e40`) | **137/180 (0.761)** | **18/33** | **4/11** | **85/98** | **30/38** |
| product hybrid **on** P54 (`7653135`) | **140/180 (0.778)** | **20/33** | **4/11** | **85/98** | **31/38** |
| product hybrid **on** P58 (`4817e11`) | **141/180 (0.783)** | **21/33** | **4/11** | **85/98** | **31/38** |
| product hybrid **on** P59 (`2111b3b`) | **143/180 (0.794)** | **23/33** | **4/11** | **85/98** | **31/38** |
| product hybrid **on** P61 (`ee2baa6`) | **144/180 (0.800)** | **24/33** | **4/11** | **85/98** | **31/38** |
| product hybrid **on** P62 (`c02d70a`) | **145/180 (0.806)** | **25/33** | **4/11** | **85/98** | **31/38** |
| industry search+harness (reader-off pin) | 62/180 (0.344) | 10/33 | 3/11 | 27/98 | 22/38 |

MH **24→25**. OD **held 4**. SH **held 85**. Temporal **held 31**. Product overall still leads this-VM industry 62/180 on the labeled product lane — still not a Mem0 same-pin.

Item flips vs P61: **+1 / −0 = net +1**. Unique losses: **none**.

Named recovery (generic S2, not a LoCoMo-named rule):

- `conv-26-q39` Caroline community participation: leftover transition-courage line → **mentorship program, LGBTQ art show, LGBTQ activist group, LGBT pride event** (joined/organizing/host/article-participating-in objects, then attended; gold activist / pride parades / art show / mentoring accepted as paraphrase).

Held: P61 beneficiary join (dog shelter, homeless, children's hospital); P59 dest-class (Audrey collars/tags/toys/beds; James swim/frisbee/skateboard); leftover covering (soda and candy); Jolene snakes Susie+Seraphim; Maria dogs Coco+Shadow; food-set joins (q18 suggestion list; q37 meals without pie).

Still miss (next generic product, not covering): who-told marriage (`conv-49-q78`); food-set completeness vs WRITE-missing gold; OD 4/11; WRITE 4; polar surfing; Phuket where; companionship leftover (do not cover).

## Failure ledger (35 misses)

| Primary | P61 | P62 |
| --- | ---: | ---: |
| RETRIEVAL_MISS | 10 | 9 |
| PROOF_MISS | 14 | 14 |
| READER_MISS | 7 | 7 |
| WRITE_MISS | 4 | 4 |
| HARNESS_ERROR | 1 | 1 |

Unique leftover losses remain **none**. RETRIEVAL 10→9 is q39 flipping off the leftover courage line.

Largest P62 cells: `single-hop:PROOF_MISS` 8, `open-domain:PROOF_MISS` 5, `multi-hop:READER_MISS` 4, `temporal:RETRIEVAL_MISS` 3, `multi-hop:RETRIEVAL_MISS` 3. WRITE 4 — do not merge #133.

## What this says

1. "In what ways … community" questions were an S2 enumerate bug: leftover covering answered with a recency-top courage slogan while joined/attended/organizing/mentorship facts were already in the frozen store. Recovering those objects onto an activity hop, ranking joined/organizing/host ahead of attended compiler junk (`attended back`, `attended not-so-great`), and keeping the join against leftover covering recovers that set.
2. 145/180 is 80.6% on this sample, still far from 90% (**162/180**). Remaining mass is who-told lists, incomplete food gold (WRITE), OD hypotheticals, relative dates, and polar/where misses. Do not add leftover covering for companionship or peaceful moments.
3. Do not add an LGBTQ/activist/parade gold dictionary. Cues are joined / organizing / host / article participating-in / mentorship program / attended.

Report: `locomo-s0-diag-mh-135-p62-product-recall-s1-1a5a7c` (summary JSON + failure ledger in this folder). Auto smoke JSON/md dumps are not committed (secret scanner).
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
{
"tenant_prefix": "diag-mh-135",
"runs": [
{
"lane": "product-recall",
"run_id": "locomo-s0-diag-mh-135-p62-product-recall-s1-1a5a7c",
"accuracy": 0.8055555555555556,
"correct": 145,
"total": 180,
"by_group": {
"temporal": {
"correct": 31,
"total": 38
},
"multi-hop": {
"correct": 25,
"total": 33
},
"open-domain": {
"correct": 4,
"total": 11
},
"single-hop": {
"correct": 85,
"total": 98
}
},
"failure_histogram": {
"total": 35,
"by_primary": {
"PROOF_MISS": 14,
"RETRIEVAL_MISS": 9,
"READER_MISS": 7,
"WRITE_MISS": 4,
"HARNESS_ERROR": 1
},
"by_group": {
"single-hop:PROOF_MISS": 8,
"open-domain:PROOF_MISS": 5,
"multi-hop:READER_MISS": 4,
"temporal:RETRIEVAL_MISS": 3,
"multi-hop:RETRIEVAL_MISS": 3,
"open-domain:RETRIEVAL_MISS": 2,
"temporal:WRITE_MISS": 2,
"single-hop:READER_MISS": 2,
"temporal:READER_MISS": 1,
"multi-hop:WRITE_MISS": 1,
"single-hop:RETRIEVAL_MISS": 1,
"single-hop:WRITE_MISS": 1,
"temporal:PROOF_MISS": 1,
"single-hop:HARNESS_ERROR": 1
}
}
}
]
}
74 changes: 74 additions & 0 deletions docs/research/competitive/cycle-closeout.md
Original file line number Diff line number Diff line change
Expand Up @@ -5784,3 +5784,77 @@ Last Brainy pin **4/20**. Not re-run.

**One step:** generic **S2 list completeness remainder** — community participation lists (`conv-26-q39`) and who-told lists (`conv-49-q78`) whose gold objects are in the frozen store. Do not fish food-set completeness against WRITE-missing sandwich snacks / Beef Merlot. Then **S2b OD** (still 4/11; 0/4 diagnostic). Then **S1 WRITE** with re-ingest. Then **S5 industry** (62/180 on this tenant). Do **not** queue leftover covering from this 180. Do not chase camping-peaceful. Do not invent Sunday. Do not steal across speakers. Do not special-case Scout. Do not add companionship covering. Isolated leftover covering is saturating. Remaining 36: RETRIEVAL 10, PROOF 14, READER 7, WRITE 4, HARNESS 1. Fair Mem0 180 waits on quota reset 2026-09-01. n=1540 only at S6. Do not merge #133. Do not write SOTA. Kill list unchanged. Start: [handover-sota-agent-2026-08-21.md](../handover-sota-agent-2026-08-21.md).

## 2026-08-26 — P62 community participation sets (145/180; not leftover covering)

Pin: [locomo-s0-diag-mh-135-p62-20260826.md](../../benchmarks/artifacts/locomo-s0-diag-mh-135-p62-20260826.md).
Run `locomo-s0-diag-mh-135-p62-product-recall-s1-1a5a7c`. Product SHA **`c02d70a`**.
Honesty stop **`0c33eb8`** stays on `dev`; this increment is **S2 enumerate**, not covering.

**90% honesty:** 90% on this 180 is **162/180**. 90% on public LoCoMo is **n=1540**, last pin **11.4%**.
This row is **145/180**. It is **not** 90%. It is **not** beating Mem0.

### Landed

Product: in-what-ways / ways + community questions enumerate activity objects recovered from listed memories with joined / organizing / host / article participating-in / mentorship-program cues, then attended. Compiler `attended back` / `attended not-so-great` rows are thin-stopped and ranked after primary slots so they cannot fill the hop cap before art-show/activist/mentorship objects. Dual-entity join stays off. Leftover covering keeps a ≥2 activity join. Named-community token filter still runs. **No** LoCoMo-named rules. **No** leftover-covering detector. **No** LGBTQ/activist/parade gold dictionary.

Pin docs: this section; [docs/benchmarks/README.md](../../benchmarks/README.md) P62 row.

**Do not merge** leftover-covering PRs **#133**, **#131**, **#143**, **#145**.

### Own pins

Same 180, seed 1, 10 convos, fail-closed skip-ingest, tenant `diag-mh-135`. Hybrid on. Dataset SHA `79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4`.

| Pin | SHA | Overall | MH | OD | SH | temporal |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| Product reader off | `453a929` | **19/180** | **12/33** | 0/11 | 5/98 | 2/38 |
| P28 | `454fbb3` | **113/180** | 18/33 | 4/11 | 61/98 | 30/38 |
| P53 (covering) | `ae15e40` | **137/180** | 18/33 | 4/11 | 85/98 | 30/38 |
| P54 (entity-scoped counts) | `7653135` | **140/180 (0.778)** | **20/33** | **4/11** | **85/98** | **31/38** |
| P58 (historical typed-set lists) | `4817e11` | **141/180 (0.783)** | **21/33** | **4/11** | **85/98** | **31/38** |
| P59 (dest-class lists) | `2111b3b` | **143/180 (0.794)** | **23/33** | **4/11** | **85/98** | **31/38** |
| P61 (beneficiary org sets) | `ee2baa6` | **144/180 (0.800)** | **24/33** | **4/11** | **85/98** | **31/38** |
| **P62 (community participation sets)** | **`c02d70a`** | **145/180 (0.806)** | **25/33** | **4/11** | **85/98** | **31/38** |
| Industry search+harness | same tenant | **62/180** | 10/33 | 3/11 | 27/98 | 22/38 |
| Full n=1540 product `/recall` | `1b5ab3e` | **175/1540 = 11.4%** | 7.4% MH | 5.2% OD | 10.5% SH | 19.0% temporal |
| 1×30 conv-26 | `1b5ab3e` | **21/30 (70%)** | 10/10 | **0/4** | — | 11/16 |
| LME-20 | `1b5ab3e` | **4/20** | | | | |

Unique losses vs P61: **none**. Gain: Caroline community participation (`conv-26-q39`). Industry **62/180**. n=1540 and 1×30 **not** re-run; do not replace README 11.4% / 70%.

### Competitor compare (detailed)

#### 1. LoCoMo — trail (open-domain); this 180 is not public LoCoMo

Public LoCoMo is **n=1540 / 11.4%**. This 180 is a stratified diagnostic. **OD 4/11** (0/4 diagnostic still). Do **not** write lead from 145/180 vs Mem0 11/30 (handicapped, different pin). Published Mem0 **92.5%** (their harness, top-k 200, n=1540) is **context**, never a scoreboard row. Fair Mem0 180 waits on quota **2026-09-01**.

**Trailing axis (open-domain):** product mechanism still missing is **S2b** membership/hypothesis + **S1** write coverage (Xenoblade, yoga 2020, Phuket, Wolves). PoR: compiler coverage then re-ingest; not leftover covering.

**Leading axes (must not regress):** OpMem 13/13, marketing 17/17, 1×30 MH 10/10. P61 beneficiary join, P59 dest-class lists, P54 counts, and P53 covering holds held on live `/recall`.

#### 2. OpMem — lead (stale pin)

Last **13/13**. Last Mem0 Platform ops pin **10/13**. **Lead ops.** Not re-run this increment.

#### 3. Marketing vertical — lead (stale pin)

Last **17/17** vs Mem0 empirical **4/17**. **Lead governed vertical.**

#### 4. LME-20 — no pin this cycle

Last Brainy pin **4/20**. Not re-run.

#### 5. Graphiti / Zep — no pin

**No same-pin.**

**Mem0 OSS** was not re-measured. Platform fair 180 is **quota-blocked** until 2026-09-01.

### Why

`"In what ways is Caroline participating in the LGBTQ community?"` recovered a recency-top courage slogan because leftover covering beat an empty/wrong typed list. The four gold classes were already in listed memories as joined activist group, attended pride, organizing/hosting an art show, and a mentorship program. Recovering those objects onto an activity hop, ranking joined/organizing/host ahead of attended compiler junk, and keeping the join against leftover covering is the product mechanism.

### Next

**One step:** generic **S2 list completeness remainder** — who-told lists (`conv-49-q78`) whose gold objects are partly in the frozen store (work friends + extended family; do not invent Sam-as-told from in-dialogue). Do not fish food-set completeness against WRITE-missing sandwich snacks / Beef Merlot. Then **S2b OD** (still 4/11; 0/4 diagnostic). Then **S1 WRITE** with re-ingest. Then **S5 industry** (62/180 on this tenant). Do **not** queue leftover covering from this 180. Do not chase camping-peaceful. Do not invent Sunday. Do not steal across speakers. Do not special-case Scout. Do not add companionship covering. Isolated leftover covering is saturating. Remaining 35: RETRIEVAL 9, PROOF 14, READER 7, WRITE 4, HARNESS 1. Fair Mem0 180 waits on quota reset 2026-09-01. n=1540 only at S6. Do not merge #133. Do not write SOTA. Kill list unchanged. Start: [handover-sota-agent-2026-08-21.md](../handover-sota-agent-2026-08-21.md).

Loading
Loading