Conversation
Add docs/research/opmem/PUBLICATION_READINESS.md with reproducible protocol, claims/evidence table, independent 13/13 vs 10/13 review, missing work, threats to validity, artifact instructions, manuscript outline, and PR #185-#187 overlap map. Fix provably stale 12-task wording in opmem-spec, public-bench-ladder, mem0-parity-gaps, and master-plan; banner July opmem post as pre-upd03. Co-authored-by: aria <aria@engramhq.com>
Add brainy-recall and brainy-supersede adapter variants for manuscript ablations. Introduce run_opmem_paper_experiments.py with run manifests and optional OPMEM_ALLOW_INFRA_EXIT for diagnostic harness runs. Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Paid vendor runs require cost estimate plus approval (not a spend ban). Block Mem0 OSS/Zep/Letta/LangMem on missing adapters and worksheets. Treat 9/10 with infra errors as invalid runs, not comparable scores. Replace placeholder bib entries with verified arXiv/URL records. Co-authored-by: aria <aria@engramhq.com>
Add mem0-oss, zep, letta, and langmem lane adapters with deterministic namespaces, per-task reset, settling hooks, and redacted response logs. Dry-run token instrumentation and spend worksheets use pinned gpt-4.1-mini and embedding-3-small rates; live runs gated by OPMEM_EXTERNAL_LIVE. Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Wire tiktoken payload budgets (mem0 extraction prompt included) into dry-run output. Add run_opmem_external_live.py with preflight ceilings, smoke/full modes, and live Mem0 OSS + LangMem adapters (Postgres/pgvector). Co-authored-by: aria <aria@engramhq.com>
Record mem0-oss 9/13 and langmem 8/13 live pins, blocked zep/letta lanes, payload worst-case worksheet, run manifests, and updated manuscript PDF. Co-authored-by: aria <aria@engramhq.com>
Implement Letta 0.11.7 pip server adapter (opmem_facts block mapping), start script, pinned requirements, and full 13-task run (5/13, 0 infra). Document Zep blocked artifact when ZEP_API_KEY is absent; update manuscript and external-lanes setup for independent reproduction. Co-authored-by: aria <aria@engramhq.com>
ZEP_API_KEY absent from VM env and CLOUD_AGENT_INJECTED_SECRET_NAMES; smoke/full Zep runs remain blocked with artifacts; no score claimed. Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Migrate adapter to thread/graph SDK; free-tier preflight; smoke+full artifacts opmem-phase2-zep-full (0 infra). Update manuscript matrix and Phase 2 summary. Co-authored-by: aria <aria@engramhq.com>
Publication scoring from opmem-phase2-zep-full artifact only; raw 3/13 in footnote. No harness re-run or scoring code changes. Co-authored-by: aria <aria@engramhq.com>
Remove stale blocked/pending Zep-Letta wording; limitations and future work match Table tab:external-lanes (9/13, 8/13, 5/13, 3/9*). PDF rebuild only. Co-authored-by: aria <aria@engramhq.com>
Update Table 4 caption to state each lane ran the full 13-task script and Zep is scored on nine supported tasks with the existing asterisk footnote. Document hand-maintained external-lanes.tex in generated/README.md. Rebuild main.pdf for publication. Co-authored-by: aria <aria@engramhq.com>
Add xurl \artifact macro and sloppy evidence block; use breakable paths for live-full JSON names and Zep fixture ids. Rebuild PDF. Co-authored-by: aria <aria@engramhq.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Docs-only follow-up on OpMem manuscript Phase 2 external lanes: Table 4 caption now states each lane ran the full 13-task script and Zep is scored on 9 supported tasks (asterisk footnote unchanged in substance). Rebuilt
main.pdffor visual QA.Changes
docs/research/opmem/paper/generated/external-lanes.tex— caption wording (no score changes).docs/research/opmem/paper/generated/README.md— notesexternal-lanes.texis hand-maintained (generate-opmem-paper-tex.pydoes not emit this fragment).docs/research/opmem/paper/main.pdf— rebuilt PDF.Test plan
make pdfindocs/research/opmem/paper/pdftotextscan: noTODOor??in PDF text extractRisk / follow-ups
None — documentation only; no harness re-runs or score changes.