Skip to content

Measure production memory delivery with auditable paired comparisons - #4

Merged
RodCor merged 16 commits into
mainfrom
codex/brain-benchmark-hardening
Sep 8, 2026
Merged

RodCor merged 16 commits into
mainfrom
codex/brain-benchmark-hardening

Conversation

@RodCor

@RodCor RodCor commented Sep 7, 2026

Copy link
Copy Markdown
Owner

BrainBenchmark now compares persistent production MCP delivery and retains the evidence an agent actually receives. Paired observations include rankings, capsules, optional answerability metadata, latency, serialized response bytes, and supported Windows process-memory measurements.

  • Verify effective model, rerank cutoff, guard configuration, session isolation, and binary/fixture fingerprints independently for each side.
  • Alternate repeat order, retain failures and unpaired cases, and report stale-query denominators explicitly.
  • Preserve compatibility with historical binaries lacking answerability metadata; fix dated compressed-memory matching and timed-out descendant cleanup.
  • Update README and changelog with the evaluation contract and public report.

The latest frozen comparison retained 24/27 positive hits while reducing unwanted injections from 15/18 to 3/18. Exact metadata matched 36/45 cases in both repeats. These are evidence-delivery results, not generated-answer accuracy or a new overall BrainBench score.

Validation of the implementation: 132 Rust tests and 18 Python tests passed; optimized harness build and paired runs completed. This publication commit only changes documentation. The harness uses parent-checkout path dependencies and requires the companion Kimetsu hardening branch.

Companions: Kimetsu implementation and audit, website documentation.

@RodCor
RodCor merged commit 79a6091 into main Sep 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant