From 5b7838aeeac41fff7aa9bc93a062aa4c69d4cca3 Mon Sep 17 00:00:00 2001 From: RodCor Date: Wed, 9 Sep 2026 00:08:26 -0300 Subject: [PATCH] Present verified v2.8.0 memory metrics across website and docs --- content/docs/changelog.mdx | 20 ++++++++++- .../docs/how-kimetsu-works/configuration.mdx | 7 ++-- .../docs/memory-benchmark/answerability.mdx | 35 +++++++++++++++---- content/docs/memory-benchmark/brainbench.mdx | 2 +- content/docs/memory-benchmark/index.mdx | 10 ++++-- src/app/(home)/page.tsx | 11 +++--- 6 files changed, 66 insertions(+), 19 deletions(-) diff --git a/content/docs/changelog.mdx b/content/docs/changelog.mdx index e6b6e1a..c3052bf 100644 --- a/content/docs/changelog.mdx +++ b/content/docs/changelog.mdx @@ -12,7 +12,11 @@ onward the project follows SemVer normally: patch releases are bug-fix-only, minor releases are backward-compatible additions, and breaking changes require a major bump. -## Unreleased +## v2.8.0: Durable memory and scoped answerability + +Merged into main; release publication pending as of September 9, 2026. The +[metric summary](/docs/memory-benchmark/answerability) preserves the measured +pre-release build fingerprints and limitations. ### Memory evidence and hardening @@ -23,6 +27,8 @@ breaking changes require a major bump. - The [paired answerability study](/docs/memory-benchmark/answerability) retained 24/27 hits and reduced unwanted injections 15/18 → 3/18 on its new synthetic fixture. Exact metadata matched 36/45 in both repeats; p95 was 376.6 → 386.6 ms. + Mean MCP response bytes increased 613.5 → 651.2 (+6.2%); p95 increased + 2.7%. These measure evidence delivery, not final-answer accuracy. Compound retrieval and unsupported-subject fallback remain incomplete, and the guard remains disabled by default. @@ -31,6 +37,18 @@ breaking changes require a major bump. - Updated transitive `h2` from 0.4.14 to 0.4.16, fixing `RUSTSEC-2026-0258` (unbounded empty HTTP/2 DATA frames). +- Reject redirected repository roots and Windows path aliases; validate existing + brain state paths before initialization shortcuts. +- Enforce HTTPS and disable redirects for signed Bedrock requests; validate + region labels and encode model IDs as path segments. +- Replace yanked `der` 0.8.0 with 0.8.2. + +### Release validation + +- 1,500 workspace tests passed with six intentionally ignored; 132 benchmark + Rust tests passed after the dependency refresh. +- Release notes now support titled changelog headings and reject empty output. + ## v2.7.0: Retrieval that knows when to stay silent This release hardens the semantic broker around the cases that ordinary diff --git a/content/docs/how-kimetsu-works/configuration.mdx b/content/docs/how-kimetsu-works/configuration.mdx index 8084ff0..df9c860 100644 --- a/content/docs/how-kimetsu-works/configuration.mdx +++ b/content/docs/how-kimetsu-works/configuration.mdx @@ -5,14 +5,15 @@ description: "Every knob in project.toml, the off-switches, and the environment Every knob in `project.toml`, the off-switches, and the environment overrides. -## Unreleased fact guard +## v2.8.0 fact guard (opt-in) On builds containing structured fact evidence, `broker.explicit_fact_guard` defaults to `false`. Enable with `kimetsu config set broker.explicit_fact_guard true`. It adds scoped supported/missing/conflicting evidence without extra model calls; injected context still consumes tokens. See [results, storage behavior and -limitations](/docs/memory-benchmark/answerability). This setting is unreleased -and is deliberately separate from the published configuration example below. +limitations and publication status](/docs/memory-benchmark/answerability). +Requires v2.8.0 or a source build containing the merged changes; it is separate +from the historical configuration example below. Project config lives in `/project.toml`: diff --git a/content/docs/memory-benchmark/answerability.mdx b/content/docs/memory-benchmark/answerability.mdx index 4feca60..73a7502 100644 --- a/content/docs/memory-benchmark/answerability.mdx +++ b/content/docs/memory-benchmark/answerability.mdx @@ -1,12 +1,30 @@ --- -title: "Structured answerability (unreleased)" +title: "Structured answerability (v2.8.0)" description: "Paired evidence-delivery measurements, configuration and remaining gaps for the opt-in fact guard." --- -This is **unreleased, opt-in work**, not a new overall BrainBench score or a -claim about generated-answer accuracy. It adds source-bound configuration facts +These are **v2.8.0 opt-in evidence-delivery results**, not a new overall +BrainBench score or a claim about generated-answer accuracy. It adds source-bound configuration facts and reports which requested attributes are supported, missing or conflicting. +**Release status checked September 9, 2026:** the v2.8.0 implementation is +[merged](https://github.com/RodCor/kimetsu/pull/45), while the latest published +GitHub release is still v2.7.0. These measurements use implementation `3ae8329` +and harness `2c74dad`; they are not a new run of a tagged v2.8.0 binary or a +v2.7.0-versus-v2.8.0 comparison. + +## v2.8.0 metric summary + +Frozen structured-fact fixture: 45 cases, repeated twice. + +| Metric | Baseline | Candidate | Interpretation | +|---|---|---|---| +| Unwanted injections | 15/18 (83.3%) | 3/18 (16.7%) | 80% fewer failures; 66.7 percentage points lower | +| Positive retrieval hits | 24/27 (88.9%) | 24/27 (88.9%) | No positive-hit loss | +| Exact evidence metadata | Not emitted | 36/45 (80%) per repeat | Not generated-answer accuracy | +| Subsequent-query p95 | 376.6 ms | 386.6 ms | +10.0 ms (+2.7%) | +| Mean MCP response bytes | 613.5 | 651.2 | +37.8 bytes (+6.2%) | + ## Paired results — September 7, 2026 The campaign contains 688 observations over 299 distinct scenario/query cases. @@ -50,8 +68,8 @@ On a build containing this change, enable it with: kimetsu config set broker.explicit_fact_guard true ``` -The default is `false`; set it back to `false` to disable it. Published packages -may not contain this unreleased feature. An older warm daemon must be restarted +The default is `false`; set it back to `false` to disable it. Use v2.8.0 or +a source build containing the merged changes; v2.7.0 packages lack this feature. An older warm daemon must be restarted with the updated build to use its new behavior. Schema 15 maintains a rebuildable SQLite projection bound to claim revisions, @@ -76,8 +94,11 @@ and cannot certify corpus-wide conflict detection. No grammar, parameters or expectations changed after these results. Retrieval per requested attribute and safer unsupported-subject handling remain follow-up work. -The tested implementation passed 1,470 workspace tests (six ignored), 132 -benchmark Rust tests, 18 Python tests and six release CLI/MCP probes. +The measured implementation passed 1,470 workspace tests (six ignored), 132 +benchmark Rust tests, 18 Python tests and six release CLI/MCP probes. Separate +v2.8.0 release-preparation validation passed **1,500 workspace tests**, six +ignored, plus **132 benchmark Rust tests** after the dependency refresh. +[Release validation and security scope](https://github.com/RodCor/kimetsu/blob/785c3df/docs/audits/2026-09-08-release-security.md). Source: [full report, frozen fixtures, raw observations and hashes](https://github.com/RodCor/kimetsu/blob/281c6df/docs/audits/2026-09-07-structured-facts.md). diff --git a/content/docs/memory-benchmark/brainbench.mdx b/content/docs/memory-benchmark/brainbench.mdx index 7e7443a..e035958 100644 --- a/content/docs/memory-benchmark/brainbench.mdx +++ b/content/docs/memory-benchmark/brainbench.mdx @@ -24,7 +24,7 @@ Brain Quality Index of 80.0%**. By capability: ## v2.6.0 re-measurement (264 scenarios) -For the newer, unreleased paired evidence-delivery work, see +For the v2.8.0 paired evidence-delivery measurements, see [Structured answerability](/docs/memory-benchmark/answerability). Its synthetic 45-query result is separate from the historical overall indices on this page. diff --git a/content/docs/memory-benchmark/index.mdx b/content/docs/memory-benchmark/index.mdx index be1c984..b410b1e 100644 --- a/content/docs/memory-benchmark/index.mdx +++ b/content/docs/memory-benchmark/index.mdx @@ -9,9 +9,13 @@ them rather than take our word for it. ## The headline numbers -The [unreleased structured-answerability study](/docs/memory-benchmark/answerability) +The [v2.8.0 structured-answerability study](/docs/memory-benchmark/answerability) reports a separate paired experiment: unwanted injections 15/18 → 3/18, with -24/27 positive hits retained. It does not replace the published scores below. +24/27 positive hits retained and exact evidence metadata matching 36/45 cases +per repeat. P95 changed from 376.6 to 386.6 ms; mean response size from 613.5 +to 651.2 bytes. The linked study records build provenance and release status. +It does not replace the historical published scores below or remeasure them +on v2.8.0. | benchmark | result | |-----------|--------| @@ -24,7 +28,7 @@ reports a separate paired experiment: unwanted injections 15/18 → 3/18, with | stale-hit rate | **0.091** (was 0.500 on flat retrieval) | | cost per solved task | **~13x cheaper** than a no-brain baseline | -Every result uses `jina-v2-base-code` + the `ms-marco-tinybert-l-2-v2` +The historical results in the table use `jina-v2-base-code` + the `ms-marco-tinybert-l-2-v2` cross-encoder reranker unless noted, and the memory pipeline makes zero LLM calls: the reader in the public benchmarks answers questions, it never stores or retrieves. diff --git a/src/app/(home)/page.tsx b/src/app/(home)/page.tsx index 3777520..53c51a0 100644 --- a/src/app/(home)/page.tsx +++ b/src/app/(home)/page.tsx @@ -43,7 +43,7 @@ const questions = [ { question: "What do the new benchmark numbers prove?", answer: - "They measure delivery of stored evidence on a frozen synthetic fixture. They do not measure whether an agent gives a correct final answer. The structured-fact guard is unreleased and opt-in, and compound questions can still miss an attribute.", + "They measure delivery of stored evidence on a frozen synthetic fixture. They do not measure whether an agent gives a correct final answer. The v2.8.0 structured-fact guard is opt-in, and compound questions can still miss an attribute.", }, ]; @@ -153,7 +153,7 @@ export default function HomePage() { Better memory knows when evidence is missing.

- Our latest structured-answerability experiment reduced unwanted + The structured-answerability study for v2.8.0 reduced unwanted injections while preserving positive retrieval hits.

@@ -172,7 +172,7 @@ export default function HomePage() {

Structured fact guard

- Unreleased · Opt-in + v2.8.0 · Opt-in
@@ -196,6 +196,7 @@ export default function HomePage() { {[ ["Unwanted injections ↓", "15/18", "3/18"], ["Positive retrieval hits", "24/27", "24/27"], + ["Exact evidence metadata", "Not emitted", "36/45 (80%)"], ["P95 latency", "376.6 ms", "386.6 ms"], ["Mean response bytes", "613.5", "651.2"], ].map(([label, before, after]) => ( @@ -213,7 +214,9 @@ export default function HomePage() {

80% fewer unwanted injections; p95 latency increased by about - 2.7%. This feature remains disabled by default. + 2.7%. This feature remains disabled by default. Measurements use + the pre-release implementation; see the study for build + fingerprints and publication status.