Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion content/docs/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,11 @@ onward the project follows SemVer normally: patch releases are
bug-fix-only, minor releases are backward-compatible additions, and
breaking changes require a major bump.

## Unreleased
## v2.8.0: Durable memory and scoped answerability

Merged into main; release publication pending as of September 9, 2026. The
[metric summary](/docs/memory-benchmark/answerability) preserves the measured
pre-release build fingerprints and limitations.

### Memory evidence and hardening

Expand All @@ -23,6 +27,8 @@ breaking changes require a major bump.
- The [paired answerability study](/docs/memory-benchmark/answerability) retained
24/27 hits and reduced unwanted injections 15/18 → 3/18 on its new synthetic
fixture. Exact metadata matched 36/45 in both repeats; p95 was 376.6 → 386.6 ms.
Mean MCP response bytes increased 613.5 → 651.2 (+6.2%); p95 increased
2.7%. These measure evidence delivery, not final-answer accuracy.
Compound retrieval and unsupported-subject fallback remain incomplete, and
the guard remains disabled by default.

Expand All @@ -31,6 +37,18 @@ breaking changes require a major bump.
- Updated transitive `h2` from 0.4.14 to 0.4.16, fixing
`RUSTSEC-2026-0258` (unbounded empty HTTP/2 DATA frames).

- Reject redirected repository roots and Windows path aliases; validate existing
brain state paths before initialization shortcuts.
- Enforce HTTPS and disable redirects for signed Bedrock requests; validate
region labels and encode model IDs as path segments.
- Replace yanked `der` 0.8.0 with 0.8.2.

### Release validation

- 1,500 workspace tests passed with six intentionally ignored; 132 benchmark
Rust tests passed after the dependency refresh.
- Release notes now support titled changelog headings and reject empty output.

## v2.7.0: Retrieval that knows when to stay silent

This release hardens the semantic broker around the cases that ordinary
Expand Down
7 changes: 4 additions & 3 deletions content/docs/how-kimetsu-works/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,15 @@ description: "Every knob in project.toml, the off-switches, and the environment

Every knob in `project.toml`, the off-switches, and the environment overrides.

## Unreleased fact guard
## v2.8.0 fact guard (opt-in)

On builds containing structured fact evidence, `broker.explicit_fact_guard`
defaults to `false`. Enable with `kimetsu config set broker.explicit_fact_guard true`.
It adds scoped supported/missing/conflicting evidence without extra model calls;
injected context still consumes tokens. See [results, storage behavior and
limitations](/docs/memory-benchmark/answerability). This setting is unreleased
and is deliberately separate from the published configuration example below.
limitations and publication status](/docs/memory-benchmark/answerability).
Requires v2.8.0 or a source build containing the merged changes; it is separate
from the historical configuration example below.

Project config lives in `<project>/project.toml`:

Expand Down
35 changes: 28 additions & 7 deletions content/docs/memory-benchmark/answerability.mdx
Original file line number Diff line number Diff line change
@@ -1,12 +1,30 @@
---
title: "Structured answerability (unreleased)"
title: "Structured answerability (v2.8.0)"
description: "Paired evidence-delivery measurements, configuration and remaining gaps for the opt-in fact guard."
---

This is **unreleased, opt-in work**, not a new overall BrainBench score or a
claim about generated-answer accuracy. It adds source-bound configuration facts
These are **v2.8.0 opt-in evidence-delivery results**, not a new overall
BrainBench score or a claim about generated-answer accuracy. It adds source-bound configuration facts
and reports which requested attributes are supported, missing or conflicting.

**Release status checked September 9, 2026:** the v2.8.0 implementation is
[merged](https://github.com/RodCor/kimetsu/pull/45), while the latest published
GitHub release is still v2.7.0. These measurements use implementation `3ae8329`
and harness `2c74dad`; they are not a new run of a tagged v2.8.0 binary or a
v2.7.0-versus-v2.8.0 comparison.

## v2.8.0 metric summary

Frozen structured-fact fixture: 45 cases, repeated twice.

| Metric | Baseline | Candidate | Interpretation |
|---|---|---|---|
| Unwanted injections | 15/18 (83.3%) | 3/18 (16.7%) | 80% fewer failures; 66.7 percentage points lower |
| Positive retrieval hits | 24/27 (88.9%) | 24/27 (88.9%) | No positive-hit loss |
| Exact evidence metadata | Not emitted | 36/45 (80%) per repeat | Not generated-answer accuracy |
| Subsequent-query p95 | 376.6 ms | 386.6 ms | +10.0 ms (+2.7%) |
| Mean MCP response bytes | 613.5 | 651.2 | +37.8 bytes (+6.2%) |

## Paired results — September 7, 2026

The campaign contains 688 observations over 299 distinct scenario/query cases.
Expand Down Expand Up @@ -50,8 +68,8 @@ On a build containing this change, enable it with:
kimetsu config set broker.explicit_fact_guard true
```

The default is `false`; set it back to `false` to disable it. Published packages
may not contain this unreleased feature. An older warm daemon must be restarted
The default is `false`; set it back to `false` to disable it. Use v2.8.0 or
a source build containing the merged changes; v2.7.0 packages lack this feature. An older warm daemon must be restarted
with the updated build to use its new behavior.

Schema 15 maintains a rebuildable SQLite projection bound to claim revisions,
Expand All @@ -76,8 +94,11 @@ and cannot certify corpus-wide conflict detection. No grammar, parameters or
expectations changed after these results. Retrieval per requested attribute and
safer unsupported-subject handling remain follow-up work.

The tested implementation passed 1,470 workspace tests (six ignored), 132
benchmark Rust tests, 18 Python tests and six release CLI/MCP probes.
The measured implementation passed 1,470 workspace tests (six ignored), 132
benchmark Rust tests, 18 Python tests and six release CLI/MCP probes. Separate
v2.8.0 release-preparation validation passed **1,500 workspace tests**, six
ignored, plus **132 benchmark Rust tests** after the dependency refresh.
[Release validation and security scope](https://github.com/RodCor/kimetsu/blob/785c3df/docs/audits/2026-09-08-release-security.md).

Source: [full report, frozen fixtures, raw observations and hashes](https://github.com/RodCor/kimetsu/blob/281c6df/docs/audits/2026-09-07-structured-facts.md).

2 changes: 1 addition & 1 deletion content/docs/memory-benchmark/brainbench.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Brain Quality Index of 80.0%**. By capability:

## v2.6.0 re-measurement (264 scenarios)

For the newer, unreleased paired evidence-delivery work, see
For the v2.8.0 paired evidence-delivery measurements, see
[Structured answerability](/docs/memory-benchmark/answerability). Its synthetic
45-query result is separate from the historical overall indices on this page.

Expand Down
10 changes: 7 additions & 3 deletions content/docs/memory-benchmark/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,13 @@ them rather than take our word for it.

## The headline numbers

The [unreleased structured-answerability study](/docs/memory-benchmark/answerability)
The [v2.8.0 structured-answerability study](/docs/memory-benchmark/answerability)
reports a separate paired experiment: unwanted injections 15/18 → 3/18, with
24/27 positive hits retained. It does not replace the published scores below.
24/27 positive hits retained and exact evidence metadata matching 36/45 cases
per repeat. P95 changed from 376.6 to 386.6 ms; mean response size from 613.5
to 651.2 bytes. The linked study records build provenance and release status.
It does not replace the historical published scores below or remeasure them
on v2.8.0.

| benchmark | result |
|-----------|--------|
Expand All @@ -24,7 +28,7 @@ reports a separate paired experiment: unwanted injections 15/18 → 3/18, with
| stale-hit rate | **0.091** (was 0.500 on flat retrieval) |
| cost per solved task | **~13x cheaper** than a no-brain baseline |

Every result uses `jina-v2-base-code` + the `ms-marco-tinybert-l-2-v2`
The historical results in the table use `jina-v2-base-code` + the `ms-marco-tinybert-l-2-v2`
cross-encoder reranker unless noted, and the memory pipeline makes zero LLM
calls: the reader in the public benchmarks answers questions, it never stores
or retrieves.
Expand Down
11 changes: 7 additions & 4 deletions src/app/(home)/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ const questions = [
{
question: "What do the new benchmark numbers prove?",
answer:
"They measure delivery of stored evidence on a frozen synthetic fixture. They do not measure whether an agent gives a correct final answer. The structured-fact guard is unreleased and opt-in, and compound questions can still miss an attribute.",
"They measure delivery of stored evidence on a frozen synthetic fixture. They do not measure whether an agent gives a correct final answer. The v2.8.0 structured-fact guard is opt-in, and compound questions can still miss an attribute.",
},
];

Expand Down Expand Up @@ -153,7 +153,7 @@ export default function HomePage() {
Better memory knows when evidence is missing.
</h2>
<p className="mt-4 leading-7 text-fd-muted-foreground">
Our latest structured-answerability experiment reduced unwanted
The structured-answerability study for v2.8.0 reduced unwanted
injections while preserving positive retrieval hits.
</p>
<p className="mt-4 text-sm leading-6 text-fd-muted-foreground">
Expand All @@ -172,7 +172,7 @@ export default function HomePage() {
<div className="flex flex-wrap items-center justify-between gap-3 border-b border-fd-border p-5">
<h3 className="font-semibold">Structured fact guard</h3>
<span className="rounded-full border border-fd-border px-3 py-1 text-xs text-fd-muted-foreground">
Unreleased · Opt-in
v2.8.0 · Opt-in
</span>
</div>
<table className="w-full text-sm">
Expand All @@ -196,6 +196,7 @@ export default function HomePage() {
{[
["Unwanted injections ↓", "15/18", "3/18"],
["Positive retrieval hits", "24/27", "24/27"],
["Exact evidence metadata", "Not emitted", "36/45 (80%)"],
["P95 latency", "376.6 ms", "386.6 ms"],
["Mean response bytes", "613.5", "651.2"],
].map(([label, before, after]) => (
Expand All @@ -213,7 +214,9 @@ export default function HomePage() {
</table>
<p className="border-t border-fd-border p-5 text-xs leading-5 text-fd-muted-foreground">
80% fewer unwanted injections; p95 latency increased by about
2.7%. This feature remains disabled by default.
2.7%. This feature remains disabled by default. Measurements use
the pre-release implementation; see the study for build
fingerprints and publication status.
</p>
</div>
</div>
Expand Down
Loading