From 07006b6ff6936a7192d3e2f410e72e4ae2f692f9 Mon Sep 17 00:00:00 2001 From: RodCor Date: Wed, 9 Sep 2026 00:08:18 -0300 Subject: [PATCH] Document v2.8.0 metrics and completed CodeQL triage --- README.md | 2 +- docs/answerability.md | 27 ++++++++++++++ docs/audits/2026-09-08-release-security.md | 43 +++++++++++++++++++--- 3 files changed, 66 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 299d651..8707abf 100644 --- a/README.md +++ b/README.md @@ -202,7 +202,7 @@ ingest, TLS, Prometheus metrics, and a server-side reranker. Full setup in ## Docs -- **[v2.8.0 answerability](docs/answerability.md)**: opt-in scoped facts, +- **[v2.8.0 answerability and metrics](docs/answerability.md#v280-metrics)**: opt-in scoped facts, partial/conflicting evidence, configuration and limits. On the new 45-query synthetic fixture, unwanted injections fell **15/18 → 3/18**, with **24/27 hits retained**; this is not a new overall benchmark score. [Measured report](docs/audits/2026-09-07-structured-facts.md). diff --git a/docs/answerability.md b/docs/answerability.md index 5dac79e..e799ad7 100644 --- a/docs/answerability.md +++ b/docs/answerability.md @@ -20,6 +20,33 @@ Schema 15 adds a rebuildable SQLite fact projection. Facts bind to the memory's Projection maintenance adds write/storage work even when the guard is disabled; ordinary retrieval skips fact hydration while it is disabled. No extra model calls are required. Delivered evidence and metadata still consume agent context tokens. MCP output admission includes serialized metadata in its byte upper bound; that bound is not a measured tokenizer count. +## v2.8.0 metrics + +These measurements cover the implementation included in v2.8.0. They were +recorded on September 7 using implementation `3ae8329` and harness `2c74dad`, +against the preceding answerability build with the guard enabled on both sides. +They are not a fresh tagged-release run or a v2.7.0-versus-v2.8.0 comparison. + +| Metric (45 synthetic cases, two repeats) | Baseline | Candidate | +|---|---:|---:| +| Unwanted injections | 15/18 (83.3%) | 3/18 (16.7%) | +| Positive retrieval hits | 24/27 (88.9%) | 24/27 (88.9%) | +| Exact evidence metadata | Not emitted | 36/45 (80%) per repeat | +| Subsequent-query p95 | 376.6 ms | 386.6 ms | +| Mean MCP response bytes | 613.5 | 651.2 | + +Unwanted injections decreased by 80% relatively, or 66.7 percentage points. +P95 increased by 10.0 ms (2.7%), and response size by 37.8 bytes (6.2%). Deltas use unrounded measurements; +the table displays rounded values. +Across the three fixtures, 688 observations covered 299 distinct scenario/query +cases without positive-hit losses. The short synthetic runs do not establish +a statistically significant performance change or general answer accuracy. + +Separate release-preparation validation passed 1,500 workspace tests (six +ignored) and 132 benchmark Rust tests after dependency updates. See the +[release review](audits/2026-09-08-release-security.md) for that validation; +it does not change the original benchmark provenance. + ## Measured results and limits On the frozen 45-query synthetic fixture, unwanted injections fell from **15/18 to 3/18**, with **24/27 positive hits retained**. Exact metadata matched **36/45** cases in each of two repeats: 33/42 direct fact questions and three broad controls. P95 was **376.6 → 386.6 ms**; mean MCP result bytes were **613.5 → 651.2**. The development and prior answerability fixtures had no positive-hit losses. diff --git a/docs/audits/2026-09-08-release-security.md b/docs/audits/2026-09-08-release-security.md index f917d4d..470cf26 100644 --- a/docs/audits/2026-09-08-release-security.md +++ b/docs/audits/2026-09-08-release-security.md @@ -56,11 +56,9 @@ cannot enter the remote catalog, and verifies the excluded calls return the remote-mode denial without creating host configuration directories. No scanner rule or language is disabled. -This is a source review and regression-test disposition, not a report that -GitHub has closed the alerts. No alerts were dismissed through the API during -this review. The updated branch needs a fresh CodeQL scan, and the reviewed -false positives still need a maintainer disposition in GitHub before the alert -list can be described as cleared. +At the original September 8 review, no alerts had been dismissed through the +API. The branch still needed a fresh CodeQL scan and maintainer disposition. +The September 9 follow-up below records the completed alert review. Path checks protect against existing symlinks/junctions and filesystem aliases. Configured repository IDs that use reserved device names or trailing dots now @@ -108,3 +106,38 @@ Formatting and workspace/all-target Clippy with `-D warnings` pass for both the release feature set above and `--no-default-features`. Validation used Rust/Cargo 1.97.0 and cargo-audit 0.22.2. The manifest/lockfile consistency check covered all seven workspace packages and every existing inter-crate version pin. + + +## September 9 follow-up: code-scanning disposition + +After PR #45 merged, main commit `6a54f2e` passed all CI jobs and CodeQL analyses. +The remaining 46 open alerts were inspected against Rust SARIF analysis +`1745269927` and dismissed as **false positives**, each with a rationale in +GitHub. A subsequent API query returned **zero open code-scanning alerts**; +Dependabot also returned zero open alerts. No scanning rules were disabled. + +- **44 path findings** (#5–#48): every reported path starts with the HTTP + handler and taints `State.data_dir`. That path comes from the + operator's `--data` option, is canonicalized during startup, and is held in + `Arc`; requests cannot supply or mutate it. Repository IDs remain + validated before joining, and redirected repository/state paths are checked. +- Of these, **41 host-tool paths** additionally cannot execute through the + remote router because the allowlist rejects them before `call_tool`. + The other three are #21, #34 and #44 in repository/state initialization. +- **Two logging findings** (#2 and #3) concern a stored conversation correlation + identifier and an environment variable name, respectively. Neither field + contains the authentication secret inferred by the scanner. + +The Axum modeling issue is corroborated by GitHub's +[September 3 upstream fix](https://github.com/github/codeql/commit/d8e57bd223566df5bc236c0fdc54f12e222ab781), +which excludes state-field taint reads while retaining request-body taint. +Alert #49 for the same pattern had already been dismissed before this follow-up. +The HTTPS finding #4 no longer appears among main's open findings after the +transport hardening. These dispositions concern the reported flows, not a +claim that all possible vulnerabilities have been eliminated. + +The remote suite was rerun with embeddings, TLS and host integration features: +**36 passed, zero failed**. Main's Rust implementation and dependency lockfile +match the previously validated release candidate. A fresh dependency audit +again returned zero known vulnerabilities; the two informational maintenance +notices described above remain.