Skip to content

fix(eval): name the real cause of a noise refusal and stop faking gain_over_noise (#1089) - #1090

Merged
Birfy merged 1 commit into
mainfrom
fix/1089-noise-refusal-cause
Nov 14, 2026
Merged

Birfy merged 1 commit into
mainfrom
fix/1089-noise-refusal-cause

Conversation

@Birfy

@Birfy Birfy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

Closes #1089

build_trust_evidence printed one "inside the noise floor" sentence for three different refusal causes. This branches the note by cause and stops gain_over_noise from dividing by 1e-9.

  • Too few repeats (repeats < 2): the note says the floor cannot be estimated from N baseline repeat(s) instead of printing a one-point "floor".
  • Gain inside a measurable floor: the note prints baseline mean +0.500 ± 0.200, not an absolute interval placed next to a delta.
  • Floor narrower than min_effect: the note names min_effect and the floor half-width.
  • Every branch keeps the pinned phrase not distinguishable from noise.
  • gain_over_noise is 0.0 when the floor is undefined (fewer than 2 repeats or zero spread). The docstring says 0.0 there means undefined. distinguishable is unchanged.
  • One changelog fragment. No new public symbol.

Tests: tests/test_noise_calibration.py (parametrised undefined-floor cases + measurable-ratio case; the zero-spread > 1.0 assertion becomes == 0.0), tests/test_trust_evidence_noise.py (one-repeat, below-min_effect, ± band, passing zero-spread).

Local: 167 passed / 13 skipped on every test file touching noise/gain_over_noise; tools.changelog_fragments check and tools.gen_api_docs --check OK. tests/test_plugin_host_cli.py::test_present_host_passes fails on the box on plain main as well (environment), so it's unrelated.

@Birfy

Birfy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS

Tip SHA: 6e63c7b385ed2180a4fae9593114573b7ccd5f22
Ran: tests touching noise / gain_over_noise (167 passed, 13 skipped), python -m tools.changelog_fragments check, python -m tools.gen_api_docs --check, risk_gate (medium).
Not run: full pytest -q to completion on the box (stopped at test_plugin_host_cli.py::test_present_host_passes, which also fails on plain main locally; CI covers the full suite), mkdocs build --strict (CI).

Review: distinguishable is unchanged. Each of the three refusal causes gets its own sentence, all keep not distinguishable from noise. gain_over_noise is 0.0 exactly when the floor is undefined.

@Birfy
Birfy enabled auto-merge (squash) November 14, 2026 22:32
@Birfy

Birfy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS
Tip SHA: 6e63c7b
Ran: full pytest -q on the tip (fresh clone): all pass except 3 box-environment failures that fail identically on plain main (test_integrations.py::test_the_launcher_install_writes_ignores_a_shadowing_cwd, ::test_the_launcher_survives_a_bare_agentdescent_directory_too, test_plugin_host_cli.py::test_present_host_passes); targeted noise/trust-evidence/diagnose tests 53 passed; python -m tools.changelog_fragments check OK; python -m tools.gen_api_docs --check OK; risk_gate.py from main → T2.
Not run: mkdocs build --strict locally (CI docs / build passed on this tip).

Correctness

  • gain_over_noise now returns 0.0 when len(baseline_runs) < 2 or half_width <= 0, instead of dividing by 1e-9. This is exactly the undefined set from bug(eval): noise refusal names the wrong cause and prints an absolute band against a delta #1089, and distinguishable / bar are untouched, so the verdicts stay the same.
  • _noise_refusal_note branches in the same order the verdict logic uses: repeats < 2 first (that is the len(baseline_runs) >= 2 guard), then half_width < min_effect (where bar = max(half_width, min_effect) is the effect-size bar), then the measurable-floor case. noise.repeats is len(baseline_runs), and noise.noise_floor is the same (lo, hi) the half-width comes from, so the note can't drift from the calculation. One edge case: at exact equality half_width == min_effect the floor branch is used. The two bars are equal there, so either wording is accurate.
  • Every branch keeps the pinned phrase not distinguishable from noise. The ± band around baseline_mean fixes the old mix of an absolute interval and a delta.

Design fit / scope: private helper, no new public symbol, docstring says 0.0 means undefined, one changelog fragment. This matches the issue scope.

Tests: these cover all four probe cases (A–D) from #1089, plus the one existing assertion that changes (> 1.0 → == 0.0), with a comment explaining it. Good.

Nit (non-blocking): 0.0 now means both "no positive gain" and "undefined ratio". The docstring documents this. A later change could expose None/NaN instead if consumers need to tell the two apart, but that would be an interface change and is out of scope here.

@Birfy
Birfy merged commit c94e9d6 into main Nov 14, 2026
9 checks passed
@Birfy
Birfy deleted the fix/1089-noise-refusal-cause branch November 14, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(eval): noise refusal names the wrong cause and prints an absolute band against a delta

1 participant