You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(eval): name the real cause of a noise refusal and stop faking gain_over_noise (#1089) - #1090
build_trust_evidence printed one "inside the noise floor" sentence for three different refusal causes. This branches the note by cause and stops gain_over_noise from dividing by 1e-9.
Too few repeats (repeats < 2): the note says the floor cannot be estimated from N baseline repeat(s) instead of printing a one-point "floor".
Gain inside a measurable floor: the note prints baseline mean +0.500 ± 0.200, not an absolute interval placed next to a delta.
Floor narrower than min_effect: the note names min_effect and the floor half-width.
Every branch keeps the pinned phrase not distinguishable from noise.
gain_over_noise is 0.0 when the floor is undefined (fewer than 2 repeats or zero spread). The docstring says 0.0 there means undefined. distinguishable is unchanged.
Local: 167 passed / 13 skipped on every test file touching noise/gain_over_noise; tools.changelog_fragments check and tools.gen_api_docs --check OK. tests/test_plugin_host_cli.py::test_present_host_passes fails on the box on plain main as well (environment), so it's unrelated.
Tip SHA: 6e63c7b385ed2180a4fae9593114573b7ccd5f22
Ran: tests touching noise / gain_over_noise (167 passed, 13 skipped), python -m tools.changelog_fragments check, python -m tools.gen_api_docs --check, risk_gate (medium).
Not run: full pytest -q to completion on the box (stopped at test_plugin_host_cli.py::test_present_host_passes, which also fails on plain main locally; CI covers the full suite), mkdocs build --strict (CI).
Review: distinguishable is unchanged. Each of the three refusal causes gets its own sentence, all keep not distinguishable from noise. gain_over_noise is 0.0 exactly when the floor is undefined.
Verdict: PASS
Tip SHA: 6e63c7b
Ran: full pytest -q on the tip (fresh clone): all pass except 3 box-environment failures that fail identically on plain main (test_integrations.py::test_the_launcher_install_writes_ignores_a_shadowing_cwd, ::test_the_launcher_survives_a_bare_agentdescent_directory_too, test_plugin_host_cli.py::test_present_host_passes); targeted noise/trust-evidence/diagnose tests 53 passed; python -m tools.changelog_fragments check OK; python -m tools.gen_api_docs --check OK; risk_gate.py from main → T2.
Not run: mkdocs build --strict locally (CI docs / build passed on this tip).
_noise_refusal_note branches in the same order the verdict logic uses: repeats < 2 first (that is the len(baseline_runs) >= 2 guard), then half_width < min_effect (where bar = max(half_width, min_effect) is the effect-size bar), then the measurable-floor case. noise.repeats is len(baseline_runs), and noise.noise_floor is the same (lo, hi) the half-width comes from, so the note can't drift from the calculation. One edge case: at exact equality half_width == min_effect the floor branch is used. The two bars are equal there, so either wording is accurate.
Every branch keeps the pinned phrase not distinguishable from noise. The ± band around baseline_mean fixes the old mix of an absolute interval and a delta.
Design fit / scope: private helper, no new public symbol, docstring says 0.0 means undefined, one changelog fragment. This matches the issue scope.
Tests: these cover all four probe cases (A–D) from #1089, plus the one existing assertion that changes (> 1.0 → == 0.0), with a comment explaining it. Good.
Nit (non-blocking): 0.0 now means both "no positive gain" and "undefined ratio". The docstring documents this. A later change could expose None/NaN instead if consumers need to tell the two apart, but that would be an interface change and is out of scope here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1089
build_trust_evidenceprinted one "inside the noise floor" sentence for three different refusal causes. This branches the note by cause and stopsgain_over_noisefrom dividing by1e-9.repeats < 2): the note says the floor cannot be estimated from N baseline repeat(s) instead of printing a one-point "floor".baseline mean +0.500 ± 0.200, not an absolute interval placed next to a delta.min_effect: the note namesmin_effectand the floor half-width.not distinguishable from noise.gain_over_noiseis0.0when the floor is undefined (fewer than 2 repeats or zero spread). The docstring says0.0there means undefined.distinguishableis unchanged.Tests:
tests/test_noise_calibration.py(parametrised undefined-floor cases + measurable-ratio case; the zero-spread> 1.0assertion becomes== 0.0),tests/test_trust_evidence_noise.py(one-repeat, below-min_effect, ± band, passing zero-spread).Local: 167 passed / 13 skipped on every test file touching noise/gain_over_noise;
tools.changelog_fragments checkandtools.gen_api_docs --checkOK.tests/test_plugin_host_cli.py::test_present_host_passesfails on the box on plain main as well (environment), so it's unrelated.