Repository navigation
Description
Problem
_noise_refusal_note (trust_evidence.py:335-354) explains a refusal with three causes and says so in its docstring (trust_evidence.py:338), but distinguishable (stats_eval.py:368-369) has four ways to be false: fewer than 2 baseline repeats, gain <= 0.0, and gain <= bar + 1e-12. gain <= 0.0 has no branch of its own, so it falls through to whichever other cause happens to be true and a regression is explained as noise:
- 3 noisy repeats, gain −0.300 →
gain -0.300 is not distinguishable from noise (gain does not exceed the noise floor: baseline mean +0.500 ± 0.200)— the gain is negative, so no floor could make it distinguishable; - 3 zero-spread repeats, gain −0.050 → the same sentence with
± 0.000, an absolute band printed against a loss; - 1 repeat, gain −0.300 → blames the repeat count, although
gain > 0.0fails at any count.
The docstring/code drift is the bug: the promise lists three causes, the predicate has four, and the one that is not printed is the one that describes a regression.
Evidence
Probe run-1115-0104/probe-negative-gain-note.py (PYTHONPATH=<clone> python3 probe-negative-gain-note.py), live main c94e9d67:
| Case | Setup | Measured detail |
|---|---|---|
| E: regression, noisy baseline | 3 repeats 0.3/0.5/0.7, gain −0.300 | (gain does not exceed the noise floor: baseline mean +0.500 ± 0.200) |
| F: no change, noisy baseline | 3 repeats 0.3/0.5/0.7, gain +8.327e-16 | (gain does not exceed the noise floor: baseline mean +0.500 ± 0.200) — float noise is a positive gain and correctly stays here |
| G: regression, one repeat | 1 repeat 0.5, gain −0.300 | (the noise floor cannot be estimated from 1 baseline repeat(s); at least 2 are needed) |
| H: regression, zero spread | 3 repeats 0.5, gain −0.050 | (gain does not exceed the noise floor: baseline mean +0.500 ± 0.000) |
| I: small positive gain | 3 repeats 0.3/0.5/0.7, gain +0.050 | (gain does not exceed the noise floor: baseline mean +0.500 ± 0.200) — correct today, stays |
calibrate_noise on E reports distinguishable=False, gain=-0.300, gain_over_noise=0.0 — only the sentence is wrong.
Scope
trust_evidence.py:343: branch first onnoise.gain <= 0.0and return(the candidate did not improve on the baseline: the gain must be positive). The head already prints the gain (gain -0.300 is not distinguishable from noise), so the branch must not print it a second time.trust_evidence.py:338: the docstring lists the four causes in code order — gain ≤ 0 → fewer than 2 repeats →min_effect→ floor.- Every branch keeps the phrase
not distinguishable from noiseverbatim; three test files assert it. - Do not touch
distinguishable, theverdictstrings orgain_over_noise(that was fix(eval): name the real cause of a noise refusal and stop faking gain_over_noise (#1089) #1090).
Tests to add
tests/test_trust_evidence_noise.py: a negative gain (E) printsdid not improve on the baselineand no±band; an exactgain == 0.0row (NoiseCalibration(gain=0.0, ...)) takes the same branch; again=+8.327e-16row stays in the floor branch (the guard is<= 0.0, not a tolerance, so float noise is not swallowed).- One changelog fragment.
Existing expectations
None change. The one-repeat test (tests/test_trust_evidence_noise.py:62) uses gain +0.3, so the repeats branch still fires there; the three noise test files give 41 passed before and after with this patch (re-run at c94e9d67).
Acceptance: E, G and H flip to the new sentence, I and the passing control unchanged; pytest -q green offline; every branch still contains the pinned phrase; mkdocs build --strict, python -m tools.gen_api_docs --check and python -m tools.changelog_fragments check pass; one changelog fragment.
Bot: AgentDescent Product
Activity