Skip to content

bug(eval): a regression is explained as a noise-floor problem — the note has no branch for gain <= 0 #1091

Open

Description

@HugoXK

Problem

_noise_refusal_note (trust_evidence.py:335-354) explains a refusal with three causes and says so in its docstring (trust_evidence.py:338), but distinguishable (stats_eval.py:368-369) has four ways to be false: fewer than 2 baseline repeats, gain <= 0.0, and gain <= bar + 1e-12. gain <= 0.0 has no branch of its own, so it falls through to whichever other cause happens to be true and a regression is explained as noise:

  • 3 noisy repeats, gain −0.300 → gain -0.300 is not distinguishable from noise (gain does not exceed the noise floor: baseline mean +0.500 ± 0.200) — the gain is negative, so no floor could make it distinguishable;
  • 3 zero-spread repeats, gain −0.050 → the same sentence with ± 0.000, an absolute band printed against a loss;
  • 1 repeat, gain −0.300 → blames the repeat count, although gain > 0.0 fails at any count.

The docstring/code drift is the bug: the promise lists three causes, the predicate has four, and the one that is not printed is the one that describes a regression.

Evidence

Probe run-1115-0104/probe-negative-gain-note.py (PYTHONPATH=<clone> python3 probe-negative-gain-note.py), live main c94e9d67:

Case Setup Measured detail
E: regression, noisy baseline 3 repeats 0.3/0.5/0.7, gain −0.300 (gain does not exceed the noise floor: baseline mean +0.500 ± 0.200)
F: no change, noisy baseline 3 repeats 0.3/0.5/0.7, gain +8.327e-16 (gain does not exceed the noise floor: baseline mean +0.500 ± 0.200) — float noise is a positive gain and correctly stays here
G: regression, one repeat 1 repeat 0.5, gain −0.300 (the noise floor cannot be estimated from 1 baseline repeat(s); at least 2 are needed)
H: regression, zero spread 3 repeats 0.5, gain −0.050 (gain does not exceed the noise floor: baseline mean +0.500 ± 0.000)
I: small positive gain 3 repeats 0.3/0.5/0.7, gain +0.050 (gain does not exceed the noise floor: baseline mean +0.500 ± 0.200) — correct today, stays

calibrate_noise on E reports distinguishable=False, gain=-0.300, gain_over_noise=0.0 — only the sentence is wrong.

Scope

  • trust_evidence.py:343: branch first on noise.gain <= 0.0 and return (the candidate did not improve on the baseline: the gain must be positive). The head already prints the gain (gain -0.300 is not distinguishable from noise), so the branch must not print it a second time.
  • trust_evidence.py:338: the docstring lists the four causes in code order — gain ≤ 0 → fewer than 2 repeats → min_effect → floor.
  • Every branch keeps the phrase not distinguishable from noise verbatim; three test files assert it.
  • Do not touch distinguishable, the verdict strings or gain_over_noise (that was fix(eval): name the real cause of a noise refusal and stop faking gain_over_noise (#1089) #1090).

Tests to add

  • tests/test_trust_evidence_noise.py: a negative gain (E) prints did not improve on the baseline and no ± band; an exact gain == 0.0 row (NoiseCalibration(gain=0.0, ...)) takes the same branch; a gain=+8.327e-16 row stays in the floor branch (the guard is <= 0.0, not a tolerance, so float noise is not swallowed).
  • One changelog fragment.

Existing expectations

None change. The one-repeat test (tests/test_trust_evidence_noise.py:62) uses gain +0.3, so the repeats branch still fires there; the three noise test files give 41 passed before and after with this patch (re-run at c94e9d67).

Acceptance: E, G and H flip to the new sentence, I and the passing control unchanged; pytest -q green offline; every branch still contains the pinned phrase; mkdocs build --strict, python -m tools.gen_api_docs --check and python -m tools.changelog_fragments check pass; one changelog fragment.

Bot: AgentDescent Product

Activity

  1. added
    bugSomething isn't working
    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategies
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions