diff() never flags a leak in a newly added gold case

[royalpinto007]

Desired outcome

diff() reports a leak in a gold case that did not exist in the "before" report.

Why it matters

app/evaluate.py:

new_leaks = [k for k in sorted(set(a)) if a[k].leaked and not b.get(k, a[k]).leaked]

The default for the missing-key lookup is a[k] itself. So for a case present only in the "after" report, the expression becomes a[k].leaked and not a[k].leaked, which is always False. A brand new gold case that leaks a forbidden document is never listed in new_leaks, and since verdict is decided by if new_leaks: first, the run reports NO CHANGE or IMPROVED instead of LEAK.

This matters more here than it would elsewhere. The module docstring says leak_rate "is the only one that can end you" and passed is written so "a leak fails the case outright". Adding a gold case is exactly what a contributor does when they suspect a leak, and that is precisely the path where the diff stays silent about it. pass_rate and leak_rate in the summary do move, but the headline verdict, the thing CI and a reviewer look at, does not.

regressed and fixed are intentionally scoped to set(b) & set(a), which is correct for those. Only new_leaks iterates all of a and then defeats itself with the default.

Steps

1. Change the default so an absent baseline counts as not leaked, for example:

   new_leaks = [k for k in sorted(a) if a[k].leaked and k in b and not b[k].leaked]

   if new cases should be excluded, or

   new_leaks = [k for k in sorted(a) if a[k].leaked and not (k in b and b[k].leaked)]

   if a leaking new case should be reported, which reads as the intended behaviour.

2. Add a test in tests/test_evaluate.py alongside the existing diff tests: a case present only in after and leaking must produce verdict == "LEAK".

Claiming this

Comment below to claim it. A reply usually comes within a day.