What is wrong
cmd_check runs every check over every run twice.
agentrace/cli.py line 71, inside the per-run loop:
then again at line 90, after the loop:
has_high = any(f.severity == "high" for r in runs for f in analyse(r))
analyse (in agentrace/checks.py) executes all seven checks, several of which run re.finditer over the full result text (check_unverified_claim, check_url_without_verification). On a real session, the README cites 152 runs with 257,721 result characters, so this doubles the regex work for a value that was already available.
There is also a subtle behaviour question worth settling in the same change: the --severity filter at line 73 narrows what is printed, but line 90 ignores it. So agentrace check --severity low --strict can still exit 1 because of a high finding the user explicitly filtered out.
Steps
- In
cmd_check, compute the findings once. Keep a running flag or collect the per-run findings into a list before the --severity filter is applied.
- Replace line 90 with a check against that value, so
analyse is called exactly once per run.
- Decide what
--strict should mean when --severity is given, and make the code and the --strict help text at line 168 agree. Exiting on findings the user asked not to see is the surprising option.
- Optional but welcome: a test in
tests/test_agentrace.py that calls agentrace.cli.main(["--file", str(fixture), "check", "--strict"]) against tests/fixtures/session.jsonl and asserts the exit code.
Contained to one function, no new dependencies. Comment below to claim it; a reply usually comes within a day.
What is wrong
cmd_checkruns every check over every run twice.agentrace/cli.pyline 71, inside the per-run loop:then again at line 90, after the loop:
analyse(inagentrace/checks.py) executes all seven checks, several of which runre.finditerover the full result text (check_unverified_claim,check_url_without_verification). On a real session, the README cites 152 runs with 257,721 result characters, so this doubles the regex work for a value that was already available.There is also a subtle behaviour question worth settling in the same change: the
--severityfilter at line 73 narrows what is printed, but line 90 ignores it. Soagentrace check --severity low --strictcan still exit 1 because of a high finding the user explicitly filtered out.Steps
cmd_check, compute the findings once. Keep a running flag or collect the per-run findings into a list before the--severityfilter is applied.analyseis called exactly once per run.--strictshould mean when--severityis given, and make the code and the--stricthelp text at line 168 agree. Exiting on findings the user asked not to see is the surprising option.tests/test_agentrace.pythat callsagentrace.cli.main(["--file", str(fixture), "check", "--strict"])againsttests/fixtures/session.jsonland asserts the exit code.Contained to one function, no new dependencies. Comment below to claim it; a reply usually comes within a day.