Skip to content

check runs every check twice per run, and --strict ignores --severity #5

Description

@royalpinto007

What is wrong

cmd_check runs every check over every run twice.

agentrace/cli.py line 71, inside the per-run loop:

findings = analyse(r)

then again at line 90, after the loop:

has_high = any(f.severity == "high" for r in runs for f in analyse(r))

analyse (in agentrace/checks.py) executes all seven checks, several of which run re.finditer over the full result text (check_unverified_claim, check_url_without_verification). On a real session, the README cites 152 runs with 257,721 result characters, so this doubles the regex work for a value that was already available.

There is also a subtle behaviour question worth settling in the same change: the --severity filter at line 73 narrows what is printed, but line 90 ignores it. So agentrace check --severity low --strict can still exit 1 because of a high finding the user explicitly filtered out.

Steps

  1. In cmd_check, compute the findings once. Keep a running flag or collect the per-run findings into a list before the --severity filter is applied.
  2. Replace line 90 with a check against that value, so analyse is called exactly once per run.
  3. Decide what --strict should mean when --severity is given, and make the code and the --strict help text at line 168 agree. Exiting on findings the user asked not to see is the surprising option.
  4. Optional but welcome: a test in tests/test_agentrace.py that calls agentrace.cli.main(["--file", str(fixture), "check", "--strict"]) against tests/fixtures/session.jsonl and asserts the exit code.

Contained to one function, no new dependencies. Comment below to claim it; a reply usually comes within a day.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions