Repository navigation
fix(agent): mark agent infrastructure and simulated-user errors as unscored - #702
omkargaikwad23 wants to merge 5 commits into
Conversation
…scored Agent CLI quota and server errors, and simulated-user errors, now set generated_error instead of being scored as failures. A simulated-user error no longer skips the remaining scenarios of an evalset. The analyzer leaves rows with generated_error out of non-binary averages, because scorers return 0 for them.
The noop simulated user of the agent gRPC proxy flow returns an empty reply by design, so treating it as an error stopped multi-turn proxy scenarios after turn 1. Only a simulated-user exception now sets generated_error.
|
/gcbrun |
| # cause these, so the trial is unscored instead of failed. Status codes must | ||
| # follow a word such as "HTTP" or "error", so a traceback line number such as | ||
| # "line 503" does not match. | ||
| _INFRA_ERROR_RE = re.compile( |
There was a problem hiding this comment.
The regex searches all of stderr, so real failures get marked as infra errors: gemini-cli's "Attempt 1 failed with status 429" retry log, errors from the tool being tested (UNAVAILABLE, 500), and the gRPC proxy's error_message. Match only the model API's error formats, ideally with one check per agent CLI.
There was a problem hiding this comment.
Fixed. I removed the stderr regex. Each CLI now checks only its own final model API error, so retry logs, tool errors, and gRPC proxy errors no longer match. Other generators, including the gRPC proxy, count every error as a failure.
| # "line 503" does not match. | ||
| _INFRA_ERROR_RE = re.compile( | ||
| r"\b(?:HTTP|[Ss]tatus|[Cc]ode|[Ee]rror)\b\D{0,10}\b(?:429|50[0234])\b" | ||
| r"|Too Many Requests|RESOURCE_EXHAUSTED|UNAVAILABLE" |
There was a problem hiding this comment.
Real infra errors are missed: 529, ERROR 429, status_code=503, HTTP/1.1 and HTTP/2 status lines, OpenAI's wording, and Claude/Codex errors that appear on stdout. Use re.IGNORECASE.
There was a problem hiding this comment.
Fixed. Claude Code now uses the API error category (rate_limit, server_error), which covers 529. Codex now reads the final turn.failed message, including OpenAI wording such as "at capacity" and "high demand". Both read stdout, and the match is case-insensitive.
| conversation_history, | ||
| result.stdout | ||
| ) | ||
| try: |
There was a problem hiding this comment.
On a simulated-user quota error, QueryGenerator.generate returns "", so this try never fires. Raise on an empty reply unless the generator is NOOPGenerator.
There was a problem hiding this comment.
Fixed. An empty or None reply is now saved as a simulated user error. The noop simulated users still continue the conversation.
…derr The stderr regex marked real failures as infrastructure errors: Gemini CLI retry logs, errors from the tool under test, and gRPC proxy error messages. It also missed errors that Claude Code and Codex CLI write to stdout. AgentCliGenerator.model_api_error() now returns the final model API error if it is a quota or server error. Gemini CLI reads the "[API Error: ...]" result event, Claude Code reads the rate_limit and server_error categories of top-level assistant messages, and Codex CLI reads the turn.failed message. The default returns None, so agy_cli and agent_grpc_proxy errors still count as failures. An empty or None simulated-user reply is now an error, because QueryGenerator.generate returns "" after a quota error and ClaudeGenerator returns None. The noop simulated users still continue the conversation.
What this does
Some agent runs fail for reasons outside the tool we test. This PR marks those runs as errors instead of failures.
Two kinds of errors are covered:
429or503. Each CLI checks its own error output:[API Error: ...]message.rate_limitorserver_errormessage from the main agent.turn.failedmessage.In both cases, the conversation stops and the error is saved in the
generated_errorcolumn. A timeout still counts as a failure.Why
Other changes
noopsimulated user always replies with empty text, so its empty reply is not an error.Testing
None, an agent 404 error (counts as a failure), and an agent 429 error from a fake Vertex server (counts as an error). Logs: https://paste.googleplex.com/6755399436658858 (Google-internal)Not covered
agy_cliandagent_grpc_proxyerrors still count as failures.summary.csvstill counts them. A later PR fixes this.