Skip to content

fix(sieve): stop classing every undeclared exec exit as network (#562) - #564

Merged
mlieberman85 merged 2 commits into
darnitdevorg:mainfrom
mlieberman85:fix-562-exec-error-class
Oct 7, 2026
Merged

mlieberman85 merged 2 commits into
darnitdevorg:mainfrom
mlieberman85:fix-562-exec-error-class

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Summary

When an exec step's command exited with an undeclared code, _classify_exec_failure labelled it error_class=network unless stderr matched an auth or rate-limit pattern. Local failures read as network problems as a result. In one full test run, 232 of 242 error_class=network warnings came from this path. The most common causes were:

  • grep finding no .github/workflows/ (105);
  • commands that exited with nothing on stderr (50);
  • fatal: not a git repository (35).

With this change:

  • New error class unexpected_exit: a command exited with a code its step did not declare, and nothing in its output identified the cause.
  • Classification, first match wins:
    • rate limit -> rate_limit;
    • auth (adds "not logged in") -> auth;
    • exit 127, "command not found", or <executable>: not found -> missing_tool;
    • recognizable connection failures -> network. These are DNS, refused or reset connections, unreachable hosts, connection timeouts, and TLS or certificate errors. An HTTP status reply such as git's "The requested URL returned error: 404" is excluded, because the server answered;
    • anything else -> unexpected_exit.
  • The step message names the command and exit code, plus a stderr excerpt of at most 200 characters on one line, or "(no stderr)".
  • Unchanged: the classification for MCP, gh_api, timeouts, and crashes.

After the change, the same test run has 10 error_class=network warnings, all from MCP.

Spec changes were made first:

  • docs/architecture/framework-design.md: section 3.3 gains a decision table and a scenario; section 5.2 adds the class to the list.
  • specs/036-tier2-error-class/contracts/error-class-addendum-562.md: unexpected_exit replaces network as the catch-all for undeclared exec exits.

Closes #562

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Anything that filtered on error_class == "network" for undeclared exec exits sees unexpected_exit now; the CHANGELOG notes this under Changed.

Framework Changes Checklist

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Testing

  • Tests pass locally (uv run pytest tests/ -v): 5210 passed, 29 skipped
  • Added tests for new functionality (if applicable)
  • Linting passes (uv run ruff check .)

New tests:

  • a 36-row classifier table test;
  • tests for the step message: command, exit code, and stderr excerpt;
  • rendering tests for summary JSON, SARIF, the attestation predicate, markdown, and CLI text.

Also passing: the corpus gates, and the integration tests with GitHub Actions environment variables set (334 passed).

AI assistance

  • No AI assistance was used
  • AI assistance was used

Claude (Claude Code, claude-opus-5-5) made this change: spec, code, and tests. This description was also drafted with Claude. Commits carry an Assisted-by: Claude:claude-opus-5-5 trailer.

Additional Notes

An undeclared exit keeps the step INCONCLUSIVE. When nothing else concludes, the control ends WARN with the generic message. The new cause is in the WARN log line and the step's pass history.

🤖 Generated with Claude Code

…its (darnitdevorg#562)

An exec step whose command exits with a code the step does not declare
was classed network whenever stderr matched no rate-limit or auth
pattern, though most such exits (grep on a missing directory, git
outside a repository) never touch the network. framework-design 3.3 now
gives the decision table: rate_limit, auth, missing_tool (exit 127 or
command not found), network only for a connection error on stderr, and
otherwise the new class unexpected_exit; the step's message names the
exit code, command, and start of stderr. Section 5.2 lists the class.
The feature 036 contract gets an addendum recording the change.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
…itdevorg#562)

_classify_exec_failure now takes the exit code and executable. After the
rate-limit and auth patterns (auth also matches "not logged in"), exit
127, "command not found", or a stderr line ending "<executable>: not
found" is missing_tool; a connection error on stderr (name resolution,
refused/reset/timed-out connection, unreachable host, TLS or certificate
failure, git "unable to access" without an HTTP status) is network;
anything else is the new ErrorClass unexpected_exit. The step's message
carries the exit code, the command, and the first 200 characters of
stderr on one ASCII line. The step stays INCONCLUSIVE as before.

Reports, SARIF, JSON, CLI text, and the attestation predicate render the
class name they are given; tests pin that for unexpected_exit. The test
that expected network for git exiting 128 now expects unexpected_exit.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
@mlieberman85
mlieberman85 merged commit af80b64 into darnitdevorg:main Oct 7, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

exec steps label any unexpected exit code error_class=network

1 participant