From 14c1395177e5521a39664a0985b2398e7dfcfc5f Mon Sep 17 00:00:00 2001 From: Michael Lieberman Date: Tue, 6 Oct 2026 21:57:21 -0400 Subject: [PATCH 1/2] docs(spec): define error class unexpected_exit for undeclared exec exits (#562) An exec step whose command exits with a code the step does not declare was classed network whenever stderr matched no rate-limit or auth pattern, though most such exits (grep on a missing directory, git outside a repository) never touch the network. framework-design 3.3 now gives the decision table: rate_limit, auth, missing_tool (exit 127 or command not found), network only for a connection error on stderr, and otherwise the new class unexpected_exit; the step's message names the exit code, command, and start of stderr. Section 5.2 lists the class. The feature 036 contract gets an addendum recording the change. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Michael Lieberman --- docs/architecture/framework-design.md | 24 +++++++++-- .../contracts/error-class-addendum-562.md | 40 +++++++++++++++++++ 2 files changed, 61 insertions(+), 3 deletions(-) create mode 100644 specs/036-tier2-error-class/contracts/error-class-addendum-562.md diff --git a/docs/architecture/framework-design.md b/docs/architecture/framework-design.md index abe743d0..f97d0d1a 100644 --- a/docs/architecture/framework-design.md +++ b/docs/architecture/framework-design.md @@ -1,8 +1,8 @@ # Darnit Framework Design Specification -> **Version**: 1.0.0-alpha.12 +> **Version**: 1.0.0-alpha.13 > **Status**: Authoritative -> **Last Updated**: 2026-10-04 +> **Last Updated**: 2026-10-06 This specification defines the authoritative design of the Darnit framework, including the sieve orchestrator, TOML schema, built-in pass types, remediation actions, and plugin protocol. @@ -440,6 +440,18 @@ These are the step type's declared `settings` (section 3.0.3); any other key fai **Broken measurements**: a command whose binary is not installed is ERROR, class `missing_tool`; a timeout is ERROR, class `timeout`. Neither is FAIL. +**Undeclared exit codes**: an exit code in neither `pass_exit_codes` nor `fail_exit_codes` concludes nothing; the step is evidence only and carries a class taken from the command's exit code and stderr, first match wins (#562): + +| Evidence | Class | +|----------|-------| +| A rate-limit message (`api rate limit exceeded`, `secondary rate limit`, `abuse detection mechanism`) | `rate_limit` | +| An authentication message (`http 401`, `bad credentials`, `requires authentication`, `gh auth login`, `authentication token`, `not logged in`) | `auth` | +| Exit code 127, or stderr says the command was not found | `missing_tool` | +| A connection error: name resolution, connection refused, reset, or timed out, network or host unreachable, a TLS or certificate failure, or git's `unable to access` | `network` | +| Anything else | `unexpected_exit` | + +The step's message names the command, the exit code, and the start of stderr, so an operator can act on an `unexpected_exit` without rerunning the command. + #### Scenario: CEL expression evaluated - **WHEN** an `exec` step has an `expr` field and the exit code gives PASS or FAIL - **THEN** the expression MUST be evaluated after the command, and the step result MUST follow the outcome rules of section 3.7 @@ -450,6 +462,11 @@ These are the step type's declared `settings` (section 3.0.3); any other key fai - **THEN** the handler MUST evaluate the exit code against `pass_exit_codes` and `fail_exit_codes` - **AND** an exit code in neither list MUST NOT give PASS or FAIL +#### Scenario: Undeclared exit code with no identified cause +- **WHEN** an `exec` command exits with a code in neither list and its stderr shows no rate limit, authentication failure, missing command, or connection error (for example `grep` exiting 2 on a missing directory, or `git` exiting 128 outside a repository) +- **THEN** the step MUST carry class `unexpected_exit`, not `network` +- **AND** its message MUST name the command, the exit code, and an excerpt of stderr + ### 3.4 regex Handler **Purpose**: Regex-based content analysis @@ -1236,7 +1253,7 @@ Result fields beyond `status`: |-------|-------------| | `authority` | `dispositive` (a step concluded within its effective set), `suggestive` (no step concluded, or a model finding), `asserted` (a person confirmed it) | | `concluded_by` | Step handler name, `llm_judgment`, `confirmation`, `inferred_from`, or `none` | -| `error` | `{class, cause}`; present on every ERROR. Classes include `auth`, `rate_limit`, `unavailable`, `missing_tool`, `evaluation`, and the feature 036 classes | +| `error` | `{class, cause}`; present on every ERROR. Classes include `auth`, `rate_limit`, `unavailable`, `missing_tool`, `evaluation`, `unexpected_exit` (#562), and the feature 036 classes | | `pending` | `{kind}`; present on every PENDING | | `candidate` | Present with `pending.kind = "confirmation"`: `verdict = "pass"`, `reasoning`, `cited_evidence`, `model`, `model_version`, `evidence_digest`, `source` (`harness` or `mcp_agent`) | | `confirmation` | Present on a PASS from a confirmed candidate: `confirmed_by`, `confirmed_at`, `expires_at` | @@ -2301,6 +2318,7 @@ The following requirements have been superseded: by the handler dispatch archite | Version | Date | Changes | |---------|------|---------| +| 1.0.0-alpha.13 | 2026-10-06 | Error class `unexpected_exit`: an `exec` step whose undeclared exit code has no identified cause no longer reports `network`; exit code 127 reports `missing_tool` (Sections 3.3, 5.2; #562) | | 1.0.0-alpha.12 | 2026-10-04 | Repository-level `.baseline.toml` is no longer read: one notice points at `darnit config migrate`, framework selection only by `--framework` or a tool argument (Sections 2.3, 10.5, 14.4, 15.1; Appendix C) | | 1.0.0-alpha.11 | 2026-10-04 | Close remaining false-PASS paths (feature 044): an expression that cannot be evaluated or is not boolean makes the step ERROR, expression names per step type with usable `project` values and a repository-aware `file_exists`, load-time expression reference check (Section 3.7); step registration declares `settings` and `expression_names`, plugins cannot replace a registered step type, and unknown control keys, unknown step keys, and unregistered step types fail loading (Sections 2.3, 3.0.3); reproducibility step types conclude only FAIL (Sections 3.0.1, 12); `expr_decides`, an expression that decides on a handler PASS (Sections 3.0.3, 3.7); `gh_api` `evidence_fields`, required for personal records (Section 3.8); `file_must_exist` replaced by the registered `file_exists` (Section 3.2); field tables corrected to what each handler reads (Sections 3.3-3.5) | | 1.0.0-alpha.10 | 2026-10-02 | Remediation safety (feature 043): plan/apply protocol and single writer (Section 4.2), `platform_setting` (4.5), exec `effects`/`offline` and no platform state from exec (4.4), `file_create.project_reference` (4.3), remediation policy, digest-bound approval, outcomes, re-check, run manifest, and version-control rules (Section 15); removed `api_call`, `requires_confirmation`, `dry_run_supported`, `dry_run_command` (Appendix C) | diff --git a/specs/036-tier2-error-class/contracts/error-class-addendum-562.md b/specs/036-tier2-error-class/contracts/error-class-addendum-562.md new file mode 100644 index 00000000..5a222b66 --- /dev/null +++ b/specs/036-tier2-error-class/contracts/error-class-addendum-562.md @@ -0,0 +1,40 @@ +# Addendum: `unexpected_exit` replaces `network` as the exec catch-all + +**Amends**: [error-class.md](error-class.md) sections 1 and 2.1 +**Issue**: #562 +**Authoritative text**: `docs/architecture/framework-design.md` section 3.3 ("Undeclared exit codes") and section 5.2 + +## Why + +Section 2.1 made `network` the class for any undeclared exit code whose stderr matched no pattern. Most such exits never touch the network: `grep` exiting 2 on a missing directory, `git` exiting 128 outside a repository, a binary the shell could not find. Reports and logs then told operators to check their network for a local condition. + +## 1. The enum + +One value is added. `ErrorClass` and `ERROR_CLASSES` are edited together, as section 1 requires. + +| Value | Meaning | +|---|---| +| `unexpected_exit` | A command exited with a code its step did not declare, and nothing in its output identified the cause. | + +`network` keeps its other producers (MCP handshake failure and unusable server, section 2.2) and, for `exec`, now means only that stderr shows a connection error. + +## 2.1 `exec_handler` (replaces the non-zero-exit rows) + +Checked in this order on an exit code in neither `pass_exit_codes` nor `fail_exit_codes`; first match wins. Matching is a case-insensitive substring match on stderr. + +| Condition | `error_class` | +|---|---| +| stderr matches `_GH_RATE_LIMIT_PATTERNS` | `rate_limit` | +| stderr matches `_GH_AUTH_PATTERNS` (now also `not logged in`) | `auth` | +| exit code 127, stderr matches `_MISSING_TOOL_PATTERNS` (`command not found`), or a stderr line ends `: not found` | `missing_tool` | +| stderr matches `_NETWORK_PATTERNS` (name resolution, connection refused, reset, or timed out, failed to connect, network or host unreachable, TLS/SSL or certificate errors, git `unable to access 'http...`) | `network` | +| anything else, including empty stderr | `unexpected_exit` | + +The timeout, missing-binary (`FileNotFoundError`), pass, and declared-fail rows are unchanged. The step stays INCONCLUSIVE as before; only its class changes. + +The step's message names the exit code, the command, and the first 200 characters of stderr on one line, with non-ASCII characters replaced, so the cause is actionable from the report. The message is what the section 7 WARN log line and the step's pass-history entry carry. + +## Unchanged + +- `gh_api` status classification, the MCP mapping (section 2.2), the orchestrator's `crashed` (section 2.3), and context auto-detect (section 2.4). +- Propagation (section 3), CEL preservation (section 4), output surfaces (section 5), and validation (section 6). Every surface renders the class name it is given, so no surface needs a new mapping. From f77fc14bb6b18ff1762b2fee07cda70b2b110f86 Mon Sep 17 00:00:00 2001 From: Michael Lieberman Date: Tue, 6 Oct 2026 21:57:21 -0400 Subject: [PATCH 2/2] fix(sieve): stop classing every undeclared exec exit as network (#562) _classify_exec_failure now takes the exit code and executable. After the rate-limit and auth patterns (auth also matches "not logged in"), exit 127, "command not found", or a stderr line ending ": not found" is missing_tool; a connection error on stderr (name resolution, refused/reset/timed-out connection, unreachable host, TLS or certificate failure, git "unable to access" without an HTTP status) is network; anything else is the new ErrorClass unexpected_exit. The step's message carries the exit code, the command, and the first 200 characters of stderr on one ASCII line. The step stays INCONCLUSIVE as before. Reports, SARIF, JSON, CLI text, and the attestation predicate render the class name they are given; tests pin that for unexpected_exit. The test that expected network for git exiting 128 now expects unexpected_exit. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Michael Lieberman --- CHANGELOG.md | 8 ++ docs/SECURITY_GUIDE.md | 2 + .../darnit/src/darnit/core/error_class.py | 58 ++++---- .../src/darnit/sieve/builtin_handlers.py | 82 +++++++++-- .../sieve/test_error_class_classification.py | 129 +++++++++++++++++- .../test_error_class_output_surfaces.py | 72 ++++++++++ 6 files changed, 310 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 64d79092..4d77c248 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -448,6 +448,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `operator_config`, `trust`, `ignored_repository_settings`, `unknown_assertions`, `summary`, `results`, and `warnings` when present) instead of a bare list of results. +- An `exec` step whose command exits with a code the step does not declare + reports error class `unexpected_exit` instead of `network` unless stderr + shows a connection error (name resolution, refused or reset connection, + unreachable host, TLS or certificate failure). Exit code 127 or "command + not found" reports `missing_tool`. The step's message gives the exit code, + the command, and the start of stderr. Before, `grep` exiting 2 on a missing + directory or `git` exiting 128 outside a repository was reported as a + network failure (#562). ### Fixed diff --git a/docs/SECURITY_GUIDE.md b/docs/SECURITY_GUIDE.md index 58b79fed..77f12b10 100644 --- a/docs/SECURITY_GUIDE.md +++ b/docs/SECURITY_GUIDE.md @@ -568,6 +568,8 @@ When reviewing a framework TOML or plugin change, treat any new `existence = tru A step that could not measure returns ERROR with a class and cause: `auth` (401/403), `rate_limit` (429 or a rate-limit 403), `unavailable` (5xx, transport failure, undeclared status), `missing_tool` (an absent binary or required MCP server), or `evaluation`. ERROR never concludes FAIL: later steps still run, and if none concludes the control ends ERROR. A platform response proves failure only when the `gh_api` step lists its status in `fail_on_status` (for example 404 for "no branch protection"), and a rate limit never does. A token that cannot read a setting therefore shows up as ERROR `[auth]`, not as a failing project. ERROR is non-compliant. +An `exec` command that exits with a code its step does not declare concludes nothing either. Its class comes from the exit code and stderr: `rate_limit`, `auth`, `missing_tool` (exit 127 or "command not found"), `network` only when stderr shows a connection error, and otherwise `unexpected_exit`, whose message gives the exit code, the command, and the start of stderr. + ### PASS Candidates Confirmed by the Operator Controls that need judgment of document content end `PENDING` (`pending.kind = "llm_judgment"`) after the deterministic steps. A judgment comes from the harness's model step or from a coding agent through the `submit_judgment` MCP tool; both follow the same rules: diff --git a/packages/darnit/src/darnit/core/error_class.py b/packages/darnit/src/darnit/core/error_class.py index 2a7668ad..dd11c04f 100644 --- a/packages/darnit/src/darnit/core/error_class.py +++ b/packages/darnit/src/darnit/core/error_class.py @@ -1,6 +1,7 @@ """Environmental failure classification. -Feature 036. See specs/036-tier2-error-class/contracts/error-class.md. +Feature 036. See specs/036-tier2-error-class/contracts/error-class.md and +its #562 addendum, contracts/error-class-addendum-562.md. When a sieve pass cannot run to completion -- the network is unreachable, the auth token expired, a subprocess timed out, a required binary is @@ -16,31 +17,34 @@ check ran and the repository does not comply" -- that stays a bare FAIL/WARN with no ``error_class``. -============= =============================================================== -Value Meaning -============= =============================================================== -network Host unreachable, DNS failure, TLS error, MCP server unusable, - or any non-zero subprocess exit whose stderr matched no - more-specific pattern. -auth HTTP 401, bad credentials, expired token, "requires - authentication", or MCP plugin signature-verification failure. -timeout Subprocess exceeded its ``timeout`` budget, or an MCP tool - call exceeded ``MCP_DEFAULT_TIMEOUT_SECONDS``. -rate_limit GitHub primary or secondary rate limit, or abuse-detection - throttle. -not_found A control names an MCP server the operator never configured. - (Before feature 041 this also covered an absent binary; step - results now report that as ``missing_tool``.) -crashed A handler raised an unexpected exception, or an MCP tool - returned unparseable output. The handler did not complete - cleanly. -missing_tool Feature 041. A tool the step needs (``gh``, a command's - binary, a required MCP server executable) is not installed. -unavailable Feature 041. The platform API answered with a 5xx or a status - the step did not declare, or the request never got a response. -evaluation Feature 041. The step ran but its result could not be - evaluated (for example a CEL ``expr`` error over a response). -============= =============================================================== +=============== ============================================================= +Value Meaning +=============== ============================================================= +network Host unreachable, DNS failure, connection refused or reset, + TLS error, or MCP server unusable. An ``exec`` step reports it + only when stderr shows a connection error (#562). +auth HTTP 401, bad credentials, expired token, "requires + authentication", or MCP plugin signature-verification failure. +timeout Subprocess exceeded its ``timeout`` budget, or an MCP tool + call exceeded ``MCP_DEFAULT_TIMEOUT_SECONDS``. +rate_limit GitHub primary or secondary rate limit, or abuse-detection + throttle. +not_found A control names an MCP server the operator never configured. + (Before feature 041 this also covered an absent binary; step + results now report that as ``missing_tool``.) +crashed A handler raised an unexpected exception, or an MCP tool + returned unparseable output. The handler did not complete + cleanly. +missing_tool Feature 041. A tool the step needs (``gh``, a command's + binary, a required MCP server executable) is not installed, + or a command exited 127 or said "command not found". +unavailable Feature 041. The platform API answered with a 5xx or a status + the step did not declare, or the request never got a response. +evaluation Feature 041. The step ran but its result could not be + evaluated (for example a CEL ``expr`` error over a response). +unexpected_exit #562. A command exited with a code its step did not declare, + and nothing in its output identified the cause. +=============== ============================================================= Two names are exported because ``typing.Literal`` is erased at runtime and enforces nothing on its own. ``ErrorClass`` gives static-analysis @@ -74,6 +78,7 @@ "missing_tool", "unavailable", "evaluation", + "unexpected_exit", ] # Runtime-checkable companion to ``ErrorClass``. See module docstring for @@ -89,6 +94,7 @@ "missing_tool", "unavailable", "evaluation", + "unexpected_exit", ) ) diff --git a/packages/darnit/src/darnit/sieve/builtin_handlers.py b/packages/darnit/src/darnit/sieve/builtin_handlers.py index ae72b361..8f66eb16 100644 --- a/packages/darnit/src/darnit/sieve/builtin_handlers.py +++ b/packages/darnit/src/darnit/sieve/builtin_handlers.py @@ -54,11 +54,12 @@ # Feature 036: environmental-failure classification # ============================================================================= -# GitHub-only stderr patterns for v0 (clarify Q4). The `exec` handler sees -# only stdout/stderr/exit-code -- it has no access to response headers -- so -# classification is substring matching against `gh` CLI stderr shape. Other -# exec targets (git, curl, syft, cosign) fall through to `network`; per-target -# pattern packs are a follow-up if real audits show they are needed. +# The `exec` handler sees only stdout/stderr/exit-code -- it has no access to +# response headers -- so classification is substring matching against stderr. +# The rate-limit and auth patterns follow `gh` CLI stderr shape (clarify Q4). +# A failure no pattern identifies is `unexpected_exit`, not `network` (#562): +# most undeclared exits (grep on a missing directory, git outside a +# repository) never touch the network. # # Rate-limit is checked BEFORE auth: GitHub answers 403 for both rate limits # and permission failures, and the rate-limit body is the more specific signal. @@ -74,22 +75,75 @@ "requires authentication", "gh auth login", "authentication token", + "not logged in", ) -def _classify_exec_failure(stderr: str) -> ErrorClass: - """Classify a non-zero-exit subprocess failure from its stderr. +_MISSING_TOOL_EXIT_CODE = 127 + +_MISSING_TOOL_PATTERNS: tuple[str, ...] = ("command not found",) + +_NETWORK_PATTERNS: tuple[str, ...] = ( + "could not resolve host", + "name or service not known", + "temporary failure in name resolution", + "nodename nor servname provided", + "no such host", + "connection refused", + "connection reset", + "connection was reset", + "network is unreachable", + "no route to host", + "connection timed out", + "failed to connect to", + "i/o timeout", + "error connecting to", + "tls handshake", + "ssl_connect", + "gnutls_handshake", + "ssl certificate problem", + "certificate verify failed", + "x509:", + "server certificate verification failed", + "unable to access 'http", +) + +# git's "unable to access" also wraps an HTTP status the server sent back; +# an answer from the server is not a connection failure. +_HTTP_RESPONSE_PATTERNS: tuple[str, ...] = ("the requested url returned error",) + +_STDERR_EXCERPT_CHARS = 200 + + +def _classify_exec_failure(stderr: str, exit_code: int | None = None, executable: str | None = None) -> ErrorClass: + """Classify an undeclared-exit subprocess failure from its exit code and stderr. Only called on paths where the command did not complete as expected. Callers must NOT invoke this for a declared ``fail_exit_codes`` hit -- that is a check that ran and concluded, not an environmental failure. + ``executable`` lets a shell's ": not found" count as a missing tool + without reading every "not found" (an HTTP 404, say) as one. """ haystack = (stderr or "").lower() if any(p in haystack for p in _GH_RATE_LIMIT_PATTERNS): return "rate_limit" if any(p in haystack for p in _GH_AUTH_PATTERNS): return "auth" - return "network" + if exit_code == _MISSING_TOOL_EXIT_CODE or any(p in haystack for p in _MISSING_TOOL_PATTERNS): + return "missing_tool" + if executable: + name = re.escape(os.path.basename(executable).lower()) + if re.search(rf"(^|[\s:]){name}: not found\s*$", haystack, re.MULTILINE): + return "missing_tool" + if any(p in haystack for p in _NETWORK_PATTERNS) and not any(p in haystack for p in _HTTP_RESPONSE_PATTERNS): + return "network" + return "unexpected_exit" + + +def _excerpt(text: str) -> str: + """The start of ``text`` on one ASCII line, for an operator-facing cause.""" + line = " ".join((text or "").split())[:_STDERR_EXCERPT_CHARS] + return line.encode("ascii", "replace").decode("ascii") def _log_environmental_failure( @@ -369,10 +423,14 @@ def exec_handler(config: dict[str, Any], context: HandlerContext) -> HandlerResu ) else: # Undeclared exit code: we cannot tell whether the check concluded. - # Classify from stderr so the operator can distinguish a rate limit - # or expired token from a genuine non-compliance signal. - error_class = _classify_exec_failure(evidence["stderr"]) - message = f"Command exited with unexpected code {proc.returncode}" + # Classify from exit code and stderr so the operator can distinguish + # a rate limit or expired token from a genuine non-compliance signal. + error_class = _classify_exec_failure(evidence["stderr"], proc.returncode, resolved_cmd[0]) + excerpt = _excerpt(evidence["stderr"]) or "(no stderr)" + message = ( + f"Command exited with unexpected code {proc.returncode}: " + f"{_excerpt(' '.join(resolved_cmd))}; stderr: {excerpt}" + ) _log_environmental_failure(context.control_id, "exec", error_class, message) return HandlerResult( status=HandlerResultStatus.INCONCLUSIVE, diff --git a/tests/darnit/sieve/test_error_class_classification.py b/tests/darnit/sieve/test_error_class_classification.py index 25a34eee..2f20d106 100644 --- a/tests/darnit/sieve/test_error_class_classification.py +++ b/tests/darnit/sieve/test_error_class_classification.py @@ -84,6 +84,17 @@ def test_each_valid_value_constructs(self, value: str) -> None: ) assert result.error_class == value + @pytest.mark.unit + def test_unexpected_exit_is_a_known_class(self) -> None: + """#562: the Literal and the frozenset are edited together.""" + from typing import get_args + + from darnit.core.error_class import ErrorClass + + assert "unexpected_exit" in ERROR_CLASSES + assert "unexpected_exit" in get_args(ErrorClass) + assert set(get_args(ErrorClass)) == set(ERROR_CLASSES) + @pytest.mark.unit def test_none_is_always_allowed_including_on_pass(self) -> None: result = HandlerResult(status=HandlerResultStatus.PASS, message="ok") @@ -178,8 +189,8 @@ def test_rate_limit_wins_over_auth_when_both_could_match( assert result.error_class == "rate_limit" @pytest.mark.unit - def test_unmatched_stderr_falls_back_to_network(self, tmp_path: Path) -> None: - """FR-005a: non-GitHub / unrecognized failure is the network bucket.""" + def test_unmatched_stderr_is_unexpected_exit_not_network(self, tmp_path: Path) -> None: + """#562: an unrecognized failure no longer claims a network cause.""" with patch( "darnit.sieve.builtin_handlers.subprocess.run", return_value=_proc(128, stderr="fatal: not a git repository"), @@ -188,7 +199,51 @@ def test_unmatched_stderr_falls_back_to_network(self, tmp_path: Path) -> None: {"handler": "exec", "command": ["git", "rev-parse", "HEAD"]}, _ctx(tmp_path), ) - assert result.error_class == "network" + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert result.error_class == "unexpected_exit" + + @pytest.mark.unit + def test_unexpected_exit_message_names_command_code_and_stderr(self, tmp_path: Path) -> None: + """#562: the cause is actionable without rerunning the command.""" + stderr = "grep: .github/workflows/: No such file or directory\n" + "x" * 500 + "\u2603" + with patch( + "darnit.sieve.builtin_handlers.subprocess.run", + return_value=_proc(2, stderr=stderr), + ): + result = exec_handler( + {"handler": "exec", "command": ["grep", "-r", "uses:", ".github/workflows/"]}, + _ctx(tmp_path), + ) + assert result.error_class == "unexpected_exit" + assert "grep -r uses: .github/workflows/" in result.message + assert "code 2" in result.message + assert "grep: .github/workflows/: No such file or directory x" in result.message + assert "\n" not in result.message + assert result.message.isascii() + assert "x" * 300 not in result.message + + @pytest.mark.unit + def test_unexpected_exit_with_empty_stderr_says_so(self, tmp_path: Path) -> None: + with patch( + "darnit.sieve.builtin_handlers.subprocess.run", + return_value=_proc(3), + ): + result = exec_handler({"handler": "exec", "command": ["some-checker"]}, _ctx(tmp_path)) + assert result.error_class == "unexpected_exit" + assert "code 3" in result.message + assert "some-checker" in result.message + assert "no stderr" in result.message + + @pytest.mark.unit + def test_exit_127_is_missing_tool(self, tmp_path: Path) -> None: + """#562: a shell or env wrapper reports a missing binary as exit 127.""" + with patch( + "darnit.sieve.builtin_handlers.subprocess.run", + return_value=_proc(127, stderr="env: 'scorecard': No such file or directory"), + ): + result = exec_handler({"handler": "exec", "command": ["env", "scorecard"]}, _ctx(tmp_path)) + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert result.error_class == "missing_tool" @pytest.mark.unit def test_success_carries_no_error_class(self, tmp_path: Path) -> None: @@ -229,6 +284,74 @@ def test_clean_definitive_failure_carries_no_error_class( assert result.error_class is None +class TestClassifyExecFailure: + """#562: the undeclared-exit decision table, first match wins.""" + + @pytest.mark.unit + @pytest.mark.parametrize( + ("stderr", "exit_code", "expected"), + [ + ("gh: API rate limit exceeded for user ID 1234.", 1, "rate_limit"), + ("You have exceeded a secondary rate limit.", 1, "rate_limit"), + ("gh: HTTP 403: You have exceeded a secondary rate limit.", 1, "rate_limit"), + ("gh: Bad credentials (HTTP 401)", 1, "auth"), + ("To get started with GitHub CLI, please run: gh auth login", 4, "auth"), + ("You are not logged into any GitHub hosts.", 4, "auth"), + ("", 127, "missing_tool"), + ("bash: scorecard: command not found", 1, "missing_tool"), + ("sh: 1: scorecard: not found", 2, "missing_tool"), + ("fatal: unable to access 'https://github.com/o/r/': Could not resolve host: github.com", 128, "network"), + ("curl: (6) Could not resolve host: example.com", 6, "network"), + ("ssh: Could not resolve hostname github.com: Name or service not known", 255, "network"), + ("ssh: github.com: Temporary failure in name resolution", 255, "network"), + ( + "ssh: Could not resolve hostname github.com: nodename nor servname provided, or not known", + 255, + "network", + ), + ("dial tcp: lookup api.github.com: no such host", 1, "network"), + ("curl: (7) Failed to connect to localhost port 1: Connection refused", 7, "network"), + ("fetch-pack: unexpected disconnect: Connection reset by peer", 128, "network"), + ("connect: Network is unreachable", 1, "network"), + ("ssh: connect to host github.com port 22: No route to host", 255, "network"), + ("ssh: connect to host github.com port 22: Connection timed out", 255, "network"), + ("curl: (28) Failed to connect to example.com port 443: Operation timed out", 28, "network"), + ("dial tcp 140.82.112.6:443: i/o timeout", 1, "network"), + ("error connecting to api.github.com", 1, "network"), + ("curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.com:443", 35, "network"), + ("net/http: TLS handshake timeout", 1, "network"), + ("SSL certificate problem: unable to get local issuer certificate", 60, "network"), + ("x509: certificate signed by unknown authority", 1, "network"), + ("server certificate verification failed. CAfile: none CRLfile: none", 128, "network"), + ("error: RPC failed; curl 56 OpenSSL SSL_read: Connection was reset, errno 10054", 128, "network"), + ("fatal: unable to access 'https://example.com/r/': gnutls_handshake() failed", 128, "network"), + ( + "fatal: unable to access 'https://example.com/r/': The requested URL returned error: 404", + 128, + "unexpected_exit", + ), + ("grep: .github/workflows/: No such file or directory", 2, "unexpected_exit"), + ("fatal: not a git repository (or any of the parent directories): .git", 128, "unexpected_exit"), + ("error: No such remote 'upstream'", 2, "unexpected_exit"), + ("gh: Not Found (HTTP 404)", 1, "unexpected_exit"), + ("", 1, "unexpected_exit"), + ("", None, "unexpected_exit"), + ], + ) + def test_table(self, stderr: str, exit_code: int | None, expected: str) -> None: + from darnit.sieve.builtin_handlers import _classify_exec_failure + + assert _classify_exec_failure(stderr, exit_code, "scorecard") == expected + + @pytest.mark.unit + def test_not_found_names_only_the_executable(self) -> None: + """A bare "not found" about some other thing is not a missing tool.""" + from darnit.sieve.builtin_handlers import _classify_exec_failure + + assert _classify_exec_failure("gh: Not Found (HTTP 404)", 1, "gh") == "unexpected_exit" + assert _classify_exec_failure("sh: 1: gh: not found", 1, "/usr/bin/gh") == "missing_tool" + + class TestExecHandlerWarnLogging: """Contract section 7: environmental failures log at WARN, not DEBUG.""" diff --git a/tests/darnit_baseline/test_error_class_output_surfaces.py b/tests/darnit_baseline/test_error_class_output_surfaces.py index 7eb416b3..5d7fb794 100644 --- a/tests/darnit_baseline/test_error_class_output_surfaces.py +++ b/tests/darnit_baseline/test_error_class_output_surfaces.py @@ -169,3 +169,75 @@ def test_predicate_still_carries_authority_alongside(self) -> None: """Regression guard: the feature-025 field is untouched.""" controls = self._controls_by_id(self._predicate([ENV_FAILURE])) assert controls["OSPS-LE-02.02"]["authority"] == "dispositive" + + +class TestUnexpectedExitSurfaces: + """#562: every surface carries `unexpected_exit` as-is, never as `network`.""" + + UNEXPECTED = _result("OSPS-BR-01.02", status="WARN", error_class="unexpected_exit") + ERRORED = { + **_result("OSPS-BR-01.02", status="ERROR", error_class="unexpected_exit"), + "error": { + "class": "unexpected_exit", + "cause": "Command exited with unexpected code 2: grep -r uses: .github/workflows/; stderr: grep: " + ".github/workflows/: No such file or directory", + }, + } + + @pytest.mark.unit + def test_summary_json(self) -> None: + from darnit_baseline.tools import _compact_result + + assert _compact_result(self.UNEXPECTED)["error_class"] == "unexpected_exit" + + @pytest.mark.unit + def test_sarif(self) -> None: + from darnit_baseline.formatters.sarif import result_to_sarif_result + + out = result_to_sarif_result(self.UNEXPECTED, rule_index=0, local_path="/tmp/x", repo="r") + assert out["properties"]["errorClass"] == "unexpected_exit" + + @pytest.mark.unit + def test_predicate(self) -> None: + from darnit_baseline.attestation.predicate import build_assessment_predicate + + predicate = build_assessment_predicate( + owner="o", + repo="r", + commit="a" * 40, + ref="refs/heads/main", + level=1, + results=[self.ERRORED], + project_config=None, + adapters_used=["builtin"], + ) + control = next(c for c in predicate["controls"] if c["id"] == "OSPS-BR-01.02") + assert control["error_class"] == "unexpected_exit" + assert control["error"]["class"] == "unexpected_exit" + assert "unexpected code 2" in control["error"]["cause"] + + @pytest.mark.unit + def test_markdown(self) -> None: + from darnit.tools.audit import format_result_contract_markdown, format_results_markdown + + md = format_results_markdown( + owner="o", + repo="r", + results=[self.UNEXPECTED], + summary={"PASS": 0, "FAIL": 0, "WARN": 1, "N/A": 0, "ERROR": 0, "total": 1}, + compliance={1: False}, + level=1, + ) + line = next(ln for ln in md.splitlines() if "OSPS-BR-01.02" in ln) + assert "[unexpected_exit]" in line + assert "network" not in md + + contract = "\n".join(format_result_contract_markdown(self.ERRORED)) + assert "`unexpected_exit`" in contract + assert "No such file or directory" in contract + + @pytest.mark.unit + def test_cli_text(self) -> None: + from darnit.cli import format_result_text + + assert "[unexpected_exit]" in format_result_text(self.UNEXPECTED)