diff --git a/CHANGELOG.md b/CHANGELOG.md index 64d79092..4d77c248 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -448,6 +448,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `operator_config`, `trust`, `ignored_repository_settings`, `unknown_assertions`, `summary`, `results`, and `warnings` when present) instead of a bare list of results. +- An `exec` step whose command exits with a code the step does not declare + reports error class `unexpected_exit` instead of `network` unless stderr + shows a connection error (name resolution, refused or reset connection, + unreachable host, TLS or certificate failure). Exit code 127 or "command + not found" reports `missing_tool`. The step's message gives the exit code, + the command, and the start of stderr. Before, `grep` exiting 2 on a missing + directory or `git` exiting 128 outside a repository was reported as a + network failure (#562). ### Fixed diff --git a/docs/SECURITY_GUIDE.md b/docs/SECURITY_GUIDE.md index 58b79fed..77f12b10 100644 --- a/docs/SECURITY_GUIDE.md +++ b/docs/SECURITY_GUIDE.md @@ -568,6 +568,8 @@ When reviewing a framework TOML or plugin change, treat any new `existence = tru A step that could not measure returns ERROR with a class and cause: `auth` (401/403), `rate_limit` (429 or a rate-limit 403), `unavailable` (5xx, transport failure, undeclared status), `missing_tool` (an absent binary or required MCP server), or `evaluation`. ERROR never concludes FAIL: later steps still run, and if none concludes the control ends ERROR. A platform response proves failure only when the `gh_api` step lists its status in `fail_on_status` (for example 404 for "no branch protection"), and a rate limit never does. A token that cannot read a setting therefore shows up as ERROR `[auth]`, not as a failing project. ERROR is non-compliant. +An `exec` command that exits with a code its step does not declare concludes nothing either. Its class comes from the exit code and stderr: `rate_limit`, `auth`, `missing_tool` (exit 127 or "command not found"), `network` only when stderr shows a connection error, and otherwise `unexpected_exit`, whose message gives the exit code, the command, and the start of stderr. + ### PASS Candidates Confirmed by the Operator Controls that need judgment of document content end `PENDING` (`pending.kind = "llm_judgment"`) after the deterministic steps. A judgment comes from the harness's model step or from a coding agent through the `submit_judgment` MCP tool; both follow the same rules: diff --git a/docs/architecture/framework-design.md b/docs/architecture/framework-design.md index abe743d0..f97d0d1a 100644 --- a/docs/architecture/framework-design.md +++ b/docs/architecture/framework-design.md @@ -1,8 +1,8 @@ # Darnit Framework Design Specification -> **Version**: 1.0.0-alpha.12 +> **Version**: 1.0.0-alpha.13 > **Status**: Authoritative -> **Last Updated**: 2026-10-04 +> **Last Updated**: 2026-10-06 This specification defines the authoritative design of the Darnit framework, including the sieve orchestrator, TOML schema, built-in pass types, remediation actions, and plugin protocol. @@ -440,6 +440,18 @@ These are the step type's declared `settings` (section 3.0.3); any other key fai **Broken measurements**: a command whose binary is not installed is ERROR, class `missing_tool`; a timeout is ERROR, class `timeout`. Neither is FAIL. +**Undeclared exit codes**: an exit code in neither `pass_exit_codes` nor `fail_exit_codes` concludes nothing; the step is evidence only and carries a class taken from the command's exit code and stderr, first match wins (#562): + +| Evidence | Class | +|----------|-------| +| A rate-limit message (`api rate limit exceeded`, `secondary rate limit`, `abuse detection mechanism`) | `rate_limit` | +| An authentication message (`http 401`, `bad credentials`, `requires authentication`, `gh auth login`, `authentication token`, `not logged in`) | `auth` | +| Exit code 127, or stderr says the command was not found | `missing_tool` | +| A connection error: name resolution, connection refused, reset, or timed out, network or host unreachable, a TLS or certificate failure, or git's `unable to access` | `network` | +| Anything else | `unexpected_exit` | + +The step's message names the command, the exit code, and the start of stderr, so an operator can act on an `unexpected_exit` without rerunning the command. + #### Scenario: CEL expression evaluated - **WHEN** an `exec` step has an `expr` field and the exit code gives PASS or FAIL - **THEN** the expression MUST be evaluated after the command, and the step result MUST follow the outcome rules of section 3.7 @@ -450,6 +462,11 @@ These are the step type's declared `settings` (section 3.0.3); any other key fai - **THEN** the handler MUST evaluate the exit code against `pass_exit_codes` and `fail_exit_codes` - **AND** an exit code in neither list MUST NOT give PASS or FAIL +#### Scenario: Undeclared exit code with no identified cause +- **WHEN** an `exec` command exits with a code in neither list and its stderr shows no rate limit, authentication failure, missing command, or connection error (for example `grep` exiting 2 on a missing directory, or `git` exiting 128 outside a repository) +- **THEN** the step MUST carry class `unexpected_exit`, not `network` +- **AND** its message MUST name the command, the exit code, and an excerpt of stderr + ### 3.4 regex Handler **Purpose**: Regex-based content analysis @@ -1236,7 +1253,7 @@ Result fields beyond `status`: |-------|-------------| | `authority` | `dispositive` (a step concluded within its effective set), `suggestive` (no step concluded, or a model finding), `asserted` (a person confirmed it) | | `concluded_by` | Step handler name, `llm_judgment`, `confirmation`, `inferred_from`, or `none` | -| `error` | `{class, cause}`; present on every ERROR. Classes include `auth`, `rate_limit`, `unavailable`, `missing_tool`, `evaluation`, and the feature 036 classes | +| `error` | `{class, cause}`; present on every ERROR. Classes include `auth`, `rate_limit`, `unavailable`, `missing_tool`, `evaluation`, `unexpected_exit` (#562), and the feature 036 classes | | `pending` | `{kind}`; present on every PENDING | | `candidate` | Present with `pending.kind = "confirmation"`: `verdict = "pass"`, `reasoning`, `cited_evidence`, `model`, `model_version`, `evidence_digest`, `source` (`harness` or `mcp_agent`) | | `confirmation` | Present on a PASS from a confirmed candidate: `confirmed_by`, `confirmed_at`, `expires_at` | @@ -2301,6 +2318,7 @@ The following requirements have been superseded: by the handler dispatch archite | Version | Date | Changes | |---------|------|---------| +| 1.0.0-alpha.13 | 2026-10-06 | Error class `unexpected_exit`: an `exec` step whose undeclared exit code has no identified cause no longer reports `network`; exit code 127 reports `missing_tool` (Sections 3.3, 5.2; #562) | | 1.0.0-alpha.12 | 2026-10-04 | Repository-level `.baseline.toml` is no longer read: one notice points at `darnit config migrate`, framework selection only by `--framework` or a tool argument (Sections 2.3, 10.5, 14.4, 15.1; Appendix C) | | 1.0.0-alpha.11 | 2026-10-04 | Close remaining false-PASS paths (feature 044): an expression that cannot be evaluated or is not boolean makes the step ERROR, expression names per step type with usable `project` values and a repository-aware `file_exists`, load-time expression reference check (Section 3.7); step registration declares `settings` and `expression_names`, plugins cannot replace a registered step type, and unknown control keys, unknown step keys, and unregistered step types fail loading (Sections 2.3, 3.0.3); reproducibility step types conclude only FAIL (Sections 3.0.1, 12); `expr_decides`, an expression that decides on a handler PASS (Sections 3.0.3, 3.7); `gh_api` `evidence_fields`, required for personal records (Section 3.8); `file_must_exist` replaced by the registered `file_exists` (Section 3.2); field tables corrected to what each handler reads (Sections 3.3-3.5) | | 1.0.0-alpha.10 | 2026-10-02 | Remediation safety (feature 043): plan/apply protocol and single writer (Section 4.2), `platform_setting` (4.5), exec `effects`/`offline` and no platform state from exec (4.4), `file_create.project_reference` (4.3), remediation policy, digest-bound approval, outcomes, re-check, run manifest, and version-control rules (Section 15); removed `api_call`, `requires_confirmation`, `dry_run_supported`, `dry_run_command` (Appendix C) | diff --git a/packages/darnit/src/darnit/core/error_class.py b/packages/darnit/src/darnit/core/error_class.py index 2a7668ad..dd11c04f 100644 --- a/packages/darnit/src/darnit/core/error_class.py +++ b/packages/darnit/src/darnit/core/error_class.py @@ -1,6 +1,7 @@ """Environmental failure classification. -Feature 036. See specs/036-tier2-error-class/contracts/error-class.md. +Feature 036. See specs/036-tier2-error-class/contracts/error-class.md and +its #562 addendum, contracts/error-class-addendum-562.md. When a sieve pass cannot run to completion -- the network is unreachable, the auth token expired, a subprocess timed out, a required binary is @@ -16,31 +17,34 @@ check ran and the repository does not comply" -- that stays a bare FAIL/WARN with no ``error_class``. -============= =============================================================== -Value Meaning -============= =============================================================== -network Host unreachable, DNS failure, TLS error, MCP server unusable, - or any non-zero subprocess exit whose stderr matched no - more-specific pattern. -auth HTTP 401, bad credentials, expired token, "requires - authentication", or MCP plugin signature-verification failure. -timeout Subprocess exceeded its ``timeout`` budget, or an MCP tool - call exceeded ``MCP_DEFAULT_TIMEOUT_SECONDS``. -rate_limit GitHub primary or secondary rate limit, or abuse-detection - throttle. -not_found A control names an MCP server the operator never configured. - (Before feature 041 this also covered an absent binary; step - results now report that as ``missing_tool``.) -crashed A handler raised an unexpected exception, or an MCP tool - returned unparseable output. The handler did not complete - cleanly. -missing_tool Feature 041. A tool the step needs (``gh``, a command's - binary, a required MCP server executable) is not installed. -unavailable Feature 041. The platform API answered with a 5xx or a status - the step did not declare, or the request never got a response. -evaluation Feature 041. The step ran but its result could not be - evaluated (for example a CEL ``expr`` error over a response). -============= =============================================================== +=============== ============================================================= +Value Meaning +=============== ============================================================= +network Host unreachable, DNS failure, connection refused or reset, + TLS error, or MCP server unusable. An ``exec`` step reports it + only when stderr shows a connection error (#562). +auth HTTP 401, bad credentials, expired token, "requires + authentication", or MCP plugin signature-verification failure. +timeout Subprocess exceeded its ``timeout`` budget, or an MCP tool + call exceeded ``MCP_DEFAULT_TIMEOUT_SECONDS``. +rate_limit GitHub primary or secondary rate limit, or abuse-detection + throttle. +not_found A control names an MCP server the operator never configured. + (Before feature 041 this also covered an absent binary; step + results now report that as ``missing_tool``.) +crashed A handler raised an unexpected exception, or an MCP tool + returned unparseable output. The handler did not complete + cleanly. +missing_tool Feature 041. A tool the step needs (``gh``, a command's + binary, a required MCP server executable) is not installed, + or a command exited 127 or said "command not found". +unavailable Feature 041. The platform API answered with a 5xx or a status + the step did not declare, or the request never got a response. +evaluation Feature 041. The step ran but its result could not be + evaluated (for example a CEL ``expr`` error over a response). +unexpected_exit #562. A command exited with a code its step did not declare, + and nothing in its output identified the cause. +=============== ============================================================= Two names are exported because ``typing.Literal`` is erased at runtime and enforces nothing on its own. ``ErrorClass`` gives static-analysis @@ -74,6 +78,7 @@ "missing_tool", "unavailable", "evaluation", + "unexpected_exit", ] # Runtime-checkable companion to ``ErrorClass``. See module docstring for @@ -89,6 +94,7 @@ "missing_tool", "unavailable", "evaluation", + "unexpected_exit", ) ) diff --git a/packages/darnit/src/darnit/sieve/builtin_handlers.py b/packages/darnit/src/darnit/sieve/builtin_handlers.py index ae72b361..8f66eb16 100644 --- a/packages/darnit/src/darnit/sieve/builtin_handlers.py +++ b/packages/darnit/src/darnit/sieve/builtin_handlers.py @@ -54,11 +54,12 @@ # Feature 036: environmental-failure classification # ============================================================================= -# GitHub-only stderr patterns for v0 (clarify Q4). The `exec` handler sees -# only stdout/stderr/exit-code -- it has no access to response headers -- so -# classification is substring matching against `gh` CLI stderr shape. Other -# exec targets (git, curl, syft, cosign) fall through to `network`; per-target -# pattern packs are a follow-up if real audits show they are needed. +# The `exec` handler sees only stdout/stderr/exit-code -- it has no access to +# response headers -- so classification is substring matching against stderr. +# The rate-limit and auth patterns follow `gh` CLI stderr shape (clarify Q4). +# A failure no pattern identifies is `unexpected_exit`, not `network` (#562): +# most undeclared exits (grep on a missing directory, git outside a +# repository) never touch the network. # # Rate-limit is checked BEFORE auth: GitHub answers 403 for both rate limits # and permission failures, and the rate-limit body is the more specific signal. @@ -74,22 +75,75 @@ "requires authentication", "gh auth login", "authentication token", + "not logged in", ) -def _classify_exec_failure(stderr: str) -> ErrorClass: - """Classify a non-zero-exit subprocess failure from its stderr. +_MISSING_TOOL_EXIT_CODE = 127 + +_MISSING_TOOL_PATTERNS: tuple[str, ...] = ("command not found",) + +_NETWORK_PATTERNS: tuple[str, ...] = ( + "could not resolve host", + "name or service not known", + "temporary failure in name resolution", + "nodename nor servname provided", + "no such host", + "connection refused", + "connection reset", + "connection was reset", + "network is unreachable", + "no route to host", + "connection timed out", + "failed to connect to", + "i/o timeout", + "error connecting to", + "tls handshake", + "ssl_connect", + "gnutls_handshake", + "ssl certificate problem", + "certificate verify failed", + "x509:", + "server certificate verification failed", + "unable to access 'http", +) + +# git's "unable to access" also wraps an HTTP status the server sent back; +# an answer from the server is not a connection failure. +_HTTP_RESPONSE_PATTERNS: tuple[str, ...] = ("the requested url returned error",) + +_STDERR_EXCERPT_CHARS = 200 + + +def _classify_exec_failure(stderr: str, exit_code: int | None = None, executable: str | None = None) -> ErrorClass: + """Classify an undeclared-exit subprocess failure from its exit code and stderr. Only called on paths where the command did not complete as expected. Callers must NOT invoke this for a declared ``fail_exit_codes`` hit -- that is a check that ran and concluded, not an environmental failure. + ``executable`` lets a shell's ": not found" count as a missing tool + without reading every "not found" (an HTTP 404, say) as one. """ haystack = (stderr or "").lower() if any(p in haystack for p in _GH_RATE_LIMIT_PATTERNS): return "rate_limit" if any(p in haystack for p in _GH_AUTH_PATTERNS): return "auth" - return "network" + if exit_code == _MISSING_TOOL_EXIT_CODE or any(p in haystack for p in _MISSING_TOOL_PATTERNS): + return "missing_tool" + if executable: + name = re.escape(os.path.basename(executable).lower()) + if re.search(rf"(^|[\s:]){name}: not found\s*$", haystack, re.MULTILINE): + return "missing_tool" + if any(p in haystack for p in _NETWORK_PATTERNS) and not any(p in haystack for p in _HTTP_RESPONSE_PATTERNS): + return "network" + return "unexpected_exit" + + +def _excerpt(text: str) -> str: + """The start of ``text`` on one ASCII line, for an operator-facing cause.""" + line = " ".join((text or "").split())[:_STDERR_EXCERPT_CHARS] + return line.encode("ascii", "replace").decode("ascii") def _log_environmental_failure( @@ -369,10 +423,14 @@ def exec_handler(config: dict[str, Any], context: HandlerContext) -> HandlerResu ) else: # Undeclared exit code: we cannot tell whether the check concluded. - # Classify from stderr so the operator can distinguish a rate limit - # or expired token from a genuine non-compliance signal. - error_class = _classify_exec_failure(evidence["stderr"]) - message = f"Command exited with unexpected code {proc.returncode}" + # Classify from exit code and stderr so the operator can distinguish + # a rate limit or expired token from a genuine non-compliance signal. + error_class = _classify_exec_failure(evidence["stderr"], proc.returncode, resolved_cmd[0]) + excerpt = _excerpt(evidence["stderr"]) or "(no stderr)" + message = ( + f"Command exited with unexpected code {proc.returncode}: " + f"{_excerpt(' '.join(resolved_cmd))}; stderr: {excerpt}" + ) _log_environmental_failure(context.control_id, "exec", error_class, message) return HandlerResult( status=HandlerResultStatus.INCONCLUSIVE, diff --git a/specs/036-tier2-error-class/contracts/error-class-addendum-562.md b/specs/036-tier2-error-class/contracts/error-class-addendum-562.md new file mode 100644 index 00000000..5a222b66 --- /dev/null +++ b/specs/036-tier2-error-class/contracts/error-class-addendum-562.md @@ -0,0 +1,40 @@ +# Addendum: `unexpected_exit` replaces `network` as the exec catch-all + +**Amends**: [error-class.md](error-class.md) sections 1 and 2.1 +**Issue**: #562 +**Authoritative text**: `docs/architecture/framework-design.md` section 3.3 ("Undeclared exit codes") and section 5.2 + +## Why + +Section 2.1 made `network` the class for any undeclared exit code whose stderr matched no pattern. Most such exits never touch the network: `grep` exiting 2 on a missing directory, `git` exiting 128 outside a repository, a binary the shell could not find. Reports and logs then told operators to check their network for a local condition. + +## 1. The enum + +One value is added. `ErrorClass` and `ERROR_CLASSES` are edited together, as section 1 requires. + +| Value | Meaning | +|---|---| +| `unexpected_exit` | A command exited with a code its step did not declare, and nothing in its output identified the cause. | + +`network` keeps its other producers (MCP handshake failure and unusable server, section 2.2) and, for `exec`, now means only that stderr shows a connection error. + +## 2.1 `exec_handler` (replaces the non-zero-exit rows) + +Checked in this order on an exit code in neither `pass_exit_codes` nor `fail_exit_codes`; first match wins. Matching is a case-insensitive substring match on stderr. + +| Condition | `error_class` | +|---|---| +| stderr matches `_GH_RATE_LIMIT_PATTERNS` | `rate_limit` | +| stderr matches `_GH_AUTH_PATTERNS` (now also `not logged in`) | `auth` | +| exit code 127, stderr matches `_MISSING_TOOL_PATTERNS` (`command not found`), or a stderr line ends `: not found` | `missing_tool` | +| stderr matches `_NETWORK_PATTERNS` (name resolution, connection refused, reset, or timed out, failed to connect, network or host unreachable, TLS/SSL or certificate errors, git `unable to access 'http...`) | `network` | +| anything else, including empty stderr | `unexpected_exit` | + +The timeout, missing-binary (`FileNotFoundError`), pass, and declared-fail rows are unchanged. The step stays INCONCLUSIVE as before; only its class changes. + +The step's message names the exit code, the command, and the first 200 characters of stderr on one line, with non-ASCII characters replaced, so the cause is actionable from the report. The message is what the section 7 WARN log line and the step's pass-history entry carry. + +## Unchanged + +- `gh_api` status classification, the MCP mapping (section 2.2), the orchestrator's `crashed` (section 2.3), and context auto-detect (section 2.4). +- Propagation (section 3), CEL preservation (section 4), output surfaces (section 5), and validation (section 6). Every surface renders the class name it is given, so no surface needs a new mapping. diff --git a/tests/darnit/sieve/test_error_class_classification.py b/tests/darnit/sieve/test_error_class_classification.py index 25a34eee..2f20d106 100644 --- a/tests/darnit/sieve/test_error_class_classification.py +++ b/tests/darnit/sieve/test_error_class_classification.py @@ -84,6 +84,17 @@ def test_each_valid_value_constructs(self, value: str) -> None: ) assert result.error_class == value + @pytest.mark.unit + def test_unexpected_exit_is_a_known_class(self) -> None: + """#562: the Literal and the frozenset are edited together.""" + from typing import get_args + + from darnit.core.error_class import ErrorClass + + assert "unexpected_exit" in ERROR_CLASSES + assert "unexpected_exit" in get_args(ErrorClass) + assert set(get_args(ErrorClass)) == set(ERROR_CLASSES) + @pytest.mark.unit def test_none_is_always_allowed_including_on_pass(self) -> None: result = HandlerResult(status=HandlerResultStatus.PASS, message="ok") @@ -178,8 +189,8 @@ def test_rate_limit_wins_over_auth_when_both_could_match( assert result.error_class == "rate_limit" @pytest.mark.unit - def test_unmatched_stderr_falls_back_to_network(self, tmp_path: Path) -> None: - """FR-005a: non-GitHub / unrecognized failure is the network bucket.""" + def test_unmatched_stderr_is_unexpected_exit_not_network(self, tmp_path: Path) -> None: + """#562: an unrecognized failure no longer claims a network cause.""" with patch( "darnit.sieve.builtin_handlers.subprocess.run", return_value=_proc(128, stderr="fatal: not a git repository"), @@ -188,7 +199,51 @@ def test_unmatched_stderr_falls_back_to_network(self, tmp_path: Path) -> None: {"handler": "exec", "command": ["git", "rev-parse", "HEAD"]}, _ctx(tmp_path), ) - assert result.error_class == "network" + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert result.error_class == "unexpected_exit" + + @pytest.mark.unit + def test_unexpected_exit_message_names_command_code_and_stderr(self, tmp_path: Path) -> None: + """#562: the cause is actionable without rerunning the command.""" + stderr = "grep: .github/workflows/: No such file or directory\n" + "x" * 500 + "\u2603" + with patch( + "darnit.sieve.builtin_handlers.subprocess.run", + return_value=_proc(2, stderr=stderr), + ): + result = exec_handler( + {"handler": "exec", "command": ["grep", "-r", "uses:", ".github/workflows/"]}, + _ctx(tmp_path), + ) + assert result.error_class == "unexpected_exit" + assert "grep -r uses: .github/workflows/" in result.message + assert "code 2" in result.message + assert "grep: .github/workflows/: No such file or directory x" in result.message + assert "\n" not in result.message + assert result.message.isascii() + assert "x" * 300 not in result.message + + @pytest.mark.unit + def test_unexpected_exit_with_empty_stderr_says_so(self, tmp_path: Path) -> None: + with patch( + "darnit.sieve.builtin_handlers.subprocess.run", + return_value=_proc(3), + ): + result = exec_handler({"handler": "exec", "command": ["some-checker"]}, _ctx(tmp_path)) + assert result.error_class == "unexpected_exit" + assert "code 3" in result.message + assert "some-checker" in result.message + assert "no stderr" in result.message + + @pytest.mark.unit + def test_exit_127_is_missing_tool(self, tmp_path: Path) -> None: + """#562: a shell or env wrapper reports a missing binary as exit 127.""" + with patch( + "darnit.sieve.builtin_handlers.subprocess.run", + return_value=_proc(127, stderr="env: 'scorecard': No such file or directory"), + ): + result = exec_handler({"handler": "exec", "command": ["env", "scorecard"]}, _ctx(tmp_path)) + assert result.status == HandlerResultStatus.INCONCLUSIVE + assert result.error_class == "missing_tool" @pytest.mark.unit def test_success_carries_no_error_class(self, tmp_path: Path) -> None: @@ -229,6 +284,74 @@ def test_clean_definitive_failure_carries_no_error_class( assert result.error_class is None +class TestClassifyExecFailure: + """#562: the undeclared-exit decision table, first match wins.""" + + @pytest.mark.unit + @pytest.mark.parametrize( + ("stderr", "exit_code", "expected"), + [ + ("gh: API rate limit exceeded for user ID 1234.", 1, "rate_limit"), + ("You have exceeded a secondary rate limit.", 1, "rate_limit"), + ("gh: HTTP 403: You have exceeded a secondary rate limit.", 1, "rate_limit"), + ("gh: Bad credentials (HTTP 401)", 1, "auth"), + ("To get started with GitHub CLI, please run: gh auth login", 4, "auth"), + ("You are not logged into any GitHub hosts.", 4, "auth"), + ("", 127, "missing_tool"), + ("bash: scorecard: command not found", 1, "missing_tool"), + ("sh: 1: scorecard: not found", 2, "missing_tool"), + ("fatal: unable to access 'https://github.com/o/r/': Could not resolve host: github.com", 128, "network"), + ("curl: (6) Could not resolve host: example.com", 6, "network"), + ("ssh: Could not resolve hostname github.com: Name or service not known", 255, "network"), + ("ssh: github.com: Temporary failure in name resolution", 255, "network"), + ( + "ssh: Could not resolve hostname github.com: nodename nor servname provided, or not known", + 255, + "network", + ), + ("dial tcp: lookup api.github.com: no such host", 1, "network"), + ("curl: (7) Failed to connect to localhost port 1: Connection refused", 7, "network"), + ("fetch-pack: unexpected disconnect: Connection reset by peer", 128, "network"), + ("connect: Network is unreachable", 1, "network"), + ("ssh: connect to host github.com port 22: No route to host", 255, "network"), + ("ssh: connect to host github.com port 22: Connection timed out", 255, "network"), + ("curl: (28) Failed to connect to example.com port 443: Operation timed out", 28, "network"), + ("dial tcp 140.82.112.6:443: i/o timeout", 1, "network"), + ("error connecting to api.github.com", 1, "network"), + ("curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.com:443", 35, "network"), + ("net/http: TLS handshake timeout", 1, "network"), + ("SSL certificate problem: unable to get local issuer certificate", 60, "network"), + ("x509: certificate signed by unknown authority", 1, "network"), + ("server certificate verification failed. CAfile: none CRLfile: none", 128, "network"), + ("error: RPC failed; curl 56 OpenSSL SSL_read: Connection was reset, errno 10054", 128, "network"), + ("fatal: unable to access 'https://example.com/r/': gnutls_handshake() failed", 128, "network"), + ( + "fatal: unable to access 'https://example.com/r/': The requested URL returned error: 404", + 128, + "unexpected_exit", + ), + ("grep: .github/workflows/: No such file or directory", 2, "unexpected_exit"), + ("fatal: not a git repository (or any of the parent directories): .git", 128, "unexpected_exit"), + ("error: No such remote 'upstream'", 2, "unexpected_exit"), + ("gh: Not Found (HTTP 404)", 1, "unexpected_exit"), + ("", 1, "unexpected_exit"), + ("", None, "unexpected_exit"), + ], + ) + def test_table(self, stderr: str, exit_code: int | None, expected: str) -> None: + from darnit.sieve.builtin_handlers import _classify_exec_failure + + assert _classify_exec_failure(stderr, exit_code, "scorecard") == expected + + @pytest.mark.unit + def test_not_found_names_only_the_executable(self) -> None: + """A bare "not found" about some other thing is not a missing tool.""" + from darnit.sieve.builtin_handlers import _classify_exec_failure + + assert _classify_exec_failure("gh: Not Found (HTTP 404)", 1, "gh") == "unexpected_exit" + assert _classify_exec_failure("sh: 1: gh: not found", 1, "/usr/bin/gh") == "missing_tool" + + class TestExecHandlerWarnLogging: """Contract section 7: environmental failures log at WARN, not DEBUG.""" diff --git a/tests/darnit_baseline/test_error_class_output_surfaces.py b/tests/darnit_baseline/test_error_class_output_surfaces.py index 7eb416b3..5d7fb794 100644 --- a/tests/darnit_baseline/test_error_class_output_surfaces.py +++ b/tests/darnit_baseline/test_error_class_output_surfaces.py @@ -169,3 +169,75 @@ def test_predicate_still_carries_authority_alongside(self) -> None: """Regression guard: the feature-025 field is untouched.""" controls = self._controls_by_id(self._predicate([ENV_FAILURE])) assert controls["OSPS-LE-02.02"]["authority"] == "dispositive" + + +class TestUnexpectedExitSurfaces: + """#562: every surface carries `unexpected_exit` as-is, never as `network`.""" + + UNEXPECTED = _result("OSPS-BR-01.02", status="WARN", error_class="unexpected_exit") + ERRORED = { + **_result("OSPS-BR-01.02", status="ERROR", error_class="unexpected_exit"), + "error": { + "class": "unexpected_exit", + "cause": "Command exited with unexpected code 2: grep -r uses: .github/workflows/; stderr: grep: " + ".github/workflows/: No such file or directory", + }, + } + + @pytest.mark.unit + def test_summary_json(self) -> None: + from darnit_baseline.tools import _compact_result + + assert _compact_result(self.UNEXPECTED)["error_class"] == "unexpected_exit" + + @pytest.mark.unit + def test_sarif(self) -> None: + from darnit_baseline.formatters.sarif import result_to_sarif_result + + out = result_to_sarif_result(self.UNEXPECTED, rule_index=0, local_path="/tmp/x", repo="r") + assert out["properties"]["errorClass"] == "unexpected_exit" + + @pytest.mark.unit + def test_predicate(self) -> None: + from darnit_baseline.attestation.predicate import build_assessment_predicate + + predicate = build_assessment_predicate( + owner="o", + repo="r", + commit="a" * 40, + ref="refs/heads/main", + level=1, + results=[self.ERRORED], + project_config=None, + adapters_used=["builtin"], + ) + control = next(c for c in predicate["controls"] if c["id"] == "OSPS-BR-01.02") + assert control["error_class"] == "unexpected_exit" + assert control["error"]["class"] == "unexpected_exit" + assert "unexpected code 2" in control["error"]["cause"] + + @pytest.mark.unit + def test_markdown(self) -> None: + from darnit.tools.audit import format_result_contract_markdown, format_results_markdown + + md = format_results_markdown( + owner="o", + repo="r", + results=[self.UNEXPECTED], + summary={"PASS": 0, "FAIL": 0, "WARN": 1, "N/A": 0, "ERROR": 0, "total": 1}, + compliance={1: False}, + level=1, + ) + line = next(ln for ln in md.splitlines() if "OSPS-BR-01.02" in ln) + assert "[unexpected_exit]" in line + assert "network" not in md + + contract = "\n".join(format_result_contract_markdown(self.ERRORED)) + assert "`unexpected_exit`" in contract + assert "No such file or directory" in contract + + @pytest.mark.unit + def test_cli_text(self) -> None: + from darnit.cli import format_result_text + + assert "[unexpected_exit]" in format_result_text(self.UNEXPECTED)