diff --git a/README.md b/README.md index 338b386..9e6a401 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,7 @@ It produces a review, not a cleanup script. Default operations are **read-only** > **Status:** Pre-release read-only preview for trusted repositories; **no version has been tagged or published**. Installation and native discovery depend on agent/runtime configuration. A Full review is not a security certification or physical-device verification. -**Pre-evidence-reconciliation baseline:** [PR #14](https://github.com/raiseCatError/freview/pull/14) merged at [`b4e9de58`](https://github.com/raiseCatError/freview/commit/b4e9de581107b8b78b4a710a5be93fa4e90510e8), with [99 Python tests passing on both Ubuntu and macOS](https://github.com/raiseCatError/freview/actions/runs/38983887532). User-provided field evidence reports native invocation in Antigravity CLI, Claude Code, Codex and OpenCode; this is **not** four independently re-executed Full audits, and OpenCode's full five-gate acceptance matrix remains unobserved. - -**In-progress enhancement:** [draft PR #16](https://github.com/raiseCatError/freview/pull/16) introduces optional specification inventory, verification-receipt consistency checks, evidence-aware reports and deliberately deceptive metadata fixtures. It remains pre-release; CI and author review are not independent runtime-agent acceptance. +**Integrated on `main`:** [evidence reconciliation #16](https://github.com/raiseCatError/freview/pull/16), [optional operations #17](https://github.com/raiseCatError/freview/pull/17), [structured adversarial verdicts #18](https://github.com/raiseCatError/freview/pull/18), [verification isolation #19](https://github.com/raiseCatError/freview/pull/19), and [mutation quality / bounded autonomy #20](https://github.com/raiseCatError/freview/pull/20). All five have been merged with their development history retained; [the integrated CI run](https://github.com/raiseCatError/freview/actions/runs/39019289770) passed on Ubuntu and macOS. **This is still pre-release:** real provider, mutation workload and physical-host acceptance are separate from synthetic CI. **Dogfooding:** [final pre-release freview-on-freview review](docs/audits/2026-10-17-final-pre-release-self-review.md) ยท [historical original audit](docs/freview-reviews-freview.md). The newer source review is a bounded, hosted-evidence **partial-audit**, not native invocation in the author's agent session. @@ -83,7 +81,7 @@ The second command inspects a supplied JSON receipt and never runs its test comm ## Optional operations: everything off by default -The preview [freview ops companion](skills/freview/references/optional-ops.md) adds **separately enabled** specification/plan generation, worker CLI invocation, isolated test execution, fresh-process **structured JSON** adversarial reviewer verdicts (up to 3 bounded repair cycles), mutation-tool delegation, exact-commit baseline regression sensitivity, local Git hook guards and human shipping handoff. These are **not part of the default read-only audit**. No untrusted repository configuration can silently authorize execution, install tools or publish releases. +The [freview ops companion](skills/freview/references/optional-ops.md) adds **separately enabled** specification/plan generation, worker CLI invocation, isolated test execution, fresh-process **structured JSON** adversarial reviewer verdicts (up to 3 bounded repair cycles), mutation-tool delegation, exact-commit baseline regression sensitivity, local Git hook guards and human shipping handoff. These are **not part of the default read-only audit**. No untrusted repository configuration can silently authorize execution, install tools or publish releases. ```sh # No config = all modules disabled, no execution or tokens @@ -101,7 +99,7 @@ A completed reviewer process cannot approve itself with plain text: freview vali The ops companion now understands Stryker JSON and mutmut CI stats **as actual mutation evidence**, rather than trusting a mutator's exit code alone. It reports detected, surviving/no-coverage, pending and unknown mutants, plus a bounded sample of survivor locations. Missing, stale, interrupted or malformed reports are inconclusive; a separately enabled score gate can block a handoff. The portable mutmut runner invokes an **already-installed** mutmut CLI and exports its stats without a shell. -Worker/reviewer loops additionally enforce a **global subprocess budget** and stop when a worker makes no source progress or repeated findings remain unchanged. All behavior is explicitly configurable; all modules remain OFF by default. Scratch checkpoints retain evidence without automatically resuming or signing an audit. See the [mutation and autonomy guide](skills/freview/references/mutation-autonomy.md) and [configuration example](examples/freview-ops.example.json). +Worker/reviewer loops additionally enforce a **global subprocess budget** and stop when a worker makes no source progress or repeated findings remain unchanged. All behavior is explicitly configurable; all modules remain OFF by default. Scratch checkpoints retain evidence without automatically resuming or signing an audit. The canonical [operations-policy schema](schemas/freview-ops.schema.json) is bundled with the installed skill; the separate [audit configuration schema](schemas/freview-config.schema.json) remains non-executable. See the [mutation and autonomy guide](skills/freview/references/mutation-autonomy.md) and [configuration example](examples/freview-ops.example.json). ## Review modes @@ -179,6 +177,8 @@ The helper reports current Git metadata, worktrees, branch/upstream divergence, Optionally put a validated `.freview.json` at your repository root. Declare your development/release branches, default audit mode, excluded categories, optional human QA gates, bounded audit limits and specialist preferences. The config is **data only**, never executable hooks. See [configuration](skills/freview/references/configuration.md), [JSON Schema](schemas/freview-config.schema.json) and [NMSh example](examples/nmsh.freview.json). Explicit user instructions override project preferences. +For the separately invoked **operations companion**, use `.freview-ops.json` and its [strict v1 operations schema](schemas/freview-ops.schema.json). It is distinct from `.freview.json`: it contains optional executable argv and therefore requires an explicitly trusted project, a clean exact-commit checkout, a caller-selected scratch directory, and a separate `--allow-execute` permission before anything runs. Neither config silently opts you into costly agents, mutation tools or shipping. + ## Report semantics and installation checks Report comparisons now call missing findings `absent_from_new_report` rather than `resolved`. Absence of a finding **does not prove remediation**; verify actual source and CI evidence against exact refs. JSON schema validation checks structure, **not whether a tool really ran**. The report loader bounds reads before parsing and checks timestamps are UTC. Private paths and branch names remain possible in reports; review documents before publication. diff --git a/schemas/freview-ops.schema.json b/schemas/freview-ops.schema.json new file mode 100644 index 0000000..7f0dca4 --- /dev/null +++ b/schemas/freview-ops.schema.json @@ -0,0 +1,249 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/raiseCatError/freview/blob/main/schemas/freview-ops.schema.json", + "title": "freview optional operations policy v1", + "description": "Explicitly opted-in trusted-project operations. This schema describes configuration, not permission to execute; the runtime validator also enforces stage and subprocess safety rules.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version" + ], + "properties": { + "schema_version": { + "type": "integer", + "const": 1 + }, + "modules": { + "type": "object", + "additionalProperties": false, + "properties": { + "specs": { + "type": "boolean", + "default": false + }, + "planning": { + "type": "boolean", + "default": false + }, + "implementation": { + "type": "boolean", + "default": false + }, + "tests": { + "type": "boolean", + "default": false + }, + "adversarial": { + "type": "boolean", + "default": false + }, + "mutation": { + "type": "boolean", + "default": false + }, + "regression": { + "type": "boolean", + "default": false + }, + "hooks": { + "type": "boolean", + "default": false + }, + "shipping": { + "type": "boolean", + "default": false + } + }, + "description": "Opt-in modules. All disabled unless explicitly enabled. A repository config alone does not grant permission to execute." + }, + "commands": { + "type": "object", + "additionalProperties": false, + "properties": { + "planning": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "implementation": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "tests": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "adversarial": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "mutation": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "regression": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + } + }, + "description": "Trusted executable argv; providing a command does not enable its module." + }, + "limits": { + "type": "object", + "additionalProperties": false, + "properties": { + "timeout_seconds": { + "type": "integer", + "minimum": 1, + "maximum": 600, + "default": 120 + }, + "max_rounds": { + "type": "integer", + "minimum": 1, + "maximum": 3, + "default": 3 + }, + "max_output_bytes": { + "type": "integer", + "minimum": 1024, + "maximum": 65536, + "default": 16384 + } + } + }, + "pass_env": { + "type": "array", + "uniqueItems": true, + "maxItems": 8, + "default": [], + "items": { + "enum": [ + "HOME", + "XDG_CONFIG_HOME", + "CODEX_HOME", + "CLAUDE_CONFIG_DIR", + "ANTHROPIC_API_KEY", + "OPENAI_API_KEY", + "GOOGLE_API_KEY", + "GEMINI_API_KEY" + ] + }, + "description": "Explicit environment forwarding still needs a separate runtime approval." + }, + "mutation_quality": { + "type": "object", + "additionalProperties": false, + "properties": { + "format": { + "type": "string", + "enum": [ + "auto", + "stryker", + "mutmut" + ], + "default": "auto" + }, + "mode": { + "type": "string", + "enum": [ + "advisory", + "gate" + ], + "default": "advisory" + }, + "min_score": { + "type": "number", + "minimum": 0, + "maximum": 100, + "default": 60 + }, + "min_valid_mutants": { + "type": "integer", + "minimum": 1, + "maximum": 10000, + "default": 1 + }, + "max_survivors": { + "type": "integer", + "minimum": 0, + "maximum": 50, + "default": 12 + } + } + }, + "autonomy": { + "type": "object", + "additionalProperties": false, + "properties": { + "max_stage_runs": { + "type": "integer", + "minimum": 1, + "maximum": 32, + "default": 12 + }, + "stop_on_no_progress": { + "type": "boolean", + "default": true + }, + "max_repeated_findings": { + "type": "integer", + "minimum": 1, + "maximum": 3, + "default": 1 + }, + "require_mutation_for_shipping": { + "type": "boolean", + "default": false + } + } + } + } +} diff --git a/skills/freview/assets/schemas/freview-ops.schema.json b/skills/freview/assets/schemas/freview-ops.schema.json new file mode 100644 index 0000000..7f0dca4 --- /dev/null +++ b/skills/freview/assets/schemas/freview-ops.schema.json @@ -0,0 +1,249 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/raiseCatError/freview/blob/main/schemas/freview-ops.schema.json", + "title": "freview optional operations policy v1", + "description": "Explicitly opted-in trusted-project operations. This schema describes configuration, not permission to execute; the runtime validator also enforces stage and subprocess safety rules.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version" + ], + "properties": { + "schema_version": { + "type": "integer", + "const": 1 + }, + "modules": { + "type": "object", + "additionalProperties": false, + "properties": { + "specs": { + "type": "boolean", + "default": false + }, + "planning": { + "type": "boolean", + "default": false + }, + "implementation": { + "type": "boolean", + "default": false + }, + "tests": { + "type": "boolean", + "default": false + }, + "adversarial": { + "type": "boolean", + "default": false + }, + "mutation": { + "type": "boolean", + "default": false + }, + "regression": { + "type": "boolean", + "default": false + }, + "hooks": { + "type": "boolean", + "default": false + }, + "shipping": { + "type": "boolean", + "default": false + } + }, + "description": "Opt-in modules. All disabled unless explicitly enabled. A repository config alone does not grant permission to execute." + }, + "commands": { + "type": "object", + "additionalProperties": false, + "properties": { + "planning": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "implementation": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "tests": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "adversarial": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "mutation": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + }, + "regression": { + "type": "array", + "minItems": 1, + "maxItems": 32, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 2048, + "pattern": "^[^\\u0000\\u000A]+$" + }, + "description": "Explicit argv, never a shell snippet; additional command safety checks occur at runtime." + } + }, + "description": "Trusted executable argv; providing a command does not enable its module." + }, + "limits": { + "type": "object", + "additionalProperties": false, + "properties": { + "timeout_seconds": { + "type": "integer", + "minimum": 1, + "maximum": 600, + "default": 120 + }, + "max_rounds": { + "type": "integer", + "minimum": 1, + "maximum": 3, + "default": 3 + }, + "max_output_bytes": { + "type": "integer", + "minimum": 1024, + "maximum": 65536, + "default": 16384 + } + } + }, + "pass_env": { + "type": "array", + "uniqueItems": true, + "maxItems": 8, + "default": [], + "items": { + "enum": [ + "HOME", + "XDG_CONFIG_HOME", + "CODEX_HOME", + "CLAUDE_CONFIG_DIR", + "ANTHROPIC_API_KEY", + "OPENAI_API_KEY", + "GOOGLE_API_KEY", + "GEMINI_API_KEY" + ] + }, + "description": "Explicit environment forwarding still needs a separate runtime approval." + }, + "mutation_quality": { + "type": "object", + "additionalProperties": false, + "properties": { + "format": { + "type": "string", + "enum": [ + "auto", + "stryker", + "mutmut" + ], + "default": "auto" + }, + "mode": { + "type": "string", + "enum": [ + "advisory", + "gate" + ], + "default": "advisory" + }, + "min_score": { + "type": "number", + "minimum": 0, + "maximum": 100, + "default": 60 + }, + "min_valid_mutants": { + "type": "integer", + "minimum": 1, + "maximum": 10000, + "default": 1 + }, + "max_survivors": { + "type": "integer", + "minimum": 0, + "maximum": 50, + "default": 12 + } + } + }, + "autonomy": { + "type": "object", + "additionalProperties": false, + "properties": { + "max_stage_runs": { + "type": "integer", + "minimum": 1, + "maximum": 32, + "default": 12 + }, + "stop_on_no_progress": { + "type": "boolean", + "default": true + }, + "max_repeated_findings": { + "type": "integer", + "minimum": 1, + "maximum": 3, + "default": 1 + }, + "require_mutation_for_shipping": { + "type": "boolean", + "default": false + } + } + } + } +} diff --git a/skills/freview/references/configuration.md b/skills/freview/references/configuration.md index 1835f33..ada7e87 100644 --- a/skills/freview/references/configuration.md +++ b/skills/freview/references/configuration.md @@ -50,3 +50,7 @@ The project may document checks/QA requirements in existing `AGENTS.md`, contrib ### Future schema changes Version the schema and add migrations only with review. Reject unknown configuration versions rather than silently guessing. Do not auto-create `.freview.json` in someone else's repository. + +## Separate opt-in operations policy + +The independent `.freview-ops.json` file is **not** an extension of `.freview.json`. Its [versioned JSON Schema](../assets/schemas/freview-ops.schema.json), [safe default example](../../../examples/freview-ops.example.json), and [operations manual](optional-ops.md) cover deliberately enabled worker/reviewer commands, tests, mutation quality, execution budgets, and local Git hooks. Its contents never grant execution privileges by themselves: using an executable module requires an explicit user-requested workflow, a trusted checkout, an approved scratch location and the appropriate execution/credential flags. Regular Standard and Full audits remain read-only. diff --git a/skills/freview/references/optional-ops.md b/skills/freview/references/optional-ops.md index 332527c..8794ca6 100644 --- a/skills/freview/references/optional-ops.md +++ b/skills/freview/references/optional-ops.md @@ -4,7 +4,7 @@ ## Modules and defaults -All nine modules are **disabled** unless a project explicitly enables them: `specs`, `planning`, `implementation`, `tests`, `adversarial`, `mutation`, `regression`, `hooks`, and `shipping`. This policy is read separately from `.freview.json` to avoid changing the established read-only audit contract. +All nine modules are **disabled** unless a project explicitly enables them: `specs`, `planning`, `implementation`, `tests`, `adversarial`, `mutation`, `regression`, `hooks`, and `shipping`. This policy is read separately from `.freview.json` to avoid changing the established read-only audit contract. The canonical [operations schema](../assets/schemas/freview-ops.schema.json) defines accepted settings and their defaults. Its structure is separately tested against the runtime validator; **schema validity is not execution authorization**. | Module | When explicitly enabled | Boundaries | | --- | --- | --- | diff --git a/tests/test_ops_schema_contract.py b/tests/test_ops_schema_contract.py new file mode 100644 index 0000000..c89c248 --- /dev/null +++ b/tests/test_ops_schema_contract.py @@ -0,0 +1,69 @@ +"""Standardization contract for freview's separate opt-in operations policy. + +The built-in schema must reflect the validator; tests do not install jsonschema. +""" +import importlib.util +import json +from pathlib import Path +import unittest + +ROOT = Path(__file__).resolve().parents[1] +SCRIPTS = ROOT / "skills" / "freview" / "scripts" +POLICY_SCHEMA = ROOT / "schemas" / "freview-ops.schema.json" +PACKAGED_SCHEMA = ROOT / "skills" / "freview" / "assets" / "schemas" / "freview-ops.schema.json" + +spec = importlib.util.spec_from_file_location("freview_ops_standardization", SCRIPTS / "ops.py") +ops = importlib.util.module_from_spec(spec) +spec.loader.exec_module(ops) + + +class OperationsSchemaContract(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.schema = json.loads(POLICY_SCHEMA.read_text(encoding="utf-8")) + + def test_packaged_schema_matches_repository_copy(self): + self.assertEqual(POLICY_SCHEMA.read_bytes(), PACKAGED_SCHEMA.read_bytes()) + + def test_schema_is_strict_and_versioned(self): + self.assertEqual(self.schema["$schema"], "https://json-schema.org/draft/2020-12/schema") + self.assertEqual(self.schema["properties"]["schema_version"]["const"], 1) + self.assertFalse(self.schema["additionalProperties"]) + self.assertIn("schema_version", self.schema["required"]) + self.assertEqual(set(self.schema["properties"]), { + "schema_version", "modules", "commands", "limits", "pass_env", + "mutation_quality", "autonomy"}) + + def test_modules_and_stages_match_runtime_exactly(self): + modules = self.schema["properties"]["modules"]["properties"] + commands = self.schema["properties"]["commands"]["properties"] + self.assertEqual(set(modules), set(ops.MODULES)) + self.assertEqual(set(commands), set(ops.STAGES)) + self.assertTrue(all(props.get("default") is False for props in modules.values())) + self.assertFalse(self.schema["properties"]["modules"]["additionalProperties"]) + self.assertFalse(self.schema["properties"]["commands"]["additionalProperties"]) + + def test_defaults_and_runtime_validation_match(self): + sample = json.loads((ROOT / "examples" / "freview-ops.example.json").read_text()) + clean = ops.validate_policy(sample) + self.assertFalse(any(clean["modules"][name] for name in + ("implementation", "tests", "adversarial", "mutation", + "regression", "hooks", "shipping"))) + self.assertEqual(clean["mutation_quality"], sample["mutation_quality"]) + self.assertEqual(clean["autonomy"], sample["autonomy"]) + for field in ("mutation_quality", "autonomy"): + names = self.schema["properties"][field]["properties"] + self.assertEqual(set(names), set(clean[field])) + self.assertTrue(all(names[name].get("default") == clean[field][name] + for name in names)) + + def test_readme_no_longer_calls_merged_pr_a_draft(self): + readme = (ROOT / "README.md").read_text(encoding="utf-8") + self.assertNotIn("In-progress enhancement:", readme) + self.assertIn("evidence reconciliation", readme.lower()) + self.assertIn("optional operations", readme.lower()) + self.assertIn("read-only", readme.lower()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_skill_layout.py b/tests/test_skill_layout.py index 580be54..eb683c0 100644 --- a/tests/test_skill_layout.py +++ b/tests/test_skill_layout.py @@ -30,6 +30,7 @@ def test_references_and_helper_are_bundled(self): self.assertTrue((SKILL_DIR / "scripts" / tool).is_file()) for asset in ("assets/schemas/freview-config.schema.json", "assets/schemas/freview-report.schema.json", + "assets/schemas/freview-ops.schema.json", "assets/catalog/specialists.json", "assets/templates/github-actions/freview-daily.yml"): self.assertTrue((SKILL_DIR / asset).is_file()) @@ -38,6 +39,7 @@ def test_bundled_assets_match_repository_documents(self): for package_path, source_path in ( ("assets/schemas/freview-config.schema.json", "schemas/freview-config.schema.json"), ("assets/schemas/freview-report.schema.json", "schemas/freview-report.schema.json"), + ("assets/schemas/freview-ops.schema.json", "schemas/freview-ops.schema.json"), ("assets/catalog/specialists.json", "catalog/specialists.json"), ("assets/templates/github-actions/freview-daily.yml", "templates/github-actions/freview-daily.yml"),