verification.md

Verification

The toolkit counts something as verified only after it has run it. This page explains what that means for each stage, when a stage stops instead of continuing, and how checks are reported.

The rules

1. Execution is the evidence. A fix read in the diff is present, not verified. cc-verify reports what ran, what passed, what failed, and what couldn't be verified, and why.
2. Unverified stays unverified. It is reported with its concrete reason and carried into the verdict and residual risk. "Environment unavailable" with no cause isn't an acceptable reason.
3. Nothing is invented. Commands come from what the project defines (lockfiles, manifests, task runners, CI). If a command doesn't exist, it isn't run.
4. Verification doesn't edit code unless you ask.
5. A pass that couldn't run isn't a passed check. A missing delegated skill is reported as a degraded pass.
6. Compacted output isn't evidence. A stage judges the whole diff and cites uncompacted output. When it can't obtain either, the affected part is unverified. See Complete evidence.

Running the suite across agent hosts

Tests that exercise worktree selection or executor dispatch clear inherited HERDR_*, CODEX_*, CLAUDE_*, and ORCA_* variables, plus CODE_CYCLE_HOME, XDG_CONFIG_HOME, and APPDATA. A test that needs one of those settings sets it explicitly in its own scope. CODEX_HOME points to an empty temporary directory during those modules so Codex auth checks cannot fall through to the developer's home directory. This keeps a developer's host selection, credential path, and telemetry location from changing unrelated test behavior. Herdr is optional when running the suite: with HERDR_ENV unset, worktree selection uses Git directly; dedicated Herdr tests opt into the flag and mock the command.

Credential-read confinement (E7)

The unit suite checks every native stage profile's credential rules, explicit exceptions, canonical symlink targets, and each missing-denial failure in probe_review_confinement. The installed Codex confinement probe and srt prerequisite probe also test synthetic credential stores without reading host secrets. Claude settings with inherited allowRead are refused because that setting could cancel a denial.

On a host that supports native sandbox execution, run the opt-in E7 matrix:

CODE_CYCLE_LIVE_CREDENTIAL_PROBE=1 python3 -m unittest discover -s tests -p test_credential_reads.py

This requires installed codex and srt. It exercises read/write, publishing, registry and Docker-grant profile combinations on both executors, prints only READABLE/denied states for synthetic stores, and fails on any unexpected read. Publishing profiles must allow synthetic gh credentials and deny the five other stores. No host credential contents are accessed. Preserve the output and platform/version information as boundary evidence. macOS/Windows still need platform validation before removing their Codex preparation fallback.

Do not opt in inside a native stage whose outer sandbox cannot support nested Codex or srt. The test skips there; record that boundary as not_verified, and keep unit results separate from real sandbox evidence. Static-only recovery when the sandbox is unavailable does not establish credential confinement; verification remains disabled under the existing recovery policy.

Codex Git metadata sandbox

Writing Codex dispatches from linked worktrees need some of the worktree's Git metadata, which lives outside the workspace: Git stores objects, refs, and reflogs in the common directory, and the selected worktree's index, HEAD, and pseudorefs in its private directory. Read-only dispatches receive none of it.

No side effects. Sandbox setup and teardown leave .git exactly as it was, apart from the Git changes the stage makes on purpose. Codex's Linux sandbox mounts each filesystem rule on its destination and creates a missing destination as an empty file, which outlives the dispatch: an empty commondir makes Git treat an ordinary checkout as a broken linked worktree, and an empty MERGE_HEAD reports a merge in progress (#229). So no rule names a path that does not exist when the profile is built, and nothing is cleaned up afterwards, because a sweep of .git cannot tell a synthetic file from one Git or another dispatch created meanwhile.

Read-only common root. The common directory is read-only. Write access goes only to its objects, refs, and logs directories and to the worktree's private directory; those are also the only --add-dir roots. The read-only root protects, without naming them, config, hooks, packed-refs, commondir, gitdir, modules, other worktrees' pointers, and the main worktree's HEAD, index, lock files, and merge, rebase, bisect, and sequencer state. Rules then make read-only only what exists: the private commondir and gitdir, config.worktree and its lock, the workspace's .git gitfile, and the main worktree's logs/HEAD, refs/bisect, refs/worktree, and refs/rewritten with their reflogs, which sit below the writable refs and logs. Git follows commondir and gitdir to find configuration and hooks, and the gitfile keeps a later dispatch's grants on this worktree, so all three must also stay readable; the probe reads them. Lock files stay creatable inside the writable directories, which is how Git updates refs.

What works and what does not. commit, fetch, push, and loose ref updates work. Anything that must rewrite a root-level file fails inside the sandbox: deleting a packed ref, pack-refs, garbage collection, and fetching into a shallow clone, which updates shallow. Writing Codex publish dispatches set GIT_CONFIG_PARAMETERS with gc.auto=0 and maintenance.auto=false so automatic maintenance does not try. The common refs remain writable for publication across branches; this filesystem boundary does not restrict ref names, so stage instructions and GitHub branch protections remain part of the publication boundary.

Routing away. A dangerous absent path cannot be protected without creating it, so before dispatch the runtime routes the stage to another eligible publisher, or stops that publishing operation with the reason when none is available, when:

• extensions.worktreeConfig is on and the worktree's config.worktree or config.worktree.lock is absent;
• the effective core.hooksPath, read from local and worktree-specific configuration and resolved from the worktree's top level when relative, lies inside a writable Git directory. The worktree's private hooks is not a trigger, because Git never reads it, and the default common hooks sits in the read-only root. A hooks path inside the writable source workspace stays outside these rules;
• the stage would publish from the main worktree, whose index and HEAD live in the read-only common root. The cycle only grants Git metadata to linked worktrees, so this case arises only through a direct call.

Absent main-worktree refs/bisect, refs/worktree, refs/rewritten, and their reflogs are accepted and left out.

Codex's own carve-outs. Codex itself protects .git, .codex, .agents, and .aws below every writable root, and has been observed creating them as empty directories in objects, refs, and logs. They come from the CLI rather than from these rules, and Git ignores those dot-prefixed names in all three directories.

Testing it. tests/test_codex_git_sandbox.py runs the real profile through a fake codex that creates every missing rule destination, as the Linux sandbox does, and compares .git before, during, and after the probe and the dispatch. It also runs commit, fetch, push, and a loose ref update with the common root made read-only by directory permissions. A live Codex CLI is not available in CI. To check a supported CLI by hand on Linux, run a writing publish stage from a linked worktree, then check that find "$(git rev-parse --git-common-dir)" -newer <marker> lists nothing outside objects, refs, logs, and the worktree's private directory, where <marker> is a file touched just before the dispatch.

Before granting linked-worktree metadata, the runtime ignores Git directory, worktree, index, and object-directory environment overrides. It also checks that the workspace .git file, private gitdir and commondir pointers, and Git's resolved paths agree and that the private directory is registered below the selected common directory's worktrees directory.

Before dispatching any writing publish stage to Codex, the runtime runs a deterministic local command with codex sandbox and the same filesystem rules. It writes and verifies temporary markers in each granted root, checks that a uniquely named marker cannot be created in the common root, worktrees, or hooks, and that each existing protected file refuses writes, without a model turn or network access. It never opens an absent protected name, and it removes only markers that still carry its own token, after success, failure, timeout, or interruption. A failed probe routes the stage to another eligible publisher before dispatch and records the specific failure in the route reason. Land issue #203 before issue #190, whose executor refactor reorganizes the dispatch path this fix covers.

Complete evidence

A host can change what an agent sees of a command's output. A command-rewriting hook may compact git diff, test runs, or other output before the agent reads it, and the only trace is a marker in the middle of the text. A review that approves a shortened diff has not read the change it approved.

Measured on this repository with RTK 0.48, whose Claude Code hook turns git diff into rtk git diff transparently (git diff b9dbc54 91ba1dd, the PR #74 merge):

Command	Lines
git diff through the hook	495, with markers such as ... (100 additions truncated)
raw (rtk proxy git diff)	764
git diff --output=<file> through the hook	764 (the file is written by Git, unfiltered)

In the same test, gh comment reads and exit codes were intact. The rule has two layers:

• The runtime supplies the diff. When run_cycle dispatches review, resolve, or rereview, it writes git diff <base>...HEAD itself, through subprocess and --output, which no host hook rewrites. The base is the first of change_bases_of that resolves, the same resolution the change signals use. The file goes in a fresh directory under evidence/, beside the telemetry database (CODE_CYCLE_HOME, or the user configuration directory). That is outside the workspace and Claude's disposable review clone. It is also outside the system temporary directory, which Codex's workspace-write sandbox may write. If CODE_CYCLE_HOME puts it inside the workspace, /tmp, or $TMPDIR, no file is written and the prompt says so. The prompt names the file, the base, merge base and head SHAs, the line count, and the SHA-256. The file is removed when the stage ends. When no base resolves, no file is written and the prompt says so; the stage never receives a partial file.

  How far the evidence is protected depends on the executor:

  Executor	Reads the file	Can the stage change it?
  Codex, any stage	Its sandbox reads the whole file system	No; the operating system sandbox writes only its workspace and temporary directories
  Claude, non-writing stage	Granted the directory with --add-dir	Not prevented. Claude has no sandbox here, and an added directory is editable
  Claude, writing stage	Not granted; falls back to the skill's rules	Not granted, but not sandboxed either

  For every executor the file is read-only, and after the stage the runtime checks its SHA-256 again. A changed or missing file stops the cycle as stage_not_completed before the stage's verdict is read. What this can't catch is a Claude stage that changes the file and restores it before it exits. For a review stage that matters, because the cycle acts on its verdict. No runtime can prove which bytes an agent read. For Codex the sandbox makes this case impossible. For Claude it is an accepted limit: Claude reviews remain trusted, with their capabilities unchanged and no secondary confirmation required. They are not runtime-verifiable against an edit to the evidence that is restored before the stage exits.

• The evidence skills state the rule. The eight evidence skills carry the shared ## Workspace tools and evidence section, including its ### Complete evidence subsection, for manual runs and for evidence other than the diff. A truncation or summary marker, or output shorter than its own header counts, makes the output incomplete. Diffs are read from a file Git wrote, or through the host's raw mode. When complete output can't be obtained, the affected part is unverified, never checked.

This section decides whether output is complete. Evidence levels decides who produced it; the two don't overlap.

Verification plan and tests version 2

New reports use tests.version: 2 and tests.checks, replacing the legacy top-level tests.evidence list. This is separate from the unchanged code-host checks.items object. The enclosing result supplies head_sha.

{
  "version": 2,
  "conclusion": "verified",
  "boundary": "not_required",
  "checks": [{
    "id": "configured-test", "command": "python3 -m unittest",
    "executor": "codex", "profile": "deep_coder", "context": "project",
    "required": true, "criteria": ["AC-1"], "level": "test",
    "provenance": "project", "reason": "configured test command",
    "head_sha": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "result": "pass", "cause": "", "attempts": [],
    "evidence": [{"exit_code": 0, "executed": 14}]
  }]
}

Every entry in a check's evidence and attempts arrays must be a JSON object. A non-object entry (including a string) is malformed; the runtime rejects the report and caps the conclusion at not_verified rather than coercing text into evidence.

Freeze each check before execution. A result names the same check ID, command, executor/profile and context. The runtime supplies frozen check identities to later code stages. Copy that identity, including its owning executor/profile, even when resolve or rereview is dispatched with another profile or executor. A check requiring a different execution context gets a new ID and does not replace the existing obligation. Required checks originate in project policy and criterion links; an agent may promote, never demote. Record an unexecuted check as result: unverified, with cause, evidence and actual recovery attempts; do not invent an exit code. Results on another HEAD stay in history but cannot prove the current change. An uncovered criterion remains open. Required failure caps tests.conclusion at failed; required unverified or open criteria cap it at not_verified. These caps never improve an agent conclusion.

Before implementation, the runtime reads the GitHub work-item body once per cycle and parses one explicit criteria section. Accepted headings are exactly Acceptance criteria and Criterios de aceptación, case insensitive, as a Markdown heading (# through ######) or bold label. A trailing colon or period is allowed inside a bold label or after it. The section is a flat list at column zero using -, +, *, 1. or 1) markers and optional Markdown checkboxes; another heading or a bold label ends it. Punctuated labels may include prose on the same line, such as **Label.** text or **Label:** text. Bold markup inside a list item does not end the section. Empty, missing, duplicate, placeholder or unparseable sections produce deterministic needs_refinement before any implementation, even with issue review off or difficulty 1. They are insufficient specification, never a failed test. Existing issues are not migrated. .github/ISSUE_TEMPLATE/work-item.md supplies the grammar for new work. IDs are AC-n in order, scoped to the execution-plan revision and content hash. Plane and Jira return not_evaluable with reason criteria reader unavailable. Their cycles stop as BLOCKED with stop_reason: readiness_unconfirmed before any agent dispatch, regardless of issue-review mode or difficulty. Editing the work item cannot clear this stop: a provider reader must become available (Plane reading belongs to #238-A; Jira needs its own reader). This differs from parsed criteria needing refinement; neither unavailable nor unmet criteria count as satisfied.

For one release, omitted tests.version or version 1 accepts tests.evidence. The compatibility interval starts with the first published release supporting version 2 and ends at the next published release. With runtime obligations, the adapter matches exact commands and preserves missing required checks as explicit unverified results; absence never creates a pass. Legacy standalone reports retain their historical evidence classification. Version 2 evidence is normalized into a private compatibility view for tests_basis, boundary_verified, telemetry and stats; this view is not a second result contract. Malformed or unsupported reports cap at not_verified. Stages that omit tests or report tests.reason: no_code_change without outcome fields record no new test outcome and preserve the previous cycle-level outcome. Explicit reports (including checks, evidence, conclusion, passed or ran) still receive the deterministic caps, even with no_code_change; a required boundary run can still be recorded as missing for completed code work.

The native driver's final barrier prevents READY_FOR_MANUAL_MERGE while a required check is unverified, criteria remain uncovered, or required security verification was omitted. It stops once with HUMAN_INTERVENTION and stop_reason: verification_unverified. Failed checks retain their failure; blocked, omitted, missing-tool, inconclusive and environment causes remain distinguishable metadata. CI_PASSED never turns an unverified local check into pass. See Execution plan for persistence and receipt identity.

Resume with --from rereview --pr <n>. When the current HEAD, recoverable plan and trusted receipt agree on a verification stop, the runtime dispatches only cc-verify, once, in a detached disposable clone without remotes. It passes only pending required checks and retains passing results. It does not dispatch the rereview agent or republish a stage comment. A still-unavailable environment stops again without a retry loop. A moved HEAD, changed criteria, unbound command hash or unrecoverable plan takes ordinary reconciliation instead. This does not add a CLI start point or change START_ORDER/CYCLE_STARTS.

Hotfix requires hotfix-reproduction and hotfix-regression, both required, with distinct focused commands proposed by the implementer. The configured general suite satisfies neither identity. A missing proposal stops with needs_refinement; a defined check that cannot execute stays unverified. The implementer commits the reproduction before the fix. Under the reproduction check's history, it reports the defect failure's SHA, command, environment and cause: defect, preserving its output artifact. Evidence carries test_sha256, context_sha256 and output_sha256; the passing result on the fixed HEAD must have the same test and environment fingerprints. An import or environment failure is not a defect reproduction. Multiple results share one check ID; only evidence for the current HEAD determines its current result. Changing the test requires a new justified and revalidated proof, not replacing the old one.

Evidence levels

Every recorded test outcome has one tests_basis token. The functional conclusion is separate: it describes the agent's judgement, while the basis says who produced the evidence that judgement rests on.

tests_basis	Meaning	Example	Source
claimed	No executed command is attached	"tests": { "passed": true }	The agent
agent_reported	The result includes a command, exit status, and count, as reported by the agent	pytest tests/test_x.py → exit 0, 14 run	The agent; auditable in the comment, not proof
runtime_observed	The runtime observed the fact itself	Executor exit code, head SHA, or a command the runtime ran	The runtime
externally_verified	A system the agent does not control owns the fact	Checks on the pushed head SHA read by the runtime	The code host

Evidence entries use level: static or level: test to describe the command; these are command categories, not tests_basis values. cc-verify emits the entries and one conclusion token: verified, verified_with_reservations, not_verified, or failed. The conclusion is the agent's judgement and does not raise its evidence level. Missing or contradictory evidence never becomes a verified test outcome. Older telemetry rows without tests_basis are read as claimed.

Boundary verification

Some changes need one run where the code is actually used, not only a green suite. cc-verify decides it with the security gate's shape:

boundary_required = deterministic_rule OR verifier_judgement

The rule fires on changed paths classified as api, auth, database, or migrations, the same classes as the touches_* change signals (stage_signals.boundary_rule_areas). The verifier may add CLI entry points, subprocess and adapter code, provider integrations, and runtime configuration; it cannot remove what the rule fired. cc-verify lists what counts as a boundary run for each area, and boundary runs write only to the disposable workspace.

A required boundary run appears as an evidence entry with level: boundary:

{
  "conclusion": "verified",
  "boundary": "required",
  "evidence": [
    { "level": "test", "command": "python3 -m unittest tests.test_config", "exit_code": 0, "executed": 9 },
    { "level": "boundary", "command": "python3 -m app.cli sync --config config.yml --dry-run", "exit_code": 0, "executed": null }
  ]
}

A non-zero boundary command fails the test outcome like any other command. An expected failure, such as the malformed-configuration case, is recorded as a command that checks the failure and exits 0 when it occurs. When the run is required and no successful boundary entry exists, the runtime records boundary_verified: false and caps a verified conclusion at verified_with_reservations; the reason (environment, cost, or out_of_scope) is listed under Not verified, where reviewers already read unverified items and residual risk. When the driver observed the change, the runtime applies the path rule itself, so a verifier that states not_required for a migration still gets boundary_verified: false. A stage that completed code work but reported no test outcome also records boundary_verified: false when a run was required. Without a test report, a review, a blocked stage, or a no_code_change resolution records nothing. boundary: not_required records nothing new. The runtime never runs a boundary check.

What each stage verifies

Stage	Verifies
cc-implement-issue	The narrowest related tests first, then the repository's required verification through cc-verify
cc-initial-review	Reproduces actionable findings when the environment allows it. Findings are marked as observed at runtime or found by code inspection.
cc-resolve-comments	Reproduces each original finding, applies the fix, and confirms it no longer reproduces. Then cc-verify over the fixes and related regressions.
cc-rereview	Reproduces every finding claimed as fixed. Only those it could verify become resolved.

Depth follows the change:

• Authorization: both the allowed and the denied case.
• API contracts: status, payload, relevant headers, and error responses.
• Frontend: the affected flow in a browser, with console and network checks, desktop and reduced viewports, and loading, empty, error, disabled, and success states.
• Persisted data: migration state is checked before tests run.

Stop conditions

cc-verify and cc-run stop and ask instead of marking something "not verified" and continuing when:

• runtime configuration is missing (for example, only an .env.example exists);
• a required database or service is unavailable;
• a needed port is held by an unknown process;
• migrations are pending that could change the result;
• permission to run a needed command is missing.

When one of these could be cleared by creating something (a config from its template, a local database, a results directory), the skill describes exactly what it would create and asks first.

Before invoking cc-verify, the cycle skills preflight these conditions without changing state. If preflight fails, verification isn't attempted, and the reason is recorded. In a resolution thread that reads, for example:

Fix applied. Verification pending: environment unavailable (PostgreSQL on :5432 not running).

Where verification runs

Stages that only verify or start services write their artefacts to a disposable workspace, never to the source tree. Reviews are read-only. See Role workspace policy.

cc-run starts services in dependency order and reports each one's process, PID, port, URL, and stop command. It never picks another port silently and never kills a process it didn't start.

CI reporting

Every review reports the checks the repository actually defines, by their real names, and every state that isn't a pass: pending, failed, cancelled, skipped, and missing. A check the repository normally runs that didn't run on this PR is a finding, not a pass. When a repository policy skips a suite (for example, end-to-end tests on draft PRs), the review states it. skipped is never reported as passed.

For GitHub the reviewers use:

gh pr view <n> --json isDraft,headRefName,baseRefName,labels,statusCheckRollup
gh pr checks <n>          # output kept even on non-zero exit: it is evidence

Stages that push wait for their head

A local pass isn't the change request's result. cc-implement-issue after it opens the PR, and cc-resolve-comments after it pushes a fix, wait for the checks of the head SHA they pushed, never an earlier run's, while any is queued or running. The wait is bounded by the repository's own timeout, or about 15 minutes when it states none. Their shared section Checks of the pushed head holds the rule.

Checks of the pushed head	Result
every required check passed	the local result stands
failed within the stage's scope (its own code or tests)	the stage fixes it, pushes and waits again, at most two rounds unless the repository sets its own limit; these rounds aren't cycle iterations
failed outside the stage's scope (infrastructure, an unrelated flaky suite)	PARTIALLY_RESOLVED for a resolution, BLOCKED for an implementation, naming the check and why
still pending at the timeout	the local status stands, and every pending check is named; none is reported as passed

Before pushing, the stage tests the way CI will when that's cheap: no reliance on the global Git identity, ~/.gitconfig, cached credentials, or anything else only the developer's machine has. The summary states which environment assumptions its local run shares with CI and which it doesn't.

The published summary lists the final head's checks by name and state with the SHA they ran on. The structured result carries them as checks: head_sha, the counts passed, failed (failed, cancelled, or expected but missing) and pending, and the named items.

In telemetry

When the runtime drives the cycle, each tests_passed value is stored beside its tests_basis; the telemetry gate rejects a row that has one without the other. A boolean passed without evidence is claimed. A well-formed evidence list with at least one successful test entry and no failing command is agent_reported. Missing or contradictory evidence fails closed. A stage that reports ran: false records no test outcome. A justified no-code resolution uses "tests": { "ran": false, "reason": "no_code_change" }, never passed: true.

cc-stats groups outcomes by basis. It labels claimed as claimed and agent_reported as reported; it uses “verified” for test outcomes only when their basis is runtime_observed or externally_verified. Agent conclusion tokens are shown separately from those evidence levels. It also shows the boundary verification rate among the cycles that required a boundary run.

A verdict row also records checks_passed, checks_failed and checks_pending when the stage reported all three counts for its own head. Counts for a different head, or an incomplete set, record nothing. cc-stats sets each implementation and resolution head against its CI, so it can show how often a RESOLVED head was actually green. The check names stay in the PR comment. See Telemetry.

---

← Review lifecycle · ↑ Documentation index · Routing and models →