feat(manifest): run manifest coverage contract for review (#367) - #520
feat(manifest): run manifest coverage contract for review (#367)#520Gongyl01 wants to merge 9 commits into
Conversation
First slice of issue alibaba#367 (run manifest coverage contract): the data model and state machine only. Not yet wired into the agent or CLI, so existing review/scan output is unchanged. Introduce the versioned, immutable RunManifest (schema ocr.run-manifest/v1) and a concurrency-safe ManifestBuilder that tracks per-file coverage (selected/completed/reused/failed/waived) and freezes into a terminal state. - terminal state derived solely from coverage sets, never comments/warnings (complete/partial/failed/skipped) - Finalize sweeps any undecided selected item to failed/unknown so no item is silently dropped - single-mutex builder: first terminal state wins, frozen after Finalize, nil-receiver safe - fixed failure classification enum with an unknown catch-all - redaction floor on failure/waive reasons (strip secrets, cap length) as a single write entry so callers cannot bypass it - 22 unit tests, race-clean Refs: issue alibaba#367 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address findings from the concurrency / JSON-contract / PR#306-coupling adversarial review of the manifest data model (still slice 1; not wired to agent or CLI). - SetSweepClass: Finalize can classify undispatched items as cancelled/budget instead of a blanket unknown (the one real model gap the review found) - ItemID(fingerprint)=SHA-256 canonical mint helper; an item_id is never a raw fingerprint, keeping the resume cross-reference explicit and mix-ups caught - sanitizeReason: strip control/ANSI chars, coerce valid UTF-8, redact quoted secret values, guarantee single line - Finalize returns deep-copied coverage slices so the frozen snapshot is never aliased across the two outlets - RegisterSelected: nil-safe (lazy-init map) + documents that only the post-deletion/post-filter dispatchable set may be registered +7 unit tests (29 total), race-clean. Refs: issue alibaba#367 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ure (shard ②d) - Freeze per-mode input identity (mode + resolved_base/head + exact_range + source_artifact_sha256) via diff.ResolveInput/commitParents, and repository identity via RemoteIdentity/canonicalRemote (credential-free). - Add rule_config_sha256 and runtime_config_sha256 over an allowlist of non-secret fields using a length-prefixed SHA-256 framework (no tokens/URLs). - Replace SetRunLevelFailure(bool) with structured SetRunFailure(class, reason) and set ManifestInput.mode; fill execution.* (ocr version, provider, model, concurrency, config hashes). - Thread error returns through Finalize/WriteSessionEnd (main review path surfaces them; skip/all-failed/scan paths hardened in follow-up). - Tests: manifest_hash, canonical_config, git_resolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lassification Merged review themes A/B/E from the 07-22 consolidated assessment. Theme A — Finalize / session_end delivery errors no longer swallowed: - agent.go no-files path returns the Finalize error instead of nil (A1) - agent.go loadDiffs failure joins the Finalize error via errors.Join (A2) - session.Finalize uses sync.Once + cached finalizeErr: written exactly once, concurrency-safe, and every caller replays the same result so a retry cannot falsely report success (A3) - scan/agent.go wires both Finalize call sites to surface the error (A4) Theme B — canonicalRemote rewritten (internal/diff/git.go): - keep the port (u.Host, not u.Hostname) so endpoints differing only by port stay distinct (B1) - split scp syntax on the first ':' so an '@' inside the path survives (B2) - recognize local/file/Windows/UNC remotes and omit identity rather than misparsing a path as a host (B3; local-remote policy still open) Theme E — main_task-empty is now a sentinel (errMainTaskEmpty) classified via errors.Is instead of matching error text. Theme D (TOCTOU) deferred to shard 4 per issue alibaba#367 open-issues OI-12. Tests: go build ./... + go vet + go test ./... all green (23 pkgs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mal path The success-path Finalize wiring used `ferr != nil && err == nil`, so when the review (or scan) failed AND session_end also failed to persist, the persistence error was dropped and only the dispatch error surfaced — the caller never learned the session/manifest was not saved. Join both with errors.Join when both occur (matching the loadDiffs path), so a persistence failure is always reported even alongside a dispatch failure. This closes the last gap in the OI-10 contract. - internal/agent/agent.go: review normal path - internal/scan/agent.go: scan normal path (+ errors import) Tests: go build ./... + go vet + go test ./... all green (23 pkgs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 使用冻结 manifest 统一 review JSON、文本与退出状态\n- session CLI 和 viewer 展示五集合覆盖并兼容 legacy/aborted\n- 补充本地 mock、跨出口一致性及安全验收用例
验收用例:configuration 分类(run 级 sweep + item 级映射)、budget/timeout/panic 混合 partial 隔离、跨出口一致性改为规范化原始字节比对、flag 校验失败无产物断言。 代码修复:sanitizeReason 先剥控制字符再脱敏(堵控制字节绕过)、失败项异分类二次标记报冲突错误、source_artifact_sha256 按 item_id 去重并稳定排序、sortItems 改 SliceStable 对齐设计用词。 全仓 go test 23 包通过。
# Conflicts: # internal/llmloop/loop.go
覆盖 issue alibaba#367 验收标准 provider transition:resume 时 provider/model 改变后,子 manifest 记录当前值而非继承父运行,并经 parent_run_id 链接父会话以支持审计。用 mock client,不依赖真实 provider key。
|
🔍 OpenCodeReview found 7 issue(s) in this PR.
|
| if manifest != nil { | ||
| out.Status = string(manifest.TerminalState) | ||
| out.Message = manifestMessage(manifest, len(comments)) |
There was a problem hiding this comment.
Breaking API change: When a manifest is present, the JSON status field changes from "success" to the manifest's TerminalState value (e.g., "complete", "partial", "failed"). Downstream consumers that check for status == "success" will no longer match successful reviews. Consider documenting this as a breaking change in release notes, or maintaining backward compatibility by mapping StateComplete → "success" if there are existing consumers of this JSON output.
| if manifest != nil && manifest.TerminalState == session.StateFailed { | ||
| return errors.New("review failed: run manifest terminal state is failed") | ||
| } |
There was a problem hiding this comment.
When runErr == nil but manifest.TerminalState == StateFailed, the error message lacks diagnostic detail. Consider including the RunFailure classification and reason (when available) or the count of failed items so the stderr message gives the user actionable context without requiring them to re-run with JSON output.
For example:
if manifest.RunFailure != nil {
return fmt.Errorf("review failed (%s): %s", manifest.RunFailure.Classification, manifest.RunFailure.Reason)
}
return fmt.Errorf("review failed: %d of %d selected item(s) failed",
len(manifest.Coverage.Failed), len(manifest.Coverage.Selected))| if s.RunManifest != nil { | ||
| return string(s.RunManifest.TerminalState) | ||
| } |
There was a problem hiding this comment.
If a persisted manifest has an empty or unrecognized TerminalState (e.g., due to data corruption or a future schema version), this will return an empty string, resulting in a blank status column in the table and detail views. Consider adding a fallback for safety:
if s.RunManifest != nil {
if ts := s.RunManifest.TerminalState; ts != "" {
return string(ts)
}
return "unknown"
}This is defensive — Finalize always sets a valid state, but data loaded from disk could be malformed.
| // StopEmptyRounds — the model returned no usable tool result for too many | ||
| // consecutive rounds. Not a declared budget; unclassifiable. | ||
| StopEmptyRounds | ||
| // StopCompression — context compression exceeded its threshold, so the loop | ||
| // could not continue. Token/context driven but not a declared budget. | ||
| StopCompression |
There was a problem hiding this comment.
Missing test coverage: there are no tests verifying that StopEmptyRounds or StopCompression are returned from their respective trigger points. The StopMaxRounds path is tested in TestRunPerFile_MaxToolRequestsWithoutTaskDoneDoesNotComplete, but the other two stop reasons should have analogous tests to prevent regressions.
For StopEmptyRounds: configure a fake client that returns tool calls with no valid results for maxConsecutiveEmptyRounds consecutive rounds, then assert stop == StopEmptyRounds.
For StopCompression: configure a scenario where addNextMessage returns false, then assert stop == StopCompression.
| for _, bi := range b.items { | ||
| if bi.state == stateSelected { | ||
| bi.state = stateFailed | ||
| bi.item.Classification = sweepClass | ||
| if bi.item.Reason == "" { | ||
| bi.item.Reason = sweepReason | ||
| } | ||
| } | ||
| } | ||
|
|
||
| cov := b.buildCoverageLocked() | ||
| if err := b.validateLocked(cov); err != nil { | ||
| return RunManifest{}, err | ||
| } |
There was a problem hiding this comment.
Bug: The sweep loop mutates items from stateSelected to stateFailed before validation. If validateLocked subsequently returns an error, the builder is left unfrozen but the sweep mutations are irreversible — those items can never be re-transitioned (the transition method rejects non-selected states). This means:
- A retried
Finalizeafter fixing the validation issue will find nostateSelecteditems to sweep, so the sweep classification/reason from the first (failed) attempt is permanently baked in. - Any
Mark*calls between the failed and retriedFinalizefor swept items will return errors rather than applying the correct outcome.
Consider either (a) moving the sweep after validation (build coverage, validate structural invariants that don't depend on the sweep, then sweep and rebuild), or (b) snapshotting item states before the sweep and restoring them if validation fails.
| func readJSONLLines(r io.Reader, visit func([]byte)) error { | ||
| reader := bufio.NewReader(r) | ||
| for { | ||
| line, err := reader.ReadBytes('\n') | ||
| if len(line) > 0 { | ||
| visit(line) | ||
| } | ||
| switch err { | ||
| case nil: | ||
| continue | ||
| case io.EOF: | ||
| return nil | ||
| default: | ||
| return err | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Bug: When ReadBytes('\n') encounters a non-EOF error (e.g., disk I/O error), it can return partial bytes along with the error. The current code visits these partial bytes before checking the error. In peekSession, this overwrites lastLine with corrupt data, potentially replacing a valid previous last line. If the file's last complete line was a valid session_end record but a subsequent partial read occurs, the session_end detection will silently fail because the partial data won't parse as JSON.
The fix is to only visit lines when there is no error, or at minimum skip visiting when the error is not io.EOF:
Suggestion:
| func readJSONLLines(r io.Reader, visit func([]byte)) error { | |
| reader := bufio.NewReader(r) | |
| for { | |
| line, err := reader.ReadBytes('\n') | |
| if len(line) > 0 { | |
| visit(line) | |
| } | |
| switch err { | |
| case nil: | |
| continue | |
| case io.EOF: | |
| return nil | |
| default: | |
| return err | |
| } | |
| } | |
| } | |
| func readJSONLLines(r io.Reader, visit func([]byte)) error { | |
| reader := bufio.NewReader(r) | |
| for { | |
| line, err := reader.ReadBytes('\n') | |
| if err != nil { | |
| if err == io.EOF { | |
| // Process any remaining bytes before EOF (last line without trailing newline). | |
| if len(line) > 0 { | |
| visit(line) | |
| } | |
| return nil | |
| } | |
| // Non-EOF error: discard partial data to avoid corrupting downstream state. | |
| return err | |
| } | |
| visit(line) | |
| } | |
| } |
| // fixed token ceiling. session_end embeds the complete run manifest and can | ||
| // legitimately exceed the former 10 MiB scanner limit on very large reviews. | ||
| func readJSONLLines(r io.Reader, visit func([]byte)) error { | ||
| reader := bufio.NewReader(r) |
There was a problem hiding this comment.
Performance: The stated motivation for this refactor is to handle lines exceeding the former 10 MiB scanner limit. However, bufio.NewReader uses a default 4096-byte buffer. For very large JSONL lines (multi-megabyte manifest records), ReadBytes will repeatedly grow its internal buffer through many small allocations and copies, causing significant GC pressure. Consider using bufio.NewReaderSize with a larger initial buffer to reduce allocation overhead for the expected large-line use case.
Suggestion:
| reader := bufio.NewReader(r) | |
| reader := bufio.NewReaderSize(r, 64*1024) |
Summary
Closes #367.
Adds a versioned, immutable
RunManifesttoocr reviewthat records per-item coverage (selected/completed/reused/failed/waived) and an authoritative terminal state (complete/partial/failed/skipped). On successful persistence, CLI JSON andsession_end.run_manifestexpose the same frozen manifest, so a partial review can no longer be mistaken for ordinary success or a complete no-findings result.Built on top of PR #306's resumable sessions; this is a narrow output-contract follow-up, not a parallel persistence system.
internal/session/manifest.go:RunManifestvalue object (schemaocr.run-manifest/v1),ManifestBuilder(single state table, unified transition entry,RegisterSelected/MarkCompleted/MarkReused/MarkFailed/MarkWaived),SealSelected(closes the selected denominator before dispatch),Finalize(elapsed) (RunManifest, error)with hard validation, structuredRunFailure+RunFailureClassenum, andsanitizeReasonredaction floor.internal/agent/agent.go: a pre-dispatch pass (registerCoverage) registers all post-filter, non-deleted items and seals them before resume reuse or goroutine dispatch;markCompleted/markReused/markFailedrecord per-item outcomes;SetRunFailurerecords run-level stop causes at the trigger source (inputfailure onloadDiffs,internalfailure onregisterCoverage).initManifestinitializes requested input and execution metadata; resolved input and repository identity are captured from the actual run and attached before finalization.internal/diff/git.go:ResolveInputfreezes resolved base/head/exact range per input mode;RemoteIdentity/canonicalRemoteproduce a credential-free repository identity by canonicalizing the origin URL tohost[:port]/path(dropping userinfo/query/fragment) and SHA-256ing the result; local remotes omit identity.internal/session/history.go:NewManifestBuildermounts the builder onSessionHistory;Finalizenow returnserror(was void) and usessync.Oncesosession_endis written exactly once while replaying the first error on every call.internal/session/persist.go:WriteSessionEndwritessession_endas the last physical JSONL record, embeds the frozen manifest underrun_manifest, and surfaces write/flush/close errors.cmd/opencodereview: review JSON exposesmanifest; top-levelstatususes the four terminal states (scanstays legacy); text output no longer showsLooks good to me.forpartial/failed;session list/showand the viewer prefersession_end.run_manifestand fall back to legacy without fakingcomplete.Design (key invariants)
Finalize. On successful persistence, CLI JSON andsession_end.run_manifestserialize the same frozen manifest value. JSON output no longer derives review status from warnings.SealSelectedcloses the selected denominator after the pre-dispatch pass; resume-reused and to-be-dispatched items enter the same frozen set, soselected = completed ∪ reused ∪ failed ∪ waivedalways holds.run_failureis recorded at the trigger source (input= diff resolution,internal= scheduler/invariant). It is never inferred fromctx.Err(). Contract-level recorders and sweeping forcancelled, run-leveltimeout, and run-levelbudgetare in place but have no live source in this release (no new SIGINT handler, global deadline, or run-level budget pool).completemay have zero or many findings.sanitizeReasonprovides a second redaction floor for URL credentials, Bearer/Basic tokens, credential-like assignments, and control characters. CLI output suppresses rawsubtask_errorwarnings when a manifest is present. Existing session checkpoint and conversation persistence behavior is unchanged.Finalize/WriteSessionEndreturn errors up the stack across all exit paths—the no-files path, theloadDiffs-failure path, the normal dispatch path, and the scan path. On the normal path, dispatch and persistence errors are reported viaerrors.Join. A persistence failure does not rewrite the frozen manifest; it surfaces as a delivery error with a non-zero exit code.parent_run_id; immutable-ref drift rejection is unchanged and remains out of scope.Terminal state
completefailedempty, norun_failurepartial0 < failed < selected, norun_failurefailedrun_failurepresentskippedrun_failurewaiveditems count as covered, so a run containing onlycompleted/reused/waiveditems iscomplete. Findings count never affects the state.scanis unchanged and continues to emit the legacystatusvalues; the two commands do not yet share the same status contract.How to test
Checklist
go build ./...succeedsgo test ./internal/session/... ./internal/agent/... ./cmd/opencodereview/...greengo vet ./...andgo fmt ./...cleaninput/internal), per-item timeout/budget/panic, skipped, resume (parent_run_id+ reused), provider transition (the resumed child records its current provider/model, linksparent_run_id, and preserves checkpoint reuse), cancellation (contract-level:run_failure=cancelledsweeps pending items tofailed(cancelled)),SealSelectedlifecycle, conflicting/idempotent transitions, invalid failure class, empty waiver reason, cross-exit manifest consistency, legacy/aborted session display, and security redactionmanifestandsession_end.run_manifestserialize the same frozen manifest valueOut of scope (deliberately)
scanis not wired to the v1 manifest (Non-Goal; it passes a nil manifest andemitRunResultis nil-safe) and keeps its legacystatusvalues.waivedoutput semantics.cancelled, run-leveltimeout, and run-levelbudgethave contract and unit tests but no live source in this release—no new SIGINT handler, global deadline, or run-level budget pool is added. The recorders, sweep, andFinalizecontract are complete so they activate automatically when those run controls are added later.parent_run_id; ref-drift rejection and "diff by immutable SHA" are deferred.complete.