feat(report)!: target commit, remediation, simplified Markdown, and an always-numeric spend estimate (#1268)

[aviggiano]

Summary

Implements #1268. The final report now:

• names the evaluated commit in a Commit row;
• no longer renders low-level diagnostic sections in report.md;
• ends every production issue with a ### Remediation section and renders backtick spans in finding prose as inline code;
• always shows a numeric Estimated spend, taken from accounting v4, with a trailing + when the amount is probably low.

The prompt, report contract, renderer, gates, ultrafuzz report diagnostics, docs and tests change together.

1. Commit. The Source run ID row is replaced by - Commit: <hash>. report.json run_metadata.target_commit is required: the run's source revision (run.json source_revision, the commit every task worktree is created from), a lowercase 40-hex string, or null when the run recorded none. null renders as - Commit: `none` (no Git commit was recorded for the evaluated target). source_run_id and source_run_ids stay in report.json only.
2. Simpler Markdown. report.md no longer renders ## Scoped coverage evidence or ## Artifact validation warnings.
   • report.json keeps coverage_evidence and run_metadata.artifact_validation_warnings.
   • A fixed sentence after the Run summary says when scoped coverage could not be measured, or was measured but incomplete.
   • The final-report gate now rejects a scoped coverage section or an exact-scope coverage score in report.md (REPORT_COVERAGE_EVIDENCE_MARKDOWN_UNEXPECTED) instead of requiring the section.
   • The public bundle requires both artifact-validation-warnings.json and .md whenever the report carries warnings.
3. Remediation and inline code.
   • Every production issue ends with ### Remediation: the source finding's recommendation, or a fixed sentence saying no remediation was recorded.
   • The report stage only carries recommendation. On a production row, adding, changing or omitting it is a verification error. On a non-production row, adding one is an error and changing or omitting one is a warning.
   • Finding prose renders backtick spans as inline code. Link, list, heading, HTML and fence injection stay neutralized. Index anchors use the visible title.
   • The public projection redacts private paths at more boundaries, so prose that now renders more literally cannot reveal one.
4. Spend. See "Spend rule" below.

What changed vs the original PR

• One estimator. The run.json#spend_estimate estimator is removed. There is no imputation, no default attempt usage, and no self-imputation of the report task. restated_run_metadata and the host-only projection copy are gone too. The amount is always accounting v4's accounting.cumulative.estimated_spend_usd, the same number ultrafuzz stats, evals and Modal read.
• Fallback prices live inside accounting v4. The versioned table ultrafuzz.fallback-pricing.2026-10-05 holds the models.dev list prices of that date for gpt-5.5, claude-opus-4-8, kimi-k3 and deepseek-v4-pro. v4 uses it for a model the catalog left unpriced, whether the catalog is available, unavailable or disabled. It looks models up through the same routes, so gateway and unknown IDs stay unpriced. The rates are pinned in pricing_catalog.model_prices, and pricing_catalog.fallback records the table and the models it priced.
• Missing usage is counted, not invented. run.json gains attempts_without_usage (attempts, cumulative_count), the executed agent attempts with no usage event of their own, including source runs. report.json run_metadata gains the optional counts attempts_without_usage and unpriced_attempts.
• Spend form is $X or $X+. Following the owner decision of 2026-10-06, report.md carries no note about excluded attempts. The structured counts stay in run.json and report.json, and they decide the +.
• target_commit comes from the run's source revision. refactor!: remove campaign data governance (#1269) #1275 removed the data-governance record the original PR read. The 64-hex form is gone.
• Recommendation is carry-only as an error for production rows, which fixes Greptile finding 2 (a changed recommendation could publish).
• Greptile finding 1 (a continuation without local usage lost its source spend) no longer applies. spend_estimate is gone, v4's cumulative accounting already includes source runs, and a continuation without its own accounting uses its source run's accounting.
• Path redaction. Home, private-directory, drive-letter and UNC paths are redacted at the same boundaries as rooted paths. A differential fuzz found 433 inputs that leaked at the original PR head. All 433 are fixed, and no new leaks or fail-closed projections appeared.
• ultrafuzz report diagnostics accept $0.00, never expect unavailable, and expect the + whenever run.json says the amount is probably low.
• Merged current unstable.

Spend rule

• Amount. Accounting v4's estimated_spend_usd, with two decimals from one cent up and four to ten decimals below one cent (for example $38.72 and $0.0012). It is $0.00 when nothing was priced and never unavailable.

• Pricing order. For each attempt's latest usage event, v4 uses the first of: the cost the adapter recorded, the price from the model's catalog route, or the fallback table.

• The +. The spend ends in + exactly when at least one of these holds:

  • partial_pricing is true;
  • attempts_without_usage is present (some attempts recorded no usage);
  • unpriced_attempts is present (some usage could not be priced).

  The + means "there is probably more". With no +, everything recorded was priced and every attempt recorded usage.

• Where it is enforced.

  • reportSpendIsProbablyLow and reportEstimatedSpend in @ultrafuzz/artifacts are the only place the + is decided.
  • The report-start projection, the terminal and unchecked restatements, and ultrafuzz report all use them. The restatements set the + again after they restate the counts.
  • The report@3 schema enforces the rule in both Ajv (if/then/else) and Zod. The pattern is ^\$(?:0|[1-9][0-9]*)\.[0-9]{2,10}\+?$.
  • The renderer requires the spend row to equal run_metadata.estimated_spend exactly.

• Accounting v4's own label is unchanged. Its estimated_spend ($x+ or unavailable) stays in run.json for stats, and benchmark analysis passes it through as before. It never reaches the report.

- Models used: `gpt-5.5`
- Tokens used: `41,203,118`
- Estimated spend: `$38.72+`

Breaking changes

• ultrafuzz/report@3 and the run.json schema change in place:
  • run_metadata.target_commit is required.
  • The spend pattern changes, and the schema checks the + against the counts.
  • run.json gains attempts_without_usage and pricing_catalog.fallback.
• Reports written before this release no longer verify. They also no longer render as unchecked reports, because they fail the schema.
• A controller from an earlier release rejects a run.json that this release has synchronized. Finish a run launched on an earlier release with that release.
• EVMbench grades the copied report.md, so its scores are not comparable across this release.
• Projects scaffolded earlier should delete .ultrafuzz/prompts/review/final-report.md and rerun ultrafuzz init.
• v4 now looks up newly priced models through route-exclusive catalog routes, so a model that an aggregator used to price can stay unpriced unless the fallback table lists it.

Tests

• Contract. Ajv and Zod agree on the spend pattern and on the + rule for each of the three conditions (contract-fixtures.test.ts). The shared formatter's output is valid with and without +.
• Renderer. The spend row is rendered verbatim, with or without +. An unavailable, annotated, mismatched or +-inconsistent spend is rejected, and the counts are never rendered. The public projection is a fixed point (final-report-markdown.test.ts).
• Projection.
  • The report-start projection adds + for partial pricing, unpriced attempts, attempts without usage, and accounting that priced nothing, and adds none for fully priced accounting (generated-workflow-verifier.test.ts).
  • Terminal and unchecked presentations restate the counts and set the + again, including dropping a report-start + when the whole run was fully priced (terminal-report-projection.test.ts, unverified-report.test.ts, runtime.test.ts).
• Gates. The coverage-score scans do not read $38.72+, $0.00+ or $0.0008+ as a coverage score (artifact-gates.test.ts).
• CLI. ultrafuzz report flags a missing + and a spend above accounting, and accepts $0.00+ and a fully priced spend without + (cli.test.ts). The campaign e2e checks that report.md shows the v4 amount with + for the SIGKILLed attempts.
• Gates run. pnpm -w format:check, lint, lint:strict:ci, docs:check, typecheck and knip all pass.
• Test results.
  • artifacts: 335/335.
  • runtime: renderer, projection, unchecked, metrics and pricing 119/119; final-report gates 36/36; runtime.test.js report publication, republication and accounting tests 7/7.
  • generated-workflow-verifier: 122/124. The 2 failures ("a missing or unrenderable prompt fails only its own task at assert-task-inputs" and "generated Smithers verification marker root must be a canonical directory") fail the same way with the test code from before this change on Node 24.9.
  • CLI report tests: 7/7. Campaign e2e (campaign-resume): 1/1.
  • modal: 129/129. evals: 103/103. prompts: 6/6. dashboard: 8/8.

Closes #1268.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no outstanding blocking finding was established.

Findings

1.  Source spend disappears ▶

Fix with agent prompt

### Issue 1
packages/runtime/src/workflow-sync.ts:undefined-1493
When a continuation has a source run but no local usage or executed agent attempts, this return skips the source-run estimate. The continuation’s synchronized `run.json` then has no `spend_estimate` or spend lineage, even if the source run has a persisted estimate.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR changes final-report presentation and accounting, and the latest revision replaces selector-filtered prompt authorities with per-attempt indexes of admitted ancestor outputs.

• Reports gain a target commit and remediation sections while removing low-level Markdown sections.
• Estimated spend uses accounting v4 and marks potentially incomplete amounts with +.
• The revised prompt input index includes dynamically generated ancestors.

_{Reviews (3) · Last reviewed commit: "Merge branch 'unstable' into fix/1268-fi..."}

[greptile-apps]

Source spend disappears When a continuation has a source run but no local usage or executed agent attempts, this return skips the source-run estimate. The continuation’s synchronized run.json then has no spend_estimate or spend lineage, even if the source run has a persisted estimate.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/workflow-sync.ts
Line: 1493

Comment:
**Source spend disappears** When a continuation has a source run but no local usage or executed agent attempts, this return skips the source-run estimate. The continuation’s synchronized `run.json` then has no `spend_estimate` or spend lineage, even if the source run has a persisted estimate.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Comments Outside Diff

These findings could not be posted inline.

•  Valid snapshot gets accounting warning packages/cli/src/commands/report.ts:98 ▶

  While a run is active, ultrafuzz report --require-verified can show the verified agent report, which contains the accounting snapshot from when the report task started. This check instead uses the current run.json. If a later attempt adds missing or unpriced usage, the current spend requires a + that the valid earlier snapshot does not have, producing a misleading REPORT_ACCOUNTING_MISMATCH warning.