fix(dashboard): make Phase 0.5 evidence activation-grade by brokemac79 · Pull Request #1156 · openclaw/clawsweeper

brokemac79 · 2026-08-13T13:13:49Z

Summary

make 15-minute and one-hour GitHub-egress completeness depend on evidence actually evicted from each rollup kind, not merely the time an old-row cap eviction ran
add durable, bounded, privacy-safe publication-transition causes and independent reconciliation for publish, supersession, semantic dedupe, retry, dead letter, and refresh
preserve unattempted credential-circuit deferrals without charging publication retry budget
keep terminal batch outcomes at their existing failure depth after an earlier retry
preserve the active credential pool on fallback batch completion/release, including the owner-isolated target_app path

This is the CSW-127 Phase 0.6 remediation derived from the Phase 0.5 monitor. It does not activate Phase 1 and does not change admission, capacity, schedules, retry ceilings/delays, credential ownership, gates, or rollback behavior.

Final head: 9e984a44ec946557896ca69558a00a7cf929fe9c
Rebased base/current main: 6494ab5eb285cc2993d4679d6723e4b2486de99c
Live deployed dashboard at final local proof: 6494ab5eb285cc2993d4679d6723e4b2486de99c

Problem

Phase 0.5 proved two activation blockers:

rollup_window_complete=false and therefore query_complete=false persisted in otherwise small, untruncated 15-minute and one-hour views. The store recorded when cap pruning ran, so evicting old rows could incorrectly poison newer windows.
A sixth-recovery cohort ended in four retry_exhausted dead letters, but the public surface lacked durable stage/revision/pool/recovery attribution needed to distinguish real retry amplification, circuit deferral, supersession, and refresh.

The monitor also observed a separate owner-isolated target_app closure-proof 403 path. This PR does not alter that read path or merge it into repository_actions; it only ensures fallback publication telemetry retains the active closed pool classification.

Implementation

Honest rollup completeness

persist the greatest timestamp actually evicted for five-minute rollups, hourly rollups, and rate-limit observations
compare those per-kind watermarks with the aligned query boundary
track per-kind exact eviction counts and expose whether legacy counts are exact
migrate old diagnostics conservatively; if legacy state cannot prove what was evicted, completeness fails closed
preserve missing, late, truncated, and genuinely in-window eviction signals without synthesizing conservation

Publication-cause reconciliation

retain five-minute transition causes in SQLite for 48 hours, cap public output at 256 closed-dimension rows, and expose truncation
attribute stage, completion kind/reason, revision relation, credential pool, recovery cause, backoff class, and attempt bucket
reconcile independent flow counts for publish, supersession, semantic dedupe, retry, dead letter, and refresh; attribution_complete fails closed on mismatch, truncation, or unattributed rows
propagate pool class and attempted state through batch publication completion
preserve the underlying failure reason in a retry-exhausted cause while retaining the established retry_exhausted DLQ reason

Causally supported behavior corrections

an unattempted quota/circuit deferral remains a backoff cause and does not consume publication retry budget
terminal batch outcomes (published, superseded, deferred) preserve the existing failure depth instead of being counted as another failed attempt
fallback complete and release retain the active circuit scope instead of silently defaulting target_app observations to repository_actions

Current-main reconciliation

The branch was rebased once, conflict-free, onto exact main 6494ab5eb285cc2993d4679d6723e4b2486de99c, which includes merged PR #1169. Range-diff shows the original nine Phase 0.6 commits are patch-equivalent. Two additional commits reconcile proof and fix the fallback pool attribution found by committed review.

PR	State / exact disposition	Reconciliation
#1155	merged `dc738b3845655ad36f91ea9584d90abdd4df3ca3`	comment-router throttle cursor; different files, preserved
#1157	merged `4398fe9c9fb599a83aaef5304b4ab39e359273da`	runner-label infrastructure; independent
#1158	merged `c175faa03a95208d82bca5ec3e2335ccc9d5dc11`	apply folder throttle deferral; independent
#1159	merged `c4751080cf1466a45df3a8073ad0a07d8488c911`	GraphQL activity read reduction; telemetry-adjacent, preserved
#1160	merged `82d9ba983abdb5738c138e4741c52960fa8aa019`	apply comment-sync throttle deferral; independent
#1161	merged `c9eda83c29d222bddf917b6a8c5ef27254887ec2`	apply read-generation reuse; independent
#1162	merged `746cb858bf00fc0fdf37c7f9471b935054a25bab`	hosted Linux sandbox proof infrastructure; preserved
#1163	merged `6a358d239c1e67cd38159d70de18eecbd9a5d844`	R2 exact-review bundle receipt cache; separate store/routes, preserved
#1164	merged `dc6762092a509828f8a9acebae23d342f92efeed`	durable ETag broker; `broker_lookup` and `conditional_response` remain in completeness totals
#1165	open draft `93ecdfc3937f7988667da7f59e784f4c2fe1930d` at reconciliation	sandbox-failure proof; not in main, no Phase 0.6 dependency
#1166	merged `386730fc41ae0b1b63a7dbbd648404be6bd65c66`	apply close-mode throttle yielding; independent
#1167	merged `bf762f3f6f9e8ee89fc31a260053dabe257a7103`	webhook materialized read model; additive queue routes/store preserved
#1168	merged `e17e09425b604aeb6db9fb56494f640a9454ec97`	throttled-publication retry classification preserved
#1169	merged/current main `6494ab5eb285cc2993d4679d6723e4b2486de99c`	bounded per-run webhook operational-health confirmation, exact recheck, unknown-health, and phantom-run eviction semantics preserved; shared docs file reconciled without semantic overlap

No landed logic is duplicated. Phase 0.6 uniquely contributes per-kind cap-eviction watermarks, bounded publication transition/cause reconciliation, retry-neutral credential-circuit deferrals, and correct closed-pool attribution through fallback completion.

Compatibility, privacy, and Bay

existing v1/v2 GitHub-egress fields remain intact; new metadata is additive
public cause dimensions are closed and bounded and expose no repository, item, credential, run, artifact, lease, error fingerprint, or exact revision identity
target_app and repository_actions owner isolation remains intact
typed exits, command resumption, reset-plus-jitter, pool ownership, retry ceilings/delays, rollback shadow behavior, and existing queue semantics are preserved
PR fix: evict phantom queued runs from operational health #1169's webhook operational-health/status semantics remain unchanged
Bay remains observer-only; this PR adds health context and no dispatch, retry, replay, reset, acknowledgement, or gate control

Validation

focused telemetry, queue, batch-publication, and batch-CLI suite — 235/235 passed
full Linux pnpm run check in the exact-code container — passed
corepack pnpm run check:docs — passed
git diff --check — passed
dirty Codex review — clean after the fallback pool-attribution correction
committed Codex review against 6494ab5eb285cc2993d4679d6723e4b2486de99c — no actionable findings
local ClawSweeper --local-range against 6494ab5eb285cc2993d4679d6723e4b2486de99c — completed at exact head, keep_open, high confidence

Real Behavior Proof

Claim: short windows remain complete when only older evidence was evicted, overlapping windows fail closed, retry plus refresh causes reconcile through restart, and fallback pool scope remains owner-correct without identity leakage.
Exercised surface: signed Worker ingest and queue routes, public observer routes, real Wrangler Worker, real SQLite ExactReviewQueue Durable Object, production row caps, and Worker restart.
Scenario/fixture: synthetic local publication members produce a same-revision state-contention retry and artifact refresh; fallback completion/release exercise target_app; 50,000 rollup rows and 10,000 rate observations are seeded before cap eviction.
Command/environment: Docker-backed Crabbox provider=local-container, lease cbx_b188010777d1 (swift-krill-16ae), node:24-bookworm, wrangler@4.107.0 --local. The container reconstructed an immutable git archive and asserted code tree 84f9205a89d0fafdd20e9796affb7b67bd534fe2 for code commit 92d91e812f810c5bf9f7e0f470cb017533aee43c before execution.
Observed result: exit 0; lease stopped; 235/235 focused tests; retry and refresh causes reconciled; two rollup rows and one rate row evicted; 15-minute and one-hour query_complete=true; overlapping six-hour view incomplete; restart preserved watermarks and causes; fallback target_app scope preserved; privacy assertions passed; full Linux pnpm run check passed. Coverage: 81.49% lines, 74.08% branches, 87.35% functions.
Artifact/trace: docs/proof/csw-127-phase0-6/ contains the contract, reproducible harness, sanitized receipt, artifact hashes, and limits.
Limits: synthetic local fixtures only; no production, GitHub, workflow, queue, DLQ, gate, schedule, capacity, deployment, or credential mutation. A stable production cohort starts only after an authorized merge deploys successfully.

Artifact SHA-256 receipts are recorded in container-receipt.json, including runtime receipt D890DFA935CE8A8DCE76A93651DFD3AA79030E266869C4E67DA054838DFA2FDB and redacted Wrangler log 53A55DECCBA450985FFADBCC7D864D14C2C2ECF24C049877B458A6BCDB33149E.

Two discarded Docker leases stopped before test execution: cbx_42cd4a3fdb12 rejected a CRLF-mounted wrapper, and cbx_35f8c4b178f6 rejected ignored-file contamination through the worktree bind. The successful run used the immutable archive harness and the same expected Git tree.

Review findings resolved

The first rebased review found terminal batch outcomes advancing the attempt bucket after an earlier failure. The fix aligns the batch path with direct completion and adds a retry-then-publish regression.
The committed review found fallback rate-limit completion losing target_app ownership. Commit 92d91e812f810c5bf9f7e0f470cb017533aee43c carries the active circuit scope through both batch complete and release; the complete container proof was rerun afterward.

Phase 1 and unrelated work

Draft PR #1145 remains independent, untouched, default-disabled, and inactive. Phase 0.6 does not activate it or establish Phase 1 readiness.
fix(publication): stop repository pool on first throttle #1145 must later reconcile current main, the landed read/cache/retry work, and this Phase 0.6 contract before any new review or activation decision.
#1110 remains a separate conflicting draft for adaptive hot-review allocation; it is not incorporated here.
The owner-isolated target_app closure-proof 403/read-fallback behavior remains a separate follow-up and is not folded into repository_actions circuit semantics.

Risks, rollout, and rollback

The SQLite changes are additive and migrate legacy state conservatively. A rollback to current main retains the existing rollback shadow and ignores additive public fields; new cause rows and exact eviction metadata can remain dormant. The main residual risk is production-cohort behavior, so Phase 1 activation remains separately authorized even if the post-merge Phase 0.6 cohort passes.

ClawSweeper review

What this changes

This PR makes GitHub-egress completeness depend on the evidence actually evicted and adds bounded, privacy-safe publication transition reconciliation to the exact-review dashboard.

Regression provenance

Possible regression — probable (reviewed change; reproduction). No predecessor PR is attributed.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep open for normal maintainer merge review: current main does not contain the eviction-boundary or publication-cause fixes, and the branch has coherent exact-head Worker/SQLite proof with no actionable correctness or security finding. The durable-schema and public-observer expansion still warrants controlled rollout observation.

Priority: P2
Reviewed head: 9e984a44ec946557896ca69558a00a7cf929fe9c

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	Strong production-shaped proof and focused coverage support a correct patch, while durable-state and public-contract rollout remain meaningful operational review points.
Proof confidence	🦞 diamond lobster (5/6) ✨ media proof bonus	Sufficient (linked_artifact): The PR includes an exact-tree Docker-backed Worker and SQLite Durable Object receipt showing after-fix cap, restart, reconciliation, fallback-pool, privacy, and broad-check results.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (linked_artifact): The PR includes an exact-tree Docker-backed Worker and SQLite Durable Object receipt showing after-fix cap, restart, reconciliation, fallback-pool, privacy, and broad-check results.
Evidence reviewed	5 items	Eviction completeness implementation: The branch records per-kind cap eviction boundaries before deleting the oldest rows, then bases public query completeness on those evidence timestamps rather than cleanup time. Durable cause reconciliation implementation: The branch adds a bounded transition-cause table and reconciles its public rows against independent publication-flow counters. Exact-head behavior proof: The committed receipt identifies the rebased base, tested code commit and immutable tree, then records a successful real Wrangler Worker plus SQLite Durable Object run with cap eviction, restart, reconciliation, privacy, and fallback-pool assertions.
Findings	None	None.
Security	None	None.

How this fits together

The exact-review queue receives publication completion events and signed GitHub-egress telemetry, persists bounded operational facts in a SQLite Durable Object, and exposes observer-only status to dashboard consumers. This change improves how retained telemetry determines query completeness and how publication outcomes are attributed downstream.

flowchart LR
  A[Signed egress telemetry] --> B[Rollup and rate-limit store]
  B --> C[Eviction watermarks]
  D[Queue and batch completions] --> E[Publication cause buckets]
  C --> F[Observer API]
  E --> F
  F --> G[Dashboard and Bay observers]

Before merge

Resolve merge risk (P1) - The Durable Object schema and observer response gain new persisted/public fields, so legacy migration behavior should be observed after deployment.
Resolve merge risk (P1) - A defect in transition accounting could misstate publication health or completeness even though it does not change admission, retry ceilings, or activation policy.
Complete next step (P2) - No actionable repair finding remains; this PR needs normal maintainer merge review and deployment observation rather than an automated fix branch.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Changed surface	19 files, +2,333 / -34 lines	The PR spans durable queue state, public telemetry, batch clients, tests, and proof material.
Runtime and regression coverage	runtime +804/-28, tests +596/-0	The substantial durable-state change is paired with focused queue, Worker-runtime, telemetry, and batch-client regression coverage.

Merge-risk options

Maintainer options:

Merge with rollout observation (recommended)
Merge the additive telemetry contract and verify the deployed Durable Object's migrated observer output and attribution-complete state.
Pause for production migration evidence
Hold the PR if maintainers require evidence from an existing durable-state migration before accepting the new observer contract.

Technical review

Best possible solution:

Land the additive observer changes through the normal deployment path, then verify migrated dashboard responses and attribution completeness before treating the new telemetry as activation evidence.

Do we have a high-confidence way to reproduce the issue?

Yes. The committed Worker/SQLite harness provides a focused cap-eviction, restart, batch-completion, and public-observer scenario; source inspection also identifies the former cleanup-time completeness mechanism.

Is this the best way to solve the issue?

Yes. Recording actual eviction boundaries and bounded closed-dimension causes fixes the observed attribution gap without enabling Phase 1 or changing queue policy.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 6494ab5eb285.

Labels

Label justifications:

P2: This is a bounded internal dashboard and queue-telemetry correctness fix with no demonstrated active user-facing outage.
merge-risk: 🚨 compatibility: Persisted diagnostics and observer response fields change while legacy Durable Object state is migrated.
merge-risk: 🚨 availability: The exact-review queue's Durable Object records new transition facts during publication completion.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (linked_artifact): The PR includes an exact-tree Docker-backed Worker and SQLite Durable Object receipt showing after-fix cap, restart, reconciliation, fallback-pool, privacy, and broad-check results.
proof: sufficient: Contributor real behavior proof is sufficient. The PR includes an exact-tree Docker-backed Worker and SQLite Durable Object receipt showing after-fix cap, restart, reconciliation, fallback-pool, privacy, and broad-check results.

Evidence

What I checked:

Eviction completeness implementation: The branch records per-kind cap eviction boundaries before deleting the oldest rows, then bases public query completeness on those evidence timestamps rather than cleanup time. (dashboard/github-egress-telemetry.ts:691, 9e984a44ec94)
Durable cause reconciliation implementation: The branch adds a bounded transition-cause table and reconciles its public rows against independent publication-flow counters. (dashboard/exact-review-queue.ts:8289, 9e984a44ec94)
Exact-head behavior proof: The committed receipt identifies the rebased base, tested code commit and immutable tree, then records a successful real Wrangler Worker plus SQLite Durable Object run with cap eviction, restart, reconciliation, privacy, and fallback-pool assertions. (docs/proof/csw-127-phase0-6/container-receipt.json:3, 9e984a44ec94)
Current-main history: Current main history identifies the merged telemetry and quota-handoff work that owns the touched surfaces; the branch is not redundant with those commits. (dashboard/exact-review-queue.ts:1, 56cb78d60734)
Security scope check: The reviewed range changes queue, telemetry, tests, docs, and proof files; it does not modify workflows, dependency manifests, lockfiles, package publishing metadata, or secret handling. (9e984a44ec94)

Likely related people:

Martin Cleary: Current-main history attributes the egress telemetry and exact-review quota-handoff foundations to the merged commits for the affected paths. (role: introduced merged telemetry and quota-recovery behavior; confidence: high; commits: ac340908bf69, 56cb78d60734; files: dashboard/github-egress-telemetry.ts, dashboard/exact-review-queue.ts)
Peter Steinberger: Recent current-main changes on both affected dashboard surfaces include queue materialization and egress telemetry work. (role: recent adjacent contributor; confidence: high; commits: bf762f3f6f9e, dc6762092a50; files: dashboard/exact-review-queue.ts, dashboard/github-egress-telemetry.ts)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

After deployment, record one migrated observer check confirming retained legacy state reports conservative completeness and reconciled causes.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (4 earlier review cycles)

reviewed 2026-08-13T19:32:56.677Z sha 531dcb2 :: needs maintainer review before merge. :: none
reviewed 2026-08-14T11:51:57.983Z sha 531dcb2 :: needs maintainer review before merge. :: none
reviewed 2026-08-14T17:34:21.678Z sha 3b8feda :: needs maintainer review before merge. :: none
reviewed 2026-08-14T17:39:34.359Z sha 3b8feda :: needs maintainer review before merge. :: none

brokemac79 · 2026-08-14T17:29:05Z

@clawsweeper re-review

clawsweeper · 2026-08-14T17:29:09Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

clawsweeper · 2026-08-14T17:36:40Z

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

brokemac79 · 2026-08-14T19:16:23Z

@clawsweeper re-review

clawsweeper · 2026-08-14T19:16:26Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

State: Complete
Detail: The durable review result and its route handoff completed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/31832892783
Updated: 2026-08-14T19:21:50.947Z

brokemac79 force-pushed the codex/csw-127-phase0-6 branch from 531dcb2 to 3b8feda Compare August 14, 2026 17:28

brokemac79 marked this pull request as ready for review August 14, 2026 17:36

brokemac79 added 12 commits August 14, 2026 19:37

fix(dashboard): make phase 0.5 evidence activation-grade

14f6f45

docs(proof): record phase 0.6 container evidence

2f49749

fix(dashboard): reconcile publication refresh causes

ecb7007

test(proof): exercise refresh reconciliation boundary

a1a0b5d

test(proof): match refresh no-backoff dimension

086b29b

docs(proof): record final phase 0.6 evidence

377df05

docs(proof): refresh phase 0.6 rebase evidence

59d4242

fix(telemetry): preserve batch terminal retry depth

3f3793d

docs(proof): record corrected phase 0.6 proof

ef640c5

docs(proof): reconcile phase 0.6 with merged health fix

029df04

fix(telemetry): preserve fallback circuit scope

92d91e8

docs(proof): record fallback scope proof

9e984a4

brokemac79 force-pushed the codex/csw-127-phase0-6 branch from 3b8feda to 9e984a4 Compare August 14, 2026 19:16

brokemac79 merged commit 9fb6d6b into main Aug 14, 2026
11 of 12 checks passed

brokemac79 deleted the codex/csw-127-phase0-6 branch August 14, 2026 19:54

This was referenced Aug 15, 2026

fix: contain public worker status metadata #1171

Merged

feat(bay): chart GitHub throttle history #1174

Merged

Uh oh!

Conversation

brokemac79 commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Implementation

Honest rollup completeness

Publication-cause reconciliation

Causally supported behavior corrections

Current-main reconciliation

Compatibility, privacy, and Bay

Validation

Real Behavior Proof

Review findings resolved

Phase 1 and unrelated work

Risks, rollout, and rollback

Related

Uh oh!

clawsweeper Bot commented Aug 13, 2026

Uh oh!

brokemac79 commented Aug 13, 2026

Uh oh!

clawsweeper Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ClawSweeper review

What this changes

Regression provenance

Merge readiness

Review scores

Verification

How this fits together

Before merge

Security

Review metrics

Merge-risk options

Technical review

Labels

Evidence

Rank-up moves

Rating scale

Workflow

History

Uh oh!

brokemac79 commented Aug 14, 2026

Uh oh!

clawsweeper Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Aug 14, 2026

Uh oh!

brokemac79 commented Aug 14, 2026

Uh oh!

clawsweeper Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

brokemac79 commented Aug 13, 2026 •

edited

Loading

clawsweeper Bot commented Aug 13, 2026 •

edited

Loading

clawsweeper Bot commented Aug 13, 2026 •

edited

Loading

clawsweeper Bot commented Aug 14, 2026 •

edited

Loading

clawsweeper Bot commented Aug 14, 2026 •

edited

Loading