bazel l1 [fork 6]: owed-upload ledger — shedding becomes deferral, not loss

[shreyas-blacksmith]

Summary

Restores the L1 ⊆ MinIO subset invariant (design ledger 30) that lossy write-through shedding violated. FindMissingBlobs answers local-first, so a blob whose backend upload was shed was reported present to every client forever — never re-uploaded, never reaching MinIO: invisible to rehydration, new ring members, and footprint accounting. Surfaced as P1 by the 2026-08-01 staging load experiment (~1.9M sheds during the write staircase).

• Owed ledger (cache/s3proxy/owed.go): queue-full sheds in Put and terminal upload failures in UploadFile (failed PUT, breaker refusal) record the upload identity (kind + hash + resolved prefix + resolved bucket). Bounded at 2^18 entries per backend (~50 MiB; at capacity new debt is rejected with a counter — older entries sit closer to LRU eviction, which self-resolves them). Snapshotted atomically beside the cache dir every sweep pass, reloaded at startup (≤15 s loss window).
• Sweeper: re-enqueues owed items through the normal upload queue — same workers, same breaker (ExecuteNoProbe), same outcome accounting — and yields absolutely: runs only while the breaker is closed and the queue is under half capacity. An owed blob already evicted locally settles as void: it is then honestly missing everywhere and the normal client re-upload heals both tiers.
• Coalescing: duplicate enqueues (same identity) are dropped at the door against an in-flight set. Matrix fan-out delivers the same missing blob from every host at once; MinIO's create-if-absent would 412 all but one; each queued duplicate pins an open FD — the queue's true scarce resource.
• Settlement: created or already_exists settles debt; terminal failure re-records it; the in-flight key is released on terminal outcome.
• Wiring: standalone-only (WithOwedLedgerDir from config/proxy.go; BlobSource back-reference injected in main.go after disk-cache construction — metricsDecorator promotes the method). The host-side embedded proxy keeps best-effort semantics by omission.
• New series: bazel_remote_s3_owed_uploads (gauge — the live L1⊆MinIO divergence), owed_sweep_total{result}, owed_rejected_total, upload_queue_coalesced_total.

Deliberately NOT owed: items in the queue at process stop (the roll contract's bounded loss — owing every enqueue would make the ledger a high-churn queue mirror), and host-tier forwards (hosts own no durable tier).

Test plan

• go test ./cache/s3proxy/ ./cache/disk/ ./config/ — new tests: shed→owed→sweep→repay round-trip against fake S3, duplicate coalescing, breaker-open and busy-queue sweep yield, evicted-blob void settlement, snapshot reload + corrupt-snapshot recovery, capacity rejection, raw-bytes/prefix-scoped OpenOwedBlob
• Staging re-test 2026-08-03 (goal-state 5f9076a, both L1 nodes + fleet on train tip): all bars pass — backlog 40,719 == drops, rejected 0; drain to zero in 40.3 min (sweeper-cap-bound at 1,024/min); 40/40 shed digests convergence-proven in MinIO; mid-backlog restart resumed from snapshot and drained; read p99 1.50 ms during drain vs 1.38 ms baseline. Known residual (documented roll contract): in-queue items are not restart-persistent (~85k lost at the test restart); SIGTERM drain-to-ledger is the candidate fast-follow.

Made with Cursor

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled. (Staging)}