Skip to content

perf(blobs): stream downloads, single-scan, and in-flight memory budget - #174

Merged
beinan merged 2 commits into
mainfrom
blob-memory-optimization
Jul 23, 2026
Merged

perf(blobs): stream downloads, single-scan, and in-flight memory budget#174
beinan merged 2 commits into
mainfrom
blob-memory-optimization

Conversation

@beinan

@beinan beinan commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

Rollout artifact blobs (binary_payload) are stored as an inline LargeBinary
column (deliberately — the MemWAL LSM scanner can't materialize blob-v2 columns),
so every blob is fully materialized in RAM at each hop:

  • Upload: whole request body buffered (JSON base64 ≈+33%, or per multipart
    part) then copied into an Arrow buffer via LargeBinaryBuilder::finish().
  • Download: get_blobtake_rows materializes the Arrow buffer, then
    .to_vec() copies to an owned Vec<u8> (≈2× blob size), then Body::from
    hands the whole thing to the HTTP layer as one frame.

With a 1 GiB per-request ceiling and no concurrency cap, N concurrent large
requests needed ≈2 × size × N bytes and could OOM the worker. For >100 MB blobs
this is the real risk under concurrency.

This PR does the three lower-risk mitigations and documents the larger step.

Changes

1. Streaming downloads. Worker fetch_rollout_blob and master
download_experiment_blob now send the payload as 256 KiB chunked frames
(blob_stream_body) instead of one Body::from(Vec<u8>). Frames are refcounted
Bytes slices of one allocation (no per-frame copy); removes the extra full-blob
copy in the send queue and lets a slow client apply backpressure. Content-Length
still set.

2. Single-scan record+blob. New RolloutStore::get_record_with_blob returns
(record, payload) from one base-first scan, replacing the master download's
get_by_id + get_blob double scan over the same shard. Keeps base-first + the
NotFound-tolerant WAL fallback.

3. In-flight blob-byte budget. Process-wide BlobBudget
(ROLLOUT_MAX_INFLIGHT_BLOB_BYTES, 0=disabled). Uploads reserve Content-Length
before buffering; downloads hold the reservation through the streamed send.
Over-budget requests get 503 OVERLOADED (+rollout_blob_budget_rejections_total
metric) instead of allocating. Bounds concurrency, not max blob size — a lone
oversized request is admitted when idle.

4. Proposal only (not implemented): specs/rollout-blob-streaming.md documents
the memory model and the next step — lance blob-v2 offloaded columns + BlobFile
range reads for true read-from-storage-in-frames — with the LSM-scanner blocker
and a two-sub-step rollout. Corresponds to a separate future PR.

Tests

  • BlobBudget admission/release + lone-oversized-request behavior.
  • blob_stream_body chunk-boundary reassembly (byte-for-byte) + reservation
    released when the stream drains.
  • content_length header parsing/defaulting.
  • get_record_with_blob across un-merged WAL, merged base, and miss.
  • Existing blob fetch tests still pass. cargo fmt --all -- --check and
    cargo clippy --all-targets -D warnings clean on all three crates.

Notes

  • Streaming download does not yet remove the single in-RAM Vec<u8> from
    get_blob — that needs item 4 (range reads). This PR removes the extra copy
    and caps concurrency.
  • Budget is opt-in (default 0); no behavior change unless configured.

beinan and others added 2 commits July 23, 2026 01:04
Large artifact blobs (>100MB) were fully materialized in RAM at every hop
and, with a 1GiB per-request ceiling and no concurrency cap, N concurrent
requests could OOM the worker. This bounds and smooths that memory use:

- Stream blob downloads as 256KiB chunked frames (refcounted Bytes slices,
  no per-frame copy) on both worker fetch_rollout_blob and master
  download_experiment_blob, instead of one Body::from(Vec<u8>) frame. Removes
  the extra full-blob copy in the send path and lets slow clients backpressure.
- Add RolloutStore::get_record_with_blob: one base-first scan returning
  (record, payload), replacing the master download's get_by_id + get_blob
  double scan.
- Add a process-wide in-flight blob-byte budget (ROLLOUT_MAX_INFLIGHT_BLOB_BYTES,
  0=disabled). Uploads reserve Content-Length before buffering; downloads hold
  the reservation through the streamed send. Over-budget requests get 503
  OVERLOADED instead of allocating. Bounds concurrency, not max blob size.
- specs/rollout-blob-streaming.md documents the memory model and proposes the
  larger next step (blob-v2 BlobFile range reads) not implemented here.

Tests: BlobBudget admission/release + lone-oversized, chunk-boundary
reassembly with reservation release, content-length parsing,
get_record_with_blob across WAL/base/miss. fmt + clippy --all-targets clean.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
@beinan
beinan merged commit 15e81d9 into main Jul 23, 2026
9 checks passed
@beinan
beinan deleted the blob-memory-optimization branch July 23, 2026 01:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant