Add an optional Foyer-backed data-file cache - #73
Conversation
d7c1d5c to
195de5f
Compare
195de5f to
ceb8f35
Compare
ceb8f35 to
570a3d0
Compare
570a3d0 to
b1d20b5
Compare
b1d20b5 to
a39bc8b
Compare
a39bc8b to
ecc5caa
Compare
ecc5caa to
a4c7130
Compare
There was a problem hiding this comment.
The eager-stream blocker is fixed: cached gets now return before payload I/O, fetch at most one cache block per poll, and stop reading when the consumer drops the stream. The bounded-read-ahead, repeated-scan, and prior regression coverage passes at this head.
The remaining risk is operational: a cold multi-block stream issues one sequential origin range request per cache block, and concurrent cold readers still do not coalesce misses. Normal large scanner reads retain the existing coalesced get_ranges path, but cold streamed reads should be benchmarked with the chosen block size before broad rollout; keep sessions long-lived and cached bytes disposable.
### What problem does this PR solve? Issue Number: None Related PR: lance-format/lance-c#73 Problem Summary: Refresh the bundled Lance-C PR apache#73 patch from head a39bc8b08cd1367b7445cb9acf3309faa1f3d610 to a4c71309ddb76ad79808e3e8f7797bcd9bfc174a. The updated cache implementation extends coverage to whole-object and single-range reads while preserving the existing v0.1.9 plus PR apache#73 then PR apache#74 patch order. ### Release note None ### Check List (For Author) - Test: No need to test (the refreshed PR apache#73 and existing PR apache#74 patches apply in order to a clean v0.1.9 archive with zero fuzz and no rejects or backup files; compilation and tests were not requested) - Behavior changed: Yes. The bundled PR apache#73 data-cache patch is refreshed to its latest upstream head. - Does this need documentation: No
Issue Number: None Related PR: lance-format/lance-c#73 Problem Summary: Refresh the bundled Lance-C PR apache#73 patch from head a39bc8b08cd1367b7445cb9acf3309faa1f3d610 to a4c71309ddb76ad79808e3e8f7797bcd9bfc174a. The updated cache implementation extends coverage to whole-object and single-range reads while preserving the existing v0.1.9 plus PR apache#73 then PR apache#74 patch order. None - Test: No need to test (the refreshed PR apache#73 and existing PR apache#74 patches apply in order to a clean v0.1.9 archive with zero fuzz and no rejects or backup files; compilation and tests were not requested) - Behavior changed: Yes. The bundled PR apache#73 data-cache patch is refreshed to its latest upstream head. - Does this need documentation: No
Issue Number: None Related PR: lance-format/lance-c#73 Problem Summary: Refresh the bundled Lance-C PR apache#73 patch from head a39bc8b08cd1367b7445cb9acf3309faa1f3d610 to a4c71309ddb76ad79808e3e8f7797bcd9bfc174a. The updated cache implementation extends coverage to whole-object and single-range reads while preserving the existing v0.1.9 plus PR apache#73 then PR apache#74 patch order. None - Test: No need to test (the refreshed PR apache#73 and existing PR apache#74 patches apply in order to a clean v0.1.9 archive with zero fuzz and no rejects or backup files; compilation and tests were not requested) - Behavior changed: Yes. The bundled PR apache#73 data-cache patch is refreshed to its latest upstream head. - Does this need documentation: No
#71 not merge to main branch, only as patch.