Repository navigation
Conversation
Member
|
@andersglindstrom Could you review this? |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
3 open findings
What changed in this PR
Fixes stream adapter buffering behavior so exact-fit positive read requests yield remaining buffered bytes before pulling the next source chunk, preventing IPC record batches from arriving one chunk late at chunk boundaries.
Changes:
- Adjusted adapter generator loop conditions to handle positive
size === bufferLengthreads without prefetching another chunk. - Added unit tests verifying schema/batch reads don’t trigger an extra pull for iterable/async-iterable sources and that null/zero reads advance correctly.
| File | Description |
|---|---|
src/io/adapters.ts |
Updates adapter yield loop conditions for Iterable/AsyncIterable/DOM/Node stream sources to avoid prefetching on exact-fit reads. |
test/unit/ipc/reader/stream-reader-tests.ts |
Adds deterministic regression tests for chunk-boundary behavior and ByteStream null/zero read semantics. |
🧠 Review effort: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
93
to
97
| if (done || size <= bufferLength) { | ||
| do { | ||
| ({ cmd, size } = yield byteRange()); | ||
| } while (size < bufferLength); | ||
| } while (size < bufferLength || (size > 0 && size === bufferLength)); | ||
| } |
Comment on lines
192
to
196
| if (done || size <= bufferLength) { | ||
| do { | ||
| ({ cmd, size } = yield byteRange()); | ||
| } while (size < bufferLength); | ||
| } while (size < bufferLength || (size > 0 && size === bufferLength)); | ||
| } |
Comment on lines
+80
to
+96
| async function* chunks() { | ||
| for (const message of messages) { pulled++; yield message; } | ||
| // the byte stream stays open, like a WebSocket or network stream | ||
| await new Promise(() => { }); | ||
| } | ||
|
|
||
| const reader = await (await RecordBatchReader.from(chunks())).open(); | ||
| const iterator = reader[Symbol.asyncIterator](); | ||
|
|
||
| // reading the schema should not pull the next chunk | ||
| expect(pulled).toBe(1); | ||
| for (let i = 0; i < table.batches.length; i++) { | ||
| const { value } = await iterator.next(); | ||
| expect(value).toEqualRecordBatch(table.batches[i]); | ||
| expect(pulled).toBe(i + 2); | ||
| } | ||
| await iterator.return!(null); |
Copilot stopped reviewing on behalf of
kou due to an error
October 14, 2026 02:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's Changed
Yield buffered bytes before the next source chunk for positive exact-fit requests in
src/io/adapters.ts.Record batches no longer arrive one chunk late when an IPC message ends exactly at a source chunk boundary.
This applies to Iterable, AsyncIterable, DOM ReadableStream, and Node ReadableStream.
The condition keeps the existing
<case and adds equality only for positive sizes.Null and zero requests retain their existing behavior.
test/unit/ipc/reader/stream-reader-tests.tsadds deterministic tests for schema and batch reads through iterable sources without a pull of the next message chunk.The tests also check batch contents, finite-stream completion, and consecutive null/zero reads through iterable sources.
The latency tests fail on the unchanged baseline with
Expected: 1andReceived: 2, and pass with the fix.Closes #490.