Skip to content

Preserve selected streaming rows across concatenated epochs - #8887

Open
sjdoane wants to merge 1 commit into
huggingface:mainfrom
sjdoane:codex/preserve-concatenated-stream-subsets
Open

sjdoane wants to merge 1 commit into
huggingface:mainfrom
sjdoane:codex/preserve-concatenated-stream-subsets

Conversation

@sjdoane

@sjdoane sjdoane commented Oct 23, 2026

Copy link
Copy Markdown

After concatenating a streaming dataset selected with take(4), set_epoch(1) can change its rows from [0, 1, 2, 3] to [0, 3, 6, 9]. The same problem affects skip() and explicit shuffling after concatenation.

Propagate shuffling through vertical concatenation's shard wrappers so the existing skip/take ordering restriction is honored. This preserves the selected rows while allowing buffer shuffling within that selection. Related to the epoch handling in #8707.

Validation on macOS with Python 3.12 and CPU PyTorch:

  • New regression matrix: 16 failures and 8 passing controls before the fix; all 24 pass afterward.
  • Iterable and distributed unit tests: 714 passed, 39 skipped, 6 integration tests deselected.
  • Repository-wide make quality and git diff --check pass.

The new tests cover Python and Arrow sources, skip/take through a map wrapper, 1/2/4 shards, single/mixed concatenations, repeated epochs and explicit buffered shuffling. Full repository and hosted integration matrices were not run locally.

Implementation and tests were generated with OpenAI Codex; the checks above were executed locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant