Skip to content

Preserve all streams when interleave inputs share an iterator - #641

Open
fhgffy wants to merge 1 commit into
pytoolz:masterfrom
fhgffy:fix/interleave-shared-iterators
Open

fhgffy wants to merge 1 commit into
pytoolz:masterfrom
fhgffy:fix/interleave-shared-iterators

Conversation

@fhgffy

@fhgffy fhgffy commented Oct 2, 2026

Copy link
Copy Markdown

interleave can silently drop values from unrelated inputs when the same iterator occupies more than one input slot:

shared = iter([0, 1])
list(interleave([shared, shared, [10, 11]]))
# Current: [0, 1, 10]
# Expected: [0, 1, 10, 11]

If the shared iterator starts empty, the entire [10, 11] input is lost. The removal loop uses iterator identity to find the exhausted slot again, so another occurrence of that iterator ends the retained cycle too early.

Wrap each input occurrence in a distinct single-input itertools.chain. This preserves shared consumption while giving each slot its own identity. The existing cycle/removal algorithm and inner loop remain intact, including support for infinite inner and outer inputs.

The three regression tests cover adjacent and separated shared slots, empty and odd-length shared inputs, and infinite inputs. All three fail before the change and pass afterward.

Validation on Linux:

  • CPython 3.12.14 and 3.13.5: full package tests and doctests each pass with 271 passed, 2 skipped
  • pytest bench/ -q: 19 passed
  • Repository CI's pycodestyle command and git diff --check: passed
  • Separate round-robin oracle checks passed for 15,125 finite shared-iterator configurations, plus consumption/laziness and exception-propagation checks
  • Other Python versions and platforms were not run locally

Performance tradeoff: the wrapper adds per-input overhead, and timings vary by workload. One local median-of-five sample for ten streams of 1,000 elements (number=200) took 0.0751 s before and 0.0930 s after, about 24% slower. Uneven-length and many-short-stream samples were close to baseline (ratios 0.963 and 0.995). This fixes data loss but does not claim performance neutrality.

@fhgffy
fhgffy marked this pull request as ready for review October 2, 2026 13:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant