Skip to content

feat(interpreter): stream pipeline stages concurrently with SIGPIPE - #2541

Merged
chaliy merged 1 commit into
mainfrom
claude/fidelity-pipelines-stream
Oct 7, 2026
Merged

chaliy merged 1 commit into
mainfrom
claude/fidelity-pipelines-stream

Conversation

@chaliy

@chaliy chaliy commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Requested by Михайло · project thread

Stacked on #2538.

What changed

  • Pipeline stages now stream into each other concurrently, starting at the first stage that runs shell code (a loop, group, function, eval). Each such stage runs on a forked shell and writes into a 4 KiB pipe with backpressure. The last stage stays in the caller's shell, so its output still streams to exec_streaming callers.
  • SIGPIPE: when a reader is done (head got its lines, a while read loop broke, or the stage never reads stdin), the writer's next write fails and it ends with 141. while :; do echo x; done | head -n 2 prints two lines and sets PIPESTATUS to 141 0, as in bash. Before, the loop ran until the command limit and the whole exec failed.
  • read pulls one line and head -n/-c pulls only what it needs from a streaming pipe. Commands that never read stdin (echo, true, sleep, ...) do not wait for the producer.
  • Leading stages that are one builtin call (echo x | grep x) still run in sequence without forking, so the common case costs nothing extra.
  • concurrent_jobs(false) also turns this off.
  • New TM-DOS-124 covers runaway producers and recursive pipelines (f() { f | f; }). A streaming group counts as a subshell level, and past 4 nested levels stages run in sequence, which keeps native stack near plain function recursion.
  • L-PIPE-001 is narrowed to the remaining gaps:
    • Leading single builtins (yes | head) still produce their whole output before the next stage reads.
    • cat/grep downstream of a streaming stage read all input first.
    • The pipe is 4 KiB, not 64 KiB.

Why

This is step 2 of the bash-fidelity roadmap. producer | head and producer | while read ...; break are everyday idioms. They either ran away into limits or produced the wrong PIPESTATUS. The pipe design (a shared buffer polled on the caller's task, with no threads) is also the building block for wiring a reader-backed stdin into Terminal (L-TERM-002).

Before / After

$ bashkit -c 'while :; do echo x; done | head -n 2; echo "${PIPESTATUS[*]}"'

Before: x printed thousands of times, then resource limit exceeded: maximum command count exceeded (10000).
After: x, x, 141 0. This is identical to GNU bash 5.2.

The new spec cases in pipes-redirects.test.sh (endless producer into head, into two reads, through a middle while read stage (141 141 0), and into a loop that breaks) all match real bash locally.

Risk

  • Medium. Pipelines whose producer is a loop or function now run concurrently, so stage side effects interleave like in bash rather than strictly in order. Output order of the final stage is unchanged.
  • Stack: measured min stack for f() { f | f; } in debug builds: 1.99 MB sequential (main 1.90), 2.19 MB with streaming. The 4-level cap keeps it bounded. The existing nested $(...) canary is unchanged (1994531 bytes).
  • A finite loop writing between 4 and 64 KiB into a reader that quits early exits 141 where bash usually exits 0 (documented in L-PIPE-001).

Checklist

  • Tests added or updated (pipe unit tests, pipeline_tests, 4 bash-compared spec cases, TM-DOS-124 regressions)
  • Backward compatibility considered (concurrent_jobs(false) keeps sequential stages)

https://claude.ai/code/session_019aFikmptPc91Fj4N2iDXQA


Generated by Claude Code

@chaliy chaliy self-assigned this Oct 7, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 007395e Commit Preview URL

Branch Preview URL
Oct 07 2026, 06:22 AM

@chaliy
chaliy force-pushed the claude/fidelity-pipelines branch from a5ef256 to 9a406b9 Compare October 7, 2026 05:51
Base automatically changed from claude/fidelity-pipelines to main October 7, 2026 06:20
From the first stage that runs shell code, pipeline stages run on forked
shells over 4 KiB pipes with backpressure. A writer whose reader is gone
ends with 141, so 'while :; do echo x; done | head -n 2' stops the loop.
head and read pull only what they need. Nested streaming counts as a
subshell level and falls back to sequential stages past 4 levels
(TM-DOS-124). L-PIPE-001 narrowed.

Claude-Session: https://claude.ai/code/session_019aFikmptPc91Fj4N2iDXQA
@chaliy
chaliy force-pushed the claude/fidelity-pipelines-stream branch from 20b5dc5 to 007395e Compare October 7, 2026 06:21
@chaliy
chaliy merged commit f7e2d61 into main Oct 7, 2026
87 checks passed
@chaliy
chaliy deleted the claude/fidelity-pipelines-stream branch October 7, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant