Skip to content

fix(execution): loud post-run streams, unified runtime corpus, fail-loud build validation - #134

Merged
mvallebr merged 1 commit into
mainfrom
fix/followups-behavior
Sep 20, 2026
Merged

mvallebr merged 1 commit into
mainfrom
fix/followups-behavior

Conversation

@mvallebr

Copy link
Copy Markdown
Contributor

Summary

Behavioral follow-ups from the audit, covering #128, #129, #131 and #132 (behavioral items only; #130 DagNode split and the GWT rename churn stay separate on purpose).

Important correction to #128 first: deeper investigation (watchdog + faulthandler stack dumps, 40 hot-loop + cold-process runs) showed the sync engine never deadlocked on nested_fanout — the original repro hung because my debug script drained executor.outputs after run(). What actually hangs is that footgun, and it is what this PR fixes — plus the discovery that complex_parallel* really do deadlock both engines by design (details below).

Changes

#128 — loud post-run stream semantics

  • SyncQueueIterator now tracks why it closed: exhausted branches keep raising StopIteration (iterator protocol preserved), failed branches re-raise their error, and a branch closed before being fully consumed raises a descriptive RuntimeError instead of blocking forever on a dead pump
  • Executor cleanup terminates live fan-outs with the new FanoutStreamClosedError (core.exceptions) — a late drain of an unconsumed branch fails loud instead of hanging or silently returning a truncated stream; AsyncQueueBranch gains identical closed-state semantics plus terminate(), and async cleanup terminal-izes tracked branches (parity)
  • wait_for_workers_after_shutdown gains a log grace window (2s default): the first poll almost always sees mid-teardown workers, so the "4 workers blocked" warning fired on every healthy run (0.1s teardown) and destroyed trust in the diagnostic
  • nested_fanout pack step_results corrected to the observable contract (terminal drained EACH steps publish None)

#129 — unified runtime corpus

  • New tests/execution/test_corpus_runtime.py executes every registered pack on its own engine and asserts frozen step_results — replacing the two hand-maintained per-engine pack name tuples (the gap that let complex_parallel*/nested_fanout silently never run); the per-engine test_runner_step_results.py files are removed and the error-materializer registration tests moved into the unified module
  • Exclusions are explicit and symmetric-checked: complex_parallel + complex_parallel_mixed deadlock BOTH engines by design — their terminal step consumes sibling fan-out branches sequentially, which circular-backpressures a bounded lockstep handoff. That consumption pattern violates the lockstep contract (§1.7); mixed_fanout shows the supported zip pattern. Fixing it would require unbounded buffering, contradicting the memory model — documented + excluded instead (kept in fix(execution): nested fan-out/fan-in topology deadlocks the sync engine #128)

#131 — fail-loud build validation (non-breaking parts)

  • Include-cycle detection unified (dag_expansion.validate_no_include_cycle); the registry's instance-identity check stays separate (different semantics)
  • Include return annotations validated by type inspection instead of a name substring: aliased imports now validate, and a look-alike type (NotTheParams) no longer passes; unresolvable string annotations keep the tolerant fallback
  • get_safe_type_hints raises when a callable declares annotations that cannot be resolved — the previous silent {} disabled every validation that depends on hints; only fully-unannotated callables resolve to {} (PEP 563 still works — resolvable string annotations are resolved as before)

#132 — small smells

  • AsyncQueueBranch.put blocks on the queue properly (wait_for-bounded, re-checking closure) instead of a 1 ms hot sleep loop
  • CsvSerializer fills ragged rows with restval=""
  • Async error dispatch checks callable(err_mat) before building the context, mirroring sync
  • Both step runners replace the "output" not in locals() trick with an explicit _NOT_PRODUCED sentinel (provably identical semantics)

Tests

  • 832 passing (was 816): +24 runtime corpus cases, +8 handoff close-semantics unit tests (sync & async), +2 hints contract tests, +2 grace-window tests, +1 corpus-symmetry check for exclusions
  • Full suite, corpus spec tests, sync/async parity meta-tests, ruff format/F401 and the pre-commit coverage gate all green
  • Behavioral notes for reviewers: pipelines with declared-but-unresolvable annotations now fail at build (previously silently unvalidated — this is the fail-loud contract); draining a fan-out branch post-run now raises instead of hanging

…s, fail-loud hints

Follow-ups #128, #129, #131, #132 (behavioral items only).

#128 — post-run stream ergonomics (corrected diagnosis: no scheduler
deadlock; the original repro hung draining executor.outputs after the run):
- SyncQueueIterator tracks why it closed: exhausted branches keep
  raising StopIteration, failed branches re-raise their error, and a
  branch closed before being fully consumed raises a descriptive
  RuntimeError instead of blocking forever on a dead pump
- executor cleanup aborts live fan-outs with FanoutStreamClosedError
  (new, in core.exceptions) so a late drain of an unconsumed branch
  fails loud; AsyncQueueBranch gains the same closed-state semantics
  and terminate(); async cleanup terminal-izes tracked branches
- wait_for_workers_after_shutdown gains a log grace window (2s default):
  the first poll almost always sees mid-teardown workers, and warning
  about a 0.1s teardown destroyed trust in the diagnostic
- nested_fanout pack step_results corrected to the observable contract
  (terminal drained EACH steps publish None)

#129 — unified runtime corpus:
- new tests/execution/test_corpus_runtime.py executes every registered
  pack on its own engine and asserts frozen step_results (replaces the
  two hand-maintained per-engine pack lists; engine step_results test
  files removed; error-materializer registration tests moved here)
- exclusions are explicit and symmetry-checked: complex_parallel and
  complex_parallel_mixed deadlock BOTH engines by design — their
  terminal step consumes sibling fan-out branches sequentially, which
  circular-backpressures a bounded lockstep handoff (issue #128)

#131 — fail-loud build validation (non-breaking parts):
- include-cycle detection unified (dag_expansion.validate_no_include_cycle)
- include return-annotation validated by type inspection instead of a
  name substring (aliases validate; look-alike names no longer pass);
  unresolvable string annotations keep the tolerant fallback
- get_safe_type_hints raises when a callable declares annotations that
  cannot be resolved — silent {} disabled all DAG validation; only
  fully-unannotated callables resolve to {}

#132 — small smells:
- AsyncQueueBranch.put blocks on the queue properly (wait_for-bounded)
  instead of a 1ms hot sleep loop
- CsvSerializer fills ragged rows with restval=""
- async error dispatch checks callable(err_mat) before building the
  context, mirroring sync
- both step runners replace the "output" not in locals() trick with an
  explicit _NOT_PRODUCED sentinel (identical semantics)

Tests: 832 passing (was 816). Suite grows by the runtime corpus (24
pack/engine cases), close-semantics unit tests for both handoffs, and
the hints contract tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant