Repository navigation
fix(execution): loud post-run streams, unified runtime corpus, fail-loud build validation - #134
Merged
Merged
Conversation
…s, fail-loud hints Follow-ups #128, #129, #131, #132 (behavioral items only). #128 — post-run stream ergonomics (corrected diagnosis: no scheduler deadlock; the original repro hung draining executor.outputs after the run): - SyncQueueIterator tracks why it closed: exhausted branches keep raising StopIteration, failed branches re-raise their error, and a branch closed before being fully consumed raises a descriptive RuntimeError instead of blocking forever on a dead pump - executor cleanup aborts live fan-outs with FanoutStreamClosedError (new, in core.exceptions) so a late drain of an unconsumed branch fails loud; AsyncQueueBranch gains the same closed-state semantics and terminate(); async cleanup terminal-izes tracked branches - wait_for_workers_after_shutdown gains a log grace window (2s default): the first poll almost always sees mid-teardown workers, and warning about a 0.1s teardown destroyed trust in the diagnostic - nested_fanout pack step_results corrected to the observable contract (terminal drained EACH steps publish None) #129 — unified runtime corpus: - new tests/execution/test_corpus_runtime.py executes every registered pack on its own engine and asserts frozen step_results (replaces the two hand-maintained per-engine pack lists; engine step_results test files removed; error-materializer registration tests moved here) - exclusions are explicit and symmetry-checked: complex_parallel and complex_parallel_mixed deadlock BOTH engines by design — their terminal step consumes sibling fan-out branches sequentially, which circular-backpressures a bounded lockstep handoff (issue #128) #131 — fail-loud build validation (non-breaking parts): - include-cycle detection unified (dag_expansion.validate_no_include_cycle) - include return-annotation validated by type inspection instead of a name substring (aliases validate; look-alike names no longer pass); unresolvable string annotations keep the tolerant fallback - get_safe_type_hints raises when a callable declares annotations that cannot be resolved — silent {} disabled all DAG validation; only fully-unannotated callables resolve to {} #132 — small smells: - AsyncQueueBranch.put blocks on the queue properly (wait_for-bounded) instead of a 1ms hot sleep loop - CsvSerializer fills ragged rows with restval="" - async error dispatch checks callable(err_mat) before building the context, mirroring sync - both step runners replace the "output" not in locals() trick with an explicit _NOT_PRODUCED sentinel (identical semantics) Tests: 832 passing (was 816). Suite grows by the runtime corpus (24 pack/engine cases), close-semantics unit tests for both handoffs, and the hints contract tests.
This was referenced Sep 20, 2026
refactor(core): fail loud on silent type-hint fallbacks (get_safe_type_hints, dag_dependencies)
#131
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Behavioral follow-ups from the audit, covering #128, #129, #131 and #132 (behavioral items only; #130 DagNode split and the GWT rename churn stay separate on purpose).
Important correction to #128 first: deeper investigation (watchdog +
faulthandlerstack dumps, 40 hot-loop + cold-process runs) showed the sync engine never deadlocked onnested_fanout— the original repro hung because my debug script drainedexecutor.outputsafterrun(). What actually hangs is that footgun, and it is what this PR fixes — plus the discovery thatcomplex_parallel*really do deadlock both engines by design (details below).Changes
#128 — loud post-run stream semantics
SyncQueueIteratornow tracks why it closed: exhausted branches keep raisingStopIteration(iterator protocol preserved), failed branches re-raise their error, and a branch closed before being fully consumed raises a descriptiveRuntimeErrorinstead of blocking forever on a dead pumpFanoutStreamClosedError(core.exceptions) — a late drain of an unconsumed branch fails loud instead of hanging or silently returning a truncated stream;AsyncQueueBranchgains identical closed-state semantics plusterminate(), and async cleanup terminal-izes tracked branches (parity)wait_for_workers_after_shutdowngains a log grace window (2s default): the first poll almost always sees mid-teardown workers, so the "4 workers blocked" warning fired on every healthy run (0.1s teardown) and destroyed trust in the diagnosticnested_fanoutpackstep_resultscorrected to the observable contract (terminal drained EACH steps publishNone)#129 — unified runtime corpus
tests/execution/test_corpus_runtime.pyexecutes every registered pack on its own engine and asserts frozenstep_results— replacing the two hand-maintained per-engine pack name tuples (the gap that letcomplex_parallel*/nested_fanoutsilently never run); the per-enginetest_runner_step_results.pyfiles are removed and the error-materializer registration tests moved into the unified modulecomplex_parallel+complex_parallel_mixeddeadlock BOTH engines by design — their terminal step consumes sibling fan-out branches sequentially, which circular-backpressures a bounded lockstep handoff. That consumption pattern violates the lockstep contract (§1.7);mixed_fanoutshows the supported zip pattern. Fixing it would require unbounded buffering, contradicting the memory model — documented + excluded instead (kept in fix(execution): nested fan-out/fan-in topology deadlocks the sync engine #128)#131 — fail-loud build validation (non-breaking parts)
dag_expansion.validate_no_include_cycle); the registry's instance-identity check stays separate (different semantics)NotTheParams) no longer passes; unresolvable string annotations keep the tolerant fallbackget_safe_type_hintsraises when a callable declares annotations that cannot be resolved — the previous silent{}disabled every validation that depends on hints; only fully-unannotated callables resolve to{}(PEP 563 still works — resolvable string annotations are resolved as before)#132 — small smells
AsyncQueueBranch.putblocks on the queue properly (wait_for-bounded, re-checking closure) instead of a 1 ms hot sleep loopCsvSerializerfills ragged rows withrestval=""callable(err_mat)before building the context, mirroring sync"output" not in locals()trick with an explicit_NOT_PRODUCEDsentinel (provably identical semantics)Tests
ruff format/F401and the pre-commit coverage gate all green