Repository navigation
fix(execution): align async engine with sync engine semantics (audit follow-up) - #133
Merged
Merged
Conversation
added 4 commits
September 19, 2026 22:20
…s canonical) Body: ports the async engine onto the sync engine's observable contract. - publish() routes on compiled publish_plan.strategy and completion_policy - output contracts validated at publish (TypeError), like sync - materializer crashes propagate; mid-stream item failures keep sync prefix semantics (valid prefix + had_error) with lazy out-of-core access preserved - producer failures delivered in-band to consumers under any on_error policy (no silent truncation under CONTINUE) - PipelineStopException queued to consumers carries its cause - execute() awaits cleanup in finally; cleanup() logs instead of silent pass - terminal lazy streams stored in executor.outputs like sync - builder compiles fn_kind + async_stream_deps; runtimes consult metadata, not runtime introspection - builder compiles publish_async_fanout for single-consumer async streams (EACH consumers unroll via queue branches) - dropped dead _is_stream_output helpers Tests updated to canonical contract: annotated under-annotated async generators, fixed the async threshold helper that silently disconnected producer from consumer (broken copy of sync twin), mirrored in-band failure handoff test in sync.
…ors, fail loud on cycles
- delete the unreachable 'requires a custom materializer' validation in
_resolve_materializers (the factory identity it checked had already
been resolved away) together with its now-unused _is_builtin_type
helper; a regression test locks in that the default memory factory
handles custom item types
- collapse _validate_no_async_handlers/_validate_no_sync_handlers
(~130 near-duplicate lines) into one parameterized
_validate_handler_execution_modes; error messages preserved
- replace the vacuous _validate_declared_step_names duplicate-name call
(it passed an empty dag, so only the reserved check could fire) with
an explicit validate_reserved_step_name; duplicate detection stays in
the expanded pass where it works
- get_execution_levels now raises a descriptive ValueError when the
remaining nodes form a cycle instead of silently returning a partial
topological order (regression test added)
- delete dead _check_scalar_producer_to_iterable_consumer
- drop the __import__("typing").Union hack; reuse core
is_async_callable in composite materializers (superset: also detects
partials and async callable objects)
- add module docstring to composite.py
… drift, harden corpus hygiene - nested_fanout corpus packs existed in both engines but were never registered in _MODULES — dead specification. Both packs are now complete (frozen json_dag, execution levels, step results) and registered; the build-time corpus tests now cover the nested fan-out/fan-in topology. NOTE: executing this topology currently deadlocks the sync engine (workers block) — tracked separately; the runtime step-results assertions stay unregistered until that is fixed (see step_results explicit pack lists) - fix real sync/async drift in the linear corpus pack: the async twin silently dropped the step-level observer its sync twin declares (with the frozen observers json entry) — mirrored with an async-adapted handler - add a corpus symmetry meta-test: pack sets must match across engines (modulo the documented sync-only threadpool pack) so an orphaned or engine-missing pack can never happen silently again - test_runner_step_results (both engines): a failing iterator now propagates instead of being swallowed into a truncated 'passing' list - conftest run_pipeline fixture: document why it is sync-only (pipelines are engine-specific by contract; parity lives in the twins + corpus + meta-test) and drop the misleading dead async branch - PipelinePack.expected_dag was never populated and never read: removed - test_dag_builder_future docstring no longer claims xfail semantics for passing contract tests
- add purposeful module docstrings to the modules flagged in the audit (core build/validation modules, both executors and step runners, handoff, stats, overrides, materializers) - translate ExecutionState docstrings to English (language consistency) and make seed() fail loud on unsupported params types instead of crashing on AttributeError (plain dict now accepted) - type the previously untyped _identity / async_adapter signatures - add direct unit tests for ExecutionState (seeding, consumer-scoped keys, inputs_available) — previously covered only transitively; live under tests/execution/ since the module is engine-agnostic
This was referenced Sep 20, 2026
refactor(core): fail loud on silent type-hint fallbacks (get_safe_type_hints, dag_dependencies)
#131
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full-audit follow-up. The sync engine runs in production and is the behavioral reference; wherever the engines diverged, this PR changes the async engine to match sync — sync behavior is untouched except where the same decision was hoisted from runtime introspection into compiled DAG metadata (provably identical values, computed earlier).
The audit also surfaced concrete bugs (a validation that could never fire, a vacuous duplicate-name check, a broken async test helper that silently disconnected its producer, and real corpus drift) plus a hidden deadlock discovered by registering the orphaned
nested_fanoutcorpus packs (#128).Changes
Async → sync parity (the engine drifts are user-visible contracts):
publish()routes on the compiledpublish_plan.strategyand readscompletion_policyfromoutput_contractinstead of re-deriving fromnode.mode+ runtime output type — completion events now fire at the same point in both enginesTypeErroron mismatch), like sync; value-compiled async steps can no longer silently return streamsCONTINUE); mid-stream item failures keep the sync prefix semantics (valid prefix +had_error) while preserving lazy, out-of-core materializer accesson_errorpolicy — no more silent stream truncation underCONTINUE(mirrorsSyncFanout'sExceptionMarkerdelivery); queuedPipelineStopExceptions carry theircauseAsyncPipelineExecutor.execute()awaitscleanup()infinallyon failure (no orphaned pump tasks);cleanup()logs instead of bareexcept: passexecutor.outputslike syncBuild/run separation:
fn_kind(sync/sync_generator/async/async_generator) andasync_stream_depsinto the DAG; runtimes consult that metadata instead ofinspect.is*function/ annotation-__origin__re-derivationspublish_async_fanoutfor single-consumer async streams too — EACH-mode consumers unroll through queue branches by contract, so the single-consumer case was always queue-delivered in practice; the DAG JSON now tells the truth about itCore fixes:
_is_builtin_type; regression test locks in that the default memory factory handles custom item types_validate_no_async_handlers/_validate_no_sync_handlers) into one parameterized check; error messages preserved_validate_declared_step_namespassed an empty dag to the duplicate check (vacuous) — replaced with an explicit reserved-name check; real duplicate detection stays in the expanded passget_execution_levels()raises a descriptiveValueErroron cycles instead of silently returning a partial topological order_check_scalar_producer_to_iterable_consumerand the__import__("typing")hack; composite materializers reuse the coreis_async_callable(superset: also detects partials/async callables)Test infrastructure:
nested_fanoutcorpus packs (both engines) with full build-time specs; discovered the topology deadlocks the sync engine — tracked in fix(execution): nested fan-out/fan-in topology deadlocks the sync engine #128, runtime assertions stay unregistered until thenlineartwin had silently dropped its step-level observer (and the frozen observers JSON entry)test_runner_step_results(both engines): a failing iterator propagates instead of being swallowed into a truncated "passing" listrun_pipelinefixture is sync-only (pipelines are engine-specific by contract — test: engine-agnostic dual-engine corpus execution (unlock shared run_pipeline fixture) #129 tracks the dual-engine design)ExecutionState; module docstrings for the ~25 flagged modules; Englishstate.pydocs with fail-loudseed(); typed_identity/async_adapterDeferred items are filed as issues #128–#132.
Tests
output_contract/publish_planbefore themruff format --check+ruff check --select F401clean; pre-commit patch coverage gate passesfndirectly where the sync twin wraps it, silently disconnecting producer → consumer; the handoff failure test asserted truncation). The sync side was not weakened anywhere.