refactor[next]: [experiment] replace factory-boy factories with TypedDict option builders

[egparedes]

Experiment, not meant to be merged as is. It implements the alternative design suggested in the review of #2808, so the two can be compared on real code. It only replaces the factory-boy factories: none of the rest of the OTF refactoring stack (#2741, #2742, #2743) is included, and it branches from main.

Design

• Shared settings are keyword arguments of the builders: device (gpu / device_type), translation caching, CMake build type, unit-stride layout, and for DaCe auto-optimize and the external workspace. The builders forward them by hand to every step that needs them. Unset build type and unit-stride layout are read from config when the builder is called.

• Step-local settings are one TypedDict per step, unpacked into the step's constructor. The keys are the step's own field names, and the shared settings are not in these dicts, so they cannot be set for one step alone:

  • GTFNTranslationOptions: code_spec, enable_itir_transforms, symbolic_domain_sizes, use_max_domain_range_on_unstructured_shift
  • GTFNBuildSystemOptions: cmake_extra_flags, renew_compiledb
  • GTFNCompilationOptions: fingerprint_builder_factory, force_recompile
  • DaCeTranslationOptions: auto_optimize_args, async_sdfg_call, use_metrics, disable_itir_transforms, disable_field_origin_on_program_arguments, use_max_domain_range_on_unstructured_shift
  • DaCeCompilationOptions: add_gpu_trace_markers

  The GTFN ones live in runners.gtfn, the DaCe ones in runners.dace.workflow.factory.

• Replacing a whole step is dataclasses.replace on the built workflow or backend, and the replacement is used as given. There is no injection mechanism, no config object, and no check that steps agree with each other. Each step only validates its own settings: DaCeTranslator checks its auto_optimize_args on construction.

from gt4py.next.program_processors.runners import gtfn
from gt4py.next.program_processors.runners.dace import make_dace_backend

gtfn.make_gtfn_backend(
    gpu=True,
    name_postfix="_no_transforms",
    translation={"enable_itir_transforms": False},
)

make_dace_backend(
    gpu=False,
    translation={"use_metrics": False, "auto_optimize_args": {"blocking_size": 10}},
)

What the checkers catch

• mypy, at the call site: a misspelled key, a value of the wrong type, and a shared setting passed inside an options dict (for example translation={"device_type": ...}). The error names the argument but not the offending key.
• mypy, inside the builders: the unpacking into the step constructor is checked, so a key that no longer matches a step field is an error there.
• At runtime: a shared setting in an options dict raises TypeError: ... got multiple values for keyword argument. An auto_optimize_args key the translator derives itself (gpu, constant_symbols, unit_strides_kind) raises ValueError when the translator is created.

Differences from main

• factory-boy moves from the runtime dependencies to the test group.

• Removed: GTFNTranslationStepFactory, GTFNCompilerFactory, GTFNCompileWorkflowFactory, GTFNBackendFactory, DaCeTranslationStepFactory, DaCeCompilationStepFactory, DaCeWorkflowFactory, DaCeBackendFactory.

• New: make_gtfn_compile_workflow, make_gtfn_backend, make_dace_compile_workflow (replacing DaCeWorkflowFactory) and the options TypedDicts.

• Breaking: make_dace_backend keeps gpu and auto_optimize as positional-or-keyword arguments, and external_workspace and unstructured_horizontal_has_unit_stride as keyword-only. It gains cached_translation, cmake_build_type and compilation. The translation settings move into translation={...}, spelled with the step's field names:

  • optimization_args becomes auto_optimize_args;
  • use_zero_origin becomes disable_field_origin_on_program_arguments;
  • async_sdfg_call, use_metrics and use_max_domain_range_on_unstructured_shift keep their names.

• Renamed: run_gtfn_no_transforms is now named run_gtfn_cpu_no_transforms. It was run_gtfn_cpu, colliding with run_gtfn; refactor[next]: replace factory-boy factories with plain builders #2808 fixes the same bug.

• DaCeTranslator validates and derives its own optimization arguments.

  • It derives unit_strides_kind next to gpu and constant_symbols.
  • On construction it rejects those three keys in auto_optimize_args, whether or not auto-optimize is enabled. On main, make_dace_backend only warned when auto-optimize was off.
  • It warns about the remaining arguments when auto-optimize is off.

  These checks therefore also cover make_dace_compile_workflow and translators built directly.

• make_dace_backend only keeps what belongs to the backend: the external workspace and its transient_memory_mode. With a workspace, auto-optimize enabled and no explicit mode, the mode defaults to EXTERNAL. Without auto-optimize the mode is not used, so it is no longer set.

• async_sdfg_call defaults to True in both DaCe builders and is no longer restricted to GPU in the builder: the SDFG call helpers are already no-ops on CPU.

• Shared helpers remove the copies between the GTFN and DaCe builders: cache.persistent_translation_cache(step, backend) and backend.select_device(gpu). The device selection is the same as on main for both families (CUPY_DEVICE_TYPE or CUDA on GPU).

• Pre-built backends:

  • The GTFN ones are field-by-field identical to main, apart from the rename above.
  • The DaCe translators no longer store unit_strides_kind in auto_optimize_args, and the CPU ones store async_sdfg_call=True.
  • The generated SDFGs are unchanged, but DaCe translation-cache keys rotate once.

Compared with #2808

	#2808 (config + step builders)	This PR (TypedDict options)
Change one inner setting	translation=partial(make_gtfn_translation, enable_itir_transforms=False)	translation={"enable_itir_transforms": False}
Replace a whole step	pass any Callable[[Config], Step]; it is still cached	dataclasses.replace on the built object; caching must be re-applied by hand
Shared settings	one config object, read by each step builder	keyword arguments, forwarded by hand through each builder layer
What a builder uses	not visible in its signature	visible: shared keyword arguments plus one dict per step
Where step construction lives	one small builder per step	inline in the compile-workflow builder

Open questions

• make_dace_backend's breaking change. Its flat translation keywords are gone with no compatibility path, which affects icon4py.
• No ADR. If this design is chosen, ADR 0028 would be rewritten for it.
• No GPU CI yet. The CSCS GPU pipeline has not run on this draft.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot review overview

🔵 Needs a closer look

It is explicitly experimental, introduces breaking public APIs, and has unresolved validation, documentation, and coverage issues.

Review effort: Balanced
Findings: 4 · 2

Open (6)

• Document or avoid ROCm GTFN device selection change · New
• Test None fallbacks for direct DaCe workflow · New
• Reject derived optimization keys regardless of auto_optimize · New
• Test call-time fallback for unit-stride configuration · New
• Clarify sentence and refer to the returned workflow · New
• Document workspace inference behavior when auto_optimize is disabled · New

[egparedes]

Valid, added in 1194174: test_compile_workflow_defaults_follow_config monkeypatches UNSTRUCTURED_HORIZONTAL_HAS_UNIT_STRIDE and CMAKE_BUILD_TYPE, builds the workflow with make_dace_compile_workflow() and checks the translator and the compiler.

[egparedes]

Valid, added in 1194174: test_compile_workflow_unit_stride_default_follows_config monkeypatches the config value and checks the ExtensionGenerator built by make_gtfn_compile_workflow(). The CMake build-type fallback was already covered by test_make_gtfn_backend_build_type_config.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Three moderate findings require additional regression coverage or assertions.

Review effort: Lite
Findings: 2

Open (2)

• Test call-time fallback for unit-stride configuration
• Test None fallbacks for direct DaCe workflow

Resolved since last review (4)

• Reject derived optimization keys regardless of auto_optimize
• Document or avoid ROCm GTFN device selection change
• Document workspace inference behavior when auto_optimize is disabled
• Clarify sentence and refer to the returned workflow

Previously missed (5)

In code that hasn't changed since last review

Add regression test for external workspace without auto-optimization

src/​gt4py/​next/​program_processors/​runners/​dace/​workflow/​backend.py:103

The new external_workspace path intentionally leaves transient_memory_mode unset when auto_optimize=False, but the test suite only covers workspace inference with auto-optimize enabled. Add a regression test for the disabled-auto-optimize case to lock in the documented behavior and prevent this branch from restoring an unused EXTERNAL mode.

Test cached translation in the direct DaCe workflow builder

src/​gt4py/​next/​program_processors/​runners/​dace/​workflow/​factory.py:99

The new cached_translation branch of the direct DaCe workflow builder is not covered by a test: the DaCe builder is only called with cached_translation=False in the new builder-comparison test, while the cache assertions cover GTFN. Add a direct-builder assertion that cached_translation=True returns a CachedStep (and false returns the bare translator) so this option cannot silently stop being honored.

Update cache documentation for builder-based configuration

src/​gt4py/​next/​otf/​compilation/​cache.py:71

The cache module still documents TRANSLATION_CACHE_BACKENDS as being enabled by a workflow factory's cached_translation trait, although this PR removes those traits and adds builder functions instead. Update that comment to describe the builder option so the cache contract remains accurate.

Update module documentation to remove obsolete factory references

src/​gt4py/​next/​program_processors/​runners/​dace/​workflow/​__init__.py:19

The module docstring still says the backend module uses factory to define the workflow, but this change removes the factory-boy workflow factories and replaces them with builders. Update that sentence so the documentation does not describe a dependency and construction mechanism that no longer exists.

Correct misspelled overridden error message

src/​gt4py/​next/​program_processors/​runners/​dace/​workflow/​translation.py:383

The new error message spells overridden as overriden. Please correct the spelling and update the matching test expectations so callers receive a correctly worded validation error.

[egparedes]

Closing in favor of #2808. This experiment implemented the TypedDict-options alternative so the two designs could be compared on real code. An independent comparison recommended #2808's config + step-builder design. Its advantages: it preserves main's behavior and make_dace_backend compatibility, it keeps the shared settings in one place, it gives sharper mypy messages, and a replaced step keeps its caching. The parts of this experiment that were better have been ported to #2808: the DaCe optimization arguments are validated in DaCeTranslator itself, the translation-cache wrapping is shared in cache.persistent_translation_cache, and the step builders take the step's own field names and reach disable_itir_transforms and add_gpu_trace_markers.