ENH: Support for specific node caching

[Carwyn Pelley (cpelley)]

Utilises existing DAGrunner cache support to provide:

• Per-node cache control, extending the previously supported global on/off behaviour.
• Atomic cache writes to improve robustness and prevent partial cache files.
• Cache filenames now incorporate a tokenised extract value to guarantee uniqueness and avoid collisions.
• Additional unit test coverage.
• Dependency updates, including correcting an overly restrictive Python version upper bound that inadvertently limited support to Python 3.9 Needed updating now so as not to unnecessarily handle differences in tokenization with dask.

This change may be relevant to EPP, based on my current understanding of the caching-related issue they are investigating.

[Rob Purvis (mo-robert-purvis)]

Looks good. Tests pass as required.

[Copilot]

Copilot review overview

🟡 Changes recommended

There are correctness/security issues in the cache file path construction and the cache “atomic” write behavior, plus a dry-run caching bug and a test isolation issue that should be fixed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 1

Open (3)

• Include arguments in cache key or correct the docstring · New
• Sanitize node_id before using it in cache filenames · New
• Deep-copy nested configuration dictionaries for test isolation · New

What changed in this PR

This PR enhances DAGrunner’s caching capabilities by allowing per-node opt-in caching, improving cache file naming uniqueness, and adding/adjusting tests around cache utilization.

Changes:

• Added cache_result to plugin_executor to enable caching on a per-node basis (in addition to the existing global cache toggle).
• Updated _PickleCache pickle filename scheme to include a tokenized component and changed the write path intended to be more atomic.
• Added a unit test for single-node caching and updated existing cache-related test naming/docs.

File	Description
dagrunner/​utils/​_cache.py	Adjusts cache file naming and cache write behavior.
dagrunner/​execute_graph.py	Adds per-node caching control via cache_result and documents it.
dagrunner/​tests/​execute_graph/​test_plugin_executor.py	Renames global cache test and adds a single-node cache test + fixture.
dagrunner/​tests/​execute_graph/​test_integration.py	Updates cache utilization test docstring to reflect graph filtering behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Joshua Wiggs (JoshuaWiggs)]

Should this be configurable? Do we want this to always be the behavior that occurs when a file exists already?

[Carwyn Pelley (cpelley)]

This is an edge-case scenario.

Under normal operation, if a cache entry already exists, plugin_executor does not execute the node. Instead, it loads and returns the cached result (i.e. it wouldn't get to this exception).

This particular test covers the case where two identical nodes, called with the same input arguments, are executed concurrently from different graphs and neither finds an existing cache entry when execution begins.

In this situation, both executions proceed independently. By the time the second execution completes, the first may already have written the cache file. There is no meaningful distinction between the two results, as they are produced by the same node with the same inputs. The only difference is that they do not finish at exactly the same time.

In other words, this scenario only arises when two identical node executions, with identical input data, are running concurrently. Users who expect this pattern should likely configure a dedicated cache directory as described in the global configuration documentation so that cache doesn't collide.