From 0eebe6c824842f4cd6ba9c7e3092cc5d5f20cbdd Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Wed, 9 Sep 2026 18:11:24 -0500 Subject: [PATCH 1/2] scripts: make generated diagram figures byte-reproducible MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `gen_pipeline_diagrams.py --check` exists to catch renderer drift in CI, but it could not be trusted: one of the three figures disagreed with its own committed output about half the time, for reasons unrelated to the renderer. Two independent causes. `dj.Diagram` iterates sets of table-name strings on its emission path, so node and cluster order in the SVG follows `str.__hash__`, which Python salts per process. Two identical runs render the same picture — same coordinates, colors and edges — and emit it in a different order. The script now re-execs itself with `PYTHONHASHSEED=0` before importing datajoint, since that variable is only read at interpreter start. Filed upstream as datajoint-python#1551; the block comes out when that lands. `pydot` was unpinned, and its SVG writer changed between 3.x and 4.x — 4.0.1 drops the XML prolog and DOCTYPE, reorders `` attributes, drops cluster `` elements, and emits literal spaces where 3.x emitted ` `. None of it changes the rendering, all of it rewrites the file. Pinned to 3.0.4. Also replaces the "one collapsed edge is traversal-order dependent" caveat in the module docstring. That described pre-#1545 behavior, where a collapsed edge inherited the style of whichever foreign key in its bundle was visited first; bundle edges render uniformly as of DataJoint 2.3.3. --- pip_requirements.txt | 6 ++++++ scripts/gen_pipeline_diagrams.py | 28 ++++++++++++++++++++-------- 2 files changed, 26 insertions(+), 8 deletions(-) diff --git a/pip_requirements.txt b/pip_requirements.txt index c57ddd86..964a7866 100644 --- a/pip_requirements.txt +++ b/pip_requirements.txt @@ -11,5 +11,11 @@ mkdocs-literate-nav mkdocs-autorefs mkdocs-redirects +# Pinned so the committed diagram figures stay byte-reproducible: pydot's SVG +# writer changed between 3.x and 4.x (XML prolog, attribute order, cluster +# titles), which reshuffles every generated SVG without changing the picture. +# See scripts/gen_pipeline_diagrams.py. +pydot==3.0.4 + # DataJoint from master for tutorials git+https://github.com/datajoint/datajoint-python.git@master diff --git a/scripts/gen_pipeline_diagrams.py b/scripts/gen_pipeline_diagrams.py index 16c145ec..4d5819c2 100644 --- a/scripts/gen_pipeline_diagrams.py +++ b/scripts/gen_pipeline_diagrams.py @@ -49,14 +49,11 @@ by the pydot that produced the committed figures and as literal spaces by 4.0.1, which shows up as a whole-file diff with no visual change. Compare rendered content, not bytes, when the pydot version moves. Nothing pins pydot. -- **One collapsed edge is traversal-order dependent.** A collapsed edge inherits - the attributes of whichever foreign key in its bundle is visited first - (``diagram.py``, ``_collapse_graph``: ``if not new_graph.has_edge(...)``), with - no aggregation over the bundle. Where a bundle mixes a primary and a secondary - foreign key — ``lab -> session`` here, which bundles ``Subject -> Session`` - (primary) and ``User -> Session`` (secondary) — the edge renders solid or - dashed depending on order alone. The committed figure has it solid; this script - produces dashed. Both are outputs of the same renderer. +- **Node emission order is hash-seeded.** ``dj.Diagram`` iterates sets of + table-name strings when emitting, so node and cluster order follows the + per-process string hash. This script therefore re-execs itself with + ``PYTHONHASHSEED=0``; without that, two identical runs disagree on order while + rendering the same picture. Upstream: datajoint-python#1551. A non-empty diff after a DataJoint upgrade is the signal to review the notation and the surrounding prose together — see issue #246. @@ -68,6 +65,21 @@ import tempfile from pathlib import Path +# Re-exec with a fixed string-hash seed before anything imports datajoint. +# +# dj.Diagram iterates sets of table-name strings on its emission path, so the +# order nodes and clusters land in the SVG follows str.__hash__, which Python +# salts per process (PEP 456). The rendered picture is identical either way — +# same coordinates, colors and edges — but the bytes are not, which makes +# --check unusable and buries real notation changes in reordering noise. +# Upstream: datajoint/datajoint-python#1551. Remove this block once that lands. +# +# PYTHONHASHSEED only takes effect at interpreter start, so it cannot be set +# from inside the process. +if os.environ.get("PYTHONHASHSEED") != "0": + os.environ["PYTHONHASHSEED"] = "0" + os.execv(sys.executable, [sys.executable, *sys.argv]) + import datajoint as dj sys.path.insert(0, str(Path(__file__).resolve().parent)) From 389f611c112df34d9a1450604a278ccff9024409 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Wed, 9 Sep 2026 18:11:24 -0500 Subject: [PATCH 2/2] Regenerate diagram figures against released DataJoint 2.3.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One notation change, and it is the one datajoint-docs#266 anticipated: in `pipeline-modules-collapsed.svg` the `reference -> session` edge goes from dashed to solid. Before datajoint-python#1545 a collapsed edge took the style of an arbitrary member of its foreign-key bundle; now every bundle edge renders uniformly. #266 predicted `lab -> session` would be the visible one — the mechanism was right, the edge was not. Everything else is pydot serialization, not renderer drift. Verified rather than assumed: for all three figures the fill set, stroke set, node count, edge count and text labels are identical to the committed versions, and for the two unchanged figures the dasharray count matches too. The tier palette already matched released 2.3.3, so no colors moved. Regenerated with DataJoint 2.3.3 from PyPI, pydot 3.0.4 and graphviz 15.1.0. `--check` is now clean and stable across repeated runs. --- src/images/imaging-schema.svg | 39 ++- src/images/pipeline-modules-collapsed.svg | 115 ++++----- src/images/pipeline-modules.svg | 293 +++++++++++----------- 3 files changed, 213 insertions(+), 234 deletions(-) diff --git a/src/images/imaging-schema.svg b/src/images/imaging-schema.svg index 7bda33a1..99fc25ef 100644 --- a/src/images/imaging-schema.svg +++ b/src/images/imaging-schema.svg @@ -1,11 +1,4 @@ - - - - - + + \ No newline at end of file diff --git a/src/images/pipeline-modules-collapsed.svg b/src/images/pipeline-modules-collapsed.svg index 66ab8dc9..d1f2e8a6 100644 --- a/src/images/pipeline-modules-collapsed.svg +++ b/src/images/pipeline-modules-collapsed.svg @@ -1,11 +1,4 @@ - - - - - + + \ No newline at end of file diff --git a/src/images/pipeline-modules.svg b/src/images/pipeline-modules.svg index 213c7c77..b89778a4 100644 --- a/src/images/pipeline-modules.svg +++ b/src/images/pipeline-modules.svg @@ -1,11 +1,4 @@ - - - - - + + \ No newline at end of file