Add checkpointing support to dpsynth using etils.epath and MBI pytree serialization: - #166
Add checkpointing support to dpsynth using etils.epath and MBI pytree serialization:#166copybara-service[bot] wants to merge 1 commit into
Conversation
MuhammadNiazAli
left a comment
There was a problem hiding this comment.
The import reorganization and new checkpoint_lib import look clean. The new working_dir config field is well-documented in the docstring, including the None-disables-checkpointing behavior, which is good.
I've only reviewed swift.py here there are 3 more changed files I haven't looked at yet, so holding off on approving until I go through those too.
MuhammadNiazAli
left a comment
There was a problem hiding this comment.
The two-stage resume logic in swift.py reads well:
Full resume (model.npz + measurements.npz both exist) skips straight to synthesis using the cached model/measurements, correctly falling back to initial_measurements over the checkpointed ones when both are available (total_src = initial_measurements if initial_measurements else measurements).
Partial resume (marginals only) — loads marginals.npz if present, otherwise computes via mbi.CliqueVector.from_projectable and saves it before continuing. The assert answers is not None after load is a nice defensive touch even though exists() already gated it.
One thing I couldn't verify from the diff alone: checkpointer = checkpoint_lib.Checkpointer(self.config.working_dir) is constructed unconditionally, and the docstring says working_dir=None disables checkpointing. Worth confirming Checkpointer(None) actually makes exists() always return False and save()/load() no-ops otherwise this line would break for anyone not opting into checkpointing.
Also noticed in the full-resume branch, phase_times stays {} since none of the common.timed(...) blocks run that's expected, but if any downstream diagnostics code assumes specific phase keys are always present, it'd be worth a quick check.
test_checkpointing_saves_and_resumes in swift_test.py is a good addition — real temp dir, real save/load round-trip rather than mocking. The unused-looking from etils import epath import is that used later in the test for path handling, or is it dead? Couldn't tell from what's shown.
66a62f3 to
a2a4877
Compare
… serialization: - Checkpointer: Save/load intermediate mechanism state (exact marginals, noisy measurements, fitted models) to flat .npz files under a configurable working_dir. - SWIFT: Implemented the Stage-Gated Pipeline architecture, enabling: * Full resume from (model.npz, measurements.npz) directly to synthesis. * Resume from measurements.npz without re-measuring or consuming DP budget. * Resume from exact marginals.npz. - DiscreteMechanism: Added working_dir to DiscreteConfig, propagates working_dir to inner mechanisms, and checkpoints one_way_measurements.npz. - TabularConfig: Added working_dir to TabularConfig and propagates working_dir to discrete_mechanism. - common: Handle small domains (< 3 attributes) gracefully in compiled_workload and supporting_cliques. PiperOrigin-RevId: 972268686
a2a4877 to
2e22025
Compare
Add checkpointing support to dpsynth using etils.epath and MBI pytree serialization:
noisy measurements, fitted models) to flat .npz files under a configurable
working_dir.
to inner mechanisms, and checkpoints one_way_measurements.npz.
to discrete_mechanism.
and supporting_cliques.