From 900fc4f066aac51b0bd3e74c64480f60405523d3 Mon Sep 17 00:00:00 2001 From: Ryan McKenna Date: Fri, 4 Sep 2026 13:16:40 -0700 Subject: [PATCH] Fix ReadTheDocs build and add documentation pages on the MechanismConfig vs. CalibratedMechanism API, along with configure(zcdp_rho) vs. calibrate(epsilon, delta) patterns in the library. PiperOrigin-RevId: 976463690 --- docs/conf.py | 13 +- docs/contributors_guide.md | 64 +++++++- docs/index.md | 15 +- docs/mechanism_api.md | 300 +++++++++++++++++++++++++++++++++++++ docs/scalable_beam_api.md | 79 +++++++++- docs/sitemap.md | 13 ++ 6 files changed, 479 insertions(+), 5 deletions(-) create mode 100644 docs/mechanism_api.md diff --git a/docs/conf.py b/docs/conf.py index c15e515c..a9cda9ae 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -45,9 +45,20 @@ 'show-inheritance': True, } -# templates_path = ['_templates'] +templates_path = ['_templates'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] +# Enable MyST-Parser extensions for math rendering ($...$ and AMS environments). +myst_enable_extensions = [ + 'dollarmath', + 'amsmath', +] +myst_heading_anchors = 3 + +suppress_warnings = [ + 'misc.highlighting_failure', +] + # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output diff --git a/docs/contributors_guide.md b/docs/contributors_guide.md index 2ee1df9c..a5e2cd84 100644 --- a/docs/contributors_guide.md +++ b/docs/contributors_guide.md @@ -130,7 +130,69 @@ without changing a single line of code. testing and debugging. * **`pipeline_dp.BeamBackend`**: Executes on Apache Beam. -[PipelineDP](https://github.com/OpenMined/PipelineDP) + In open-source deployments, it allows DPSynth to run on Apache Spark + or Google Cloud Dataflow. + +### 3. Designing Pipeline Functions + +When writing new transformations in `pipeline_transformations/`, functions +**MUST** accept a `backend: PipelineBackend` instance as an argument and use its +methods exclusively for all data transformations. + +```python +from pipeline_dp import PipelineBackend + +def compute_custom_column_stats(backend: PipelineBackend, input_collection): + # input_collection is framework-agnostic (PCollection or Python list) + # Every operation MUST include a clear, descriptive stage_name + + # 1. Map: Extract column value + col_values = backend.map(input_collection, lambda row: row[2], "ExtractColumnValue") + + # 2. Filter: Remove missing entries + filtered_values = backend.filter(col_values, lambda x: x is not None, "FilterNonNull") + + # 3. Key: Pair with constant key + keyed_data = backend.map(filtered_values, lambda x: ("all_records", x), "AddKey") + + # 4. Group & Reduce: Sum values per key + grouped_data = backend.group_by_key(keyed_data, "GroupByKey") + summed_data = backend.map_values(grouped_data, sum, "SumValues") + + return summed_data +``` + +### 3. Core `PipelineBackend` Methods Available + +* `map(col, fn, stage_name)`: Apply a function to each element. +* `flat_map(col, fn, stage_name)`: Apply a function returning an iterable, + flattening results. +* `filter(col, fn, stage_name)`: Keep elements where `fn` returns `True`. +* `group_by_key(col, stage_name)`: Group key-value pairs by key. +* `map_values(col, fn, stage_name)`: Apply a function to values of key-value + pairs. +* `sum_per_key(col, stage_name)` / `reduce_per_key(col, fn, stage_name)`: + Aggregate values per key. +* `flatten(cols, stage_name)`: Merge multiple collections into one. +* `distinct(col, stage_name)`: Get unique elements. + +### 4. Critical Constraints & Safety Guardrails + +> [!CAUTION] **Never use framework-specific operations** like `beam.Map`, +> `beam.Filter`, or the Beam pipe operator (`|`) inside functions designed to be +> backend-agnostic. All transformations must route through methods of the passed +> `backend` object. +> +> **Never assume collection types**. `input_collection` will be a Python +> iterable for `LocalBackend` but a `PCollection` for `BeamBackend`. Pipeline +> logic must never invoke methods specific to `list` or `PCollection`. + +### 5. PipelineDP Integration & DP Aggregations + +While the `PipelineBackend` abstraction is used to write the framework-agnostic +data manipulation pipeline, the actual **Differentially Private Aggregations** +are powered by the `pipeline_dp.DPEngine` object from +the [PipelineDP](https://github.com/OpenMined/PipelineDP) library. When contributing new mechanisms or data transformations, verify that you diff --git a/docs/index.md b/docs/index.md index 3576de82..1c563081 100644 --- a/docs/index.md +++ b/docs/index.md @@ -77,7 +77,9 @@ interface. * **Execution**: Runs on Apache Beam (or Apache Spark in open-source deployments). -[`bin/run_data_generation.py`](../bin/run_data_generation.py). * **Documentation**: +* **Inputs/Outputs**: (Sharded) CSV and TFRecord. +* **CLI + Binary**: [`bin/run_data_generation.py`](../bin/run_data_generation.py). * **Documentation**: [Scalable PipelineBackend API Guide](scalable_beam_api.md). -------------------------------------------------------------------------------- @@ -102,6 +104,9 @@ contributor expanding the library, explore the documentation below: * **[Contributor & Architecture Guide](contributors_guide.md)**: Architectural separation, core abstractions, PipelineBackend programming rules, diagnostics, and the Tabular Evaluation framework. +* **[Mechanism API Architecture](mechanism_api.md)**: Design decisions, + the 3-step pipeline, configure vs. calibrate semantics, and tight + privacy accounting. -------------------------------------------------------------------------------- @@ -148,6 +153,14 @@ processing_lifecycle contributors_guide ``` +```{toctree} +:maxdepth: 2 +:caption: Developer documentation: +:hidden: + +mechanism_api +``` + ```{toctree} :maxdepth: 2 :caption: API Reference diff --git a/docs/mechanism_api.md b/docs/mechanism_api.md new file mode 100644 index 00000000..d7fefdc6 --- /dev/null +++ b/docs/mechanism_api.md @@ -0,0 +1,300 @@ + + +# Mechanism API Architecture: Design, Lifecycle, & Accounting + +This document covers the architectural foundations and design decisions +underlying the DPSynth Mechanism API ({mod}`dpsynth.api`). It is written for +developers extending DPSynth with new synthesis algorithms, integrating +mechanisms into data pipelines, or auditing privacy accounting correctness. + +-------------------------------------------------------------------------------- + +(design-decisions)= +## Design Decisions: `MechanismConfig` vs. `CalibratedMechanism` + +DPSynth structures all differential privacy (DP) algorithms around two core +abstractions: + +1. **{class}`~dpsynth.MechanismConfig`**: An abstract base class defining + the interface for an configuration recipe for a `CalibratedMechanism`. + Subclasses are generally lightweight frozen dataclasses defined in terms of + basic python primitives (mechanism hyper-parameters), and can hence be + serialized and restored via human- and machine-readable formats like YAML. + This abstract base class defines two key methods: `configure()` and + `calibrate()`, both of which consume domain information and a privacy + budget, and return a `CalibratedMechanism`. +2. **{class}`~dpsynth.CalibratedMechanism`**: An abstract base class + representing a runnable mechanism with concrete privacy parameters bound. + The abstract two key methods: `dp_event` and `__call__`. The former provides + an exact characterization of the mechanisms privacy properties in the + language of `dp_accounting`, and the latter allows you to run the mechanism + on actual data. The format of the data can vary between subclasses. + +### Architectural Motivation: Decoupling Blueprint from Execution + +Earlier prototypes combined hyperparameters, mutable runtime state, privacy +budget allocation, and synthesis execution into single classes. This pattern +created several fundamental issues: + +* **State Mutability & Lifecycle Bugs**: Objects transitioned through + unconfigured and configured states, where calling execution before + configuration caused runtime failures or cross-run state contamination. +* **Nullable Fields**: Privacy parameters (such as noise standard deviations + $\sigma$ or selection thresholds) were unknown at instantiation, requiring + fields to be typed as `float | None` and necessitating defensive assertions. +* **Serialization Ambiguity**: Serializing an algorithm configuration risked + accidentally capturing transient runtime artifacts or failing on + non-serializable mathematical state (such as graphical model clique + vectors). + +To resolve this, DPSynth strictly decouples the immutable **recipe** +({class}`~dpsynth.api.MechanismConfig`) from the immutable **runnable instance** +({class}`~dpsynth.api.CalibratedMechanism`): + +| Dimension | {class}`~dpsynth.api.MechanismConfig` | {class}`~dpsynth.api.CalibratedMechanism` | +| :--- | :--- | :--- | +| **Role** | Hyperparameter recipe / blueprint | Executable mechanism instance | +| **Mutability** | Frozen dataclass (immutable) | Frozen dataclass (immutable) | +| **Data Dependencies** | None (independent of data & budget) | Bound to domain & concrete budget | +| **Privacy Parameters** | None (holds only budget fractions) | Fully concrete ($\sigma$, thresholds) | +| **Serialization** | Fully serializable to/from YAML | Runtime only (not serialized) | +| **Interface** | `.configure()`, `.calibrate()` | `dp_event`, `__call__(rng, data)` | + +By enforcing this separation: + +1. **Immutability**: Once constructed, a + {class}`~dpsynth.api.CalibratedMechanism` cannot be modified or + un-calibrated. +2. **Strict Type Safety**: All parameters on + {class}`~dpsynth.api.CalibratedMechanism` are non-nullable; no defensive + assertions are required. +3. **Clean Separation of Concerns**: Algorithm designers specify *how to + partition budget* in the config, while the calibrated mechanism focuses + purely on *how to run* on data. + +-------------------------------------------------------------------------------- + +(three-step-pipeline)= +## The 3-Step Mechanism Pipeline + +All DP data generation in DPSynth follows a standardized 3-step lifecycle: + +### Step 1: Reusable Configuration + +In Step 1, the user or pipeline defines a `MechanismConfig`. This configuration +is completely independent of the dataset size, record values, and privacy +parameters. It can be safely reused across multiple datasets, shared in model +registries, or persisted in configuration files. + +```python +import dpsynth + +# Step 1: Instantiate a reusable configuration recipe. +config = dpsynth.TabularConfig( + discrete_mechanism=dpsynth.discrete_mechanisms.AIMConfig( + pgm_iters=5000, + max_marginal_size=10_000_000, + select_budget_fraction=0.5, + ), + numerical_bins=32, + init_budget_fraction=0.1, +) +``` + +Because `config` contains no private data or privacy parameters, it can be +serialized to disk via `dpsynth.to_yaml(config)`. + +### Step 2: Calibrate to Data Domain & Privacy Parameters + +In Step 2, the configuration recipe is bound to a specific domain and +target $(\varepsilon, \delta)$-DP budget via +{meth}`~dpsynth.api.MechanismConfig.calibrate`: + +1. **A Domain/Schema**: Specifies attribute classifications (categorical, + numerical, open-set), value domains, and cross-attribute constraints. +2. **A Privacy Budget $(\varepsilon, \delta)$**: Target differential privacy + parameters. + +This produces a concrete, runnable {class}`~dpsynth.api.CalibratedMechanism`: + +```python +# Define domain schema: +schema = dpsynth.Schema({ + "age": dpsynth.NumericalAttribute(min_value=18, max_value=90), + "education": dpsynth.CategoricalAttribute( + categories=["High School", "Bachelors", "Masters", "Doctorate"] + ), + "income": dpsynth.NumericalAttribute(min_value=0, max_value=250_000), +}) + +# Calibrate to target (epsilon, delta)-DP: +calibrated = config.calibrate(schema, epsilon=1.0, delta=1e-5) +``` + +### Step 3: Run on Sensitive Data + +In Step 3, the calibrated mechanism is executed on sensitive data. The +calibrated instance is directly callable ({meth}`~dpsynth.api.CalibratedMechanism.__call__`): + +```python +import numpy as np +import pandas as pd + +sensitive_df = pd.read_csv("sensitive_data.csv") + +rng = np.random.default_rng(seed=42) +result = calibrated(rng, sensitive_df) + +synthetic_df = result.synthetic_data +``` + +-------------------------------------------------------------------------------- + +(configure-vs-calibrate)= +## `configure(zcdp_rho)` vs. `calibrate(epsilon, delta)` + +A common question for developers is: *Why are there two methods for binding a +privacy budget, and how do they interact?* + +```python +class MechanismConfig(abc.ABC): + + @abc.abstractmethod + def configure( + self, domain=None, *, zcdp_rho: float, delta: float = 0.0 + ) -> CalibratedMechanism: + """Low-level primitive: map zCDP rho to concrete parameters.""" + + def calibrate( + self, domain=None, *, epsilon: float, delta: float, + ) -> CalibratedMechanism: + """High-level entry point: numerical search over rho to satisfy (ε, δ).""" +``` + +`configure(zcdp_rho)` is the abstract primitive that every `MechanismConfig` +subclass **must** implement. Its responsibilities are: + +1. **Closed-Form Noise Derivation**: Map the scalar $\rho$-zCDP budget directly + to the mechanisms natural parameters (see {ref}`natural-parameters`). Since + this mapping is often very simple, it should execute very quickly. +2. **Deterministic Budget Splitting**: Subdivide $\rho$ across different + sub-mechanisms (e.g., per-column initialization + base mechanism). + +```{important} +The CalibratedMechanism returned by `configure()` should satisfy rho-zCDP, but +this is not the tightest characterization of the privacy properties of the +mechanism! The exact `dp_event` associated with the mechanism can be obtained +via the property `CalibratedMechanism.dp_event`. + +**Exercise for the Reader:** Configure the `DirectMechanism` with zcdp_rho=1.0 +and compute epsilon for delta=1e-5 using two different methods: + +1. Using the formula $\epsilon = \rho + 2 \cdot \sqrt(\rho \cdot \ln(1/\delta))$, + or any other zCDP -> DP conversion formula. +2. Using dp_accounting directly on the `calibrated.dp_event`. + +(2) should yield a strictly smaller epsilon than (1). Understanding +this point is critical to understand the design decisions and correctness of the +`configure` and `calibrate` APIs. +``` + +### `calibrate(epsilon, delta)`: + +`calibrate()` is a concrete method defined once on the base +{class}`~dpsynth.api.MechanismConfig` class. It serves users who want to specify +a specific $(\varepsilon, \delta)$ target. The zcdp_rho intermediate +representation is mostly hidden from users of `calibrate()`. Calibrate is +implemented internally using `dp_accouning.calibrate_dp_mechanism`, using +zcdp_rho as the parameter to calibrate and the `dp_event` for what to calibrate +to. Critically, our choice to use zCDP internally for configuration +does not imply any looseness in the final accounting and calibration of our +mechanisms. + +-------------------------------------------------------------------------------- + +(zcdp-intermediate)= +## Notes on the `configure(zcdp_rho)` Design + +Using $\rho$-zCDP as the intermediate configuration parameter provides several +key practical advantages: + +* **Single-parameter Calibration**: `dp_accounting.calibrate_dp_mechanism` + expects an arbitrary function that consumes a single scalar value and + returns a `DpEvent` object. For simple and homogeneous mechanisms like + the Gaussian Mechanism, the Exponential Mechanism, or even Poisson-sampled + DP-SGD, there is usually a single parameter (e.g., $\sigma$, $\varepsilon$, + or a related quantity) that is natural to calibrate to. In `dpsynth`, our + mechanisms are heterogeneous compositions of these simpler mechanisms, so + the natural parameters are multi-dimensional, which `dp_accounting` does + not know how to calibrate to. We therefore use a single scalar parameter + to configure to across the entire `dpsynth` API, and have simple functions + to map this parameter to the natural parameter(s) of the mechanism / + sub-mechanisms. +* **Linear Budget Splitting & Arbitrary Nesting**: Unlike + $(\varepsilon, \delta)$ composition, $\rho$-zCDP composes linearly + ($\rho = \sum_i \rho_i$). A parent mechanism can divide its budget into + additive slices ($\rho_i = w_i \cdot \rho$) and pass them down to + sub-mechanisms cleanly. +* **Universal Applicability**: All mechanisms used in DPSynth admit a + well-defined (even if loose) approximate-zCDP guarantee. For example, + Gaussian queries map directly to $\rho$, while pure $\varepsilon$-DP + exponential mechanisms satisfy $\rho = \frac{1}{8}\varepsilon^2$-zCDP. + Crucially, *approximate zCDP at configuration time does not imply loose + accounting for a calibrated mechanism*: the mechanism's `dp_event` + is what `calibrate()` evaluates via tight PLD accounting. +* **Cheap, Closed-Form Parameter Derivation**: Deriving concrete noise + parameters from $\rho$ requires only simple, closed-form arithmetic (e.g., + $\sigma = \sqrt{1 / (2\rho)}$). There are no root-finding loops or + numerical convolutions inside `configure()`, keeping it fast enough to + execute hundreds of times per second during binary search calibration. + +(natural-parameters)= +## `zcdp_rho` vs. Natural Privacy Parameters + +Each DP mechanism has a "natural" parameterization reflecting its mathematical +formulation. However, these natural parameters are heterogeneous and cannot be +composed or split as cleanly: + +* **Gaussian Mechanism**: Natural parameter is the noise standard deviation + $\sigma$ (or variance $\sigma^2$). +* **Other Gaussian-DP Mechanisms**: Natural parameter is either $\sigma$ or + the GDP parameter $\mu^2$ (where $\mu = 1/\sigma$). +* **Exponential Mechanism**: Natural parameter is $\varepsilon$. +* **AIM**: Natural parameter is $\rho$. +* **DP-SGD**: Natural parameters are the tuple `(noise_multiplier, + sampling_probability, iterations)`. +* **[DP Quantiles][dp-quantiles-src]**: Natural parameter is a list of step + budgets $[\varepsilon_1, \dots, \varepsilon_k]$ across recursive + bisections, whose composite `dp_event` is a composition of exponential + mechanisms. + +[dp-quantiles-src]: https://github.com/google/dpsynth/blob/main/dpsynth/local_mode/_quantiles.py + +Because these heterogeneous parameters cannot be directly combined, +`configure()` accepts a single scalar `zcdp_rho` and translates it into each +sub-mechanisms natural parameters in closed form. The mechanism then exposes +its exact composition via its {attr}`~dpsynth.api.CalibratedMechanism.dp_event`, +enabling `calibrate()` to provide numerically tight accounting. + +## Correctness of `calibrate()` vs. `configure()` + +The correctness of `calibrate()` **does not rely on the correctness** of +`configure()`, only that it produces a `CalibratedMechanism` with an honestly +reported `dp_event`. With that being said, `dpsynth` is designed so that +calling `configure()` with a given `zcdp_rho` should produce a +`CalibratedMechanism` has a $\rho$-zCDP guarantee. This makes it safe for users +to call `configure()` directly if they want, although we encourage users to +leverage the higher-level `calibrate()` API since it's tighter. + diff --git a/docs/scalable_beam_api.md b/docs/scalable_beam_api.md index 8dbcb8e9..b818f067 100644 --- a/docs/scalable_beam_api.md +++ b/docs/scalable_beam_api.md @@ -5,7 +5,17 @@ [TOC] When your dataset is large enough to be processed on one machine (>1M records) -[`pipeline_transformations/`](../pipeline_transformations/README.md) +or when your need distributed processing, use the **Scalable Beam API**. + +There are 2 options to run: + +1. Using Library API from you code as Python Beam Pipeline. + +1. From command line. + +This interface is built on top of `pipeline_dp.PipelineBackend`, allowing the +mathematical synthesis transformations +in [`pipeline_transformations/`](../pipeline_transformations/README.md) to run seamlessly on distributed computation engines like **Apache Beam** (or Apache Spark in open-source deployments). @@ -86,7 +96,48 @@ config = data_generation.DataGenerationConfig( delta=1e-7, mechanism=data_generation.Mechanism.MST, # or AIM, SWIFT, INDEPENDENT dataset_descriptor=descriptor_object, # Schema descriptor (CSV, TFRecord) - [`bin/run_data_generation.py`](../bin/run_data_generation.py) + data_format=types.DataFormat.CSV, # Input format + output_format=types.DataFormat.CSV, # Output format (defaults to data_format) + num_out_records=100_000, # Number of synthetic records to sample +) +``` + +### Running the Pipeline in Python + +```python +import apache_beam as beam +from dpsynth import data_generation +import pipeline_dp + +# 1. Instantiate your Beam pipeline +with beam.Pipeline() as pipeline: + # 2. Load raw distributed data into a PCollection + raw_records = pipeline | "ReadRecords" >> beam.io.ReadFromTFRecord("/path/to/data.tfrecord*") + + # 3. Instantiate BeamBackend + beam_backend = pipeline_dp.BeamBackend() + + # 4. Optional: Capture diagnostic accounting and L1 loss metadata + additional_output = data_generation.AdditionalOutput() + + # 5. Execute distributed generation pipeline + synthetic_records = data_generation.generate( + input_data=raw_records, + config=config, + backend=beam_backend, + additional_output=additional_output, + ) + + # 6. Write synthetic records to distributed sink + synthetic_records | "WriteSynthetic" >> beam.io.WriteToTFRecord("/path/to/synthetic.tfrecord") +``` + +-------------------------------------------------------------------------------- + +## Command-Line Interface: `bin/run_data_generation.py` + +The standalone +binary [`bin/run_data_generation.py`](../bin/run_data_generation.py) orchestrates the complete distributed lifecycle. It handles format deduction, launches the Beam job on cluster infrastructure, and manages sink connectors. @@ -112,6 +163,30 @@ python3 bin/run_data_generation.py \ * `--domain_file`: Optional path to explicit `domain.yaml`. If omitted, the schema is deduced and populated privately on-the-fly. * `--epsilon`, `--delta`: Differential privacy budget. +* `--data_format`, `--output_format`: Must be one of `CSV` or `TFRECORD`. +* `--use_beam`: If `true`, launches a distributed + Beam job across computing clusters. If `false`, runs locally + in-process using `pipeline_dp.LocalBackend`. +* `--mechanism`: Supported options are `mst`, `aim`, `independent`, and + `swift`. +* `--attributes`: Comma-separated list of specific columns to synthesize. If + omitted, all columns are synthesized. +* `--num_out_records`: Target number of output synthetic records. If omitted, + generates a dataset approximately equal in size to the input data (privately + estimated). +* `--output_path`: Destination storage path for the synthetic data. + +#### Advanced & Diagnostic Flags + +* `--diagnostic_information_path`: Storage path where a + `DiagnosticInformation` protobuf (containing DP budget splits, operation + counts, and L1 loss metrics) will be saved. +* `--model_save_path`: Storage path where the trained `SyntheticModel` + protobuf and populated `DatasetDescriptor` will be saved. This allows + generating additional records later without re-reading sensitive source + data. +* `--aim_rounds`, `--aim_pgm_iters`, `--aim_max_model_size`: Advanced tuning + parameters specifically for the AIM mechanism. -------------------------------------------------------------------------------- diff --git a/docs/sitemap.md b/docs/sitemap.md index 173bbb94..d75b4df1 100644 --- a/docs/sitemap.md +++ b/docs/sitemap.md @@ -119,3 +119,16 @@ * [Running Evaluation (CLI)](contributors_guide.md#running-evaluation-cli) + +-------------------------------------------------------------------------------- + +
+📁 Mechanism API Architecture + +* [Design Decisions: `MechanismConfig` vs. `CalibratedMechanism`](mechanism_api.md#design-decisions) +* [The 3-Step Mechanism Pipeline](mechanism_api.md#three-step-pipeline) +* [`configure(zcdp_rho)` vs. `calibrate(epsilon, delta)`](mechanism_api.md#configure-vs-calibrate) +* [Notes on the `configure(zcdp_rho)` Design](mechanism_api.md#zcdp-intermediate) +* [`zcdp_rho` vs. Natural Privacy Parameters](mechanism_api.md#natural-parameters) + +