Fix ReadTheDocs build and add documentation pages on the MechanismConfig vs. CalibratedMechanism API, along with configure(zcdp_rho) vs. calibrate(epsilon, delta) patterns in the library.

docs/conf.py

@@ -45,9 +45,20 @@
     'show-inheritance': True,
 }
 
-# templates_path = ['_templates']
+templates_path = ['_templates']
 exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 
+# Enable MyST-Parser extensions for math rendering ($...$ and AMS environments).
+myst_enable_extensions = [
+    'dollarmath',
+    'amsmath',
+]
+myst_heading_anchors = 3
+
+suppress_warnings = [
+    'misc.highlighting_failure',
+]
+
 # -- Options for HTML output -------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 

docs/contributors_guide.md

@@ -130,7 +130,69 @@ without changing a single line of code.
     testing and debugging.
 *   **`pipeline_dp.BeamBackend`**: Executes
     on Apache Beam.
-[PipelineDP](https://github.com/OpenMined/PipelineDP)
+    In open-source deployments, it allows DPSynth to run on Apache Spark
+    or Google Cloud Dataflow.
+
+### 3. Designing Pipeline Functions
+
+When writing new transformations in `pipeline_transformations/`, functions
+**MUST** accept a `backend: PipelineBackend` instance as an argument and use its
+methods exclusively for all data transformations.
+
+```python
+from pipeline_dp import PipelineBackend
+
+def compute_custom_column_stats(backend: PipelineBackend, input_collection):
+    # input_collection is framework-agnostic (PCollection or Python list)
+    # Every operation MUST include a clear, descriptive stage_name
+
+    # 1. Map: Extract column value
+    col_values = backend.map(input_collection, lambda row: row[2], "ExtractColumnValue")
+
+    # 2. Filter: Remove missing entries
+    filtered_values = backend.filter(col_values, lambda x: x is not None, "FilterNonNull")
+
+    # 3. Key: Pair with constant key
+    keyed_data = backend.map(filtered_values, lambda x: ("all_records", x), "AddKey")
+
+    # 4. Group & Reduce: Sum values per key
+    grouped_data = backend.group_by_key(keyed_data, "GroupByKey")
+    summed_data = backend.map_values(grouped_data, sum, "SumValues")
+
+    return summed_data
+```
+
+### 3. Core `PipelineBackend` Methods Available
+
+*   `map(col, fn, stage_name)`: Apply a function to each element.
+*   `flat_map(col, fn, stage_name)`: Apply a function returning an iterable,
+    flattening results.
+*   `filter(col, fn, stage_name)`: Keep elements where `fn` returns `True`.
+*   `group_by_key(col, stage_name)`: Group key-value pairs by key.
+*   `map_values(col, fn, stage_name)`: Apply a function to values of key-value
+    pairs.
+*   `sum_per_key(col, stage_name)` / `reduce_per_key(col, fn, stage_name)`:
+    Aggregate values per key.
+*   `flatten(cols, stage_name)`: Merge multiple collections into one.
+*   `distinct(col, stage_name)`: Get unique elements.
+
+### 4. Critical Constraints & Safety Guardrails
+
+> [!CAUTION] **Never use framework-specific operations** like `beam.Map`,
+> `beam.Filter`, or the Beam pipe operator (`|`) inside functions designed to be
+> backend-agnostic. All transformations must route through methods of the passed
+> `backend` object.
+>
+> **Never assume collection types**. `input_collection` will be a Python
+> iterable for `LocalBackend` but a `PCollection` for `BeamBackend`. Pipeline
+> logic must never invoke methods specific to `list` or `PCollection`.
+
+### 5. PipelineDP Integration & DP Aggregations
+
+While the `PipelineBackend` abstraction is used to write the framework-agnostic
+data manipulation pipeline, the actual **Differentially Private Aggregations**
+are powered by the `pipeline_dp.DPEngine` object from
+the [PipelineDP](https://github.com/OpenMined/PipelineDP)
 library.
 
 When contributing new mechanisms or data transformations, verify that you

docs/index.md

@@ -77,7 +77,9 @@ interface.
 *   **Execution**: Runs
     on Apache Beam
     (or Apache Spark in open-source deployments).
-[`bin/run_data_generation.py`](../bin/run_data_generation.py). *   **Documentation**:
+*   **Inputs/Outputs**: (Sharded) CSV and TFRecord.
+*   **CLI
+    Binary**: [`bin/run_data_generation.py`](../bin/run_data_generation.py). *   **Documentation**:
     [Scalable PipelineBackend API Guide](scalable_beam_api.md).
 
 --------------------------------------------------------------------------------
@@ -102,6 +104,9 @@ contributor expanding the library, explore the documentation below:
 *   **[Contributor & Architecture Guide](contributors_guide.md)**: Architectural
     separation, core abstractions, PipelineBackend programming rules,
     diagnostics, and the Tabular Evaluation framework.
+*   **[Mechanism API Architecture](mechanism_api.md)**: Design decisions,
+    the 3-step pipeline, configure vs. calibrate semantics, and tight
+    privacy accounting.
 
 --------------------------------------------------------------------------------
 
@@ -148,6 +153,14 @@ processing_lifecycle
 contributors_guide
 ```
 
+```{toctree}
+:maxdepth: 2
+:caption: Developer documentation:
+:hidden:
+
+mechanism_api
+```
+
 ```{toctree}
 :maxdepth: 2
 :caption: API Reference