Release v0.3.0: drift-detection features, calibration metrics, and bug fixes

.github/dependabot.yml

@@ -9,3 +9,7 @@ updates:
     directory: "/" # Location of package manifests
     schedule:
       interval: "weekly"
+  - package-ecosystem: "uv"
+    directory: "/" # Location of pyproject.toml / uv.lock
+    schedule:
+      interval: "weekly"

CHANGELOG.md

@@ -0,0 +1,96 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.3.0] - 2026-08-04
+
+### Added
+
+- `cyclops.monitor`: `Detector.detect_shift_by_subgroup()` runs the fitted
+  drift tester independently on each subgroup of a target dataset (e.g.
+  age band, sex, hospital site, defined via the existing `SliceSpec`),
+  instead of only testing the aggregate population - a model can look
+  stable overall while drifting badly for a specific subgroup, which
+  matters for health-equity-aware monitoring. Includes Bonferroni
+  correction across subgroups and a minimum-sample-size guard.
+- `cyclops.monitor`: `DCTester.explain_shift()` explains a detected shift
+  using SHAP on the domain classifier trained by `tester_method="classifier"`,
+  returning features ranked by how strongly they indicate a sample belongs
+  to the shifted distribution.
+- `cyclops.evaluate.metrics.experimental`: added calibration metrics -
+  `BinaryBrierScore`/`MulticlassBrierScore` and `BinaryCalibrationError`
+  (Expected Calibration Error / Maximum Calibration Error), plus their
+  functional counterparts. Discrimination metrics like AUROC don't tell
+  you whether a predicted probability can be trusted at face value, which
+  matters for clinical risk scores that are often acted on directly.
+
+### Fixed
+
+- `cyclops.monitor`: `errorfill()` crashed on the default `color=None` because
+  matplotlib removed `ax._get_lines.prop_cycler`.
+- `cyclops.monitor`: `TSTester.test_shift()` mutated `p_val_threshold` in
+  place on every call, so the Bonferroni correction compounded across
+  repeated calls (e.g. in `Detector`'s sweep loops) instead of being
+  computed fresh each time; also fixed an `UnboundLocalError` when the
+  input isn't a plain `np.ndarray`.
+- `cyclops.monitor`: `ContextMMDWrapper` was missing the
+  `preprocess_at_init` argument in its positional argument list to
+  alibi-detect's `ContextMMDDrift`, silently shifting every later
+  argument by one slot. `ContextMMDWrapper` and `LKWrapper` now pass
+  keyword arguments so future alibi-detect signature changes fail loudly
+  instead of silently misaligning.
+- `cyclops.monitor`: `Reductor` raised `TypeError` when torchvision wasn't
+  installed and `transforms` was passed, because `isinstance(transforms,
+  Compose)` was called with `Compose is None`.
+- `cyclops.monitor`: removed `plot_label_distribution`, an unreachable,
+  untested, and uncalled function with a use-before-assignment bug.
+- `cyclops.report`: `ModelCardReport.export()` raised `IndexError` when no
+  `PerformanceMetric` had been logged.
+- `cyclops.report`: `_process_metric_name()` raised `UnboundLocalError`
+  for any metric `type` not prefixed with `Binary`/`Multiclass`/
+  `Multilabel` (e.g. a custom metric name).
+- `cyclops.report`: `export()`'s default output filename was a static
+  `model_card.html`/`.json`, so repeated calls into the same
+  `output_dir` silently overwrote prior reports and broke trend/history
+  comparisons. The default filename is now timestamped per call.
+- `cyclops.report`: `Citation.content` (raw BibTeX text) was rendered
+  with Jinja's `|safe` filter, bypassing autoescaping for no reason.
+- `cyclops.utils`: `exchange_extension()` dropped the filename entirely
+  for paths with no existing extension (e.g. `"myfile"` -> `".csv"`
+  instead of `"myfile.csv"`).
+- `cyclops.models`: `MLPModel` (the packaged `"mlp_pt"` model) could not
+  be constructed at all - an activation class was inserted into
+  `nn.Sequential` instead of an instance, the first hidden layer was
+  double-wrapped in a list, and the loop connecting hidden layers used
+  the wrong input dimension. The packaged `mlp_pt.yaml` config also
+  passed a nonexistent `layer_dim` argument left over from the RNN/GRU/
+  LSTM configs.
+- `cyclops.data`: `SliceSpec`'s datetime `day` component filter
+  (`filter_datetime`) silently matched on year instead of day of month.
+
+### Changed
+
+- `cyclops.monitor.utils`: removed ~470 lines of unused temporal-modeling
+  scaffolding (`Data`, `get_data`, `run_model`, `get_serving_data`,
+  `scale`, `daterange`, `get_obj_from_str`, `load_model`/`save_model`,
+  `print_metrics_binary`, `load_ckp`, `get_device`, `get_temporal_model`,
+  `Loader`, and a stray `__main__` demo block) that was neither exported,
+  imported elsewhere in the repo, nor tested.
+
+### CI / infra
+
+- Added a `uv` ecosystem entry to Dependabot so `pyproject.toml`/
+  `uv.lock` dependencies get automated update PRs.
+- Fixed the README's "integration tests" badge, which linked to a
+  workflow file that no longer exists; replaced with the (existing,
+  previously unlinked) unit tests badge.
+- Expanded `CONTRIBUTING.md` with environment setup, test-running, and
+  repository layout sections.
+
+[Unreleased]: https://github.com/VectorInstitute/cyclops/compare/v0.3.0...HEAD
+[0.3.0]: https://github.com/VectorInstitute/cyclops/compare/v0.2.12...v0.3.0

CONTRIBUTING.md

@@ -5,14 +5,49 @@ Thanks for your interest in contributing to cyclops!
 To submit PRs, please fill out the PR template along with the PR. If the PR
 fixes an issue, don't forget to link the PR to the issue!
 
-## Pre-commit hooks
+## Setting up your environment
 
-Once the python virtual environment is setup, you can run pre-commit hooks using:
+cyclops uses [uv](https://docs.astral.sh/uv/getting-started/installation/) to
+manage dependencies. Once uv is installed, set up a development environment
+with the test dependency group and activate it:
 
 ```bash
-pre-commit run --all-files
+uv sync --group test
+source .venv/bin/activate
+```
+
+Some modules have optional dependencies (e.g. `torch`, `xgboost`, `monai`,
+`alibi-detect`) that aren't installed by default - see the table in
+[README.md](README.md) for the full list of extras. To work on a module that
+needs one of these, install the matching extra, e.g.:
+
+```bash
+uv sync --group test --extra alibi-detect
+```
+
+Install the pre-commit hooks so code-style issues are caught before you push:
+
+```bash
+pre-commit install
+```
+
+## Running tests
+
+Run the unit test suite with:
+
+```bash
+python -m pytest -m "not integration_test"
 ```
 
+Tests marked `@pytest.mark.integration_test` require external
+infrastructure (e.g. a live database via
+[cycquery](https://github.com/VectorInstitute/cycquery)) that isn't
+available in a plain checkout, which is why they're excluded above and not
+run in CI. Only run them locally if you have that infrastructure set up.
+
+Pass `-k <pattern>` to scope a run to a subset of tests, and
+`--cov=cyclops` to see coverage for the code you changed.
+
 ## Coding guidelines
 
 For code style, we recommend the [PEP 8 style guide](https://peps.python.org/pep-0008/).
@@ -24,3 +59,27 @@ analysis. Ruff checks various rules including [flake8](https://docs.astral.sh/ru
 
 Last but not the least, we use type hints in our code which is then checked using
 [mypy](https://mypy.readthedocs.io/en/stable/).
+
+You can run all pre-commit checks (ruff, ruff-format, mypy, notebook
+stripping) against the whole repository at once with:
+
+```bash
+pre-commit run --all-files
+```
+
+## Repository layout
+
+- `cyclops/data` - dataset construction, loading, and slicing
+- `cyclops/models` - scikit-learn and PyTorch model wrappers and implementations
+- `cyclops/tasks` - task formulations (e.g. binary/multi-label classification) tying data and models together
+- `cyclops/evaluate` - metrics and fairness evaluation for clinical prediction tasks
+- `cyclops/monitor` - dataset shift / drift detection for deployed models
+- `cyclops/report` - model report card generation
+- `cyclops/utils` - small shared utilities used across the other modules
+
+Each module has a corresponding test package under `tests/cyclops/`.
+
+## Code of Conduct
+
+Participation in this project is governed by our
+[Code of Conduct](CODE_OF_CONDUCT.md).

README.md

@@ -5,7 +5,7 @@
 [![PyPI](https://img.shields.io/pypi/v/pycyclops)](https://pypi.org/project/pycyclops)
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pycyclops)
 [![code checks](https://github.com/VectorInstitute/cyclops/actions/workflows/code_checks.yml/badge.svg)](https://github.com/VectorInstitute/cyclops/actions/workflows/code_checks.yml)
-[![integration tests](https://github.com/VectorInstitute/cyclops/actions/workflows/integration_tests.yml/badge.svg)](https://github.com/VectorInstitute/cyclops/actions/workflows/integration_tests.yml)
+[![unit tests](https://github.com/VectorInstitute/cyclops/actions/workflows/unit_tests.yml/badge.svg)](https://github.com/VectorInstitute/cyclops/actions/workflows/unit_tests.yml)
 [![docs](https://github.com/VectorInstitute/cyclops/actions/workflows/docs.yml/badge.svg)](https://github.com/VectorInstitute/cyclops/actions/workflows/docs.yml)
 [![codecov](https://codecov.io/gh/VectorInstitute/cyclops/branch/main/graph/badge.svg)](https://codecov.io/gh/VectorInstitute/cyclops)
 [![docker](https://github.com/VectorInstitute/cyclops/actions/workflows/docker.yml/badge.svg)](https://hub.docker.com/r/vectorinstitute/cyclops)

cyclops/data/slicer.py

@@ -768,7 +768,7 @@ def _apply_mask(
         months = pc.month(example_values)
         mask = _apply_mask(months, month, mask)
     if day is not None:
-        days = pc.year(example_values)
+        days = pc.day(example_values)
         mask = _apply_mask(days, day, mask)
     if hour is not None:
         hours = pc.hour(example_values)

cyclops/evaluate/metrics/experimental/__init__.py

@@ -15,6 +15,13 @@
     MulticlassAveragePrecision,
     MultilabelAveragePrecision,
 )
+from cyclops.evaluate.metrics.experimental.brier_score import (
+    BinaryBrierScore,
+    MulticlassBrierScore,
+)
+from cyclops.evaluate.metrics.experimental.calibration_error import (
+    BinaryCalibrationError,
+)
 from cyclops.evaluate.metrics.experimental.confusion_matrix import (
     BinaryConfusionMatrix,
     MulticlassConfusionMatrix,
@@ -95,6 +102,9 @@
     "BinaryAveragePrecision",
     "MulticlassAveragePrecision",
     "MultilabelAveragePrecision",
+    "BinaryBrierScore",
+    "MulticlassBrierScore",
+    "BinaryCalibrationError",
     "BinaryConfusionMatrix",
     "MulticlassConfusionMatrix",
     "MultilabelConfusionMatrix",

cyclops/evaluate/metrics/experimental/brier_score.py

@@ -0,0 +1,163 @@
+"""Brier score metric."""
+
+from typing import Any, Optional
+
+from cyclops.evaluate.metrics.experimental.functional.brier_score import (
+    _binary_brier_score_compute,
+    _binary_brier_score_format_arrays,
+    _binary_brier_score_update,
+    _binary_brier_score_validate_args,
+    _binary_brier_score_validate_arrays,
+    _multiclass_brier_score_format_arrays,
+    _multiclass_brier_score_update,
+    _multiclass_brier_score_validate_args,
+    _multiclass_brier_score_validate_arrays,
+)
+from cyclops.evaluate.metrics.experimental.metric import Metric
+from cyclops.evaluate.metrics.experimental.utils.types import Array
+
+
+class BinaryBrierScore(Metric):
+    """Brier score for binary classification tasks.
+
+    The Brier score is the mean squared error between predicted
+    probabilities and the (binary) target, and is a proper scoring rule
+    for probabilistic predictions - it rewards models whose predicted
+    probabilities are well-calibrated, not just well-ranked, which matters
+    for clinical risk scores that are acted on directly.
+
+    Parameters
+    ----------
+    ignore_index : int, optional, default=None
+        Values in the target array to ignore when computing the metric.
+    **kwargs : Any
+        Additional keyword arguments common to all metrics.
+
+    Examples
+    --------
+    >>> import numpy.array_api as anp
+    >>> from cyclops.evaluate.metrics.experimental import BinaryBrierScore
+    >>> target = anp.asarray([0, 1, 1, 0])
+    >>> preds = anp.asarray([0.1, 0.9, 0.8, 0.3])
+    >>> metric = BinaryBrierScore()
+    >>> metric(target, preds)
+    Array(0.0375, dtype=float32)
+
+    """
+
+    name: str = "Brier Score"
+
+    def __init__(self, ignore_index: Optional[int] = None, **kwargs: Any) -> None:
+        super().__init__(**kwargs)
+        _binary_brier_score_validate_args(ignore_index=ignore_index)
+        self.ignore_index = ignore_index
+
+        self.add_state_default_factory(
+            "sum_squared_error",
+            lambda xp: xp.asarray(0.0, dtype=xp.float32, device=self.device),  # type: ignore
+            dist_reduce_fn="sum",
+        )
+        self.add_state_default_factory(
+            "num_obs",
+            lambda xp: xp.asarray(0.0, dtype=xp.float32, device=self.device),  # type: ignore
+            dist_reduce_fn="sum",
+        )
+
+    def _update_state(self, target: Array, preds: Array) -> None:
+        """Update the state of the metric."""
+        xp = _binary_brier_score_validate_arrays(
+            target,
+            preds,
+            ignore_index=self.ignore_index,
+        )
+        target, preds = _binary_brier_score_format_arrays(
+            target,
+            preds,
+            self.ignore_index,
+            xp=xp,
+        )
+        sum_squared_error, num_obs = _binary_brier_score_update(target, preds)
+        self.sum_squared_error += sum_squared_error  # type: ignore
+        self.num_obs += num_obs  # type: ignore
+
+    def _compute_metric(self) -> Array:
+        """Compute the binary Brier score."""
+        return _binary_brier_score_compute(
+            self.sum_squared_error,  # type: ignore
+            self.num_obs,  # type: ignore
+        )
+
+
+class MulticlassBrierScore(Metric):
+    """Brier score for multiclass classification tasks.
+
+    Computed as the mean squared error between the predicted probability
+    vector for each sample and the one-hot encoded target.
+
+    Parameters
+    ----------
+    num_classes : int
+        The number of classes in the classification task.
+    ignore_index : int, optional, default=None
+        Values in the target array to ignore when computing the metric.
+    **kwargs : Any
+        Additional keyword arguments common to all metrics.
+
+    Examples
+    --------
+    >>> import numpy.array_api as anp
+    >>> from cyclops.evaluate.metrics.experimental import MulticlassBrierScore
+    >>> target = anp.asarray([0, 1, 2])
+    >>> preds = anp.asarray(
+    ...     [[0.7, 0.2, 0.1], [0.1, 0.8, 0.1], [0.2, 0.2, 0.6]],
+    ... )
+    >>> metric = MulticlassBrierScore(num_classes=3)
+    >>> metric(target, preds)
+    Array(0.14666666, dtype=float32)
+
+    """
+
+    name: str = "Brier Score"
+
+    def __init__(
+        self,
+        num_classes: int,
+        ignore_index: Optional[int] = None,
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(**kwargs)
+        _multiclass_brier_score_validate_args(num_classes, ignore_index=ignore_index)
+        self.num_classes = num_classes
+        self.ignore_index = ignore_index
+
+        self.add_state_default_factory(
+            "sum_squared_error",
+            lambda xp: xp.asarray(0.0, dtype=xp.float32, device=self.device),  # type: ignore
+            dist_reduce_fn="sum",
+        )
+        self.add_state_default_factory(
+            "num_obs",
+            lambda xp: xp.asarray(0.0, dtype=xp.float32, device=self.device),  # type: ignore
+            dist_reduce_fn="sum",
+        )
+
+    def _update_state(self, target: Array, preds: Array) -> None:
+        """Update the state of the metric."""
+        xp = _multiclass_brier_score_validate_arrays(target, preds, self.num_classes)
+        target, preds = _multiclass_brier_score_format_arrays(
+            target,
+            preds,
+            self.ignore_index,
+            self.num_classes,
+            xp=xp,
+        )
+        sum_squared_error, num_obs = _multiclass_brier_score_update(target, preds)
+        self.sum_squared_error += sum_squared_error  # type: ignore
+        self.num_obs += num_obs  # type: ignore
+
+    def _compute_metric(self) -> Array:
+        """Compute the multiclass Brier score."""
+        return _binary_brier_score_compute(
+            self.sum_squared_error,  # type: ignore
+            self.num_obs,  # type: ignore
+        )