SCENE pipelines

Documentation: scene-collaboration.github.io/SCENE-pipelines

Shared DataJoint tables for the labs of SCENE, and the Python helpers that write to them. Every lab registers its labs, subjects, sessions and files in the same tables, keeps its own tables next to them, and links the two by key.

Migrated with full history from Base-schemas, where the earlier review discussions live.

Note

The table definitions are placeholders and may still change.

Documentation

The documentation is built from docs/ with MkDocs; mkdocs serve serves it locally (see Development). Good places to start:

• Introduction: the ideas behind the shared tables (layers, lab codes and ids, files).
• Getting started: a notebook that runs on a local database. New to DataJoint? See the DataJoint tutorials to start with.
• Guides: setup and configuration, writing data, schemas and the registry, provenance and sync.
• API reference: every table and helper, from the docstrings.

An existing lab: MLAI and MouseAR shows a real lab pipeline (Mathis lab, Mouse AR) on the shared tables.

Quickstart

Quick installation

To use the package, install the latest version on main directly from GitHub:

pip install "scene-pipelines @ git+https://github.com/SCENE-Collaboration/SCENE-pipelines.git"

Editable install

Or clone the repository and install an editable version:

git clone https://github.com/SCENE-Collaboration/SCENE-pipelines.git
cd SCENE-pipelines
pip install -e ".[dev]"

To run the unit tests (no database needed):

make test

To set up a local database and run the database tests against it:

make init          # once: copy .env.example → .env, then edit as needed
make db_up         # local MySQL in Docker
make test-db       # database tests (also starts the database if needed)

Tutorials and guides

• See Setup and configuration for a guide on setting up the environment variables and the Docker client container.
• Run our notebook Getting started to try hands-on.

Acknowledgments

We thank Prof. Mackenzie Mathis, Dr. Tanmay Nath, Dr. Gary Kane, and Dr. Mariia Popova for their early contributions to this codebase, which helped establish the foundation of the shared schemas used across pipelines.