diff --git a/README.md b/README.md index 8ce21db..7ddcb11 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,7 @@ These concerns are intentionally left to downstream tooling. ## Example (concept navigation) ```python -from omop_alchemy.model.vocabulary import ConceptView +from omop_alchemy.cdm.model.vocabulary.concept import ConceptView concept = session.get(ConceptView, 320128) # Lung cancer concept.domain.domain_id # "Condition" @@ -93,26 +93,4 @@ omop-config init omop-config configure omop_alchemy ``` -See [Configuration](docs/getting-started/configuration.md) for full details. - ---- - -## Docker Compose - -The included `docker-compose.yaml` provides a PostgreSQL database and a Python -container with the `[postgres]` extra pre-installed. Default credentials work out of the box: - -```bash -docker compose up -``` - -The `python-alchemy` service runs `omop-config configure` at startup and writes -`~/.config/omop/config.toml` on the host on first start; subsequent starts skip -configuration automatically. - -To override credentials, copy `.env.example` to `.env` and edit before starting: - -```bash -cp .env.example .env -docker compose up -``` \ No newline at end of file +See [Configuration](docs/getting-started/configuration.md) for full details. \ No newline at end of file diff --git a/docker-compose.yaml b/docker-compose.yaml deleted file mode 100644 index ce40655..0000000 --- a/docker-compose.yaml +++ /dev/null @@ -1,66 +0,0 @@ -services: - omop-cdm-db: - image: postgres:16-alpine - restart: always - environment: - POSTGRES_USER: ${OMOP_CDM_DB_USER:-omop} - POSTGRES_PASSWORD: ${OMOP_CDM_DB_PASSWORD:-omop} - POSTGRES_DB: ${OMOP_CDM_DB_NAME:-omop_cdm} - PGDATA: /var/lib/postgresql/data/pgdata - volumes: - - db_data:/var/lib/postgresql/data - ports: - - "5432:5432" - networks: - - omop-net - command: > - postgres - -c shared_buffers=512MB - -c effective_cache_size=1GB - -c work_mem=128MB - -c maintenance_work_mem=512MB - -c max_wal_size=4GB - -c min_wal_size=512MB - -c wal_buffers=16MB - -c wal_compression=zstd - -c full_page_writes=off - -c checkpoint_timeout=30min - -c synchronous_commit=off - -c max_parallel_workers_per_gather=2 - -c max_worker_processes=4 - -c max_parallel_maintenance_workers=2 - healthcheck: - test: ["CMD-SHELL", "pg_isready -U ${OMOP_CDM_DB_USER:-omop}"] - interval: 5s - timeout: 5s - retries: 5 - - python-alchemy: - build: . - restart: unless-stopped - depends_on: - omop-cdm-db: - condition: service_healthy - volumes: - - ${HOME}/.config/omop:/root/.config/omop - networks: - - omop-net - command: > - bash -c " - omop-config configure omop_alchemy - --database cdm --dialect postgresql+psycopg - --host omop-cdm-db --port 5432 - --user ${OMOP_CDM_DB_USER:-omop} - --password ${OMOP_CDM_DB_PASSWORD:-omop} - --database-name ${OMOP_CDM_DB_NAME:-omop_cdm} - --cdm-schema omop && - omop-config configure orm_loader && - sleep infinity - " - -networks: - omop-net: - name: omop-net - -volumes: - db_data: diff --git a/docs/advanced/vocabulary_load_performance.md b/docs/advanced/vocabulary_load_performance.md index fbf96bd..9ec4703 100644 --- a/docs/advanced/vocabulary_load_performance.md +++ b/docs/advanced/vocabulary_load_performance.md @@ -37,9 +37,9 @@ The next biggest bottleneck after pagination is `synchronous_commit=on` (the Pos ### Recommended settings -These settings are present in the docker-compose files for each package. If you are running PostgreSQL outside Docker, add them to `postgresql.conf` or pass them as `-c` flags. +Apply these via `postgresql.conf` or `-c` flags on whatever PostgreSQL instance you're loading into (per-package `docker-compose.yaml` files no longer exist; Docker orchestration for the OMOP stack now happens at the workspace root). -**devcontainer (omop-spires `docker-compose.override.yaml`) — 8 GB host:** +**8 GB host:** ``` synchronous_commit=off checkpoint_timeout=30min @@ -53,7 +53,7 @@ wal_compression=zstd full_page_writes=off ``` -**standalone docker-compose (OMOP_Alchemy / omop-graph) — ~4 GB host:** +**~4 GB host:** ``` synchronous_commit=off checkpoint_timeout=30min @@ -81,7 +81,7 @@ SELECT pg_reload_conf(); SHOW synchronous_commit; -- confirm: should show 'off' ``` -Settings that ARE overridden by `-c` (e.g. `checkpoint_timeout`) require a container restart to pick up the updated docker-compose value. +Settings that ARE overridden by `-c` (e.g. `checkpoint_timeout`) require a container restart to pick up an updated value. To monitor whether WAL-write stalls are happening during a load: diff --git a/docs/cli/index.md b/docs/cli/index.md index 8440343..7a51e66 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -59,29 +59,26 @@ No connection flags are injected; all configuration comes from oa_configurator. When a decorated command is invoked: -1. Loads `~/.config/omop/config.toml` via `load_stack_config()`. -2. Calls `OmopAlchemyConfig.from_stack(config)` to read package-specific settings and validate that the required `cdm_db` resource (or the `[tools.omop_alchemy] default_resource` override) is present. Raises `ConfigurationError` with a helpful message if it is missing. -3. Resolves the resource: `Resolver(config).resolve_resource("cdm_db")`. -4. Calls `.create_engine()` to build a SQLAlchemy engine with `schema_translate_map` applied. -5. Prints a command header showing the resource name, CDM schema, and run mode. -6. Calls the original function body with `(conn, engine, ...)`. -7. Catches `RuntimeError`, `SQLAlchemyError`, and `BackendNotSupportedError`; renders them as formatted errors and exits with code 1. +1. Calls `get_cdm_context()`, which loads `~/.config/omop/config.toml` (via `load_stack_config()`) and resolves whatever `OmopAlchemyConfig.cdm_db` currently names, returning `(pkg_config, resolved)`. Raises `RuntimeError` with a helpful message if no config file exists yet. +2. Calls `create_cdm_engine(resolved)` to build a SQLAlchemy engine (`resolved.create_engine()`, with `schema_translate_map` applied), with a clearer error if the PostgreSQL driver isn't installed. +3. Builds `conn` (`db_schema=resolved.schema_name`, `athena_source=pkg_config.athena_source_path`). +4. Prints a command header showing the connection, CDM schema, and run mode. +5. Calls the original function body with `(conn, engine, ...)`. +6. Catches `RuntimeError`, `SQLAlchemyError`, and `BackendNotSupportedError`; renders them as formatted errors and exits with code 1. ### Before and after Without the decorator, every command would need this boilerplate: ```python -from omop_alchemy.config import TOOL_NAME +from omop_alchemy.config import create_cdm_engine, get_cdm_context + def my_command() -> None: - stack = load_stack_config() - tool = stack.tools.get(TOOL_NAME) - resource_name = (tool.default_resource if tool else None) or "cdm_db" - resolved = Resolver(stack).resolve_resource(resource_name) - engine = resolved.create_engine() + pkg_config, resolved = get_cdm_context() + engine = create_cdm_engine(resolved) try: # actual work here - results = do_work(engine, db_schema=resolved.cdm_schema) + results = do_work(engine, db_schema=resolved.schema_name) console.print(render_results(results)) except Exception as exc: handle_error(exc) @@ -105,5 +102,5 @@ def my_command(conn, engine) -> None: | Attribute | Description | |---|---| -| `conn.db_schema` | CDM schema name from the resolved resource (e.g. `"omop"`) | -| `conn.athena_source` | Athena vocabulary CSV directory from `[tools.omop_alchemy.extra]`; `None` if not configured | +| `conn.db_schema` | CDM schema name from the resolved database (e.g. `"omop"`) | +| `conn.athena_source` | Athena vocabulary CSV directory from `[tools.omop_alchemy]`'s `athena_source_path` field; `None` if not configured | diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index 51cc229..25e6b87 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -13,36 +13,40 @@ Run the interactive configure command to set up the CDM database connection and omop-config configure omop_alchemy ``` -This prompts for connection details (host, dialect, credentials) and schema names, then -saves them under the canonical resource name `cdm_db` that all OMOP stack packages +This prompts for connection details (host, dialect, credentials) and schema name, then +saves them under the canonical database name `cdm_db` that all OMOP stack packages recognise. The resulting TOML looks like: ```toml -[databases.cdm] -dialect = "postgresql+psycopg2" +[connections.cdm] +dialect = "postgresql+psycopg" host = "localhost" port = 5432 user = "omop" password = "changeme" database_name = "omop_cdm" -[resources.cdm_db] -database = "cdm" -cdm_schema = "omop" +[databases.cdm_db] +kind = "cdm" +connection = "cdm" +schema_name = "omop" + +[tools.omop_alchemy] +cdm_db = "cdm_db" ``` You can also write or edit this file manually. ## Vocabulary loading -If you plan to load OMOP vocabulary from Athena CSV files, add the path to the package -extras section: +If you plan to load OMOP vocabulary from Athena CSV files, add the path to `[tools.omop_alchemy]`: ```toml -[tools.omop_alchemy.extra] -athena_source_path = "/path/to/athena/csvs" +[tools.omop_alchemy] +cdm_db = "cdm_db" +athena_source_path = "/path/to/athena/csvs" ``` Or set it interactively: @@ -60,63 +64,23 @@ omop-alchemy info This prints the resolved config file path, connection details, and schema. A successful run confirms that OMOP_Alchemy can reach your database. -## Docker Compose - -The included `docker-compose.yaml` spins up a PostgreSQL database and a `python-alchemy` -container. Default credentials work out of the box — no additional setup needed: - -```bash -docker compose up -``` - -The `python-alchemy` container runs `omop-config configure omop_alchemy` automatically at -startup. Your `~/.config/omop/config.toml` on the host is written on first start and -safe to re-run on subsequent starts: connection flags always apply, and any values already stored in `config.toml` are preserved for fields not explicitly provided. - -### Overriding default values - -The compose file uses built-in defaults for all database credentials. To use different -values, create a `.env` file in this directory with any of the following variables: - -| Variable | Default | Description | -|---|---|---| -| `OMOP_CDM_DB_USER` | `omop` | CDM database username | -| `OMOP_CDM_DB_PASSWORD` | `omop` | CDM database password | -| `OMOP_CDM_DB_NAME` | `omop_cdm` | CDM database name | - -Copy the example and edit as needed: - -```bash -cp .env.example .env -# edit .env -docker compose up -``` - -The `.env` file is only read by Docker Compose for variable substitution — it is not -loaded by OMOP_Alchemy at runtime. - ## Multiple instances -To configure a second CDM database (e.g. for production), use `--resource-name`: +To configure a second CDM database (e.g. for production), create it under its own name +and point the field's own flag at it: ```bash -omop-config configure omop_alchemy --resource-name cdm_db_prod +omop-config databases add cdm_db_prod --kind cdm --connection cdm_prod +omop-config configure omop_alchemy --cdm-db cdm_db_prod ``` -This creates `cdm_db_prod` without touching the existing `cdm_db`. Because two -resources now exist, configure automatically prompts you to choose the default at -the end of the same run — no second invocation needed. - -To change the default later, set `default_resource` directly in `config.toml`: - -```toml -[tools.omop_alchemy] -default_resource = "cdm_db_prod" -``` +This creates `cdm_db_prod` without touching the existing `cdm_db`. There is no "default" +toggle to flip afterward; each deployment's `configure` call names the entry it wants +directly. See the [oa-configurator integration guide](https://AustralianCancerDataNetwork.github.io/oa-configurator/integration/#multiple-environments) for the full multi-environment guide. ## Further reading -- [oa_configurator quickstart](https://AustralianCancerDataNetwork.github.io/oa-configurator/) — full config reference, multiple profiles, env var export -- [oa_configurator integration guide](https://AustralianCancerDataNetwork.github.io/oa-configurator/integration/) — Docker Compose details and multi-package setups +- [oa_configurator quickstart](https://AustralianCancerDataNetwork.github.io/oa-configurator/quickstart/): full config reference, CLI walkthrough +- [oa_configurator integration guide](https://AustralianCancerDataNetwork.github.io/oa-configurator/integration/): multi-package setups diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 48730f8..18800b3 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -17,7 +17,7 @@ engine = sa.create_engine( echo=False, ) -with so.Session(engine) as sess: +with so.Session(engine) as session: concepts = ( session.query(Concept) .filter(Concept.domain_id == "Drug") diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index 4e67a96..777747f 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -1,91 +1,33 @@ -# Quickstart (Experimental Docker Stack) +# Quickstart -**This Docker stack is intended for local experimentation, development, and exploration only.** -It is **not** hardened, secured, or tuned for production use. - -The goal is to provide a fast, reproducible environment for: -- Exploring schemas and data -- Prototyping ETL / ORM logic -- Testing materialized views, loaders, and queries -- Running notebooks against a local PostgreSQL instance - ---- - -## What this stack provides - -When started with the appropriate profile, this stack runs: - -- **PostgreSQL** (`postgres`) - - Official `postgres:18` image with bulk-load-oriented runtime tuning in compose - - Persistent storage via Docker volumes -- **Python workspace** (`python`) - - Local OMOP Alchemy source installed into a reusable container image - - PostgreSQL client tools included for direct `psql` / `pg_dump` access -- **pgAdmin** (`pgadmin`) - - Web UI for inspecting and querying PostgreSQL (optional) -- **JupyterLab** (`cava-jupyter-notebook`, optional) - - Notebook environment built from the local repo and wired to the same database - -All services communicate on a dedicated Docker bridge network (`cava-network`). - ---- +`OMOP_Alchemy` itself makes no assumptions about how PostgreSQL is provisioned: any +reachable instance works, local or otherwise. Docker orchestration for the OMOP stack +is handled at the workspace root (compose files there bring up every package's +containers as peers), not by a per-package `docker-compose.yaml` in this repo. ## Prerequisites -You’ll need: - -- Docker Desktop (or Docker Engine + Compose v2) -- `docker compose` available on your PATH -- A `.env` file in the `docker/` directory - ---- - -## Environment configuration - -Create a `.env` file alongside `docker-compose.yml`, for example: - -```env -POSTGRES_USER=postgres -POSTGRES_PASSWORD=postgres -POSTGRES_DB=cava +- A running PostgreSQL instance (any version supported by `omop_alchemy`'s SQLAlchemy dialects) +- `pip install omop-alchemy` (or an editable install from this repo) -HOST=localhost -HTTP_TYPE=http -``` - -These credentials are not secure and are intentionally simple for local use. - -### Starting the stack - -From the `docker/` directory. - -#### Database + Python workspace - -``` -docker compose up -d -``` +## Configure -#### Database + Python workspace + pgAdmin - -``` -docker compose --profile pgadmin up -d +```bash +omop-config init +omop-config configure omop_alchemy ``` -#### Database + Python workspace + Jupyter - -``` -docker compose --profile jupyter up -d -``` +See [Configuration](configuration.md) for the full field reference. --- ## Running PostgreSQL tests locally -The test suite includes PostgreSQL-specific tests that skip automatically unless a `test_cdm_db` resource is configured in `~/.config/omop/config.toml`. Tests are marked with `@pytest.mark.requires_resource(OmopAlchemyConfig.TEST_DB)` and skipped at collection time when the resource is absent — no manual filtering required. +The test suite includes PostgreSQL-specific tests that skip automatically unless a `test_cdm_db` database is configured in `~/.config/omop/config.toml`. Tests are marked with `@pytest.mark.requires_database("test_cdm_db")` and skipped at collection time when it's absent, no manual filtering required. > **This test database is destructive.** The test suite drops and recreates the entire `public` -> schema on every run. `test_cdm_db` must point to a **dedicated, empty test database** — never -> to a database that contains real data. The test suite enforces this: it will abort if the +> schema on every run. `test_cdm_db` must point to a **dedicated, empty test database**, never +> to a database that contains real data. The test suite enforces this: it fails loudly (not skips) if the > configured database is not marked `test_only = true` in your config. **Step 1 — Register a test database connection:** @@ -94,10 +36,10 @@ The test suite includes PostgreSQL-specific tests that skip automatically unless omop-config configure omop_alchemy ``` -When prompted whether to configure a test database resource, answer **Y** and supply the connection details for your dedicated test PostgreSQL instance. The resource will be saved as `test_cdm_db` with `test_only = true`. +When prompted whether to configure a test database, answer **Y** and supply the connection details for your dedicated test PostgreSQL instance. It will be saved as `test_cdm_db` with `test_only = true`. > **Note on permissions**: the test suite disables FK constraint triggers during bulk vocabulary -> loads — an operation PostgreSQL restricts to superusers. Ensure the test database user has +> loads, an operation PostgreSQL restricts to superusers. Ensure the test database user has > superuser privileges, or provision the user manually with `CREATE USER test SUPERUSER`. **Step 2 — Run the tests:** diff --git a/omop_alchemy/config.py b/omop_alchemy/config.py index 8839fe0..12e7471 100644 --- a/omop_alchemy/config.py +++ b/omop_alchemy/config.py @@ -1,15 +1,15 @@ from __future__ import annotations -from typing import ClassVar +from typing import Annotated, ClassVar import sqlalchemy as sa from pydantic import Field from oa_configurator import ( - DatabaseConfig, + CDMDatabaseConfig, PackageConfigBase, - ResourceSpec, + RefTo, Resolver, - ResolvedResource, + ResolvedDatabase, load_stack_config, ) @@ -44,36 +44,31 @@ def _missing_driver_message(url: str, exc: ModuleNotFoundError) -> str | None: class OmopAlchemyConfig(PackageConfigBase): - CDM_DB: ClassVar[ResourceSpec] = ResourceSpec( - semantic_name="cdm_db", - display_name="OMOP CDM Database", - description="Database containing the OMOP CDM tables and vocabulary.", - connection_name_hint="cdm", - ) - TEST_DB: ClassVar[ResourceSpec] = ResourceSpec( - semantic_name="test_cdm_db", - display_name="Test OMOP CDM Database", - description=( - "Dedicated PostgreSQL database for running integration tests. " - "Tests drop and recreate the entire public schema on every run." - ), - connection_name_hint="pg_test", - cdm_schema_default="public", - connection_defaults=DatabaseConfig( - dialect="postgresql+psycopg", - host="localhost", - port=55432, - user="test", - password="test", - database_name="test_db", - ), - ) + """oa-configurator config class for omop-alchemy, the CDM database owner. + + Every downstream package's own ``cdm_db``-named field shares this + database purely by naming convention (see ``RefTo``), not by importing + this class. + + Attributes + ---------- + cdm_db : str + Name of the ``[databases.*]`` entry holding the CDM database. + test_cdm_db : str, optional + Name of the ``[databases.*]`` entry holding the test CDM database, + marked ``RefTo(CDMDatabaseConfig, is_test=True)``. + + Notes + ----- + By design, this config is for internal use only and must not be + imported or resolved by any other package. + """ tool_name: ClassVar[str] = "omop_alchemy" extra_logging_namespaces: ClassVar[tuple[str, ...]] = ("orm_loader",) - required_resources: ClassVar[tuple[str, ...]] = (CDM_DB.semantic_name,) - owned_resources: ClassVar[tuple[ResourceSpec, ...]] = (CDM_DB,) - test_resources: ClassVar[tuple[ResourceSpec, ...]] = (TEST_DB,) + + cdm_db: Annotated[str, RefTo(CDMDatabaseConfig)] = "cdm_db" + test_cdm_db: Annotated[str | None, RefTo(CDMDatabaseConfig, is_test=True)] = None athena_source_path: str | None = Field( default=None, @@ -81,11 +76,12 @@ class OmopAlchemyConfig(PackageConfigBase): ) -def get_cdm_context() -> tuple[OmopAlchemyConfig, ResolvedResource]: - """Return (pkg_config, resolved_cdm_resource), loading config once. +def get_cdm_context() -> tuple[OmopAlchemyConfig, ResolvedDatabase]: + """Return (pkg_config, resolved_cdm_database), loading config once. - The resource is taken from tools.omop_alchemy.default_resource when set; - otherwise falls back to the canonical CDM_DB resource name. + The CDM database is always whatever ``OmopAlchemyConfig.cdm_db`` resolves + to -- point a deployment at a second CDM instance via that field's own + ``--cdm-db`` flag at configure time, not a call-site override. Raises ------ @@ -99,19 +95,18 @@ def get_cdm_context() -> tuple[OmopAlchemyConfig, ResolvedResource]: "No omop-alchemy configuration found. " "Run `omop-config configure omop_alchemy` to set it up." ) from exc - pkg_config = OmopAlchemyConfig.from_stack(stack) - tool = stack.tools.get(OmopAlchemyConfig.tool_name) - resource_name = (tool.default_resource if tool else None) or OmopAlchemyConfig.CDM_DB.semantic_name - resolved = Resolver(stack).resolve_resource(resource_name) + resolver = Resolver(stack) + pkg_config = resolver.resolve_package_config(OmopAlchemyConfig) + resolved = resolver.resolve_database(pkg_config.cdm_db) return pkg_config, resolved -def create_cdm_engine(resolved: ResolvedResource) -> sa.Engine: +def create_cdm_engine(resolved: ResolvedDatabase) -> sa.Engine: """Create the CDM SQLAlchemy engine with helpful PostgreSQL driver error messages.""" try: return resolved.create_engine() except ModuleNotFoundError as exc: - msg = _missing_driver_message(resolved.database.url, exc) + msg = _missing_driver_message(resolved.connection.url, exc) if msg is not None: raise RuntimeError(msg) from exc raise diff --git a/omop_alchemy/maintenance/_cli_utils.py b/omop_alchemy/maintenance/_cli_utils.py index 2abe24b..2e25f8f 100644 --- a/omop_alchemy/maintenance/_cli_utils.py +++ b/omop_alchemy/maintenance/_cli_utils.py @@ -55,7 +55,7 @@ def wrapper(**kwargs: Any) -> Any: pkg_config, resolved = get_cdm_context() engine = create_cdm_engine(resolved) conn = _ConnContext( - db_schema=resolved.cdm_schema, + db_schema=resolved.schema_name, engine_url=engine.url.render_as_string(hide_password=True), athena_source=pkg_config.athena_source_path, ) diff --git a/omop_alchemy/maintenance/cli_schema_info.py b/omop_alchemy/maintenance/cli_schema_info.py index 915a71d..9fc3de5 100644 --- a/omop_alchemy/maintenance/cli_schema_info.py +++ b/omop_alchemy/maintenance/cli_schema_info.py @@ -336,15 +336,14 @@ def collect_maintenance_info( existing_table_count: int | None = None missing_table_count: int | None = None - resource_name = OmopAlchemyConfig.required_resources[0] + db_name = OmopAlchemyConfig.model_fields["cdm_db"].default try: stack = load_stack_config() - tool = stack.tools.get(OmopAlchemyConfig.tool_name) - resource_name = (tool.default_resource if tool else None) or resource_name resolver = Resolver(stack) - resolved = resolver.resolve_resource(resource_name) - db_schema = resolved.cdm_schema - raw_url = sa.engine.make_url(resolved.database.url) + db_name = resolver.resolve_package_config(OmopAlchemyConfig).cdm_db + resolved = resolver.resolve_database(db_name) + db_schema = resolved.schema_name + raw_url = sa.engine.make_url(resolved.connection.url) engine_url = raw_url.render_as_string(hide_password=True) backend = raw_url.get_backend_name() except (FileNotFoundError, ValueError, ArgumentError, KeyError) as exc: @@ -400,7 +399,7 @@ def collect_maintenance_info( psql_path=psql_path, config_file=str(config_file), config_exists=config_file.exists(), - resource_name=resource_name, + resource_name=db_name, db_schema=db_schema, engine_url=engine_url, backend=backend, diff --git a/pyproject.toml b/pyproject.toml index d0a9734..088706b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,10 +33,10 @@ dependencies = [ "sqlalchemy>=2.0.45", "pandas>=2.0", "pyyaml>=6.0", - "oa-configurator==0.1.2", + "oa-configurator==0.1.2", # TODO: bump to >=1.0.0,<2.0.0 "typer>=0.12", "rich>=13.0", - "orm-loader==0.5.1", + "orm-loader==0.5.1", # TODO: bump to >=1.0.0,<2.0.0 ] [project.optional-dependencies] @@ -45,7 +45,7 @@ postgres = [ ] dev = [ - "oa-configurator[dev,postgres]==0.1.2", + "oa-configurator[dev,postgres]==0.1.2", # TODO: bump to >=1.0.0,<2.0.0 "ipython>=8.0", "requests>=2.33.0", "pytest>=9.0.3", diff --git a/tests/conftest.py b/tests/conftest.py index a7477c9..a796ea5 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -330,26 +330,9 @@ def pg_engine(): Resolves via OA_Configurator resource 'test_cdm_db' in ~/.config/omop/config.toml. Run: omop-config configure omop_alchemy (answer Y when asked to configure test database). """ - from oa_configurator import load_stack_config - from oa_configurator.pytest_plugin import ensure_test_db_exists, ensure_test_user_exists, resolve_test_resource + from oa_configurator.pytest_plugin import ensure_test_db_exists, ensure_test_user_exists, resolve_test_database from omop_alchemy.config import OmopAlchemyConfig - - url = resolve_test_resource(OmopAlchemyConfig.TEST_DB) - - # Safety guard: pg_session does DROP SCHEMA public CASCADE — refuse if the - # backing connection is not explicitly marked test_only in the config. - try: - stack = load_stack_config() - db_name = stack.resources[OmopAlchemyConfig.TEST_DB.semantic_name].database - if not stack.databases[db_name].test_only: - pytest.fail( - f"SAFETY ABORT: the database connection {db_name!r} backing" - f" {OmopAlchemyConfig.TEST_DB.semantic_name!r} is not marked" - f" test_only=true. Tests would DROP SCHEMA public CASCADE on" - f" a non-test database." - ) - except KeyError: - pass # resource or db not in config — resolve_test_resource will skip + url = resolve_test_database(OmopAlchemyConfig, "test_cdm_db") ensure_test_user_exists(url) ensure_test_db_exists(url) diff --git a/tests/test_config_driver.py b/tests/test_config_driver.py index f66cc73..3998936 100644 --- a/tests/test_config_driver.py +++ b/tests/test_config_driver.py @@ -89,12 +89,12 @@ def test_missing_driver_message_returns_none_for_sqlite_url(): def test_sqlite_url_not_intercepted(): """create_cdm_engine should work for sqlite without wrapping errors.""" - from oa_configurator.resolver import ResolvedDatabaseTarget - target = ResolvedDatabaseTarget(name="test", url="sqlite:///:memory:", safe_url="sqlite:///:memory:") + from oa_configurator.resolver import ResolvedConnection + target = ResolvedConnection(name="test", url="sqlite:///:memory:", safe_url="sqlite:///:memory:") from unittest.mock import MagicMock resolved = MagicMock() resolved.create_engine.return_value = target.create_engine() - resolved.database.url = "sqlite:///:memory:" + resolved.connection.url = "sqlite:///:memory:" engine = create_cdm_engine(resolved) engine.dispose() @@ -106,7 +106,7 @@ def test_create_engine_raises_runtime_for_missing_postgres_driver(monkeypatch): resolved = MagicMock() resolved.create_engine.side_effect = exc - resolved.database.url = "postgresql+psycopg://host/db" + resolved.connection.url = "postgresql+psycopg://host/db" with pytest.raises(RuntimeError, match="psycopg"): create_cdm_engine(resolved) diff --git a/tests/test_foreign_keys.py b/tests/test_foreign_keys.py index f91c1f6..1186c19 100644 --- a/tests/test_foreign_keys.py +++ b/tests/test_foreign_keys.py @@ -11,7 +11,7 @@ collect_foreign_key_trigger_status, manage_foreign_key_triggers, ) -from oa_configurator import StackConfig, DatabaseConfig +from oa_configurator import CDMDatabaseConfig, ConnectionConfig, StackConfig runner = CliRunner() @@ -75,8 +75,8 @@ def test_disable_foreign_keys_cli_fails_gracefully_for_sqlite(monkeypatch): """Test disable foreign keys cli fails gracefully for sqlite.""" cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": {"database": "db", "cdm_schema": "main"}}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config", @@ -249,8 +249,8 @@ def test_enable_foreign_keys_strict_cli_invokes_strict_management(monkeypatch): calls: dict[str, object] = {} cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": {"database": "db", "cdm_schema": "main"}}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config", @@ -361,8 +361,8 @@ def test_foreign_keys_validate_cli_invokes_validation(monkeypatch): calls: dict[str, object] = {} cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": {"database": "db", "cdm_schema": "main"}}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config", diff --git a/tests/test_fulltext.py b/tests/test_fulltext.py index 37dbd78..3deb655 100644 --- a/tests/test_fulltext.py +++ b/tests/test_fulltext.py @@ -1,7 +1,7 @@ import sqlalchemy as sa import pytest from typer.testing import CliRunner -from oa_configurator import StackConfig, DatabaseConfig +from oa_configurator import CDMDatabaseConfig, ConnectionConfig, StackConfig from omop_alchemy.backends import ( CONCEPT_NAME_TSVECTOR_COLUMN, @@ -203,8 +203,8 @@ def test_fulltext_install_cli_passes_options(monkeypatch): calls: dict[str, object] = {} cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": {"database": "db", "cdm_schema": "public"}}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="public")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config", diff --git a/tests/test_indexes.py b/tests/test_indexes.py index 7a2be4d..940f681 100644 --- a/tests/test_indexes.py +++ b/tests/test_indexes.py @@ -1,7 +1,7 @@ import pytest import sqlalchemy as sa from typer.testing import CliRunner -from oa_configurator import StackConfig, DatabaseConfig, ResourceConfig +from oa_configurator import CDMDatabaseConfig, ConnectionConfig, StackConfig from omop_alchemy.backends.sqlite import SQLiteBackend from omop_alchemy.cdm.base.indexing import OMOP_CLUSTER_INDEX_INFO_KEY, omop_index_name @@ -327,8 +327,8 @@ def test_disable_indexes_cli_invokes_management(monkeypatch): calls: dict[str, object] = {} cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": ResourceConfig(database="db", cdm_schema="main")}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config", @@ -391,8 +391,8 @@ def test_enable_indexes_cli_no_cluster_flag_passes_through(monkeypatch): calls: dict[str, object] = {} cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": ResourceConfig(database="db", cdm_schema="main")}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config", diff --git a/tests/test_load_vocab_postgres.py b/tests/test_load_vocab_postgres.py index fed6fcc..33c33c1 100644 --- a/tests/test_load_vocab_postgres.py +++ b/tests/test_load_vocab_postgres.py @@ -13,7 +13,6 @@ import sqlalchemy as sa from omop_alchemy.cdm.model.vocabulary import Concept -from omop_alchemy.config import OmopAlchemyConfig from omop_alchemy.maintenance.cli_vocab import ( _load_vocab_model_csv, load_vocab_source, @@ -59,7 +58,7 @@ def _make_concept_source( # --------------------------------------------------------------------------- -@pytest.mark.requires_resource(OmopAlchemyConfig.TEST_DB) +@pytest.mark.requires_database("test_cdm_db") def test_end_to_end_vocab_load_on_postgres(pg_session, pg_engine, tmp_path): """load_vocab_source() completes end-to-end on real Postgres via orm-loader>=0.4.0.""" source_path = _copy_fixture_source(tmp_path) @@ -74,7 +73,7 @@ def test_end_to_end_vocab_load_on_postgres(pg_session, pg_engine, tmp_path): -@pytest.mark.requires_resource(OmopAlchemyConfig.TEST_DB) +@pytest.mark.requires_database("test_cdm_db") def test_quote_mode_auto_regression_on_postgres(pg_session, pg_engine, tmp_path): """ quote_mode='auto' strips RFC-4180 double-quotes via PostgreSQL COPY. @@ -113,7 +112,7 @@ def test_quote_mode_auto_regression_on_postgres(pg_session, pg_engine, tmp_path) -@pytest.mark.requires_resource(OmopAlchemyConfig.TEST_DB) +@pytest.mark.requires_database("test_cdm_db") def test_load_vocab_model_csv_on_postgres(pg_session, tmp_path): """ _load_vocab_model_csv loads data correctly on a real PostgreSQL session. @@ -138,7 +137,7 @@ def test_load_vocab_model_csv_on_postgres(pg_session, tmp_path): -@pytest.mark.requires_resource(OmopAlchemyConfig.TEST_DB) +@pytest.mark.requires_database("test_cdm_db") def test_replace_strategy_overwrites_existing_rows(pg_session, pg_engine, tmp_path): """merge_strategy='replace' fully replaces rows with the same PKs on second load.""" concept_id = 99999 @@ -160,7 +159,7 @@ def test_replace_strategy_overwrites_existing_rows(pg_session, pg_engine, tmp_pa -@pytest.mark.requires_resource(OmopAlchemyConfig.TEST_DB) +@pytest.mark.requires_database("test_cdm_db") def test_upsert_strategy_is_non_destructive(pg_session, pg_engine, tmp_path): """merge_strategy='upsert' preserves existing rows on second load with same PKs.""" concept_id = 99998 @@ -184,7 +183,7 @@ def test_upsert_strategy_is_non_destructive(pg_session, pg_engine, tmp_path): -@pytest.mark.requires_resource(OmopAlchemyConfig.TEST_DB) +@pytest.mark.requires_database("test_cdm_db") def test_db_schema_search_path_on_postgres(pg_engine, tmp_path): """ load_vocab_source with db_schema creates vocabulary tables in the requested diff --git a/tests/test_load_vocab_source.py b/tests/test_load_vocab_source.py index 10067b1..fdface3 100644 --- a/tests/test_load_vocab_source.py +++ b/tests/test_load_vocab_source.py @@ -2,7 +2,7 @@ import pytest import sqlalchemy as sa -from oa_configurator import StackConfig, DatabaseConfig +from oa_configurator import CDMDatabaseConfig, ConnectionConfig, StackConfig from sqlalchemy.orm import sessionmaker from typer.testing import CliRunner @@ -159,9 +159,9 @@ def test_load_vocab_source_cli_uses_configured_athena_source(monkeypatch, tmp_pa athena_dir.mkdir() cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": {"database": "db", "cdm_schema": "main"}}, - tools={OmopAlchemyConfig.tool_name: {"extra": {"athena_source_path": str(athena_dir)}}}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, + tools={OmopAlchemyConfig.tool_name: {"athena_source_path": str(athena_dir)}}, ) monkeypatch.setattr( @@ -425,8 +425,8 @@ def test_load_vocab_source_cli_surfaces_database_error_detail(monkeypatch): """Test load vocab source cli surfaces database error detail.""" cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": {"database": "db", "cdm_schema": "main"}}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config", diff --git a/tests/test_truncate_tables.py b/tests/test_truncate_tables.py index 2467359..912500b 100644 --- a/tests/test_truncate_tables.py +++ b/tests/test_truncate_tables.py @@ -2,7 +2,7 @@ import sqlalchemy as sa import pytest from typer.testing import CliRunner -from oa_configurator import StackConfig, DatabaseConfig +from oa_configurator import CDMDatabaseConfig, ConnectionConfig, StackConfig from omop_alchemy.maintenance.cli import app from omop_alchemy.maintenance.cli_schema import create_missing_tables @@ -43,8 +43,8 @@ def test_truncate_tables_cli_requires_confirmation(monkeypatch): """Test truncate tables cli requires confirmation.""" cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": {"database": "db", "cdm_schema": "main"}}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config", @@ -62,8 +62,8 @@ def test_truncate_tables_cli_invokes_management(monkeypatch): calls: dict[str, object] = {} cfg = StackConfig.for_session( - databases={"db": DatabaseConfig(dialect="sqlite", database_name=":memory:")}, - resources={"cdm_db": {"database": "db", "cdm_schema": "main"}}, + connections={"db": ConnectionConfig(dialect="sqlite", database_name=":memory:")}, + databases={"cdm_db": CDMDatabaseConfig(connection="db", schema_name="main")}, ) monkeypatch.setattr( "omop_alchemy.config.load_stack_config",