diff --git a/docs/data_user_guide.md b/docs/data_user_guide.md index ff7e417..5bdfb0e 100644 --- a/docs/data_user_guide.md +++ b/docs/data_user_guide.md @@ -18,7 +18,7 @@ df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe() ## Accessing Data -When the ETL pipelines are run, the data sources (raw and/or transformed) are stored into Azure Blob Storage. You can access these datasets directly using the `datacat` interface: +Raw and transformed data produced by ETL pipelines are stored in Azure Blob Storage. You can access these datasets directly using the `datacat` interface: ```python from cfa.dataops import datacat @@ -27,12 +27,18 @@ from cfa.dataops import datacat df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe() # Get raw data as polars DataFrame -df = datacat.private.scenarios.seroprevalence.extract.get_dataframe(output="polars") +raw_df = datacat.private.scenarios.seroprevalence.extract.get_dataframe(output="polars") # Get specific version -df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe( +version_df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe( version_spec="==2025-06-03T17-56-50" ) + +# Get raw or transformed data as Polars Lazyframe +lazy_df = datacat.private.scenarios.seropervalence.extract.get_dataframe(output="pl_lazy") + +# Get reference datasets +ref_df = datacat.reference.my_reference_dataset.get_dataframe() ``` ### Dataset Access Methods @@ -138,6 +144,9 @@ vax_df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe() # Get raw data for analysis raw_vax = datacat.private.scenarios.covid19vax_trends.extract.get_dataframe() + +# Get raw or transformed data as LazyFrame +lazy_vax = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="pl_lazy") ``` ### Fetching Versions within a Range diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 0000000..a875ef2 --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,74 @@ +# CFA DataOps Glossary +This glossary provides clear, CDC-context definiitons of key technologies, tools and concepts frequently used in the **cfa-dataops** environment. It is intended to support new developers onboarding into CFA DataOps workflows. + + +## Azure Blob Storage +**Azure Blob Storage** is Microsoft Azure's cloud object storage solution used for storing large volumes of unstructured data such as CSV files, Parquet datasets, model outputs, logs, and other artifacts. + +### Why it matters in cfa-dataops + - It provides secure, scablable storage for ingestion pipelines, cleaned datasets, and analytical outputs used in CFA modeling and analytics + - Many cfa-dataops integration tests rely on Blob Storage access, which requires authenticating with 'az login --identity` + - Enables cloub-based pipelines that mirror production environments, making local-to-cloud reproducibility easier + + +## Catalog (CFA Catalog) +The **CFA Catalog** is a central structured repository of datasets used by CFA modeling teams. It provides metadata, versioning, provenance, and standardized accessibility, enabling discoverability, reproducibility, and governance. + +### Why it matters in cfa-dataops + - Ensures datasets are well-documented and versioned + - Allows analytics teams to locate authoratative ("source of truth") datasets quickly + - Supports publication workflows for modeling and public-facing data products + - Ensure reproducible analytics across CFA teams. + + +## DuckDB +DuckDB is an in-process OLAP (analytical) database designed for fast, local analytical queries. It runs inside Python and supports fast SQL queries on large data files without requiring a server. + +### Why it matters in cfa-dataops + - Supports SQL, making transformations readable and standardized + - Enables reproducible local pipelines before cloud publication + - Ideal for rapid local development and reproducible ETL workflows + - Efficient for working with large CSV/Parquet datasets locally + + +## Hypothesis +Hypothesis is a property-based testing framework for Python. Instead of manually specifying inputs, Hypothesis automatially generates input data to explore edge cases. + +### Why it matters in cfa-dataops + - Helps ensure reliability of ingestion and transformation functions + - Useful for validating data schemas or catalog consistency rules + - Integrated into cfa-dataops testing alongside pytest (unit + property-based tests, unit + randomized checks) + + +## Polars +Polars is a high-performance DataFrame library for Rust and Python, optimized for tabular data processing. + +### Why it matters in cfa-dataops + - Extremely fast for cleaning, filtering, merging, and reshaping datasets + - Offers better performance compared to pandas for large datasets + - Works seamlessly with DuckDB to deliver flexible, efficient ETL patterns + - Offers declarative query patterns and efficient lazy computation + + +## Pytest +**pytest** is a Python testing framework used to write and execute test suites, including unit tests, integration tests, and property-based tests. + +### Why it matters in cfa-dataops + - CFA DataOps uses pytest as its primary test runner, including support for: + - Discovery of test files + - Mocking with pytest-mock + - Coverage reporting + - Property-based tests via Hypothesis + - Unit tests, + - Integration tests + - pytest integrates seamlessly with uv (uv run pytest) + - supports node ID selection for running specific tests. + + +## UV +`uv` is a fast, modern Python package environment manager designed to replace slower and heavier tools sucha as pip and virtualenv. It ensures reproducible environments and predictable dependency resolution. + +### Why it matters in cfa-dataops + - uv provides reliable installs and consistent execution environments across developer machines and CI + - In cfa-dataops, uv is the recommended setup tool for running tests and syncing dependencies (uv sync, uv run pytest) + - It improves the stability of pipelines and reduces environment drift diff --git a/docs/troubleshooting-guide.md b/docs/troubleshooting-guide.md new file mode 100644 index 0000000..8113781 --- /dev/null +++ b/docs/troubleshooting-guide.md @@ -0,0 +1,254 @@ +# CFA DataOps — Troubleshooting Guide + +## Purpose + +This guide is intended to improve the user experience with cfa-dataops client capabilities. It provides guidance on troubleshooting common issues that CFA users may encounter when using **CFA dataops** toolkit, catalogs, and reporting utilities. + + +## How to Use This Guide + +1. **Start with a quick health check** (environment, access). +2. **Find your issue** in the sections below. +3. **Implement a solution**, then re‑run the minimal example(s). +4. If the issue persists, collect logs and **open a GitHub issue** in the [Repo](https://github.com/CDCgov/cfa-dataops). + + +## Quick Health Check + +- **Python environment** + + - Confirm Python 3.10+ is active and dependencies installed (pandas, polars, duckdb, papermill, etc.). + +- **Import the core namespaces** + + ```Python``` + + `from cfa.dataops import datacat` + + `print("Datasets:", datacat.__namespace_list__)` + + If imports fail, see **Environment & Installation** below. + +- **Azure Cloud access** + + If your workflow requires blob access, ensure you’re logged in, using the appropriate authorization for your context. + + `az login --identity` + +## Common Issues & Solutions + +### 1. Environment & Installation + + **Issue** + + `ModuleNotFoundError` for cfa.dataops, polars, duckdb, or pandera + +**Common causes** + + - Virtual environment not activated. + - Installed with incompatible Python version. + +**Solution** + + - Verify Python listed in pyproject.toml and install/upgrade accordingly. + - Re‑install the project using your team’s standard (e.g., uv, pip, or poetry) and re‑activate the venv. + - Re‑try minimal imports (see Quick Health Check). + + +### 2. Accessing Data — get_dataframe() Errors + + **Issue** + + Errors when loading dataframes (e.g., “no matching version”, “cannot resolve selection”). + +**Common causes** + + - No dataset versions meet your version-spec. + - Using default selection where multiple matches exist. + - Attempting to load large outputs into pandas that exceed memory. + +**Solution** + + - Preview the version that would be used before loading: + + `from cfa.dataops import datacat` + + `resolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version( version_spec=">=2025-05-01,<2025-06-01", selection="newest",)` + + `print(resolved.version)` + + `print(resolved.blob_url)` + + Then pass the same arguments to: + + `get_dataframe()` + + + - List available versions to confirm your constraints + + `datacat.private.scenarios.covid19vax_trends.load.get_versions()` + + If empty or unexpected, re‑run ETL or relax version-spec. + + - Choose appropriate output for size/performance + + \# pandas DataFrame + + `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="pandas")` + + \# polars DataFrame + + `df_pl = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="polars")` + + \# lazy polars (defer materialization) + + `df_lazy = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(output="pl_lazy")` + + Use polars/pl_lazy for large datasets to mitigate memory pressure. + + + - Advanced version filters + + - If you rely on range or pattern filters (e.g., >=..., <..., latest), confirm your string syntax matches the project’s conventions noted in release notes. + + +### 3. Authentication / Azure Blob Access + +**Issue** + + Read/write fails to blob storage; “permission denied” or timeouts. + +**Common causes** + + - Not authenticated in the current shell/session. + - Missing role or wrong subscription/tenant. + - Network constraints. + +**Solution** + + - Log in using your approved method (e.g., az login --identity) and verify subscription. + - If you use helper utilities from the companion cfa-cloudops library for authentication/workflows, ensure it’s correctly configured and you’re on a supported Python version. + - Re‑run a small read_blobs/get_versions check to validate access via the catalog (see **Data User Guide**). + + +### 4. Catalog Creation & Management + + **Issue** + + New datasets or catalogs don’t appear under datacat.__namespace_list__. + +**Common causes** + + - Catalog not installed/registered in the environment. + - Misconfigured dataset definition (TOML). + - Namespace conflicts. + +**Solution** + + - Use the catalog initialization CLI to create a standards‑compliant structure: + + `dataops_catalog_init --help` + + + Then follow the catalog creation and managing guides in docs/. + + - Validate dataset configuration (paths, names, and extract/load endpoints) and re‑install the catalog if needed. (See Managing Catalogs and Catalog Creation pages). + - Restart your Python session to refresh namespace discovery and re‑check: + + `from cfa.dataops import datacat` + + `print(datacat.__namespace_list__)` + + +### 5. Schema Validation Failures + + **Issue** + + Errors citing required columns, types, or value ranges. + + **Common causes** + + - Source feed changed upstream. + - Transform altered types unexpectedly. + - Misaligned schema definitions. + + **Solution** + + - Review the dataset’s schema expectations and correct the ETL or input source. (See Data Validation in the Data User Guide). + + - If recent changes impacted schemas, consult **Release Notes** for updates and migrate accordingly. + + +### 7. Performance & Memory + + **Issue** + + Slow dataframe operations; process killed due to memory. + + **Common causes** + + - Loading large datasets into pandas. + - Inefficient transformations and eager evaluation. + + **Solution** + + - Use Polars or DuckDB for large joins/aggregations + - Use lazy operations via output="pl_lazy" and materialize only the final result. + - Filter and project early (column & row pruning) before joins; verify results on a small sample. + + +### 8. Versioning & Reproducibility + + **Issue** + + Analyses aren’t reproducible; different runs return different data. + + **Common causes** + + - Implicit latest version selection. + - Ambiguous version-spec ranges. + + **Solution** + + - Pin exact versions using timestamp equality: + + `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe(version_spec="==2025-06-03T17-59-16")` + + - Document your version-spec and selection and store them with the analysis for auditability. + + +## Minimal Working Examples (MWE) + + - List datasets & load one + + `from cfa.dataops import datacat` + + `print(datacat.__namespace_list__)` + + `df = datacat.private.scenarios.covid19vax_trends.load.get_dataframe()` + + - Preview version before load + + `from cfa.dataops import datacat` + + `resolved = datacat.private.scenarios.covid19vax_trends.load.resolve_version(version_spec=">=2025-05-01,<2025-06-01", selection="newest",)` + + `print(resolved.version, resolved.blob_url)` + + +## When to Open a GitHub Issue + +Open an issue when: + + - You can reproduce a failure using an MWE above. + - Behavior contradicts documented APIs or release notes. + - A dataset’s schema or versioning appears inconsistent with docs. + +Provide: + + - Python version, package versions (pip list/uv pip list), OS. + - Exact code snippet and traceback. + - Dataset/catalog names and version-spec. + - Whether Azure auth was active (az login), if relevant. + +Use the repo’s [issue tracker](CDCgov/cfa-dataops) diff --git a/tests/dataops-tests.md b/tests/dataops-tests.md new file mode 100644 index 0000000..84b369e --- /dev/null +++ b/tests/dataops-tests.md @@ -0,0 +1,100 @@ +# CFA DataOps Tests + +## Overview +The cfa-dataops/tests directory contains automated checks to help ensure the reliability of **cfa-dataops** library and its supporting utilities. The suite is designed to run locally and in CI, emphasizing fast unit tests while allowing (optional) integration tests that touch cloud resources used by CFA DataOps (e.g. Azure Blob Storage). + +## Key Features of Tests Directory +**Pytest-based suite:** Leverages pytest for discovery and execution. +**Mocking support:** Uses pytest-mock to isolate external dependencies during unit testing. +**Property-based tests:** Optionally uses hypothesis to validate invariants across randomized inputs. +**Coverage instrumentation:** Configurable via .coveragerc and pytest-cov. +**Works with uv:** The ecosystem commonly runs commands through uv (e.g., uv run pytest) for consistent environments. + +## Quick Start Checklist +1. Python: install Python 3.10 or newer. + +2. Clone the repo + + `git clone https://github.com/CDCgov/cfa-dataops.git` + + `cd cfa-dataops` + +3. Set up the environment (recommended: uv) + + \# Install project dependencies using uv + + `uv sync` + +4. Authenticate to Azure (Optional) + + if you will run integration tests that touch cloud resources: + + `az login –identity` + +5. Run the tests + + \# All tests (recommended) + + `uv run pytest` + + +## Getting Started +1. Install & Setup + + #### With uv (recommended) + + \# from the repository root + + `uv sync` + + `uv run pytest` + + #### With pip (alternative) + + `python -m venv .venv` + + `source .venv/bin/activate' + + \# Windows: + + `.venv\Scripts\activate` + + `python -m pip install --upgrade pip` + + `pip install -e .` + + `pytest` + +2. Running Specific Tests + + #### Single file or node ID (pytest standard) + + `uv run pytest tests/path/to/testmodule.py::TestClass::testmethod` + + + Selecting tests via node IDs is a standard pytest feature. + - Show detailed output + + `uv run pytest -vv` + +3. Coverage (optional) + + If you’d like coverage reports: + + `uv run pytest --cov=cfa.dataops --cov-report=term-missing` + +4. Cloud-Dependent Tests (optional) + + Some tests may rely on access to CDC cloud resources. + + \# Authenticate (if applicable) + + `az login –identity` + + +## Docs for developers + Project documentation explains how data catalogs and ETL/reporting components work: + - Project documentation + - Data User Guide + - Data Developer Guide + - CLI Tools Reference