☁️ Cloud Data Platform — E-Commerce Analytics Pipeline

A production-grade batch data pipeline that extracts data from REST APIs, lands it in an S3-compatible data lake, loads it into a data warehouse, and transforms it into a Star Schema dimensional model using modern data engineering best practices.

---

🏗️ Architecture

┌──────────┐     ┌──────────────┐     ┌──────────┐     ┌──────────┐     ┌──────────────┐
│  REST    │────▶│   Apache     │────▶│  MinIO   │────▶│PostgreSQL│────▶│    dbt       │
│  API     │     │   Airflow    │     │  (S3)    │     │  (DW)    │     │  Transform   │
│          │     │              │     │          │     │          │     │              │
│ Products │     │ EL Pipeline  │     │ raw-data/│     │ raw      │     │ staging      │
│ Users    │     │ dbt Pipeline │     │          │     │ staging  │     │ intermediate │
│ Carts    │     │              │     │          │     │ analytics│     │ marts ⭐     │
└──────────┘     └──────────────┘     └──────────┘     └──────────┘     └──────────────┘

✨ Key Features

Feature	Implementation
Orchestration	Apache Airflow with TaskFlow API & CeleryExecutor
Extract & Load	Python API extraction with retry logic, S3 landing, PostgreSQL loading
Data Lake	MinIO (S3-compatible) with date-partitioned raw JSON storage
Transformations	dbt-core with 3-layer architecture (staging → intermediate → marts)
Dimensional Model	Star Schema with fact & dimension tables
Data Quality	dbt tests (unique, not_null, relationships, range, custom)
SCD Type-2	dbt snapshots tracking product dimension changes
Infrastructure	Fully containerized with Docker Compose
Testing	DAG integrity tests + API extractor unit tests
Documentation	dbt docs, data dictionary, architecture diagrams

📊 Star Schema Model

                    ┌──────────────┐
                    │ dim_customers│
                    │──────────────│
                    │ customer_key │
              ┌────▶│ full_name    │
              │     │ email        │
              │     │ customer_tier│
              │     └──────────────┘
              │
┌─────────────┤     ┌──────────────┐     ┌──────────────────┐
│ fct_orders  │     │  dim_date    │     │ fct_order_items   │
│─────────────│     │──────────────│     │──────────────────│
│ order_key   │────▶│ date_key     │◀────│ order_item_key   │
│ customer_key│     │ full_date    │     │ order_key        │
│ date_key    │     │ year/quarter │     │ product_key      │
│ total_amount│     │ is_weekend   │     │ date_key         │
│ discount_amt│     │ fiscal_year  │     │ quantity         │
└─────────────┘     └──────────────┘     │ unit_price       │
                                          │ line_total       │
                    ┌──────────────┐      └────────┬─────────┘
                    │ dim_products │               │
                    │──────────────│               │
                    │ product_key  │◀──────────────┘
                    │ title        │
                    │ category     │
                    │ brand        │
                    │ price        │
                    └──────────────┘

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• 4GB+ RAM available for Docker

One-Command Setup

# Clone the repository
git clone https://github.com/yourusername/cloud-data-platform.git
cd cloud-data-platform

# Start everything
make setup

This will:

1. Create .env from the template
2. Build the custom Airflow image
3. Start all 9 Docker services
4. Initialize the database and create the admin user
5. Create S3 buckets in MinIO

Access the Services

Service	URL	Credentials
Airflow UI	http://localhost:8080	airflow / airflow
MinIO Console	http://localhost:9001	minio_admin / minio_password
PostgreSQL	localhost:5432	warehouse / warehouse123

Run the Pipeline

1. Open the Airflow UI at http://localhost:8080
2. Unpause the el_ecommerce_pipeline DAG
3. Click Trigger DAG to run the Extract & Load pipeline
4. Once complete, unpause and trigger dbt_transform_pipeline

Useful Commands

make status          # Check service health
make logs-airflow    # Tail Airflow logs
make dbt-run         # Run dbt models manually
make dbt-test        # Run dbt tests
make test            # Run all Python tests
make shell-postgres  # Open psql shell
make clean           # Tear down everything (destructive!)

📁 Project Structure

cloud-data-platform/
├── airflow/
│   ├── Dockerfile                  # Custom Airflow image
│   ├── requirements.txt            # Python dependencies
│   ├── dags/
│   │   ├── el_ecommerce_pipeline.py    # Extract & Load DAG
│   │   └── dbt_transform_dag.py        # dbt orchestration DAG
│   └── include/extractors/
│       └── api_extractor.py        # Reusable API extraction class
│
├── dbt_project/
│   ├── models/
│   │   ├── staging/                # Clean & standardize raw data
│   │   ├── intermediate/           # Business logic & enrichment
│   │   └── marts/                  # Star Schema (dims + facts)
│   ├── macros/                     # Reusable SQL functions
│   ├── seeds/                      # Reference/lookup data
│   ├── snapshots/                  # SCD Type-2 tracking
│   └── tests/                     # Custom data quality tests
│
├── infrastructure/
│   ├── postgres/init.sql           # Schema & table initialization
│   └── minio/create-buckets.sh     # S3 bucket creation
│
├── tests/                          # Python test suite
├── docs/                           # Architecture & data dictionary
├── docker-compose.yml              # Full infrastructure stack
└── Makefile                        # One-command operations

🔧 Technology Stack

Layer	Technology	Purpose
Orchestration	Apache Airflow 2.9	DAG-based pipeline scheduling & monitoring
Extraction	Python + Requests	API data extraction with retry/pagination
Data Lake	MinIO (S3 API)	Raw data storage with date partitioning
Data Warehouse	PostgreSQL 16	Structured storage & analytical queries
Transformation	dbt-core 1.8	SQL-based ELT transformations
Message Broker	Redis 7	Celery task queue for Airflow workers
Containerization	Docker Compose	Reproducible infrastructure
Testing	pytest + dbt tests	Pipeline & data quality validation

🧪 Data Quality

Data quality is enforced at multiple levels:

1. Source Freshness: dbt checks that raw data was loaded within 24h
2. Schema Tests: Uniqueness, not-null, referential integrity on every model
3. Range Tests: Price, rating, quantity validated within expected bounds
4. Business Rules: Custom tests (e.g., no negative order totals)
5. SCD Tracking: Product changes captured via Type-2 snapshots

☁️ Cloud Migration Path

This project is designed for easy cloud deployment:

Local Component	AWS Equivalent	GCP Equivalent
MinIO	Amazon S3	Cloud Storage
PostgreSQL	Redshift / RDS	BigQuery
Airflow	MWAA	Cloud Composer
Redis	ElastiCache	Memorystore
Docker Compose	ECS / EKS	GKE

📝 License

This project is open source and available under the MIT License.