I build production data systems across AWS, Databricks, Spark, and PySpark. My work sits between platform engineering, distributed processing, and performance engineering — from designing scalable pipelines to understanding what the engine is doing underneath them.
The longer arc is deliberate: Data Platforms → Distributed Systems → Performance → Accelerated Data Systems, with semantic data and GenAI systems becoming integral layers of the stack.
Active contributions to core data engines, table formats, and query frameworks:
| Engine / Framework | Pull Request | Technical Focus & Scope | CI / Review Status |
|---|---|---|---|
| NVIDIA cuDF Spark | #16688 | Spark Catalyst multi-digit patch versioning across build tooling, CI, and test matrices | Blossom CI Passed (30+ checks green), Greptile 5/5 |
| NVIDIA cuDF Spark | #16715 | GpuCreateDataSourceTableAsSelectCommand table drop on saveAsTable overwrite (SPARK-60387) |
Greptile 5/5, Signoff Passed, Upstream Synced |
| Apache Iceberg | #4265 | Reference snapshot ID invariant validation during branch snapshot expiration (#4200) | 20/20 CI checks passed, in maintainer queue |
| Ibis Project | #12211 | Decimal mean type promotion (float64) and product precision/scale calculation (#12209) |
Semantic Release passed, DCO signed |
| Databricks dbt | #1758 | Relation string literal escaping for metadata queries against catalogs and schemas (#1747) | 100% PR Test Coverage, 8/8 CI checks passed |
A live visual representation of 7+ years of production systems evolution:
- Foundations & Academic Core: Master in Computer Applications (MCA), GEM Award for technical leadership, Python & SQL modeling.
- Enterprise Cloud Migration: Architected AWS framework (Glue, Redshift, DMS, S3) for financial datasets (Tata AMC) handling 10+ TB with 40% automation efficiency.
- High-Throughput Streaming: Databricks, PySpark, Delta Live Tables & Kafka/Kinesis real-time scoring (Acorns) delivering 40% pipeline time reduction and 30% compute cost savings.
- Accelerated Frontier: Advancing into Spark Catalyst optimization, off-heap Tungsten memory, and GPU kernel acceleration (NVIDIA CUDA & RAPIDS cuDF).
- I/O & File Pruning: Liquid clustering reduced bytes scanned by 24% across analytical predicates, measured against serverless cluster startup overheads.
- Memory & Shuffle Optimization: Mitigated high-cardinality skew via AQE dynamic split and adaptive broadcast switching, eliminating 100GB+ disk spills.
- Real-Time Stream Processing: Engineered end-to-end streaming scoring with 40% latency acceleration and 30% compute cost reduction.
- Metadata-Driven Cloud Migration: Automated migration of 10+ TB financial data to Redshift with reproducible Terraform IaC and zero-loss observability.
Incremental Ingestion · Data Quality · CDC · SCD1/SCD2 · Observability · Governance
Explore Sentinel Lakehouse Repository →
Databricks Genie AI/BI · Mosaic AI Vector Search · Incident RCA · Compound Query Routing
- Genie-Bench (Semantic Layer & Evaluation Harness): Semantic metadata compiler generating Unity Catalog table annotations, join graphs, and trusted SQL definitions with SQLGlot AST equivalence matching.
- AutoHeal-DE (Pipeline Incident RCA & Healing): Live Delta commit log analyzer detecting late-arriving partitions, bounding historical window blast radii, and generating surgical partition-pruned Delta
MERGEstatements without expensive full-table backfills. - Hybrid Compound Router: Fuses structured Databricks SQL aggregations with unstructured Mosaic AI Vector Search chunks via Reciprocal Rank Fusion (RRF k=60) with numerical consistency guardrails.
Explore Interactive Hybrid RAG & Genie Demo Notebook →
|
Flagship Production Architecture Full Medallion Lakehouse engine on Databricks with Auto Loader incremental ingestion, Delta Change Data Feed (CDF), SCD Type 1 & Type 2 dimensional models, automated Great Expectations quarantine dead-letter gates, Unity Catalog lineage, and the Sentinel-GenAI hybrid RAG engine.
|
Databricks DLT & PySpark Optimization Engine Open-source static AST optimizer and quality engine for Databricks. Features 12 diagnostic engines: Catalyst plan bloat auditing, zero-loss quarantine streams, bounded late-partition MERGE restatement, Unity Catalog migration, Liquid Clustering advisory, live workspace telemetry, and DBU cost forecasting.
|
|
Systems Internals & Execution Exploration A specialized technical laboratory exploring SQL query parsing, relational execution trees, vectorized columnar batch processing, predicate pushdown algorithms, and in-memory operator performance.
|
|
|
Self-Healing Pipelines & Incident RCA Zero-scan Delta commit log monitor detecting late-arriving partitions, bounding rolling-window blast radii, and executing surgical partition-pruned Delta MERGE reconciliations without costly full-table restatements.
|
Databricks AI/BI & Genie Space Quality Gate Automated semantic metadata compiler translating Unity Catalog tables, column metrics, and join graphs into Databricks Genie spaces with verified golden queries, SQLGlot AST validation, and automated scorecards.
|
I am interested in problems where data, systems, performance, and intelligence meet — especially data platforms, Spark and query execution, performance experiments, semantic data, GenAI workflows, and GPU-accelerated data systems.