Battle-tested Apache Spark tuning patterns with reproducible benchmarks. 10 techniques (partition pruning, broadcast joins, AQE, skew handling, Z-ORDER, and more) — each paired with measured before/after speedups runnable on a laptop.

This repository showcases how to setup a Scala Spark job on Docker and in Dataproc and execute a Broadcast Join technique.

PySpark ETL & analytics pipeline for taxi trip ETA, partitioned Parquet, windowed aggregations and performance patterns.

This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.

🚖 Ingest and analyze NYC yellow taxi data with a streamlined ETL pipeline, featuring data cleaning, analytics, and business-ready outputs.