Data Platform Modernization

📌 Objective

Modernizing legacy data pipelines (SAS-based) into scalable, cloud-ready data engineering solutions using PySpark, Snowflake, and Airflow.

---

🧠 Architecture Overview

Legacy System (SAS) ↓ Data Ingestion ↓ Transformation Layer (PySpark / Snowflake) ↓ Orchestration (Airflow) ↓ Curated Data Layer ↓ Analytics / Reporting

---

🛠️ Tech Stack

• PySpark
• Snowflake
• Apache Airflow
• SQL

---

📁 Projects Included

🔹 SAS to PySpark Transformation

Recreated SAS ETL logic using PySpark

🔹 Snowflake ELT Pipeline

Designed scalable data pipeline using Snowflake

🔹 Airflow Orchestration

Implemented DAGs for pipeline scheduling and monitoring

---

🎯 Key Highlights

• Legacy ETL modernization
• Scalable pipeline design
• Performance optimization techniques
• Data validation strategies

---

🚀 Future Enhancements

• Add streaming pipelines
• Implement data quality framework
• Integrate Delta Lake