Data Engineering Project -> Taxi Data to AWS S3

This Data Engineering project focuses on efficiently ingesting taxi data from various CSV files into AWS S3 using Magi.ai pipelines. The entire infrastructure is provisioned and managed with Terraform, ensuring scalability, reliability, and ease of deployment.

Key Features

• MageAI Pipelines Integration: Leverage the power of Magi.ai pipelines to seamlessly upload and manage taxi data from disparate sources.

• AWS S3 Storage: Utilize AWS S3 as a robust storage solution for storing and organizing large volumes of taxi-related data.

• Terraform Infrastructure as Code (IaC): Employ Terraform scripts to automate the provisioning and configuration of AWS resources, ensuring consistency and reproducibility.

Flow Diagram:

Overall Pipeline in Mage:

Fetched Data using Data Loader

Transformed and Tested Data

Loaded data to Postgres

Loaded data to S3

Loaded data in partitions

Added Trigger to run pipeline daily