Scalable Data Warehouse

Sensor Data ELT

A fully dockerized ELT pipeline project, using Airflow, PostgreSQL, DBT, and Redash.
Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Contact
4. Acknowledgements

About The Project

Fully dockerized scalable ELT pipeline using PostgreSQL, Airflow, DBT and Redash.

Built With

Tech Stack used in this project

• Airflow
• PostgreSQL
• DBT
• Redash

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• DockerCompose

Installation

1. Clone the repo and Navigate

   git clone https://github.com/nahomHmichael/Traffic-ELT-pipeline.git

2. Run

    docker-compose -f docker-compose.yaml up --build

3. Open Airflow web browser

   Navigate to `http://localhost:8081/` on the browser
   activate and trigger the `load_to_dwh_v2` dag
   activate and trigger the `dbt_orchestrate` dag

4. Access redash dashboard

   Navigate to `http://localhost:3500/` on the browser

5. Access your postgres database

   Navigate to `http://localhost:5432/` on the browser
   use `warehouse` for db name
   use `warehouse` for username
   use `warehouse` for password

Contact

Nahom Habtemichael - nahomhabtemichael@gmail.com

Acknowledgements

• 10 Academy