Local development environment for Python data projects, with Docker

This project demos setting up a local development environment for data project with Docker

Prerequisites

1. Docker and Docker Compose >= v1.27.0
2. Git

How to use the repo

git clone https://github.com/ppkhai2612/local_dev.git
cd local_dev
make up # spins up the containers
make ci # run tests and format code
make run-etl # run the ETL process
make down # spins down the containers

You can log into the warehouse DB using make warehouse to see the results of the ETL run

select * from housing.user;
\q -- quit database

You will see 10 records look like this

If you want to explore the internal environment of the loader container, you can use make sh

You can log into the Metabase dashboard by going to http://localhost:3000/

Project structure

When you clone the repo, the project structure is shown as below

local_dev/
├── containers 
│   ├── loader # build context for loader image
│   │   ├── Dockerfile
│   │   └── requirements.txt
│   └── warehouse
│       └── create_user_table.sql # script for create table in database
├── docker-compose.yml # define the required containers
├── env # define environment variables for containers
├── Makefile # define aliases for complex commands
├── README.md # project overview
├── src
│   └── loader
│       └── load_user_data.py # script for load data to
├── test # testing
│   └── integration
│       └── test_load_user_data.py
└── utils # scripts for support database connect
    ├── db.py
    ├── __init__.py
    └── sde_config.py

Loading process

For more details, please read the post: https://www.startdataengineering.com/post/local-dev/