Skip to content

Implemented normalized spatial fire occurrence ETL pipeline#153

Open
Sonu0823 wants to merge 1 commit into
mainfrom
feature/spatial-fire-etl
Open

Implemented normalized spatial fire occurrence ETL pipeline#153
Sonu0823 wants to merge 1 commit into
mainfrom
feature/spatial-fire-etl

Conversation

@Sonu0823
Copy link
Copy Markdown
Collaborator

Summary

This PR implements the FireFusion fire occurrence ETL pipeline aligned with the normalized spatial database architecture.

Changes Made

  • Added fire occurrence ingestion pipeline for 2024–2025 fire datasets
  • Implemented extraction, transformation, and loading (ETL) workflow
  • Added data cleaning and deduplication logic
  • Integrated KDTree nearest-neighbor spatial mapping
  • Replaced raw latitude/longitude storage with normalized location_id
  • Integrated with location_registry
  • Added chunked Supabase/PostgreSQL upload support
  • Added pipeline documentation and requirements configuration

Database Architecture

The pipeline uploads processed records into the fire_occurrence fact table using normalized spatial references and foreign key-compatible schema integration.

Technical Highlights

This implementation demonstrates:

  • ETL pipeline development
  • geospatial data processing
  • KDTree spatial mapping
  • PostgreSQL + Supabase integration
  • scalable batch ingestion
  • normalized database architecture

Notes

  • .env and raw datasets are excluded from version control
  • Requires existing location_registry table in Supabase
  • Uploads use chunked inserts for scalability

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant