Skip to content

data processing

Jump to bottom
Syed Ibrahim Omer edited this page Apr 12, 2026 · 5 revisions

Data Processing

Transforms raw data from data/raw/ into model-ready formats in data/processed/. Handles PDF parsing, JSON normalization, CSV consolidation, and schema standardization.

Architecture

src/processing/
├── parse.py              → PDF visa table extraction (PyMuPDF)
├── news.py               → JSON articles → Parquet + tokenization
├── merge.py              → Consolidate encounter CSVs
├── build_panel.py        → Multi-source panel construction
├── summarize.py          → FLAN-T5 cluster labeling
├── run_summarization.py  → CLI orchestrator for NLP pipeline
├── prompts.py            → Configurable prompt templates
└── utils.py              → Shared helpers, month/visa maps

Processing Steps

1. Visa PDF Parsing → PDF Parser

  • Input: 108 monthly PDFs from data/raw/visa/pdf/
  • Tool: PyMuPDF (fitz) table extraction
  • Output: data/processed/visa_master.parquet
  • Parallelized with ProcessPoolExecutor

2. News JSON → Parquet

  • Input: JSON files per country/year from data/raw/news/
  • Process: Flatten articles, parse dates (multi-source extraction), tokenize with Jina tokenizer
  • Output: data/processed/news/news_*.parquet

3. Encounter CSV Merge

  • Input: 8 CSVs from data/raw/encounter/
  • Process: Column mapping (Fiscal Year → year, month abbreviation → month_num), lazy frame concatenation (Polars)
  • Output: Unified encounter DataFrame

4. Panel Construction → Panel Construction

  • Merges visa, exchange rate, and news event data into a monthly × country panel

Data Flow

data/raw/visa/pdf/       → parse.py      → data/processed/visa_master.parquet
data/raw/news/           → news.py       → data/processed/news/news_*.parquet
data/raw/encounter/*.csv → merge.py      → merged encounter DataFrame
data/raw/trends/         → (passed through to analysis)
All processed data       → build_panel.py → data/processed/train_panel.parquet

Key Design Choices

  • Polars lazy frames for memory-efficient large dataset operations
  • Country name canonicalization via maps.json to handle variants (Dominican_Republic → Dominican Republic)
  • Parallel PDF parsing via ProcessPoolExecutor with optimal CPU count detection
  • Parquet output with LZ4 compression for efficient storage and fast reads

See Also

Wiki navigation

Quick Start

  • Project Overview — Goals, research questions, methodology, and team
  • Glossary — Key terms used throughout this wiki

Data Sources

Raw inputs that feed the prediction system.

Page Description
Visa Data US Department of State visa issuance statistics (108 monthly PDFs)
Encounter Data CBP Southwest border encounter statistics (FY2019–2026)
Google News 170K+ news articles across 15 countries × 8 topics
Google Trends Monthly search-interest time series (15 countries × 8 keywords)
Exchange Rates IMF Real Effective Exchange Rate for 6 countries

Pipeline

The end-to-end flow from raw data to production forecasts.

Page Description
Data Collection Ingestion layer: async scraping, bounded concurrency, retry logic
Data Processing PDF parsing, JSON→Parquet, encounter merging
NLP Enrichment Embedding → Clustering → Labeling → Sentiment
Panel Construction Feature engineering: 18 lag features, 6 lead targets
Training Pipeline Out-of-time train/test split, 4 architectures
Inference Pipeline Horizon-aware ensemble, production prediction flow

Models

Machine learning architectures and their roles in the ensemble.

Page Description
Random Forest cuML GPU Random Forest — best at short horizons (Lead 1–2)
LSTM MigrationLSTM — country-aware with SurgeJointLoss
Transformer MigrationTransformer — best at long horizons (Lead 5–6)
Horizon-Aware Ensemble Dynamic weighting: RF→short, Transformer→long
SurgeJointLoss Dual-objective loss: Huber + BCE for crisis detection
Jina v5 Embeddings TensorRT INT8 news article embeddings (768-dim)
Flan-T5 Summarization TensorRT INT8 cluster labeling engine

Analysis Methods

Statistical techniques driving the lead-lag and surge analysis.

Page Description
Lead-Lag Analysis Pearson correlation at 0–6 month offsets
Surge Detection Quantile-based and σ-threshold spike identification
Sentiment Analysis Rule-based lexicon scoring for migration-relevant news
Event Clustering HDBSCAN GPU clustering + LED label generation
Cross-Correlation Analysis CCF analysis, VAR benchmarking, ADF stationarity tests
Multiple Comparison Correction Benjamini-Hochberg FDR for 58 significant signals

Key Findings

What the system discovered about migration predictability.

Page Description
Event-Visa Findings News events as leading indicators (r=0.617 at 3-month lag)
Exchange Rate Findings Exchange rate signals (DR r=0.498 at 2-month lag)
Model Performance Ensemble results: F1=0.96 at Lead 1, F1=0.86 at Lead 6

Source Modules

Reference documentation for every src/ subpackage and key files.

Page Description
Main Entry Point src/main.py CLI: bootstrap, collect-live, sync-data
Collection Module src/collection/* — visa, encounter, news, trends, HF sync
Processing Module src/processing/* — parse, merge, build_panel, summarize
Analysis Module src/analysis/* — events, exchange_rate, trends_analysis, plots
Models Module src/models/* — surge_model, train_and_evaluate, inference
News Scraper Deep dive: batch decoding, checkpoint recovery, throttling
PDF Parser Deep dive: PyMuPDF table extraction, VISA_MAP normalization
TensorRT Engines Deep dive: Jina-v5, Flan-T5, LED TensorRT engines
Build Panel Detail Deep dive: lag/lead construction, forward-fill strategies
HF Sync Deep dive: bidirectional Hugging Face Hub sync

Infrastructure

Compute, reproducibility, and operational details.

Page Description
GPU Acceleration TensorRT INT8, cuML, CUDA streams, NVML profiling
Reproducibility HF bootstrap, run.sh pipeline, dependency checking

Clone this wiki locally