-
Notifications
You must be signed in to change notification settings - Fork 0
analysis module
Syed Ibrahim Omer edited this page Apr 12, 2026
·
5 revisions
src/analysis/ — statistical analysis, lead-lag correlation, and visualization.
| File | Purpose | Key Functions |
|---|---|---|
events.py |
Event detection & sentiment |
sentiment_score(), detect_surges_list(), safe_corr_list(), benjamini_hochberg(), shift_list()
|
event_visa_analysis.py |
Event → visa correlation | Loads clustered/labeled news, computes lagged correlations, surge pattern detection |
exchange_rate.py |
REER → visa analysis |
detect_exchange_shocks(), lagged correlation with visa issuances |
trends_analysis.py |
Trends → migration analysis |
load_focus_countries(), parse_trend_file(), load_trends_long(), load_visa_monthly()
|
label_events_with_led.py |
LED cluster labeling | Samples articles, builds prompts, generates 2-3 word labels, normalizes output |
plots.py |
Visualization suite |
setup_styling(), load_data(), create_dual_axis_plot(), publication-quality 300 DPI PNGs |
utils.py |
Analysis utilities |
sentiment_score(), detect_surges_list(), detect_exchange_shocks(), benjamini_hochberg(), load_exchange_monthly_lazy(), load_visa_monthly_lazy()
|
POSITIVE_WORDS: improve, growth, peace, jobs, stability
NEGATIVE_WORDS: crisis, violence, conflict, inflation, poverty, trafficking
# Event-visa lead-lag analysis
python -m src.analysis.event_visa_analysis --max-lag 6 --top-labels 8 --min-overlap 12 --min-event-months 12
# Exchange rate analysis
python -m src.analysis.exchange_rate
# Trends analysis
python -m src.analysis.trends_analysis| Analysis | Output Path |
|---|---|
| Event-visa | data/processed/production_outputs/event_visa_* |
| Sentiment | data/processed/production_outputs/event_sentiment_* |
| Exchange | data/processed/production_outputs/exchange_visa_* |
| Trends | data/processed/production_outputs/trends_* |
| Plots | data/plots/{events_vs_visas,exchange_vs_visas,trends_vs_migration}/ |
- Lead-Lag Analysis — Statistical methodology
- Sentiment Analysis — Scoring approach
- Event Clustering — HDBSCAN + labeling
- Cross-Correlation Analysis — CCF/VAR analysis
- Processing Module — Upstream data provider
- Models Module — Downstream model training
- Project Overview — Goals, research questions, methodology, and team
- Glossary — Key terms used throughout this wiki
Raw inputs that feed the prediction system.
| Page | Description |
|---|---|
| Visa Data | US Department of State visa issuance statistics (108 monthly PDFs) |
| Encounter Data | CBP Southwest border encounter statistics (FY2019–2026) |
| Google News | 170K+ news articles across 15 countries × 8 topics |
| Google Trends | Monthly search-interest time series (15 countries × 8 keywords) |
| Exchange Rates | IMF Real Effective Exchange Rate for 6 countries |
The end-to-end flow from raw data to production forecasts.
| Page | Description |
|---|---|
| Data Collection | Ingestion layer: async scraping, bounded concurrency, retry logic |
| Data Processing | PDF parsing, JSON→Parquet, encounter merging |
| NLP Enrichment | Embedding → Clustering → Labeling → Sentiment |
| Panel Construction | Feature engineering: 18 lag features, 6 lead targets |
| Training Pipeline | Out-of-time train/test split, 4 architectures |
| Inference Pipeline | Horizon-aware ensemble, production prediction flow |
Machine learning architectures and their roles in the ensemble.
| Page | Description |
|---|---|
| Random Forest | cuML GPU Random Forest — best at short horizons (Lead 1–2) |
| LSTM | MigrationLSTM — country-aware with SurgeJointLoss |
| Transformer | MigrationTransformer — best at long horizons (Lead 5–6) |
| Horizon-Aware Ensemble | Dynamic weighting: RF→short, Transformer→long |
| SurgeJointLoss | Dual-objective loss: Huber + BCE for crisis detection |
| Jina v5 Embeddings | TensorRT INT8 news article embeddings (768-dim) |
| Flan-T5 Summarization | TensorRT INT8 cluster labeling engine |
Statistical techniques driving the lead-lag and surge analysis.
| Page | Description |
|---|---|
| Lead-Lag Analysis | Pearson correlation at 0–6 month offsets |
| Surge Detection | Quantile-based and σ-threshold spike identification |
| Sentiment Analysis | Rule-based lexicon scoring for migration-relevant news |
| Event Clustering | HDBSCAN GPU clustering + LED label generation |
| Cross-Correlation Analysis | CCF analysis, VAR benchmarking, ADF stationarity tests |
| Multiple Comparison Correction | Benjamini-Hochberg FDR for 58 significant signals |
What the system discovered about migration predictability.
| Page | Description |
|---|---|
| Event-Visa Findings | News events as leading indicators (r=0.617 at 3-month lag) |
| Exchange Rate Findings | Exchange rate signals (DR r=0.498 at 2-month lag) |
| Model Performance | Ensemble results: F1=0.96 at Lead 1, F1=0.86 at Lead 6 |
Reference documentation for every src/ subpackage and key files.
| Page | Description |
|---|---|
| Main Entry Point |
src/main.py CLI: bootstrap, collect-live, sync-data |
| Collection Module |
src/collection/* — visa, encounter, news, trends, HF sync |
| Processing Module |
src/processing/* — parse, merge, build_panel, summarize |
| Analysis Module |
src/analysis/* — events, exchange_rate, trends_analysis, plots |
| Models Module |
src/models/* — surge_model, train_and_evaluate, inference |
| News Scraper | Deep dive: batch decoding, checkpoint recovery, throttling |
| PDF Parser | Deep dive: PyMuPDF table extraction, VISA_MAP normalization |
| TensorRT Engines | Deep dive: Jina-v5, Flan-T5, LED TensorRT engines |
| Build Panel Detail | Deep dive: lag/lead construction, forward-fill strategies |
| HF Sync | Deep dive: bidirectional Hugging Face Hub sync |
Compute, reproducibility, and operational details.
| Page | Description |
|---|---|
| GPU Acceleration | TensorRT INT8, cuML, CUDA streams, NVML profiling |
| Reproducibility | HF bootstrap, run.sh pipeline, dependency checking |