This project focuses on the end-to-end data engineering and statistical processing of multi-asset historical market data (2010–2025). It bridges the gap between raw financial datasets and advanced analytical modeling by implementing rigorous data cleaning, feature engineering, and dimensionality reduction techniques.
The analysis processes daily historical data for five key global and domestic assets:
- Indices: Nifty 50 (India), S&P 500 (USA)
- Currency: USD/INR Exchange Rate
- Commodities: Gold, Brent Crude Oil
Raw financial data often contains gaps due to regional market holidays. This project implements:
- Handling Missing Values: Systematic identification of nulls across all asset classes.
- Dual-Phase Imputation: Utilization of Forward-Fill (
ffill) followed by Backward-Fill (bfill) to maintain time-series continuity without introducing look-ahead bias.
To prepare the data for statistical modeling, the following metrics are derived for each asset:
-
Logarithmic Returns: Calculated as
$\ln(Price_t / Price_{t-1})$ to ensure the returns are additive over time and approximately normally distributed. -
Volatility Proxy: An intra-day range metric calculated as
$(High - Low) / Open$ to capture daily market stress.
The project moves beyond basic cleaning into exploratory factor analysis:
- Principal Component Analysis (PCA): Reducing dimensionality to identify core drivers of market movement.
- Factor Analysis (FA): Extracting latent market sentiment scores (Fear vs. Greed) by analyzing the covariance between different asset returns and volatilities.
- Python: Core processing engine.
- Pandas & NumPy: For high-performance data manipulation and mathematical operations.
- Scikit-Learn / FactorAnalyzer: For implementing dimensionality reduction and latent variable modeling.
- Matplotlib: For visualizing returns and sentiment indices over time.
data_cleaning_1.ipynb: Primary Jupyter Notebook containing the cleaning pipeline and statistical analysis.daily_market_data.csv: Historical dataset spanning 15 years of market activity.
Author: Applied Statistician & Financial Risk Enthusiast