Forecasting realized volatility for 5 US sector ETFs across 20 years of daily data. The project runs a full pipeline from data collection to deep learning, comparing statistical models, machine learning, and LSTM.
ETFs: XLF (Financials), XLK (Technology), XLE (Energy), XLV (Health Care), XLU (Utilities)
- Source: Yahoo Finance, daily OHLCV, 2004-2026
- Volatility measure: Yang-Zhang realized volatility, annualized
- Target variable: Next-day Yang-Zhang RV
- Train/Test split: 80/20, chronological, no shuffling
| Category | Features |
|---|---|
| Own RV history | Lags 1-5, rolling means at 5/10/21/63 days |
| Cross-sector | Lagged log(RV) of the other 4 sectors |
| Market | Log return (lag 1), VIX (lag 1) |
| Technical | RSI extremity (abs(RSI-50)/50), Bollinger Band Width |
| Calendar | Day-of-week (Mon-Thu), month (Jan-Nov), month-end, quarter-end flags |
All features shifted by 1 day. StandardScaler fit on training data only. LSTM drops the lag and rolling RV features since the lookback window already captures that history.
Statistical models
- Rolling window baseline (21-day)
- ARIMA with rolling one-step-ahead forecasting
- GARCH(1,1) and EGARCH(1,1) with skewed-t distribution
- HAR-RV (OLS on daily, weekly, monthly RV components)
Machine learning
- Ridge and Lasso regression
- Random Forest
- XGBoost
- SVR
All ML models use walk-forward cross-validation with 4 expanding folds and a 63-day purge gap between train and validation to prevent leakage from autocorrelated volatility. Hyperparameters tuned with Optuna.
Deep learning
- Single-layer LSTM with 21-day lookback window
- Optuna tuning for hidden size (32-256), dropout (0.1-0.5), learning rate
- ReduceLROnPlateau scheduler, early stopping with patience 15
- Trained in log(RV) space, evaluated in original RV space
| Sector | Baseline | ARIMA | GARCH | EGARCH | XGBoost | LSTM |
|---|---|---|---|---|---|---|
| Financials | 0.281 | 0.365 | 0.218 | 0.282 | 0.554 | 0.393 |
| Technology | 0.238 | 0.254 | 0.138 | 0.279 | 0.245 | 0.161 |
| Energy | 0.281 | 0.340 | 0.272 | 0.297 | 0.588 | 0.291 |
| Health Care | 0.174 | 0.274 | 0.175 | 0.140 | 0.507 | 0.332 |
| Utilities | 0.071 | 0.166 | 0.110 | 0.137 | 0.310 | 0.088 |
| Sector | Baseline | ARIMA | GARCH | EGARCH | XGBoost | LSTM |
|---|---|---|---|---|---|---|
| Financials | 0.0831 | 0.0781 | 0.0866 | 0.0830 | 0.0654 | 0.0754 |
| Technology | 0.1091 | 0.1080 | 0.1161 | 0.1062 | 0.1087 | 0.1157 |
| Energy | 0.1063 | 0.1018 | 0.1069 | 0.1051 | 0.0805 | 0.1038 |
| Health Care | 0.0617 | 0.0578 | 0.0616 | 0.0629 | 0.0476 | 0.0548 |
| Utilities | 0.0702 | 0.0665 | 0.0687 | 0.0676 | 0.0605 | 0.0693 |
Yang-Zhang captures what Parkinson misses Yang-Zhang diverges most from Parkinson during crisis periods (2008-09, 2020) when overnight gaps are large. Parkinson only uses the high-low range and misses the overnight jump component, which is exactly when volatility spikes matter most.
Volatility memory is strong and long Lag-1 ACF ranges from 0.60 to 0.73 across sectors and stays statistically significant through lag 63. Volatility today predicts volatility months from now, not just tomorrow. This is the core signal every model here exploits.
Short windows beat long windows for the baseline A 5-day rolling window consistently beats 10, 21, and 63-day windows as a baseline. This lines up with the ACF pattern, where short lags carry the most predictive signal. The one exception is Utilities, where a slightly longer window works better because its volatility is driven by slow-moving macro and regulatory factors rather than news events.
ARIMA beats the rolling baseline across all 5 sectors ARIMA picks up the linear autocorrelation structure that a simple average misses. The gains are bigger in Financials and Health Care, which have stronger long-memory, and smaller in Technology where volatility is more event-driven.
GARCH underperforms ARIMA on direct RV forecasting GARCH models return variance from returns as a proxy for realized volatility. There is a mismatch between what GARCH estimates and what we are trying to forecast. ARIMA works directly in log(RV) space, which gives it a structural edge here. That said, GARCH is still the right tool for risk management applications where you care about the full return distribution, not just point forecasts.
Leverage effect confirmed across all sectors EGARCH shows negative gamma in all 5 sectors. Negative returns amplify volatility more than positive returns of the same size. Technology and Financials show the strongest asymmetry, consistent with sentiment-driven markets. Energy and Utilities show the weakest leverage effect, consistent with their more fundamental, demand-driven dynamics.
XGBoost is the best overall model XGBoost wins on 4 of 5 sectors, with R2 of 0.554 for Financials, 0.588 for Energy, and 0.507 for Health Care. Max depth of 3 was optimal across all sectors, which tells you the predictive relationships are fairly shallow and do not require deep feature interactions.
ML beats LSTM despite the sequence structure LSTM does not outperform XGBoost on any sector despite using a 21-day lookback window. On a dataset of around 5000 rows, the added complexity of sequence modeling does not pay off. The signal is mostly a persistence problem and tree models handle it more efficiently.
LSTM early stopping was very fast Financials stopped at epoch 11, Technology at epoch 17, out of 100 max. The models learned what they could in very few passes. This suggests the 26-feature input already carries a compact, fast signal and there is not much more for the LSTM to extract over longer training.
Utilities is the hardest sector across every model R2 never exceeds 0.310 for Utilities across any model. The LSTM only gets 0.088, essentially no predictive power. Utilities volatility is driven by interest rate movements and regulatory changes, which are slow-moving macro factors that do not align well with the pattern-based approach any of these models take.
Technology has a performance ceiling Every model, from ARIMA to LSTM, clusters in the R2 range of 0.14 to 0.28 for Technology. The sector's volatility is event-driven and idiosyncratic. Its ACF decays the fastest across all sectors, meaning there is simply less temporal structure to exploit.
Volume is the top feature for Financials and Energy This was a bit surprising. For these two sectors, trading activity and volatility interact nonlinearly, and Random Forest and XGBoost pick up on it. For Health Care and Utilities, the volatility persistence signal is approximately linear and Ridge ends up generalizing better.
Lasso eliminated technical indicators in 3 of 5 sectors RSI extremity and Bollinger Band Width were zeroed out for Technology, Health Care, and Utilities. Once persistence and cross-sector features are included, technical indicators carry no incremental information in those sectors.
Calendar effects had almost no impact anywhere Day-of-week and month dummies were largely eliminated by Lasso across all sectors. There are no meaningful seasonal patterns in realized volatility for these ETFs.
Technology ignores cross-sector RV; Energy depends on it Lasso dropped cross-sector RV from Financials, Health Care, and Utilities for Technology, confirming that tech volatility is largely sector-specific. Energy kept most cross-sector features, consistent with its macro and commodity-driven co-movements.
Heavy regularization wins everywhere Across all ML models, the best hyperparameters favored regularization over complexity: high Ridge alpha, max depth 3 for XGBoost, small C for SVR. The signal-to-noise ratio in volatility forecasting is low and simpler models generalize better.
Volatility Forecast/
├── get_data.ipynb # Fetch OHLCV, compute Yang-Zhang RV
├── feature_engineering.ipynb # Build feature matrix, save to features.xlsx
├── modeling.ipynb # Rolling baseline, ARIMA, GARCH, EGARCH, HAR-RV
├── machine_learning_modeling.ipynb # Ridge, Lasso, RF, XGBoost, SVR with Optuna CV
├── deep_learning.ipynb # LSTM (local GPU)
├── deep_learning_colab.ipynb # LSTM (Google Colab)
├── data/
│ ├── realized_volatility.xlsx
│ ├── features.xlsx
│ └── cv_folds.pkl
└── results/ # Per-sector predictions, metrics, loss curves, plots
pip install -r requirements.txt
# PyTorch (GPU, CUDA 11.8)
pip install torch --index-url https://download.pytorch.org/whl/cu118
# PyTorch (CPU only)
pip install torchRun notebooks in order: get_data -> feature_engineering -> modeling -> machine_learning_modeling -> deep_learning