Summary
The Evolving Correlations notebook computes overlapping two-observation returns and differences for its PCA inputs, but later accumulates every observation when constructing the “Actual vs Predicted” paths.
This causes two separate correctness issues:
gs_quant.timeseries.returns produces decimal returns (for example, 1% is represented as 0.01), but the notebook divides those values by 100 again before compounding.
- Consecutive two-observation changes overlap, so accumulating every row double-counts periods.
Current implementation
The PCA input is built with:
frame_t[key] = returns(frame_t[key], 2) if v['type'] == 'r' else diff(frame_t[key], 2)
The displayed paths are then constructed with:
(1 + t_d[key] / 100).cumprod()
t_d[key].cumsum()
For example, consecutive return observations represent P_t / P_(t-2) - 1 and P_(t+1) / P_(t-1) - 1. Multiplying both does not reconstruct a valid price path because the intervals overlap.
Proposed approach
Preserve the two-observation changes used by the PCA, but select every second observation before constructing the displayed paths. This produces a non-overlapping sequence while retaining the notebook’s stated cross-market smoothing choice.
For return series:
sampled = changes.loc[since:].iloc[::2]
path = (1.0 + sampled).cumprod()
For difference series:
sampled = changes.loc[since:].iloc[::2]
path = sampled.cumsum()
The paths can then be normalized to 1 for return series and 0 for difference series.
The associated variables, headings, and legends should also use “PCA reconstruction” rather than “predicted”, because the calculation is an in-sample low-rank reconstruction rather than an out-of-sample forecast.
Notebook outputs
The final comparison plots were generated by the current implementation. A PR implementing this change should clear the affected saved outputs so that figures produced by the old calculation are not retained. Maintainers with the required FRED API credentials can regenerate those outputs before merge or in a follow-up change.
I plan to submit a linked PR implementing the non-overlapping path construction and terminology updates.
Summary
The
Evolving Correlationsnotebook computes overlapping two-observation returns and differences for its PCA inputs, but later accumulates every observation when constructing the “Actual vs Predicted” paths.This causes two separate correctness issues:
gs_quant.timeseries.returnsproduces decimal returns (for example, 1% is represented as0.01), but the notebook divides those values by 100 again before compounding.Current implementation
The PCA input is built with:
The displayed paths are then constructed with:
For example, consecutive return observations represent
P_t / P_(t-2) - 1andP_(t+1) / P_(t-1) - 1. Multiplying both does not reconstruct a valid price path because the intervals overlap.Proposed approach
Preserve the two-observation changes used by the PCA, but select every second observation before constructing the displayed paths. This produces a non-overlapping sequence while retaining the notebook’s stated cross-market smoothing choice.
For return series:
For difference series:
The paths can then be normalized to 1 for return series and 0 for difference series.
The associated variables, headings, and legends should also use “PCA reconstruction” rather than “predicted”, because the calculation is an in-sample low-rank reconstruction rather than an out-of-sample forecast.
Notebook outputs
The final comparison plots were generated by the current implementation. A PR implementing this change should clear the affected saved outputs so that figures produced by the old calculation are not retained. Maintainers with the required FRED API credentials can regenerate those outputs before merge or in a follow-up change.
I plan to submit a linked PR implementing the non-overlapping path construction and terminology updates.