Skip to content

Fix cumulative path construction in Evolving Correlations notebook #360

Description

@Lqz13Th

Summary

The Evolving Correlations notebook computes overlapping two-observation returns and differences for its PCA inputs, but later accumulates every observation when constructing the “Actual vs Predicted” paths.

This causes two separate correctness issues:

  1. gs_quant.timeseries.returns produces decimal returns (for example, 1% is represented as 0.01), but the notebook divides those values by 100 again before compounding.
  2. Consecutive two-observation changes overlap, so accumulating every row double-counts periods.

Current implementation

The PCA input is built with:

frame_t[key] = returns(frame_t[key], 2) if v['type'] == 'r' else diff(frame_t[key], 2)

The displayed paths are then constructed with:

(1 + t_d[key] / 100).cumprod()
t_d[key].cumsum()

For example, consecutive return observations represent P_t / P_(t-2) - 1 and P_(t+1) / P_(t-1) - 1. Multiplying both does not reconstruct a valid price path because the intervals overlap.

Proposed approach

Preserve the two-observation changes used by the PCA, but select every second observation before constructing the displayed paths. This produces a non-overlapping sequence while retaining the notebook’s stated cross-market smoothing choice.

For return series:

sampled = changes.loc[since:].iloc[::2]
path = (1.0 + sampled).cumprod()

For difference series:

sampled = changes.loc[since:].iloc[::2]
path = sampled.cumsum()

The paths can then be normalized to 1 for return series and 0 for difference series.

The associated variables, headings, and legends should also use “PCA reconstruction” rather than “predicted”, because the calculation is an in-sample low-rank reconstruction rather than an out-of-sample forecast.

Notebook outputs

The final comparison plots were generated by the current implementation. A PR implementing this change should clear the affected saved outputs so that figures produced by the old calculation are not retained. Maintainers with the required FRED API credentials can regenerate those outputs before merge or in a follow-up change.

I plan to submit a linked PR implementing the non-overlapping path construction and terminology updates.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions