Skip to content

perf(markov): vectorize PCCA+ inner simplex search -- 78x faster - #345

Merged
clonker merged 1 commit into
deeptime-ml:mainfrom
dexhunter:perf/pcca-isa-vectorize
Nov 9, 2026
Merged

clonker merged 1 commit into
deeptime-ml:mainfrom
dexhunter:perf/pcca-isa-vectorize

Conversation

@dexhunter

@dexhunter dexhunter commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR

  • This PR speeds up _pcca_connected_isa, the inner simplex step of PCCA+ in markov/tools/analysis/dense/_pcca.py.
  • It replaces the two per-row Python loops with whole-array NumPy calls: row norms, one rank-one update and one masked argmax.
  • Benchmark with 6000 states and 10 eigenvectors, plus smaller calls: 228.81 ms -> 2.92 ms, 98.7% faster.
  • python -m pytest tests/markov/tools/analysis/impl/dense/pcca_test.py passes (10 passed).

What & Why

Before, the function visited each state in a Python loop. The first loop computed the norm of each row twice to find the most outlying state. For each later representative, a second loop projected each row in turn, computed its norm and checked i not in ind[0:k].

Now one small helper, farthest, computes all row norms at once. It masks the states that are already representatives and returns the first largest positive norm. It returns 0 when no norm is positive, as the old loop did. The projection of all rows is one np.outer update. The projection after the last representative is skipped because no later step uses it. The first representative is the origin, so its projection was always zero and is not computed. The same states are selected in the same order, so chi and the rotation matrix stay identical.

Results

Situation Before After Change
6000 states, 10 clusters 197.18 ms 1.92 ms 103x faster
2000 states, 4 of 7 eigenvectors 24.21 ms 0.31 ms 78x faster
500 states, 3 clusters 4.39 ms 0.08 ms 53x faster
20 states, 2 clusters 0.134 ms 0.029 ms 4.7x faster
More clusters than eigenvectors (early ValueError) 0.6 µs 0.6 µs no slowdown (same code)

Why this is safe and easy to maintain

The change is small and only touches _pcca_connected_isa. One helper does the representative search for every step, so the selection rule is in one place. The chosen states, chi, the rotation matrix and the error messages stay the same. The tests in tests/markov/tools/analysis/impl/dense/pcca_test.py, such as test_pcca_1 and test_pcca_2, cover this function through pcca.

Validation

  • python -m pytest tests/markov/tools/analysis/impl/dense/pcca_test.py: 10 passed with and without the change.
  • Benchmark: random eigenvectors of 6000 states with 10 clusters, plus smaller calls, eigenvectors of a metastable chain and the error paths. Median time went from 228.81 ms to 2.92 ms on one pinned core. Three repeated runs gave ratios of 0.0129, 0.0128 and 0.0126. Each run also checked that the outputs match the unchanged code.
  • Reproduce the first row of the table:
import numpy as np, timeit
from deeptime.markov.tools.analysis.dense._pcca import _pcca_connected_isa
v = np.random.default_rng(0).normal(size=(6000, 10)); v[:, 0] = 1.0
print(min(timeit.repeat(lambda: _pcca_connected_isa(v, 10), number=1, repeat=7)))

Checklist

  • Make sure to include one or more tests for your change (the existing PCCA tests already cover this function; no new test added)
  • Add yourself to AUTHORS

AI assistance

This PR was prepared with AI assistance from Aiden. Candidates were tracked with Weco Observe.

Weco share link: https://dashboard.weco.ai/share/soFJ60nnElI_gJ0oasAkfKw4hTI-wya2

_pcca_connected_isa looped over every state in Python to find the most
outlying state and, for each later representative, to project each row
and compute its norm. Use whole-array row norms, one np.outer rank-one
update and a shared masked argmax helper instead. The same states are
selected in the same order, so chi and the rotation matrix are unchanged.

Benchmark (6000 states, 10 eigenvectors, plus smaller calls):
228.81 ms -> 2.92 ms.

Co-Authored-By: Aiden <aiden@weco.ai>
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.03%. Comparing base (2a57dcd) to head (eb54e62).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #345      +/-   ##
==========================================
- Coverage   94.03%   94.03%   -0.01%     
==========================================
  Files         141      141              
  Lines       11202    11200       -2     
==========================================
- Hits        10534    10532       -2     
  Misses        668      668              
Flag Coverage Δ
decomposition 48.84% <0.00%> (+<0.01%) ⬆️
markov 69.18% <100.00%> (-0.01%) ⬇️
misc 65.69% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@clonker
clonker merged commit b1b7006 into deeptime-ml:main Nov 9, 2026
76 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants