Skip to content

perf(flux): speed up pathway searches -- 70% faster - #351

Open
dexhunter wants to merge 1 commit into
deeptime-ml:mainfrom
dexhunter:perf/pathway-reachability
Open

dexhunter wants to merge 1 commit into
deeptime-ml:mainfrom
dexhunter:perf/pathway-reachability

Conversation

@dexhunter

Copy link
Copy Markdown
Contributor

TL;DR

  • Speed up ReactiveFlux.pathways for dominant-path decomposition.
  • Reuse reachable nodes as edges are added, and skip repeated endpoint sorting.
  • Combined benchmark: 73.23 ms → 21.96 ms, 70.0% faster; a 100-state call saves 43.68 ms.
  • Native tests: python3 -m pytest tests/markov/tools/flux/test_reactive_flux.py tests/markov/test_reactive_flux.py -q --tb=short — 57 passed.

What & Why

The bottleneck search rebuilt sparse graphs and repeated reachability searches for different edge cutoffs.
Now it adds edges in the existing sorted order and keeps the nodes already reached.
Duplicate or nonpositive edges keep the existing search; unchanged edge ordering preserves tied-path choices.

Capacity reads reuse DOK storage, and one helper handles singleton endpoint checks.
Numeric endpoint checks use comparisons instead of sorting arrays.

Validation

Times below are per call, measured on one isolated CPU core with one numerical-library thread.
The grid uses 100 states, nearest-neighbor transitions, and a 50% flux target.
The smaller case comes from the existing reactive-flux tests.
Grid rows separate five tied-weight samples from four varied-weight samples already measured.
Before and after columns show median times; percentage changes use the median new/old ratio for each situation.

Situation Before After Change
100-state grid, tied edge weights, 50% flux 55.16121 ms 11.80238 ms 78.4% faster
100-state grid, varied edge weights, 50% flux 55.63443 ms 11.94501 ms 78.6% faster
Five-state test case, all paths 2.08046 ms 1.14052 ms 44.9% faster
Rectangular zero-flux input 0.44623 ms 0.42274 ms 5.6% faster
Out-of-range source endpoint (control) 0.03259 ms 0.03224 ms No material change

The combined workload also checks sparse formats, multiple endpoints, tied edges, zero flux, and iteration limits.
Its three runs give these statistics:

Statistic Value
Median new/old time ratio 0.300 (70.0% faster)
95% CI of the median (bootstrap, 10,000 resamples) 0.298–0.303 (69.7–70.2% faster)
Mean ± SD of the ratio 0.300 ± 0.006
P95 ratio (slowest 5% of pairs) 0.306
Number of old/new pairs (n) 27 (3 runs)

The control does not reach the optimized search; its measured difference is within the 5% noise band.
Samples contain 50 calls for the five-state case, 20,000 for zero flux, and 2,000 for invalid endpoints.
The table divides these times by the call count.

To reproduce the tied-grid situation after installing each revision, use Python's standard benchmark command:

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 python3 -m timeit -n 10 -r 5 \
  -s 'import numpy as np; from scipy.sparse import diags, kronsum; from deeptime.markov.msm import MarkovStateModel' \
  -s 'line = diags([np.ones(9), np.ones(9)], [-1, 1]); p = kronsum(line, line).toarray()' \
  -s 'p = .4 * p / p.sum(axis=1)[:, None] + .6 * np.eye(100)' \
  -s 'flux = MarkovStateModel(p, reversible=True).reactive_flux([0], [99])' \
  'flux.pathways(fraction=.5, maxiter=30)'

The measurements above include fresh grid inputs with tied and varied edge weights, so this command reproduces one situation.

Why this is safe and easy to maintain

The change stays within five existing helpers in one module and shares endpoint membership checks.
Paths, capacities, warnings, exceptions, and input matrices remain unchanged in the checked cases.
Existing reactive-flux tests cover path ordering, capacities, major flux, and dense and sparse inputs.

Repository checklist

  • Include one or more new tests for the change; this patch uses existing tests and additional output checks.
  • Add yourself to AUTHORS.

AI assistance

This PR was prepared with AI assistance from Aiden. Candidates were tracked with Weco Observe.

Weco share link: https://dashboard.weco.ai/share/Y_bF9N3rsRnEcG_y3ARNiEBEYjk5YCpk

Reuse reachable nodes while adding flux edges in sorted order.
Reduce endpoint sorting and repeated sparse matrix copies.

Co-Authored-By: Aiden <aiden@weco.ai>
@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.92857% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.93%. Comparing base (59cfe0f) to head (a51d43d).

Files with missing lines Patch % Lines
src/deeptime/markov/tools/flux/pathways.py 83.92% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #351      +/-   ##
==========================================
- Coverage   94.03%   93.93%   -0.11%     
==========================================
  Files         141      141              
  Lines       11199    11237      +38     
==========================================
+ Hits        10531    10555      +24     
- Misses        668      682      +14     
Flag Coverage Δ
decomposition 48.68% <0.00%> (-0.17%) ⬇️
markov 69.16% <83.92%> (-0.03%) ⬇️
misc 65.48% <0.00%> (-0.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@clonker clonker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. I checked it against main on random flux networks and the results are identical, so it seems change is correct. The speedup is limited to small networks though:

  • about 4–5x at 100 states,
  • roughly break-even at 10^4 states, and
  • 15–40% slower on larger ones (200×200 grid, random networks with 2·10^5–5·10^5 edges).

Since it also keeps the old bisection as a second code path, I'd rather keep the simpler implementation unless you have an idea how to get a nice implementation that still looks like an algorithm without regressions and actual speedup.

Otherwise I'd close this one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants