When I've been reading articles about change points there's a lot of discussions about performance. But I don't see a lot of benchmarks here in ruptures.

I suggest to use airspeed velocity (used by numpy/pandas/xarray etc) and make some simple tests that loops through each of the algorithms.

asv can be difficult to setup locally (at least for me) so I suggest to add it to the CI as well to avoid complexity for new (and old) contributors:
xarray has a nice workflow that you can use (PR: pydata/xarray#5796) and I suggest to try to follow the same folder structure as xarray does.

Interesting reading:
https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/