Adding parameters to remove unnecessary changepoints

[efemeryds]

Hi! I have been using ruptures for a couple of weeks and I found useful for my case study adding two very simple parameters. I wanted to discuss if you find them meaningful in the official library as well.

I noticed that sometimes the shift that was detected took too long and made no sense in my research so I added parameter max_samples to decide on maximum number of samples between start and end of a shift, now it is naive, just deleting from final bkps samples where period exceeds the one set in int variable. Maybe it could be implemented naively and later developed, what is your opinion on that? Did you have such issues?

The second parameter concerns whether to include change point if it starts but never ends, I also had such scenarios.

Locally I added both of these solutions to _seg in window detection model.
I am quite new to the contribution itself so please clarify anything that I could misunderstand.
Cheers

[deepcharles]

Hello,

Adding a parameter such as max_samples could very well be useful for people. For Dynp and Pelt it can even provide significant speed-ups.

I will look if that can be adding easily.

For other methods (window-based, bottom-up and binseg) this might be more complicated, because that would need some arbitrary rules (maybe not but I need to check) that would be difficult to explain.

The second parameter concerns whether to include change point if it starts but never ends, I also had such scenarios.

I am not sure I understand this one. Do you mean a regime that starts and never ends? To me, that would mean that there is no more change to add.

[deepcharles]

You can also share the relevant parts of your code here. Your solution might do the trick. If that's the case, I will explain how you can integrate it to the package.

[efemeryds]

Hi there, I discovered that it is possible to address my use case in different way than what I described above. I would like to propose an idea of helper function to filter final results. I am still interested in developing the previous idea so let me know if you prefer to stick to that. But now shortly about different approach.

In my scenario I need to differentiate between shift and individual outliers. However the results from ruptures for my data after grouping them into pairs often indicated long shift where it was an individual outlier only or a really short change. What I mean by that is that in same cases two separate change points where combined together to create improbable shift that lasted even a few months.

At the beginning I thought that the solution for that is to constrain the length of shift but I think something different may be implemented. After receiving all the indexes I still group them together but then I use additional function to check if the points are individual outliers or longer shifts. Now the solution is very basic I just compare the median of sample data before the start of a proclaimed shift, in the duration and after the end of the proclaimed shift. If median between start and end is very different from what is before start and after end, then I assume it is a shift and label all period as anomalous. And if the median is not very outstanding then I assume these are two outliers and between them there is no change. Something more sophisticated may be added later on.

I wanted to ask, what do you think of this solution to filter/re-check the results afterwards? It could be additional helper function if someone has similar scenario to mine. If it does not fit the main idea of ruptures then please let me know that as well.
Cheers

[deepcharles]

This is an interesting use case, but maybe a bit too specific to be integrated in the core of ruptures.

[efemeryds]

I see, thank you for the response :)