Are there any samples of stumpi

[jh72ie]

Hi folks,
Are there any samples of stumpi where the mp parameter is being used?

mp : numpy.ndarray, default None
A pre-computed matrix profile (and corresponding matrix profile indices).
This is a 2D array of shape (len(T) - m + 1, 2 * k + 2), where the first
k columns are top-k matrix profile, and the next k columns are their
corresponding indices. The last two columns correspond to the top-1 left and
top-1 right matrix profile indices. When None (default), this array is
computed internally using stumpy.stump.

I am saving the mp so that I can pick up where i left off when i have new data. Inexplicable, it has worked for me many times but now it just doesn't at all and i cannot figure out why.

If someone has a save and load example appreciation would be much!!

tnx

[NimaSarajpoor]

@jh72ie

but now it just doesn't at all and i cannot figure out why.

Hard to tell without seeing any code and the issue/error. Do you mean you are getting an error? I tried the following example in Google Colab and it works:

# install stumpy

# imports
import numpy as np
import stumpy

# inputs
T = np.random.rand(10)
m = 3
new_value = np.random.rand()

# pre-compute mp and save it
mp = stumpy.stump(T, m)
np.save('stumpy_mp.npy', mp)

# load mp 
mp = np.load('stumpy_mp.npy', allow_pickle=True)

# stumpi
stumpi_obj = stumpy.stumpi(T, m, mp=mp)
stumpi_obj.update(new_value)

[jh72ie]

Thanks so much for answering. I’m not at my computer just now to show anything.

It has always worked after an initial stump.

It fails after trying to store the result of stumpi & later resurrecting from that.

It’s about how to reconstruct what stumpi’s mp= parameter wants or what actually gets saved.

None of the gpts have been able to solve it for me 😔

[jh72ie]

SPOILER ALERT: I think I have figured it out.

if i try to save the updated stumpi_obj in your sample (in readiness for the next time it's needed - a reboot, etc.):
np.save('stumpy_mp.npy', stumpi_obj)
I'll get this strange error:
PicklingError: Can't pickle <class 'stumpy.stumpi.stumpi'>: it's not the same object as stumpy.stumpi.stumpi
I used other ways to save but still nothing helped the situation. So, instead I save the bits separately using reconstruction: np.column_stack..

Anyway, here is an example of code (without the persist but with the _stack) that works for me if i run it multiple times in kaggle. But by maybe run 5 time, it eventually fails - just pressing run with no other changes:

import numpy as np
import stumpy

m = 3
T0 = np.random.rand(10)
updates = [0.5, 0.6, 0.7]
updates_1 = [0.4, 0.6, 0.4]
updates_2 = [0.7, 0.4, 0.6]

# original stump run before entering real-time
mp = stumpy.stump(T0, m)

# initializing real-time (maybe some time later)
stumpi_obj = stumpy.stumpi(T0, m, mp=mp)

# Incremental updates
for val in updates:
    stumpi_obj.update(val)

# Save fully updated state
T_saved = stumpi_obj.T_.copy()  # full T after updates 
P_ = stumpi_obj.P_ 
I_ = stumpi_obj.I_ 
left_I_ = getattr(stumpi_obj, "left_I_", np.full_like(P_, np.nan)) 
right_I_ = getattr(stumpi_obj, "right_I_", np.full_like(P_, np.nan)) 
mp_saved = np.column_stack([P_, I_, left_I_, right_I_])

# Reload / rehydrate
stumpi_rehydrated = stumpy.stumpi(T_saved, m, mp=mp_saved)

# Continue updating safely
for val in updates_1:
    stumpi_rehydrated.update(val)
print("Rehydrated P_:", stumpi_rehydrated.P_[:5])

# Save fully updated state
T_saved = stumpi_rehydrated.T_.copy()  # full T after updates 
P_ = stumpi_rehydrated.P_ 
I_ = stumpi_rehydrated.I_ 
left_I_ = getattr(stumpi_rehydrated, "left_I_", np.full_like(P_, np.nan)) 
right_I_ = getattr(stumpi_rehydrated, "right_I_", np.full_like(P_, np.nan)) 
mp_saved = np.column_stack([P_, I_, left_I_, right_I_])

# Reload / rehydrate
stumpi_rehydrated_1 = stumpy.stumpi(T_saved, m, mp=mp_saved) 
# Continue updating safely 
for val in updates_2:
    stumpi_rehydrated_1.update(val)
print("Rehydrated P_:", stumpi_rehydrated_1.P_[:5])

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_37/4056997510.py in <cell line: 0>()
     44 
     45 # Reload / rehydrate
---> 46 stumpi_rehydrated_1 = stumpy.stumpi(T_saved, m, mp=mp_saved)
     47 # Continue updating safely
     48 for val in updates_2:

/usr/local/lib/python3.11/dist-packages/stumpy/core.py in inner_wrapper(*args, **kwargs)
    136             is_same_params = _compare_parameters(norm, non_norm, exclude=exclude)
    137             if not is_same_params or kwargs.get("normalize", True):
--> 138                 return norm(*args, **kwargs)
    139             else:
    140                 kwargs = {k: v for k, v in kwargs.items() if k != "normalize"}

/usr/local/lib/python3.11/dist-packages/stumpy/stumpi.py in __init__(self, T, m, egress, normalize, p, k, mp, T_subseq_isconstant_func)
    245         for i in np.flatnonzero(self._left_I >= 0 & ~mask):
    246             j = self._left_I[i]
--> 247             QT = np.dot(self._T[i : i + self._m], self._T[j : j + self._m])
    248             D_square = core._calculate_squared_distance(
    249                 self._m,

ValueError: shapes (3,) and (2,) not aligned: 3 (dim 0) != 2 (dim 0)

But, the good news (probably jinx myself now) is that I think i have gotten to the bottom of it. In hindsight maybe it all looks obvious but, damn, this was difficult for me because:

1. it worked sometimes!
2. the different array types being produced by stump and stumpi provided me with red herrings
3. the funkiness of whatever np.save and save compressed were doing - more distraction
4. my own general lack of capabilities

But here it is:
stumpy.stumpi(T_saved, m, mp=mp_saved, **egress=False**)

With this setting I seem to be able to consistently recover a situation. Of course, that brings a bunch of other issues about the usefulness of doing this.

Again, I utterly appreciate you pitching in to help me out.

[NimaSarajpoor]

@jh72ie

Again, I utterly appreciate you pitching in to help me out.

Happy to help! And getting feedback from different users can help us as well to better discover the areas that require improvements.

if i try to save the updated stumpi_obj in your sample...

Thanks for sharing more information. So, you basically want to store the object or the updated mp (with egress==True) and re-use it later.

TL;DR
Try cloudpickle for storing the stumpi object. See this comment by @seanlaw for more info.

To understand what's going on in your case, read on:

But by maybe run 5 time, it eventually fails - just pressing run with no other changes:

When egress=True, the values stored in the updated mp do NOT JUST correspond to T_. and they may point to previously-observed-but-now-removed data point. Further more, an index stored in left_I_ is based on FULL data. Suppose j = left_I_[i]. This means that for the subsequence T_[i: i+m], its left nearest neighbour (within ${len(T)}$ range) is T_FULL[j:j+m]. However, stumpi does not track T_FULL in egress==True mode. When stumpi is instantiated, it needs to compute the updated left matrix profile but cannot do that with just using updated left_I_ and T_ because, as mentioned earlier, some of the values in the updated left_I_ can correspond to the historical-but-no-longer-available data.

---

As a side note:
I think a better way to think about egress==True is this: For each data point, only those neighbours whose index-wise distance are $\le{n}$ are considered. Here, n is the length of array, which remains constant throughout the update process, and we only keep the matrix profile information for the last n data points. The following three matrix profile values are NOT necessarily the same:

# inputs
# array: T
# window size: m
# new value: t

# case1
mp = stumpy.stump(np.r_[T[1:], t], m)
P1 = mp[:, 0].copy()

# case2
mp = stumpy.stump(np.r_[T, t], m)
P2 = mp[1:, 0].copy()

# case3
stumpi_obj = stump.stumpi(T, m)
stumpi_obj.update(t)
P3 = stumpi_obj.P_.copy()