problems while using stumpy.match if normalize=False

[SaVoAMP]

Hey, I don't understand what I am doing wrong but the stumpy.match function doesn't seem to work for my non-normalized input.

I have got the following function call:

query_matches = stumpy.match(
            Q=dimensions_arr[subspace, 0: 0 + m],
            T=dimensions_arr[subspace],
            M_T=sliding_mean[subspace],
            Σ_T=sliding_stddev[subspace],
            max_distance=np.inf,
            max_matches=num_rep,
            query_idx=0,
            normalize=normalize
        )

Everything is working if normalize=True but if I set this parameter to False, I get:

What I am trying to do is to compute the matrix profile, specify a query index and afterwards run the stumpy.match function to find all matches of the predefined subsequence.
Concretely I am doing the following (the normalize parameter can be set in an interactive jupyter notebook):

1. Compute the sliding mean and standard deviation of the time series:
   _, sliding_mean, sliding_stddev = core.preprocess(dimensions, m)
2. Compute the multidimensional matrix profile:
   mps, indices = stumpy.mstump(dimensions, m, normalize=normalize)
3. Specify the query subsequence (simply starting at the first index of the whole time series) and find its nearest neighbor in each dimension:

n, d = dimensions.shape
query_idx = np.full((1, d), 0)[0]
nn_idx = indices[np.arange(len(query_idx)), query_idx]

4. Compute the MDL results and the subspace for each dimension:
   mdls, subspaces = stumpy.mdl(dimensions, m, query_idx, nn_idx, normalize=normalize)
5. Get the number of relevant dimensions as well as the associated subspace:

reset_k = False
if k is None:
          k = np.argmin(mdls)
          reset_k = True
      subspace = subspaces[k]
      subspace_names = [dimensions.columns.values[s] for s in subspace]
      dimensions_arr = core._preprocess(dimensions)

6. Find the multidimensional matches of the query subsequence using the function call above

Now, if I run my notebook specifying that normalize=True, everything works fine. But when I try to set normalize=False, I get the exception shown above. What am I missing?
I was wondering if this might be a bug, but everything is working using the mmotifs function in which the same function call is used and nothing else is done with the normalize parameter. If anything it could be an error related to the input parameters that might cause an exception in the case that normalize=False. But it is probably more likely that I have made a mistake during the steps 1 to 5.

[NimaSarajpoor]

@SaVoAMP
This might help: #530

[SaVoAMP]

Are you able to save the input time series (and query) as a CSV file and drag-and-drop it into a new comment reply? This way, we can play around with it to see if we get the same thing

Thank you, here are some csv files that might be relevant:
dimensions_arr.csv
M_T.csv
Q.csv
T.csv
Σ_T.csv

Since I faced a similar issue before, I think the problem is:

Yes, thank you! I also think that this is the problem, but I'm not sure how to solve it. I was taking a look at #530 and it really seems to be a similar problem.

[seanlaw]

@SaVoAMP I haven't had a chance to dig into this too deep or use your data (thanks for sharing it) but can you try something like:

# Follow steps 1-6 as you've described above and find the subspace and query subsequence. Then

if normalize:
    # normalize == True
    _, sliding_mean, sliding_stddev = core.preprocess(dimensions, m)
    query_matches = stumpy.match(
            Q=dimensions_arr[subspace, 0: 0 + m],
            T=dimensions_arr[subspace],
            M_T=sliding_mean[subspace],
            Σ_T=sliding_stddev[subspace],
            max_distance=np.inf,
            max_matches=num_rep,
            query_idx=0,
            normalize=normalize
        )
else:
    # normalize == False
    _, T_subseq_isfinite = core.preprocess_non_normalized(dimensions, m)
    query_matches = stumpy.match(
            Q=dimensions_arr[subspace, 0: 0 + m],
            T=dimensions_arr[subspace],
            M_T=T_subseq_isfinite[subspace],  # This is likely what you need
            Σ_T=None,
            max_distance=np.inf,
            max_matches=num_rep,
            query_idx=0,
            normalize=normalize
        )

Would you mind giving this a try and seeing if it works? As you can imagine, when normalize == True, then sliding_mean and sliding_stddev are relevant. However, when normalize == False, sliding_mean and sliding_stddev are actually irrelevant for computing distances and, instead, we need to use T_subseq_isfinite. Note that M_T and Σ_T are optional parameters so, in theory (I hope???), you should've been able to do:

normalize = False

query_matches = stumpy.match(
            Q=dimensions_arr[subspace, 0: 0 + m],
            T=dimensions_arr[subspace]
            max_distance=np.inf,
            max_matches=num_rep,
            query_idx=0,
            normalize=normalize
        )

Of course, this would've resulted in some waste as T_subseq_isfinite is being recomputed every time you pass Q and T in:

stumpy/stumpy/aamp_motifs.py

Lines 384 to 385 in c01c04c

	if T_subseq_isfinite is None:
	T, T_subseq_isfinite = core.preprocess_non_normalized(T, m)

Admittedly, setting M_T = T_subseq_isfinite when normalize=False is completely non-obvious (though this is more of a hack and I only expected a handful of users to ever do this). Considering that both @SaVoAMP and @NimaSarajpoor have encountered this, I think it would make sense add some code (not sure where) to:

1. Detect that normalize=False
2. Detect that M_T is not None
3. Check that the M_T array (which is replaced with T_subseq_isfinite) is a bool array (since normalize=False) and, if not, raise an exception that explains that when normalze=False then M_T should be T_subseq_isfinite and then give an example:

_, T_subseq_isfinite = core.preprocess_non_normalized(T, m)

I haven't thought this through but I think we might be able to add a new decorator to handle this check

[seanlaw]

Additionally, I think that when normalize=False, then this also works (and should be preferred!!):

    # normalize == False
    _, T_subseq_isfinite = core.preprocess_non_normalized(dimensions, m)
    query_matches = stumpy.match(
            Q=dimensions_arr[subspace, 0: 0 + m],
            T=dimensions_arr[subspace],
            T_subseq_isfinite=T_subseq_isfinite[subspace],  # This is likely what you need
            max_distance=np.inf,
            max_matches=num_rep,
            query_idx=0,
            normalize=normalize
        )

Notice that normalize == False and stumpy.match is "smart" enough (thanks to the decorator) to recognize this and accept T_subseq_isfinite as an input parameter in place of M_T! So, when normalize == False, setting M_T = T_subseq_isfinite is identical to setting T_subseq_isfinite = T_subseq_isfinite. The former is just convenience but it's better to do the latter to avoid confusion. In other words, the function "knows" how to handle it when normalize == False and you provide T_subseq_isfinite. However, the function would fail if normalize == True and T_subseq_isfinite is provided!

[seanlaw]

New issue #637 to track this

[SaVoAMP]

Admittedly, setting M_T = T_subseq_isfinite when normalize=False is completely non-obvious (though this is more of a hack and I only expected a handful of users to ever do this). Considering that both @SaVoAMP and @NimaSarajpoor have encountered this, I think it would make sense add some code (not sure where) to:

Detect that normalize=False
Detect that M_T is not None
Check that the M_T array (which is replaced with T_subseq_isfinite) is a bool array (since normalize=False) and, if not, raise an exception that explains that when normalze=False then M_T should be T_subseq_isfinite and then give an example:

_, T_subseq_isfinite = core.preprocess_non_normalized(T, m)

I haven't thought this through but I think we might be able to add a new decorator to handle this check

I think that this sounds reasonable. The mmotifs function, for example, is clearly easier to use for the user since it is doing everything automatically and the user only has to specify whether normalize is True or False (but that is of course also easier here since there is no need to set or compute M_T and Σ_T).

[SaVoAMP]

Would you mind giving this a try and seeing if it works?

Yes, it is working! Thank you very much! :)

[seanlaw]

In case you missed it, this is actually preferred:

    _, T_subseq_isfinite = core.preprocess_non_normalized(dimensions, m)
    query_matches = stumpy.match(
            Q=dimensions_arr[subspace, 0: 0 + m],
            T=dimensions_arr[subspace],
            T_subseq_isfinite=T_subseq_isfinite[subspace],  # This is likely what you need
            max_distance=np.inf,
            max_matches=num_rep,
            query_idx=0,
            normalize=normalize
        )

Notice that we don't use M_T at all and so there is no weird "replace" happening behind the scenes.