Varying Length Time Series Classification

[christianbaumberger]

Hello All

First of all, thanks for providing this great library! I would like to classify time series which are of varying length. Does the TSDatasets can handle varying legnth of the time series? All the examples in the tutorials have equal length time series. Here is how the dataframe looks like:

within each cell of the colum "timeseries", there is a numpy array of varying length. So far I've used tshresh to extract features on the time series and a random forest as a classifier. But I would like to try the methods implemented with tsai.

Thanks in advance for your help!
Kind regards,
Christian

[vrodriguezf]

One way is to add padding to the end of your sequences to make them all the same size. You can do it calling SlidingWindow with pad_remainder=True, setting the window length to the maximum length of a time series in your dataset, and the stride equals to the window length. Additionally, you can set add_padding_feature=True to add a parallel data channel that marks whether a data point is real or padded.

[oguiza]

Hi @christianbaumberger, @vrodriguezf ,
I’m afraid SlidingWindow won’t work on your dataset because you have an array in each pandas cell.
There’s no specific functionality for varying-length time series in tsai. But Pytorch require tensors to have the same size. It’s not just a tsai requirement. There are multiple ways ti handle this: padding/ truncating, interpolating, etc. There’s a nice (paper)[https://arxiv.org/abs/1910.04341] that addresses these and other methods. Once you select a method, you’ll need to create a function and apply it to each row to get an array of shape n_samples x features (1?) x sequence_length.

[vrodriguezf]

Oh yes, I forgot to say that you need to reshape your dataframe to a tidy format in which there's a column representing the id of the series and other with the id. With that, SlidingWindowPanel should work shouldn't it?

[oguiza]

I don't know. SlidingWindow is meant for long-time series that need to be converted into multiple subsequences of the same length. If I understand correctly in this case there are multiple time series that need to be classified individually. So there's no need for chunking. The only option to use SlidingWindow would be to select the length of the longest time series, and set that as window_length. In this case, you'd get 1 time series per sample padded to the longest time series.

[christianbaumberger]

Dear vrodriquezf and oguiza, thanks a lot for your fast replies!
Of course reshaping the dataframe into the required format isn't a problem. I will try the SlidingWindow approch on monday (unfortunately I don't have time over the weekend...). On the other hand I will definitely read the suggest paper you mentioned! Once I did some experiments, I will let you know how I did it!
The reason I had this format of the dataframe so far, is that with tsfresh this worked perfectly like this!

[vrodriguezf]

@oguiza I was meaning exactly that, using SlidingWindow not to slide, but just to create padding...it's true that it's a bit of a misuse of the function, but it's something that I have done in the past

[Victordmz]

Adding to this question and reading all the previous replies, is there a way to create batches of variable length time series, and pad them in the batch themselves (to the max of the batch length, or to the max of the dataset if the model requires this)? This is because in many tutorials, the batches are standardized with TSStandardize or TSNormalize, but these don't seem to have a way to exclude the padding in their calculations, changing the zero-padding to a meaningful value, which is undesired. A solution would be to first standardize and then pad the batch, but I don't think this is currently possible with the tsai or fastai interface.

[HCWDavid]

I have done something myself related to that... Basically I pad my time series into np.NaN then standardize it (i think in that way the standardization would not take into account of the nan values), then i do nantozero or some other techniques for specific values that i want.