Issue with frequency since pandas 2.2.0 in the DeepAREstimator

[vm3538]

Description

After updating pandas from 2.1.4 to 2.2.0, I get exceptions because of the frequency.

To Reproduce

Train a DeepAREstimator

import pandas as pd
import torch

from gluonts.dataset.pandas import PandasDataset
from gluonts.dataset.split import split
from gluonts.torch import DeepAREstimator


df = pd.read_csv(
    'AirPassengers.csv',
    index_col=0,
    parse_dates=True,
)
dataset = PandasDataset(df, target="#Passengers")

training_data, _ = split(dataset, offset=-36)

model = DeepAREstimator(
    prediction_length=12, freq="M", trainer_kwargs={"max_epochs": 5}
).train(training_data)

Error message or code output

//trainer/minimal_code.py:1: DeprecationWarning:
Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
but was not found to be installed on your system.
If this would cause problems for you,
please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466

  import pandas as pd
/usr/local/lib/python3.10/site-packages/gluonts/time_feature/_base.py:243: FutureWarning: 'M' is deprecated and will be removed in a future version, please use 'ME' instead.
  offset = to_offset(freq_str)
/usr/local/lib/python3.10/site-packages/gluonts/time_feature/lag.py:104: FutureWarning: 'M' is deprecated and will be removed in a future version, please use 'ME' instead.
  offset = to_offset(freq_str)
Traceback (most recent call last):
  File "//trainer/minimal_code.py", line 20, in <module>
    ).train(training_data)
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/estimator.py", line 246, in train
    return self.train_model(
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/estimator.py", line 165, in train_model
    training_network = self.create_lightning_module()
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/deepar/estimator.py", line 382, in create_lightning_module
    return DeepARLightningModule(
  File "/usr/local/lib/python3.10/site-packages/gluonts/core/component.py", line 364, in init_wrapper
    return init(self, **all_args)
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/deepar/lightning_module.py", line 59, in __init__
    self.model = DeepARModel(**model_kwargs)
  File "/usr/local/lib/python3.10/site-packages/gluonts/core/component.py", line 364, in init_wrapper
    return init(self, **all_args)
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/deepar/module.py", line 140, in __init__
    self.lags_seq = lags_seq or get_lags_for_frequency(freq_str=freq)
  File "/usr/local/lib/python3.10/site-packages/gluonts/time_feature/lag.py", line 149, in get_lags_for_frequency
    raise Exception("invalid frequency")
Exception: invalid frequency

Changing frequency to ME won't solve the issue, since it is not supported in gluonTS.

//trainer/minimal_code.py:1: DeprecationWarning:
Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
but was not found to be installed on your system.
If this would cause problems for you,
please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466

  import pandas as pd
Traceback (most recent call last):
  File "//trainer/minimal_code.py", line 20, in <module>
    ).train(training_data)
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/estimator.py", line 246, in train
    return self.train_model(
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/estimator.py", line 165, in train_model
    training_network = self.create_lightning_module()
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/deepar/estimator.py", line 382, in create_lightning_module
    return DeepARLightningModule(
  File "/usr/local/lib/python3.10/site-packages/gluonts/core/component.py", line 364, in init_wrapper
    return init(self, **all_args)
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/deepar/lightning_module.py", line 59, in __init__
    self.model = DeepARModel(**model_kwargs)
  File "/usr/local/lib/python3.10/site-packages/gluonts/core/component.py", line 364, in init_wrapper
    return init(self, **all_args)
  File "/usr/local/lib/python3.10/site-packages/gluonts/torch/model/deepar/module.py", line 140, in __init__
    self.lags_seq = lags_seq or get_lags_for_frequency(freq_str=freq)
  File "/usr/local/lib/python3.10/site-packages/gluonts/time_feature/lag.py", line 149, in get_lags_for_frequency
    raise Exception("invalid frequency")
Exception: invalid frequency

Environment

• Operating system: debian
• Python version: 3.10
• GluonTS version: 0.14.3
• MXNet version: -
• Pandas version: 2.2.0

[lostella]

@vm3538 thanks for filing this! After a quick look, I think we may need to fix this utility function that is used internally to normalize frequency strings. The case ending in E probably needs to be added, and we should check that tests cover this.

[pantanurag555]

Aliases deprecated in pandas v2.2: https://pandas.pydata.org/docs/dev/whatsnew/v2.2.0.html
Will need to add a version check in the utility function before normalizing.

[lostella]

@vm3538 I think this was fixed in #3117 (dev) and #3121 (0.14.x) can you confirm?

[vm3538]

@lostella I can confirm that with gluonTS 0.14.4 we cannot use pandas 2.2.0, so yes the issue is fixed for now.