What Lags you use when pre-training the lag-llama?

[SpeeeedLee]

Hello, it's Arthur,
Thank you for your great work.
I would like to ask whether it is possible to get the specific lag indices you use during the pre-training or zero-shot phases.

In the Colab tutorial notebook, it is indicated that the context length is set to 32, and the maximum potential lag index could be 1092. However, the exact indices used to tokenize the 32 historical time points remain unclear to me.
Do you employ all of the 1092 lags, or is there a specific subset that is used?

Thank you!

[SpeeeedLee]

Also, I do not fully understand how the model gets the prediction for multiple future time points autoregressively at inference time.

Say I want to make 3 future predictions, and 100 trajectories, does it work as follows?

for _ in range(100):

1. Get the t-distribution parameters (Parameter_1) for the future first day.
2. Sample one data (Data_1) from t-distribution with Parameter_1.
3. Get the Parameter_2, by including the Data_1
4. Sample one data (Data_3) from t-distribution with Parameter_2.
5. trajectories.append([Data_1, Data_2, Data_3])

Additionally, it would be great if you could give me a hint on where I can find the detail code for autoregressive prediction, thanks!

[sudongwang-upc]

lag-llama/lag_llama/gluon/lightning_module.py

Lines 229 to 261 in 7454088

	for t in range(self.prediction_length):
	if self.time_feat:
	params, loc, scale = self.model(
	*args,
	past_time_feat=repeated_past_time_feat,
	future_time_feat=repeated_future_time_feat[..., : t + 1, :],
	past_target=repeated_past_target,
	past_observed_values=repeated_past_observed_values,
	use_kv_cache=self.use_kv_cache,
	)
	else:
	params, loc, scale = self.model(
	*args,
	past_time_feat=None, # repeated_past_time_feat,
	future_time_feat=None, # repeated_future_time_feat[..., : t + 1, :],
	past_target=repeated_past_target,
	past_observed_values=repeated_past_observed_values,
	use_kv_cache=self.use_kv_cache,
	)
	sliced_params = [
	p[:, -1:] for p in params
	] # Take the last timestep predicted. Each tensor is of shape (#bsz*#parallel_samples, 1)
	distr = self.model.distr_output.distribution(sliced_params, loc, scale)
	sample = distr.sample() # (#bsz*#parallel_samples, 1)
	if self.nonnegative_pred_samples:
	sample = F.relu(sample)
	future_samples.append(sample)
	repeated_past_target = torch.cat((repeated_past_target, sample), dim=1)
	repeated_past_observed_values = torch.cat(
	(repeated_past_observed_values, torch.ones_like(sample)), dim=1
	)

[ashok-arjun]

Hello, it's Arthur, Thank you for your great work. I would like to ask whether it is possible to get the specific lag indices you use during the pre-training or zero-shot phases.

In the Colab tutorial notebook, it is indicated that the context length is set to 32, and the maximum potential lag index could be 1092. However, the exact indices used to tokenize the 32 historical time points remain unclear to me. Do you employ all of the 1092 lags, or is there a specific subset that is used?

Thank you!

Hi @SpeeeedLee .

The 32 historical time series points are consecutive, and sampled before the timestep to be predicted.

The lags however are sampled possibly even beyond this 32-length context, but sparsely as denoted by the lag indices.
The figure below might be useful to clarify the difference.

As for the indices of the lags, in our experiments, we sample lags of certain frequencies, upto a certain length.

The frequencies are denoted here:

lag-llama/lag_llama/gluon/estimator.py

Line 137 in 35f62a9

lags_seq: list = ["Q", "M", "W", "D", "H", "T", "S"],

The corresponding code to sample lags is here. We use the get_lags_for_frequency function of GluonTS in our code:

lag-llama/lag_llama/gluon/estimator.py

Lines 158 to 161 in 35f62a9

	for freq in lags_seq:
	lag_indices.extend(
	get_lags_for_frequency(freq_str=freq, num_default_lags=1)
	)

To give an example, the lags of the "D" frequency (daily) frequency look like this:

[0, 7, 12, 13, 14, 19, 20, 21, 26, 27, 28, 29, 30, 55, 83, 362, 363, 364, 726, 727, 728, 1090, 1091, 1092]

As for the actual lag indices that come from all these frequencies:

[0, 7, 8, 10, 11, 12, 13, 14, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 34, 35, 36, 46, 47, 48, 50, 51, 52, 55, 57, 58, 59, 60, 61, 70, 71, 72, 83, 94, 95, 96, 102, 103, 104, 117, 118, 119, 120, 121, 142, 143, 144, 154, 155, 156, 166, 167, 168, 177, 178, 179, 180, 181, 334, 335, 336, 362, 363, 364, 502, 503, 504, 670, 671, 672, 718, 719, 720, 726, 727, 728, 1090, 1091, 1092]

For the code GluonTS uses to generate these lag indices, you can refer to the source code of the get_lags_for_frequency function.

[ashok-arjun]

Feel free to follow up if you have clarifications or close the issue if it answers your questions. Thanks!

[YuMeng2v]

Hi guys, I actually wanna run pre-trained model and I find if I want to load lags_seq from pre-trained model. For example：
LagLlamaEstimator(
ckpt_path="lag-llama.ckpt",
prediction_length=prediction_length,
context_length=context_length,
nonnegative_pred_samples=True,
aug_prob=0,
lr=5e-4,
lags_seq = estimator_args["lags_seq"],
)
And the code comes to an error and the lags_seq should be like ['M', 'D', ...str] not [0, 7, 8, 10, 11, 12, 13, 14, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 34, 35...] you give.

[arthur-b1]

Hi,

Should I adjust lags_seq based on the frequency of my time series? For daily data, is it correct to use lags_seq = ["Q", "M", "W", "D"] and exclude hour, minute, and second lags since they might not make sense for daily intervals?

Thanks for your help.

[ashok-arjun]

@YuMeng2v The lags sequence cannot be modified for a pretrained model. So if you're loading from the released model, lags_seq parameter shouldn't be passed.

If you are training your own model from scratch (not finetuning), you may set that parameter using the frequencies.

[ashok-arjun]

@arthur-b1 The model uses all lags as it is a generic model independent of the frequency. The lags cannot be modified for a trained model as the first MLP depends on the number of lags.

For your own data, ideally the most useful lag would be that of your frequency, but the other lags shouldn't affect it.

[YuMeng2v]

Thank you! I didn't pass the ckpt path to the estimator, I think it's actually pre-train?

@YuMeng2v The lags sequence cannot be modified for a pretrained model. So if you're loading from the released model, lags_seq parameter shouldn't be passed.

If you are training your own model from scratch (not finetuning), you may set that parameter using the frequencies.

[ashok-arjun]

You are passing the ckpt_path as seen in this code

Hi guys, I actually wanna run pre-trained model and I find if I want to load lags_seq from pre-trained model. For example： LagLlamaEstimator( ckpt_path="lag-llama.ckpt", prediction_length=prediction_length, context_length=context_length, nonnegative_pred_samples=True, aug_prob=0, lr=5e-4, lags_seq = estimator_args["lags_seq"], ) And the code comes to an error and the lags_seq should be like ['M', 'D', ...str] not [0, 7, 8, 10, 11, 12, 13, 14, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 34, 35...] you give.

[ashok-arjun]

Hi, just checking if this issue is resolved.