Inquiry on ML Input Data Pipeline

[kwon-encored]

@Sukh-P
Hi, I hope you’re doing well!

I had a question regarding the input datapipes in your PVNet repository. Do you apply normalization to all the input variables? I was reviewing the code but didn’t see any reference to normalization.

1. If normalization is applied, I’m curious about your approach, especially considering the multimodal nature of the data (e.g., NWP variables, satellite channels). Do you calculate the mean and normalize for each region and timeframe, or do you handle normalization individually for each variable or channel?

2. I heard that you are using LARGE one-chunk of zarr file that contains the entire region and the timestep. When normalized, are you applying that to entire "global" sample?

[Sukh-P]

Hi @kwon-encored, hope you are doing well too!

On your questions:

1. Yes we apply normalisation to each input variables that requires it, for NWP variables you can see how we apply normalisation here, we compute normalisation constants (here is gist of how this could be done) for each variable from the dataset using 200 samples
2. The normalisation is applied after the Datasets (we use xarray Datasets and xarray in general is what we use to open the zarr files) have already been sliced in time and space so the normalisation is only applied to data we actually plan on using in training.

Hope this helps! Thanks