ConvLSTM architecture for sequence-to-sequence prediction

[Renat2001]

The task: precipitation nowcasting for radar echo measurements. Predict next 16 frames given recent 4 ones.
The data: single measurement - image of shape (252, 252) with intensity values as pixels. Input Sequence - collection of time distributed images with shape (batch_size, 4, 1, 252, 252). Output Sequence has shape (batch_size, 16, 1, 252, 252).
The problem: I need simple architecture applying ConvLSTM2D

[sachinprasadhs]

Hi,

You can refer the below example to predict the next set of frames using ConvLSTM https://keras.io/examples/vision/conv_lstm/.

The above example predicts the next 10 frames.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[Renat2001]

Below code is the model implementation:

train_data = sequence(trainsequences, 16)
val_data = sequence(valsequences, 16)
inputs = keras.Input((4, 1, 252, 252))
x = layers.ConvLSTM2D(64, (3, 3), padding='same', return_sequences=True, data_format='channels_first')(inputs)
x = layers.ConvLSTM2D(64, (3, 3), padding='same', return_sequences=True, data_format='channels_first')(x)
outputs = layers.Conv3D(1, (3, 3, 3), activation='sigmoid', padding='same')(x)
model = keras.Model(inputs, outputs) 
print(model.summary())
callbacks = [keras.callbacks.ModelCheckpoint('lstm.keras', save_best_only=True)]
model.compile(optimizer='rmsprop', loss='mse')
history = model.fit(train_data, epochs=10, validation_data=val_data, callbacks=callbacks, verbose=1)

Here sequence(trainsequences, 16) generates sequences in 16 batches. This is the code of the generator:

# CUSTOM DATA GENERATOR YIELDS BATCH OF SHAPE (BATCH_SIZE, SEQUNCE_LEN, 1, 252, 252)

def sequence(sequences, batchsize):
    while True:
        for batch in np.split(sequences, range(batchsize, len(sequences), batchsize)):
            inbatch = []
            outbatch = []
            for sequence in batch:
                imgs = []
                for timestamp, file in sequence:
                    with h5py.File(file) as dataset:
                        img = np.array(dataset[timestamp]['intensity'])
                        img[img==-1e6] = 0
                        img[img==-2e6] = -1 
                        img = np.where(img<0, img, img/maxint) # APPLY MIN MAX NORMALIZATION (MIN CANCELS OUT SINCE IT'S 0)
                        imgs.append(img) 
                inbatch.append(imgs[:inseqlen])
                outbatch.append(imgs[inseqlen:])
            yield np.expand_dims(inbatch, axis=2), np.expand_dims(outbatch, axis=2)

STDOUT of model summary:

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_2 (InputLayer)        [(None, 4, 1, 252, 252)]  0         
                                                                 
 conv_lstm2d_1 (ConvLSTM2D)  (None, 4, 64, 252, 252)   150016    
                                                                 
 conv_lstm2d_2 (ConvLSTM2D)  (None, 4, 64, 252, 252)   295168    
                                                                 
 conv3d (Conv3D)             (None, 4, 64, 252, 1)     6805      
                                                                 
=================================================================
Total params: 451,989
Trainable params: 451,989
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/10

The error I am getting:

ResourceExhaustedError                    Traceback (most recent call last)
Cell In[15], line 11
      9 callbacks = [keras.callbacks.ModelCheckpoint('lstm.keras', save_best_only=True)]
     10 model.compile(optimizer='rmsprop', loss='mse')
---> 11 history = model.fit(train_data, epochs=10, validation_data=val_data, callbacks=callbacks, verbose=1)

File /opt/conda/lib/python3.10/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File /opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:52, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50 try:
     51   ctx.ensure_initialized()
---> 52   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                       inputs, attrs, num_outputs)
     54 except core._NotOkStatusException as e:
     55   if name is not None:

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No