cuda.OutOfMemoryError for reasonable size model

[xhOwenMa]

I am getting the outofmemory error when using CROWN method, specifically: I trained two models - a generator and a controller - separately and each on itself can be verified using CROWN with no problem. However, when concatenated together, I keep getting the out of memory error.

To reproduce the error: gen_control.zip

Usage:

• model is my concatenated model, its definition is in the zip file

lirpa_model = BoundedModule(model, torch.zeros(1,6), bound_opts={"conv_mode": "patches"})
ptb = PerturbationLpNorm(norm=float("inf"), x_L=X_min, x_U=X_max)
bounded_X = BoundedTensor(X, ptb)

with torch.no_grad():
    prediction = lirpa_model(bounded_X)
    print(f'Prediction: {prediction}')

    lb, ub = lirpa_model.compute_bounds(x=(bounded_X,), method='CROWN')
    print(f'bounds:')
    print(lb)
    print(ub)

Some error outputs that may also be helpful:

• on gpu:

UserWarning: Creating an identity matrix with size 1024x1024 for node BoundConvTranspose(name=/83, inputs=[/82, /27], perturbed=True). This may indicate poor performance for bound computation. If you see this message on a small network please submit a bug report.
  sparse_C = self.get_sparse_C(node, ref_intermediate)
Traceback (most recent call last):
  File ".\veri\veri.py", line 111, in <module>
    lb, ub = lirpa_model.compute_bounds(x=(bounded_X,), method='CROWN')
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 1326, in compute_bounds
    update_mask=update_mask)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 1414, in _compute_bounds_main
    self.check_prior_bounds(final)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 879, in check_prior_bounds
    self.check_prior_bounds(n)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 879, in check_prior_bounds
    self.check_prior_bounds(n)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 879, in check_prior_bounds
    self.check_prior_bounds(n)
  [Previous line repeated 24 more times]
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 898, in check_prior_bounds
    node.inputs[i], prior_checked=True)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 983, in compute_intermediate_bounds
    apply_output_constraints_to=apply_output_constraints_to)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\backward_bound.py", line 340, in backward_general
    start_shape=start_shape)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\operators\reshape.py", line 78, in bound_backward
    return [(_bound_oneside(last_lA), _bound_oneside(last_uA)), (None, None)], 0, 0
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\operators\reshape.py", line 71, in _bound_oneside
    unstable_idx=A.unstable_idx)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\patches.py", line 383, in patches_to_matrix
    A_matrix = torch.zeros(batch_size, unstable_size, input_channel, (input_x + padding[2] + padding[3]) * (input_y + padding[0] + padding[1]), device=pieces.device, dtype=pieces.dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 76.51 GiB ...

• on cpu (it would be even better if there are advices to why this is happening, thank you very much):

UserWarning: Creating an identity matrix with size 1024x1024 for node BoundConvTranspose(name=/83, inputs=[/82, /27], perturbed=True). This may indicate poor performance for bound computation. If you see this message on a small network please submit a bug report.
  sparse_C = self.get_sparse_C(node, ref_intermediate)
Traceback (most recent call last):
  File ".\veri\veri.py", line 111, in <module>
    lb, ub = lirpa_model.compute_bounds(x=(bounded_X,), method='CROWN')
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 1326, in compute_bounds
    update_mask=update_mask)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 1414, in _compute_bounds_main
    self.check_prior_bounds(final)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 879, in check_prior_bounds
    self.check_prior_bounds(n)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 879, in check_prior_bounds
    self.check_prior_bounds(n)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 879, in check_prior_bounds
    self.check_prior_bounds(n)
  [Previous line repeated 19 more times]
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 898, in check_prior_bounds
    node.inputs[i], prior_checked=True)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\bound_general.py", line 983, in compute_intermediate_bounds
    apply_output_constraints_to=apply_output_constraints_to)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\backward_bound.py", line 340, in backward_general
    start_shape=start_shape)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\operators\relu.py", line 306, in bound_backward
    upper_d = maybe_unfold_patches(upper_d, last_lA if last_lA is not None else last_uA)
  File "C:\Users\owenm\miniconda3\envs\carla\lib\site-packages\auto_LiRPA\patches.py", line 513, in maybe_unfold_patches
    d_unfolded_r = d_unfolded_r[last_A.unstable_idx[1], last_A.unstable_idx[2]]
IndexError: index 15 is out of bounds for dimension 1 with size 15