Unfreezing the model and training all layers?

[mbrz97]

Greetings!

Thank you for the amazing work. I've been trying to apply this model to my own image captioning dataset, however, when trying to unfreeze the "BERT low", "BERT high", and embedding layers, I face the error below. I did also try enabling retain_graph=true, but it just throws a different error. I would appreciate it if you could provide the steps needed for training all components from scratch.

Thank you!

File "/mnt/2T/Barzegar/LaDiC/main.py", line 232, in
l, x_t_loss, x_1_loss, prob_loss, valid_token_loss, pad_loss = train_func(model, optimizer, x, scheduler)
File "/mnt/2T/Barzegar/LaDiC/main.py", line 163, in train_func
accelerator.backward(l) #, retain_graph=True
File "/home/habib/.conda/envs/ladic/lib/python3.8/site-packages/accelerate/accelerator.py", line 2246, in backward
loss.backward(**kwargs)
File "/home/habib/.conda/envs/ladic/lib/python3.8/site-packages/torch/_tensor.py", line 521, in backward
torch.autograd.backward(
File "/home/habib/.conda/envs/ladic/lib/python3.8/site-packages/torch/autograd/init.py", line 289, in backward
_engine_run_backward(
File "/home/habib/.conda/envs/ladic/lib/python3.8/site-packages/torch/autograd/graph.py", line 769, in _engine_run_backward
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.