多卡训练失败

[youhha]

我使用4卡4090上训练maptr，nuscense数据集，出现以下报错，请问是什么原因呀
Traceback (most recent call last):
File "./tools/train.py", line 260, in
main()
File "./tools/train.py", line 249, in main
custom_train_model(
File "/workspace/code/projects/mmdet3d_plugin/bevformer/apis/train.py", line 27, in custom_train_model
custom_train_detector(
File "/workspace/code/projects/mmdet3d_plugin/bevformer/apis/mmdet_train.py", line 75, in custom_train_detector
model = MMDistributedDataParallel(
File "/usr/local/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 496, in init
dist._verify_model_across_ranks(self.process_group, parameters)
RuntimeError: replicas[0][0] in this process with sizes [80, 128] appears not to match sizes of the same param in process 0.