ValueError: Trying to set a tensor of shape torch.Size([3072, 128]) in "weight" (which has shape torch.Size([3072, 16, 2, 2])), this looks incorrect.

[DennisFaucher]

System Info / 系統信息

$ python -V
Python 3.12.3

$ uname -a
Linux nvidia-k8s 5.15.0-125-generic #135~20.04.1-Ubuntu SMP Mon Oct 7 13:56:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"

$ nvidia-smi
NVIDIA-SMI 535.216.01 Driver Version: 535.216.01 CUDA Version: 12.2

CogVideo$ python inference/cli_demo.py --prompt "A girl riding a bike." --model_path THUDM/CogVideoX1.5-5b --generate_type "t2v"

Traceback (most recent call last):
File "/home/dennis/Documents/Python/CogVideo/inference/cli_demo.py", line 179, in
generate_video(
File "/home/dennis/Documents/Python/CogVideo/inference/cli_demo.py", line 84, in generate_video
pipe = CogVideoXPipeline.from_pretrained(model_path, torch_dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dennis/.pyenv/versions/pyenv_312/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/dennis/.pyenv/versions/pyenv_312/lib/python3.12/site-packages/diffusers/pipelines/pipeline_utils.py", line 896, in from_pretrained
loaded_sub_model = load_sub_model(
^^^^^^^^^^^^^^^
File "/home/dennis/.pyenv/versions/pyenv_312/lib/python3.12/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 704, in load_sub_model
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dennis/.pyenv/versions/pyenv_312/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/dennis/.pyenv/versions/pyenv_312/lib/python3.12/site-packages/diffusers/models/modeling_utils.py", line 886, in from_pretrained
accelerate.load_checkpoint_and_dispatch(
File "/home/dennis/.pyenv/versions/pyenv_312/lib/python3.12/site-packages/accelerate/big_modeling.py", line 613, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "/home/dennis/.pyenv/versions/pyenv_312/lib/python3.12/site-packages/accelerate/utils/modeling.py", line 1780, in load_checkpoint_in_model
set_module_tensor_to_device(
File "/home/dennis/.pyenv/versions/pyenv_312/lib/python3.12/site-packages/accelerate/utils/modeling.py", line 286, in set_module_tensor_to_device
raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([3072, 128]) in "weight" (which has shape torch.Size([3072, 16, 2, 2])), this looks incorrect.

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

$ python inference/cli_demo.py --prompt "A girl riding a bike." --model_path THUDM/CogVideoX1.5-5b --generate_type "t2v"

Expected behavior / 期待表现

Successful completion

[randaller]

Same here under Windows, diffusers built from RzRz cogvideox branch.

[nitinmukesh]

@DennisFaucher

Compare pip list with this.
#509 (comment)

I had the same issue but after installing it today it got resolved (Windows 11)

[DennisFaucher]

Thank you

[zRzRzRzRzRzRzR]

Please use the latest diffusers main branch to install, I believe this issue can be resolved.

[agneet42]

hi @zRzRzRzRzRzRzR , installing diffusers from main seems to not resolve this. here is my system info, am I missing anything?

- 🤗 Diffusers version: 0.32.0.dev0
- Python version: 3.11.9
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Transformers version: 4.46.3
- Accelerate version: 1.1.1

I think its as-is with the instructions here : https://huggingface.co/THUDM/CogVideoX1.5-5B
I am using the sample script as mentioned in the link.

[zRzRzRzRzRzRzR]

Which commit is your main branch?

[joeljang]

I am facing a similar problem!

size mismatch for model.diffusion_model.mixins.patch_embed.proj.weight: copying a param with shape torch.Size([3072, 32, 2, 2]) from checkpoint, the shape in current model is torch.Size([3072, 256]).
	size mismatch for model.diffusion_model.mixins.final_layer.linear.weight: copying a param with shape torch.Size([64, 3072]) from checkpoint, the shape in current model is torch.Size([128, 3072]).
	size mismatch for model.diffusion_model.mixins.final_layer.linear.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).

[Droliven]

I am facing a similar problem!

size mismatch for model.diffusion_model.mixins.patch_embed.proj.weight: copying a param with shape torch.Size([3072, 32, 2, 2]) from checkpoint, the shape in current model is torch.Size([3072, 256]).
	size mismatch for model.diffusion_model.mixins.final_layer.linear.weight: copying a param with shape torch.Size([64, 3072]) from checkpoint, the shape in current model is torch.Size([128, 3072]).
	size mismatch for model.diffusion_model.mixins.final_layer.linear.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).

same problem, have you soved it?

[Droliven]

solved: #569