New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Assertion Error: world_size % group_num == 0 When finetune cogv1.5-i2v on distributed environment #506

Open

1 of 2 tasks

hrcheng98 opened this issue Nov 15, 2024 · 1 comment

Assignees

hrcheng98 commented Nov 15, 2024

System Info / 系統信息

Information / 问题信息

The official example scripts / 官方的示例脚本
My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

run code on 2*8 GPU
find assertion error world_size % group_num != 0

Expected behavior / 期待表现

I found group_num in loss_fn_config in yaml is 40.
Should the group_num be modified according to the machine amount I am using?

zRzRzRzRzRzRzR self-assigned this

Member

zRzRzRzRzRzRzR commented Nov 20, 2024 •

edited

Loading

we have not conducted experiments on multi-machine multi-card inference, all are single-machine. I'm sorry I couldn't solve the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment