You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to use the codebase on a multi-gpu setup and I am running into the Data Parallelism limitations where the overhead on GPU 0 renders the parallelism's usefulness subpar at best.
Could you please enable DDP in the codebase? That could be achieved easily with Microsoft's Deepspeed library due to its automatic mpi discovery.
Cheer!
The text was updated successfully, but these errors were encountered:
Thanks for the suggestion. We will consider implementing the DDP, but it's not our priority right now. You are welcome to submit a pull request on this.
Hey,
I am trying to use the codebase on a multi-gpu setup and I am running into the Data Parallelism limitations where the overhead on GPU 0 renders the parallelism's usefulness subpar at best.
Could you please enable DDP in the codebase? That could be achieved easily with Microsoft's Deepspeed library due to its automatic mpi discovery.
Cheer!
The text was updated successfully, but these errors were encountered: