training issues

[himanshugiriraj]

Hi,
After training for a while the system goes into idle state. Can anyone please suggest me the possible issues.

[MISTCARRYYOU]

If the idle state refers to only one or two threads are working, it may be caused by the slow training of one or two edges. In that case, the other edges may stop to wait for the complement of others.

[himanshugiriraj]

Hello,
thank you for your reply.
after a while the i got this:

terminate called after throwing an instance of 'c10::Error'
what(): pool INTERNAL ASSERT FAILED at "../aten/src/ATen/ParallelOpenMP.cpp":65, please report a bug to PyTorch. Invalid thread pool!
Exception raised from set_num_threads at ../aten/src/ATen/ParallelOpenMP.cpp:65 (most recent call first):

After this the model starts training but throws same error.
How much time does the model takes to train and generate results?

[MISTCARRYYOU]

Well I did not encounter this issue before, but it seems like a software version issue.
The running time depends on the aggregation rounds and training episodes you set, but generally several hours are enough.

[himanshugiriraj]

thank you once again for your prompt reply.

can you please share your exact environment details. It shall be really helpful.