Specs of gpu for training

[eric-le-12]

Hi,
I wonder if you can release the specs of GPU for training on these datasets.
thank you for your support

[kaanakan]

Hi,
we mostly used T4s or V100s for training our models. However, these GPUs are not necessary for the training process. You can change the batch size and use other GPUs.

Best,
Kaan

[eric-le-12]

Thanks for your reply,
What i was confusing is the training time, so i think the number of gpus would help.
Since for my case, training for a whole epoch would takes up to 14 hours :(
I was wondering about the number of iterations , in the paper, the authors mentioned that each epoch consists of 1000 updates, so i just want to confirm: Due to fixed iteration, is there any dataset that the model will not touch the whole training set (i assume num of iterations = len(dataset)/batch size)

[kaanakan]

Hi,

Unfortunately, the code does not support multi-gpu training. You can only use one GPU for training or evaluation.

We implemented the dataloader scheme a little different. We had a iterator object which gives the next batch in the dataloader and when there is no next batch, the iterator returns the first object in the dataloader. That is, when we first pass through the entire dataset, we start the next pass.

Number of iterations was around ~300K iterations for generic dataset, MNIST, KTH and BAIR and ~150K for batch size of 16 (2.4M video samples in total).

Best,
Kaan