is GPU training possible

[korhanpolat]

I wonder if it is possible to run

recipes/aud

scripts on GPU?

[lucasondel]

Yes and "yes but it's a bad idea". Some part of the recipe (scripts that corresponds to the subspace SHMM) are already trained on GPU. However, when the training is using the HMM forward-backward algorithm, even though it is possible to use the GPU it is be much slower than training on CPU.

[korhanpolat]

how about the HMM-VAE option? I'm trying to integrate VAE's as done in timit recipe

[lucasondel]

Well that's a tricky case. As I have written previously my HMM forward-backward implementation is not GPU friendly so it won't be very fast. On the other hand, the VAE would strongly benefits from the GPU during the training. So basically, it depends on the structure of your neural network. if it sufficiently big so that the forward-backward is the not the bottleneck, then I guess using the GPU will be fine.

I haven't tested much the HMM-VAE code but moving the model (including the HMM) to the GPU should be straightforward. You may have a look at how I train the SHMM to see how to train the model on the GPU: https://github.com/beer-asr/beer/blob/master/beer/cli/subcommands/shmm/train.py

[EomSooHwan]

I think I am having trouble with using GPU. When I run the code with --gpu I get the error message RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! I am curious if you have experienced same issue or if it's just my local problem.

[lucasondel]

Hi Eom,
Sorry for this late reply. Did you manage to solve the issue already ? If not, could you please provide me a little bit more details (where it happen in the code, which tensor in on cpu/gpu ) ?