Is there any plan to implement wav2vec 2.0?

[yogso]

Dear Shangeth, thank you very much for your repository, it has been extremely useful and I have learned a lot from it.

I have no experience with large pre-trained models but I am interested in trying wav2vec 2.0 as an encoder, and especially wav2vec 2.0 XLSR-53 which has been trained with multiple languages. Is there any short-term plan to include these models in the repository?

[shangeth]

Hi @yogso ,
If you want it ASAP, they check the hugging face and fairseq repositories for wav2vec2.0 models. Yes, I am planning to add the wav2vec2.0 encoder soon.

[shangeth]

Hey @yogso ,
I have added the wav2vec2 models(base, large, xlsr53) to the wavencoder. I will update the pip version soon. You can still use the repository version.

import wavencoder
import torch

encoder = wavencoder.models.wav2vec2.Wav2Vec2(model_type='base') # model_type='base'/'large'/'xlsr53'
x = torch.randn(2, 16000)

z = encoder.model.feature_extractor(x)
z.shape #torch.Size([2, 512, 49])

z = encoder(x)
z.shape #torch.Size([2, 49, 768])

[shangeth]

Also check this colab notebook

[yogso]

Thank you very much! I will try it!