What is the status of WAVESPLIT model

[manu0586]

Hello,
According to the paper "Attention is all you need in speech separation" , a wavesplit model was implemented with asteroid (by Samuele Cornell in branch wavesplit).
What is the status of the model ? Does it work now?
Thanks

[popcornell]

Hi,
I made it work with oracle speaker embeddings recently.
The training takes very long even with that and I am quite poor in terms of resources.
I am happy if someone is willing to help with that. I can push my changes in the same branch.

[popcornell]

E.g. I can run just 4 batch-size, each example of 0.750 seconds on a TITAN RTX and training takes more than week. This is the biggest GPU I have and I have only one.

[manu0586]

I don't really understand what you call 'speaker oracle embedding'. Is it linked to the speaker stack of the wavesplit?
If I well understood the model, wavesplit is NOT a speaker-conditioned speech separation. You don 't need the voice embedding of the speaker that you want extract. Speaker information is used in training in order to define the 'prototype' part (clustering) of the model

[popcornell]

The speaker stack outputs a speaker embedding for each frame. These are clustered in inference and used to condition a separation stack. The clustering part solves the permutation issue. In my understanding it is conditioned by speaker ID.
I can train and test on validation assuming oracle speaker embeddings instead of using the speaker stack to extract them.

[manu0586]

By doing that I think that your system becomes a target speech extraction (with N target) and not a 'blind' speech separation anymore (like wavesplit).
But maybe it's a first step to understand/implement the separation stack

[manu0586]

Moreover, by join optimization of the 2 stacks, the speaker stack is training taking into account the reconstruction loss, so the latent space (// voice embedding) is define in order to be 'good' for the separation stack . This is not the case if you use an oracle (= a voice embedding extractor optimized by speaker loss only). So I m not sure that using an oracle is a good idea.

[popcornell]

The oracle embeddings are learned as explained in the paper

[popcornell]

Yes sure it is a simplification I made just to be sure the separation stack is correct. Next step is to try the full pipeline the code is already there actually. It is a matter of debugging and running experiments.

…

On Wed, Feb 24, 2021, 9:15 PM manu0586 ***@***.***> wrote: By doing that I think that your system becomes a target speech extraction (with N target) and not a 'blind' speech separation anymore (like wavesplit). But maybe it's a first step to understand/implement the separation stack — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#453 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEO36OJ6T7MNYEAOIL2LB7TTAVM4TANCNFSM4YEUFD4Q> .