Amplitudes of the GMM

[erlebach]

Hi,

How do you ensure that the amplitudes of the GMM (pi[i]) remain positive and sum to unity, when minimizing the cost function with respect to these amplitudes? Thanks.

Gordon

[slim1017]

Hi,

1. pi remain positive:
   Empirically，we found that pi always remain positive during training (note that there is a log pi in loss function, maybe this is the reason), so we don't need to worry about the constraint pi >= 0;

2. sum to unity:
   we could normalize the pi vector to make its sum equal one after each batch or epoch or entire training. The three ways achieve similar accuracy.

The parameter pi has much less impact on model training and clustering accuracy than the mean and variance parameters of GMM.

[daib13]

Do you think using a softmax layer to produce pi is a better way or a worse way? Because sofmax layer ensures that pi is always positive and sum to 1. But it may cause problems like gradient vanishing or other problems in SGD.

[colobas]

Hi

The parameter pi has much less impact on model training and clustering accuracy than the mean and variance parameters of GMM.

Does this hold even for datasets where the clusters aren't balanced? It seems odd.

Also, if this is the case, couldn't one fix the the gmm_pis?