why mixture loss is so large?

[hdmjdp]

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 126/126 [02:48<00:00, 1.34s/it]
epoch:0, running loss:186126176.4375, average loss:1477191.8764880951, current lr:0.0001
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 126/126 [02:41<00:00, 1.28s/it]
epoch:1, running loss:167085797.546875, average loss:1326077.7583085317, current lr:0.0001
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 126/126 [02:45<00:00, 1.31s/it]
epoch:2, running loss:167197651.21875, average loss:1326965.4858630951, current lr:0.0001
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 126/126 [02:43<00:00, 1.30s/it]
epoch:3, running loss:162484302.359375, average loss:1289557.955233135, current lr:0.0001

[G-Wang]

mixture of logstics I pulled from r9r9's wavenet implementation, including the log loss and sampling code. It really doesn't do work very well with the current settings. I don't have enough compute to run abliation studies to see what training set up works. I recommend sticking to raw bit, and beta/gaussian for now.

[hdmjdp]

@G-Wang Thanks for replying. I think bits 9 maybe the best configuration.

[chaiyujin]

The mixture of logsitics loss from r9y9's implementation is reduced by 'sum'. If you use 'mean' to reduce the loss, it will be like 6~11.