mlp_ratio not adjusted in config if mlp_hidden_size is set

[Muennighoff]

🐛 Describe the bug

if mlp_hidden_size is set it overrides mlp_ratio but the resulting config will still have mlp_ratio = 4 (the default value) which is confusing when looking at the config in e.g. wandb. I think we should either have mlp_ratio default to None or overwrite the config value via mlp_hidden_size if set

Versions

latest