Request for SmolMLv1 Pretraining Code

[JT-Ushio]

Hi, thank you for the amazing work! I'm interested in reproducing the pretraining process of SmolMLv1 (135M) on the SmolLM-Corpus. However, I noticed that the repository currently only includes fine-tuning scripts. Could you please share the pretraining code or provide a tutorial? I would greatly appreciate your help.

Thank you!

[loubnabnl]

Hi, we use nanotron for the pre-training https://github.com/huggingface/nanotron/

Here's the config for training SmolLM v1 135M
https://gist.github.com/eliebak/4263659706519536b7eebfe6d9815c60
You might need to adapt the dataset part to use nanosets with tokenized datasets cc @eliebak