Skip to content

Pull requests: huggingface/datatrove

Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Assignee
Filter by who’s assigned
Sort

Pull requests list

Adding Megatron Tokenization pipeline
#304 opened Nov 14, 2024 by TJ-Solergibert Loading…
Better multilingual support
#285 opened Sep 5, 2024 by guipenedo Draft
Use spaCy tokenizer for Dutch
#284 opened Sep 4, 2024 by BramVanroy Loading…
Video support for datatrove
#271 opened Aug 21, 2024 by guipenedo Draft
Fix SENTINEL cluster
#250 opened Jul 13, 2024 by jordane95 Loading…
Mersenne prime hashing fix.
#200 opened May 28, 2024 by Apsod Loading…
Linewise filters
#125 opened Mar 14, 2024 by guipenedo Draft
ProTip! What’s not been updated in a month: updated:<2024-10-23.