Add jupyter notebook tutorial for single node mulilingual dataset#30

Merged

ryantwolf merged 36 commits intoNVIDIA:mainfrom nicoleeeluo:main

May 24, 2024

+3,868

Commits on May 20, 2024

Init commit for tutorial notebook
nicoleeeluo
committed
Fix metadata inference with pandas and dask (NVIDIA#35 )

ryantwolf
authored and
nicoleeeluo
committed
Disable PyTorch Compile Multiprocessing (NVIDIA#34 )

ryantwolf
authored and
nicoleeeluo
committed
Improve speed of AddId module (NVIDIA#36 )

ryantwolf
authored and
nicoleeeluo
committed
Make GPU dependencies optional (NVIDIA#27 )

ayushdg
authored and
nicoleeeluo
committed
Fix failing GPU tests with latest pandas bump (NVIDIA#41 )

ayushdg
authored and
nicoleeeluo
committed
Adds Nemo Curator K8s example (NVIDIA#40 )

terrykong
authored and
nicoleeeluo
committed
Move common dedup utils and remove unused code (NVIDIA#42 )

ayushdg
authored and
nicoleeeluo
committed
Fix lang id example (NVIDIA#37 )

ryantwolf
authored and
nicoleeeluo
committed
Add dataset blending tool (NVIDIA#32 )

ryantwolf
authored and
nicoleeeluo
committed
High level fuzzy duplicates module (NVIDIA#46 )

ayushdg
authored and
nicoleeeluo
committed
Fix indexing in PII Modifier (NVIDIA#55 )

ryantwolf
authored and
nicoleeeluo
committed
Disable string conversion globally (NVIDIA#56 )

ryantwolf
authored and
nicoleeeluo
committed
Fix issue NVIDIA#43 (empty files creation) and improve reading/writing speed (NVIDIA#57 )

miguelusque
authored and
nicoleeeluo
committed
[Tutorials] Add a tutorial for PEFT data curation (NVIDIA#45 )

Maghoumi
authored and
nicoleeeluo
committed
Only import PII constants during Curator import (NVIDIA#61 )

ayushdg
authored and
nicoleeeluo
committed
Deleting links
nicoleeeluo
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Update tutorials/single_node_tutorial/single_gpu_tutorial.ipynb

nicoleeeluo
and
ryantwolf
committed
Fixed typo. Update content to lastest NeMo Curator version. Added fuzzy deduplication wrapper example
nicoleeeluo
committed
Fixing Style
nicoleeeluo
committed
Updating container version
nicoleeeluo
committed
Merge branch 'main' into main
nicoleeeluo
authored
Fixing style
nicoleeeluo
committed

Commits on May 24, 2024