Code

Add dataset blending tool#32

Code

Merged

ryantwolf merged 38 commits intomainNVIDIA/NeMo-Curator:mainfrom rywolf/blendingNVIDIA/NeMo-Curator:rywolf/blendingCopy head branch name to clipboardMay 3, 2024

+752-1

Commits

Commits on Apr 2, 2024

• Add initial dataset blending function

  Show description for 4a183cf
  ryantwolf

  committed
  4a183cf

Commits on Apr 4, 2024

• Add blend unit tests

  Show description for 969123c
  ryantwolf

  committed
  969123c

• Add self parameter

  Show description for d8e7922
  ryantwolf

  committed
  d8e7922

• Fix return type of blend dataset

  Show description for fd5f339
  ryantwolf

  committed
  fd5f339

• Fix blending tests

  Show description for 1e1401b
  ryantwolf

  committed
  1e1401b

• Change assert statement for very uneven blend

  Show description for 5d966ba
  ryantwolf

  committed
  5d966ba

• Fix key error

  Show description for 64d5040
  ryantwolf

  committed
  64d5040

• Add proper proportion blending test

  Show description for 7972669
  ryantwolf

  committed
  7972669

• Add four dataset blend and clarify docs

  Show description for f603c2c
  ryantwolf

  committed
  f603c2c

Commits on Apr 22, 2024

• Merge branch 'main' into rywolf/blending

  ryantwolf

  committed
  e53c1eb

• Add shuffle module

  Show description for 203f88a
  ryantwolf

  committed
  203f88a

• Add blend example and tests

  Show description for 5899432
  ryantwolf

  committed
  5899432

• Fix random method name

  Show description for f197610
  ryantwolf

  committed
  f197610

• Wrap return type in DocumentDataset

  Show description for 2bc5a5f
  ryantwolf

  committed
  2bc5a5f

• Save result of column drop

  Show description for bb9803d
  ryantwolf

  committed
  bb9803d

• Change equality check for shuffle tests

  Show description for 414091d
  ryantwolf

  committed
  414091d

• Fix expected order after shuffle

  Show description for 88ae8af
  ryantwolf

  committed
  88ae8af

• Add more documents to shuffle test

  Show description for b016672
  ryantwolf

  committed
  b016672

• Add assert statement

  Show description for 48bb22b
  ryantwolf

  committed
  48bb22b

• Add within partition shuffle

  Show description for cd9fdcf
  ryantwolf

  committed
  cd9fdcf

Commits on Apr 23, 2024

• Refactor add rand column for shuffle

  Show description for 491f8b6
  ryantwolf

  committed
  491f8b6

• Fix filename tests

  Show description for 89009a8
  ryantwolf

  committed
  89009a8

• Add determinism handling for shuffle

  Show description for aaa6493
  ryantwolf

  committed
  aaa6493

Commits on Apr 24, 2024

• Merge branch 'main' into rywolf/blending

  ryantwolf

  committed
  22376f2

• Change numpy random function

  Show description for b1e1a42
  ryantwolf

  committed
  b1e1a42

• Fix tests with new random method

  Show description for c5f7b3e
  ryantwolf

  committed
  c5f7b3e

Commits on Apr 26, 2024

• Remove length call from blending

  Show description for 5dda670
  ryantwolf

  committed
  5dda670

Commits on Apr 29, 2024

• Improve scaling of blending function

  Show description for 6cee1fc
  ryantwolf

  committed
  6cee1fc

• Fix blend tests

  Show description for 2a1c7cb
  ryantwolf

  committed
  2a1c7cb

Commits on Apr 30, 2024

• Add blending script

  Show description for 7025d88
  ryantwolf

  committed
  7025d88

• Add additional file paths call

  Show description for b7d3f50
  ryantwolf

  committed
  b7d3f50

• Add documentation

  Show description for e7c2a9c
  ryantwolf

  committed
  e7c2a9c

• Reformat docs

  Show description for 6ddff54
  ryantwolf

  committed
  6ddff54

• Remove backticks

  Show description for ccc6e0c
  ryantwolf

  committed
  ccc6e0c

• Merge branch 'main' into rywolf/blending

  ryantwolf

  committed
  0d81ace

• Add context manager for shuffle tests

  Show description for 53fcda9
  ryantwolf

  committed
  53fcda9

Commits on May 3, 2024

• Add better deterministic shuffle path

  Show description for 2c578cb
  ryantwolf

  committed
  2c578cb

• Update documentation and reset index

  Show description for fa25b33
  ryantwolf

  committed
  fa25b33