Skip to content

DLIO v1.0

Compare
Choose a tag to compare
@zhenghh04 zhenghh04 released this 16 Nov 06:44
· 474 commits to main since this release

DLIO v1.0 Release Notes

We are excited to announce the release of DLIO 1.0! There are many new features and new enhancements compared to previous 0.0.1 version:

  • Using YAML file to configure DLIO in Hydra.cc framework; The configuration options are organized in a hierarchical way, including model, framework, workflow, dataset, train, evaluation, checkpoint, profiling. a set of YAML files for some workloads are included.
  • Data loader support enhancement:
    • Added data loader layer above data format to allow user to choose data loader and data format independently.
    • Added PyTorch data loader support. We have full PyTorch data loader support for one sample per file dataset
    • Enhanced TensorFlow tf.data loader to support for generic file format beyond tfrecord format (currently only support one sample per file case for generic data format)
  • New dataset support
    • Added support for png and jpeg formats
    • Supporting multiple subfolders for training and validation datasets.
    • Supporting generating validation dataset
  • Profiling and logging
    • Added support for iostat profiling
    • Added detailed logging info
  • Added support for validation.
  • Added post processing python script
  • Added unit tests and GitHub Actions tests.
  • User and developer documentation in github.io: https://argonne-lcf.github.io/dlio_benchmark