DLIO v1.0
DLIO v1.0 Release Notes
We are excited to announce the release of DLIO 1.0! There are many new features and new enhancements compared to previous 0.0.1 version:
- Using YAML file to configure DLIO in Hydra.cc framework; The configuration options are organized in a hierarchical way, including
model
,framework
,workflow
,dataset
,train
,evaluation
,checkpoint
,profiling
. a set of YAML files for some workloads are included. - Data loader support enhancement:
- Added data loader layer above data format to allow user to choose data loader and data format independently.
- Added PyTorch data loader support. We have full PyTorch data loader support for one sample per file dataset
- Enhanced TensorFlow tf.data loader to support for generic file format beyond tfrecord format (currently only support one sample per file case for generic data format)
- New dataset support
- Added support for png and jpeg formats
- Supporting multiple subfolders for training and validation datasets.
- Supporting generating validation dataset
- Profiling and logging
- Added support for iostat profiling
- Added detailed logging info
- Added support for validation.
- Added post processing python script
- Added unit tests and GitHub Actions tests.
- User and developer documentation in
github.io
: https://argonne-lcf.github.io/dlio_benchmark