You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Suggestion: define input pipeline where we do the following: a) read files, b) extract samples, and c) resize. DaliReader will have a init, read, and finalize API.
Suggestion Return tf.data.dataset which includes reading, extracting samples and resize. TensorflowReader will have a init, read, and finalize API.
Pytorch
The recommended way to use PyTorch is to define custom data loaders. But it has some custom image loading.
Suggested Changes
I will create separate enums for TensorflowReaders and DaliReaders we support. They will have numbers similar to our ReaderType for compatibility.
Rename our data loaders to DLIO_PYTORCH, DLIO_TENSORFLOW, and DLIO_DALI as this is our implementations.
Similarly, rename our data reads as DLIO_CSV and so on.
The new data loaders would be called NATIVE_TENSORFLOW and NATIVE_PYTORCH.
For validation our current loaders work with our DLIOReaderType. If user selects The NATIVE_TENSORFLOW then it will be validated against TensorflowReaderType and similarly for DALI.
The base classes for these reader would be different as well. We will have three baseclasses DaliBaseReader, DLIOBaseReader, PyTorchBaseReader, and TensorflowBaseReader.
The text was updated successfully, but these errors were encountered:
All data loaders support internal reading functions. I will use this issue to describe some Data loader and possible integration into dlio_benchmark.
Dali data loader
Examples: npz tfrecord
Suggestion: define input pipeline where we do the following: a) read files, b) extract samples, and c) resize. DaliReader will have a init, read, and finalize API.
TFRecord
Examples csv (experimental) and tfrecord
Suggestion Return tf.data.dataset which includes reading, extracting samples and resize. TensorflowReader will have a init, read, and finalize API.
Pytorch
The recommended way to use PyTorch is to define custom data loaders. But it has some custom image loading.
Suggested Changes
The text was updated successfully, but these errors were encountered: