Skip to content

Commit

Permalink
adding dlp logger. (#109)
Browse files Browse the repository at this point in the history
Lot of people are getting confused with iostat and we should remove it.

Co-authored-by: Huihuo Zheng <[email protected]>
  • Loading branch information
hariharan-devarajan and zhenghh04 authored Nov 20, 2023
1 parent 035f512 commit d06e63f
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 5 deletions.
18 changes: 15 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,13 @@ cd dlio_benchmark/
pip install .
dlio_benchmark ++workload.workflow.generate_data=True
```
Additionally, to generate the report `iostat` is needed and can be installed from the `sysstat` package using your package manager.

### Bare metal installation with profiler

```bash
git clone https://github.com/argonne-lcf/dlio_benchmark
cd dlio_benchmark/
pip install .[dlio_profiler]

## Container

Expand Down Expand Up @@ -74,9 +80,15 @@ If possible, one can flush the filesystem caches in order to properly capture de
```bash
sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
```
Finally, run the benchmark with ```iostat``` profiling, listing the io devices you would like to trace.
Finally, run the benchmark
```bash
mpirun -np 8 dlio_benchmark workload=unet3d
```
Finally, run the benchmark with Profiler
```bash
mpirun -np 8 dlio_benchmark workload=unet3d ++workload.workflow.profiling=True ++workload.profiling.profiler=iostat ++workload.profiling.iostat_devices=[sda,sdb]
export ENV DLIO_PROFILER_ENABLE=1
export DLIO_PROFILER_INC_METADATA=1
mpirun -np 8 dlio_benchmark workload=unet3d
```

All the outputs will be stored in ```hydra_log/unet3d/$DATE-$TIME``` folder. To post process the data, one can do
Expand Down
4 changes: 2 additions & 2 deletions dlio_benchmark/utils/utility.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,6 @@ def create_dur_event(name, cat, ts, dur, args={}):
}
return d


def get_trace_name(output_folder):
return f"{output_folder}/trace-{get_rank()}-of-{get_size()}.pfw"
return f"{output_folder}/trace-{get_rank()}-of-{get_size()}.pfw"

0 comments on commit d06e63f

Please sign in to comment.