-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathREADME
177 lines (148 loc) · 8.35 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
PIOK: Parallel I/O Kernels
=============
Suren Byna and Mark Howison
Contact: [email protected]
/***************************************************************************
PIOK is not in active development due to its dependency on H5Part (which is
not maintained actively).
A new benchmark suite, called h5bench, has been developed that uses the HDF5
API directly. h5bench adds more I/O patterns, more I/O kernels from various
HPC applications.
Go to: https://h5bench.readthedocs.io
GitHub: https://github.com/hpc-io/h5bench
***************************************************************************/
Parallel I/O is an essential part of many scientific simulation and
analysis codes on high performance computing (HPC) systems, as the codes
need to store or retrieve data to or from parallel file systems,
respectively. Identifying I/O performance bottlenecks, stressing parallel
file systems, and tuning performance of real scientific applications is
critical in using HPC systems efficiently. Instead of running large-scale
simulations that contain full-blown computation and communication phases
along with I/O, extracting the parallel I/O portions of codes reduces the
complexity in tuning I/O. Toward that goal, we extract I/O kernels from
real scientific simulations and call the set of kernels “Parallel I/O
Kernels (PIOK)”. As of September 2015, we have developed three I/O kernels
from three scientific simulations: VPIC-IO, VORPAL-IO, and GCRM-IO.
VPIC-IO (1D array): VPIC is a highly optimized and scalable particle
physics simulation developed by Los Alamos National Lab [1]. VPIC-IO uses
the H5Part API to create a file, write eight variables, and close the file.
H5Part provides a simple veneer API for issuing HDF5 calls corresponding to
a time-varying, multivariate particle data model [2]. We extracted all the
H5Part function calls of the VPIC code to form the VPIC-IO kernel. The
particle data written in the kernel is random data of oat data type. The
I/O pattern of VPIC-IO is a 1D particle array of a given number of
particles and each particle has eight variables. The kernel, by default,
writes 8M particles per MPI process for all experiments reported in this
paper.
VORPAL-IO (3D block structured grid): This I/O kernel is extracted from
VORPAL, a computational plasma framework application simulating the
dynamics of electromagnetic systems, plasmas, and rarefied as well as dense
gases, developed by TechX [3]. This benchmark uses H5Block to write
non-uniform chunks of 3D data per processor. The kernel takes 3D block
dimensions (x, y, and z) and the number of components as input.
GCRM-IO (semi structured mesh): This I/O kernel simulates I/O for GCRM, a
global atmospheric circulation model, simulating the circulations
associated with large convective clouds. This I/O kernel also uses H5Part
to perform I/O operations. The kernel performs all the GCRM I/O operations
with random data. I/O pattern of GCRM-IO corresponds to a semi-structured
geodesic mesh, where the grid resolution and sub-domain resolution are
specified as input.
[1] E. W. Bethel, J. M. Shalf, C. Siegerist, K. Stockinger, A.
Adelmann, A. Gsell, B. Oswald, and T. Schietinger. Progress on H5Part: A
Portable High Performance Parallel Data Interface for Electromagnetics
Simulations. In Proceedings of the 2007 IEEE Particle Accelerator
Conference (PAC 07). 25-29 Jun 2007, Albuquerque, New Mexico. 22nd IEEE
Particle Accelerator Conference, p.3396 , 2007.
[2] K. J. Bowers, B. J. Albright, L. Yin, B. Bergen, and T. J. T. Kwan.
Ultrahigh performance three-dimensional electromagnetic relativistic
kinetic plasma simulation. Physics of Plasmas, 15(5):7, 2008.
[3] C. Nieter and J. R. Cary. VORPAL: a versatile plasma simulation
code. Journal of Computational Physics, 196:448{472, 2004.
[4] David A. Randall et al., Global cloud resolving model (GCRM),
http://kiwi.atmos.colostate.edu/gcrm/
-------------------------------------------------------------------------
REQUIREMENTS
=============
This code has been tested with gcc on NERSC's Edison and Hopper
supercomputers. We have used gcc compiler to build them. Compiling the
kernels need HDF5 and H5Part libraries. We have compiled HDF5 with
--enable-parallel flag.
Example for compiling HDF5 1.8.14 and H5Part on Edison:
> cd <HDF5_source_dir>
> CC=cc ./configure --prefix=<HDF5_Install_dir> --disable-shared --disable-production --enable-parallel
> make
> make install
> cd <H5Part_source_dir>
> CC=cc ./configure --prefix=$PWD --with-hdf5=<HDF5_Install_dir> --enable-parallel --no-create --no-recursion
> make
> make install
-------------------------------------------------------------------------
CODE STRUCTURE
===============
The kernels are arranged in different sub-directories, in
<piok_dir>/benchmarks/read and <piok_dir>/benchmarks/write.
Currently, we have gcrmio, vorpalio, and vpicio kernels as write benchmarks; and
msi, ecog, and ardetect kernels as read benchmarks.
In <piok_dir>/benchmarks/write/gcrmio, we have the code for gcrmio kernel.
<piok_dir>/benchmarks/write/gcrmio/README file has further details on compiling,
and sample commands to run on NERSC's Edison system.
In <piok_dir>/benchmarks/write/vorpalio, we have the code for vorpalio kernel.
<piok_dir>/benchmarks/write/vorpalio/README file has further details on
compiling, and sample commands to run on NERSC's Edison system.
In <piok_dir>/benchmarks/write/vpicio, we have the code for multiple versions of
vpicio kernel.
vpicio_hdf5: Uses HDF5 calls to perform file writes.
vpicio_uni: Uses H5Part calls to perform file writes, which internally calls HDF5. _uni refers to each MPI process writing same number of particles.
vpicio_non_uni: Uses H5Part calls to perform file
writes, which internally calls HDF5.
_non_uni refers to each MPI process
writing different number of
particles. The variance in the
number of particles is controlled
by VARIABILITY constant.
vpicio_uni_md: Uses HDF5's multi-dataset write
functions. Multi-dataset functions
allow writing multiple datasets using
a single write call. THIS KERNELS NEEDS
A VERSION OF HDF5 THAT HAS MULTI-DATASET
FUNCTIONS IMPLEMENTED.
Compiling and running vpicio kernels are simple. Modify the HDF5
and H5Part paths in the corresponding Makefile and run "make".
<piok_dir>/benchmarks/write/vpicio/README file has further details on
compiling, and sample commands to run on NERSC's Edison system.
===============
Configure options
--prefix=<PATH> Specify the installation path
Default is the current directory (.), and
the executables will be installed in ./bin/.
E.g. when /tmp/mytest is provided,
the executables will be in /tmp/mytest/bin/.
--with-hdf5=<PATH> Specify the HDF5 path
--with-h5part=<PATH> Specify the H5Part path
--with-mpi=<PATH> Specify the MPI path
When MPI path is not provided, building process will use the
system default (e.g. NERSC case).
--enable-read-benchmark-only Enable read benchmark only build
Only build the read benchmarks
(i.e. benchmarks/read directory).
--enable-write-benchmark-only Enable write benchmark only build
Only build the write benchmarks
(i.e. benchmarks/write directory).
--enable-ardetection Enable ar_detection build only
Only build the ar_detection benchmarks
(i.e. benchmarks/read/ar_detection directory).
--enable-ecog Enable ecog build only
Only build the ecog benchmarks
(i.e. benchmarks/read/ecog directory).
--enable-msi Enable msi build only
Only build the MSI benchmarks
(i.e. benchmarks/read/msi directory).
--enable-gcrmio Enable gcrmio build only
Only build the gcrmio benchmarks
(i.e. benchmarks/write/gcrmio directory).
--enable-vorpalio Enable vorpalio build only
Only build the vorpalio benchmarks
(i.e. benchmarks/write/vorpalio directory).
--enable-vpicio Enable vpicio build only
Only build the vpicio benchmarks
(i.e. benchmarks/write/vpicio directory).