This is a minimalist version of SMC, a combustion code that solves
compressible Navier-Stokes equations for viscous multi-component
reacting flows.  SMC is developed at the Center for Computational
Sciences and Engineering (CCSE) at Lawrence Berkeley National
Laboratory.  It uses the BoxLib library
(https://ccse.lbl.gov/BoxLib/), also developed at CCSE.  This
minimalist version of SMC includes a stripped down version of BoxLib.
Therefore, the full version of BoxLib is not needed.

The code is mainly written in Fortran 90, with some parts in C.
Parallelization is implemented with hybrid MPI/OpenMP.  However, SMC
can also be built with pure MPI, or pure OpenMP, or neither.

GNU make is used to build the code.  The following options can be
specified in the GNUmakefile:

* MPI :=  
  This determines whether we are using the Message Passing
  Interface (MPI) library.  Leaving this option empty will disable
  MPI.  If MPI is enabled, you need to specify how to link to the MPI
  library.  You can set USE_MPI_WRAPPERS := t, then mpif90 or mpiifort
  will be used.  Or you can set "mpi_include_dir", "mpi_lib_dir", and
  "mpi_libraries" to appropriate values in GNUmakefile.

* OMP := t
  This determines whether we are using OpenMP.  Leaving this option
  empty disables OpenMP.

* NDEBUG := t
  This option determines whether we compile with support for debugging
  (usually also enabling some runtime checks). Setting NDEBUG := t
  turns off debugging.

* COMP := Intel
  Set this to your compiler of choice (e.g., GNU).  Specific options
  for a certain compiler are stored in the comps/$(COMP).mak file.   

* MIC :=
  Set this if compiling for Intel Xeon Phi.

* K_USE_AUTOMATIC := t
  This determines whether some arrays in kernels.F90 will be automatic
  or allocatable.  Automatic arrays are usually on the stack.
  When OpenMP is used, allocating memory on the stack is usually
  faster than on the heap.  However, one must make sure there is
  adequate space on the stack; otherwise a segmentation fault might
  occur.  Note that the size of the stack for threads can be adjusted
  by the OMP_STACKSIZE environment variable and the shell might also
  impose a limit on stack size.

* MKVERBOSE := t
  This determines the verbosity of the building process.

To build an executable, type "make" in the SMC directory.  Note that
the above options can also be specified in a command line such as
"make MPI=t MIC=t NDEBUG= ".  The executable will have a name like
main.GNU.debug.mpi.exe depending on the compiler and some other
options specified in the GNUmakefile or command line.  There are some
other options provided by the GNUmakefile.  You can type "make TAGS"
or "make tags" to generate tag file for Emacs or vi.  You can type
"make clean" or "make realclean" to remove files generated during the
make process.

Runtime parameters can be specified in the inputs_SMC file, which must
be placed in the directory where the executable is run.  Below are a
list of selected runtime parameters and default values.

* n_cellx = -1
* n_celly = -1
* n_cellz = -1

  Number of cells in the x, y, and z-directions.  They must be greater than 4.

* max_grid_size = 64 

  This determines the largest grid size of a box.  If max_grid_size is
  too big, there might be fewer boxes than MPI tasks, and then some
  processors will be idle.  However, if it is too small, each MPI task
  may have too many small boxes, and the performance will be affected.
  Thus, max_grid_size must be chosen according to the number of cells
  and the number of MPI tasks.  Ideally, you would like to have one
  box for each MPI task to minimize the communication cost.

* tb_split_dim = 2

  This determines how domain decomposition in some subroutines is done
  for threads.  When OpenMP is used, each box is "virtually" divided
  into "nthreads" boxes and each OpenMP thread works on one
  thread-box.  This parameter can be either 2 or 3. When it is 2, the
  domain decomposition is in y and z-directions; when it is 3, the
  domain decomposition is in 3D.

* tb_blocksize_x = -1
* tb_blocksize_y = 16
* tb_blocksize_z = 16

  SMC uses a blocking strategy.  These parameters determines blocking
  size in x, y and z-directions.  The value should be either greater
  than or equal to 4, or less than 0.  If it is less than 0, no
  blocking is used in that direction.

* verbose = 0

  This determines verbosity.

* stop_time = -1.d0

  Simulation stop time.

* max_step = 1

  Maximum number of timesteps in the simulation.

Note that there are a number of OMP COLLAPSE clauses in the code.
These tend to trigger compiler bugs.  If the code crashes, try to run
it without "COLLAPSE". 

More information about SMC can be found in the paper "High-Order
Algorithms for Compressible Reacting Flow with Complex Chemistry" by
M. Emmett, W. Zhang, and J.B. Bell (http://arxiv.org/abs/1309.7327),
and in "Optimizing for Navier-Stokes Equations" by Antonio Valles and
Weiqun Zhang (Chapter 4 of "High Performance Parallelism Pearls:
Multicore and Many-core Programming Approaches", James Reinders and
Jim Jeffers, published by Morgan Kaufmann, 2014).  If you have any
questions, please contact Weiqun Zhang ([email protected]).

**********************************************************************

As noted in the book, the best performance shown on Figure 4.19 for
the Intel Xeon Phi coprocessor is 12.99 seconds, and we found close to
publication time that a few additional Fortran compiler options allow
us to do better and achieve 9.11 seconds for the coprocessor (1.43x
improvement vs . Figure 4.19) while keeping the processor performance
the same. The compiler options for the improved performance are:
-fno-alias -noprec-div -no-prec-sqrt -fimf-precision=low
-fimf-domain-exclusion=15 -align array64byte -opt-assumesafe-padding
-opt-streaming-stores always -opt-streaming-cache-evict=0.

These new best results on coprocessor use -1, 16, 16 for the thread
blocking specified in inputs_SMC file and SIMD directives are no
longer needed in kernels.F90 file when using -fno-alias compiler
option.  The chapter discusses Figure 4.19 as best results for
coprocessor.

The files here reflect all these changes for higher performance.