Documentation inconsistencies in HPC installation of DAFoam

[uttamcadambi07]

Hi,

I have been trying to install DAFoam and its dependencies on my HPC cluster based on intel compilers.

I successfully managed to build and install DAFoam on my workstation and I can use it without any issues.
Version DAFoam v3.0.3

I have been referring to the steps mentioned in https://dafoam.github.io/mydoc_installation_source_hpc.html.

There are some inconsistencies in the documentation and I kindly request the author to clarify them on the website for the benefit of those who wish to use DAFoam.

1. DAFoam version:
   The instructions say:

Install MACH-Aero

The supported repo versions in the MACH-Aero framework for DAFoam-v3.0.3 is as follows
baseclasses 	pySpline 	pyGeo 	multipoint 	pyHyp 	cgnsUtilities 	IDWarp 	pyOptSparse 	pyOFM 	DAFoam
v1.2.0 	           v1.2.0 	   v1.5.0 	     v1.2.0 	v2.2.0 	     v2.5.0 	   v2.2.1 	     v2.3.0 	v1.2.1 	v3.0.3

Later on in the installation directions, this is mentioned

Install DAFoam

Once OpenFOAM, CGNS, and the MACH-Aero packages are installed, we can install DAFoam. Similar to OpenFOAM, we need to compile three versions of DAFoam: original, reverse-mode AD (ADR), and forward-mode AD (ADF). Download the repository and enter its code directory:

wget https://github.com/mdolab/dafoam/archive/v2.2.9.tar.gz -O dafoam.tar.gz
tar -xvf dafoam.tar.gz
cd dafoam-2.2.9

If you notice, there is a mismatch in the version that the installation instruction claims to provide documentation for (for DAFoam) and the lines of code that actually install DAFoam (documentation says v3.0.3 and installation instruction is for v2.2.9).

2. Version of petsc and nohdf5 option
   If DAFoam v3.0.3 is to be compiled, then the version of petsc cannot be 3.11-nohdf5 (I tested this and it was not compatible. There was a garbage collector file that was missing in petsc 3.11 and also there were some errors with respect to absence of hdf5). It has to be 3.14.6

3. intel compilers
   The documentation mentions that the instructions is for cluster with Intel compilers but in one of the lines of code that was asked to be modified for OpenFOAM,

cc          = icc
->
cc          = mpicc

This defaults to gcc compiler. The correct one for Intel compiler should be

cc          = icc
->
cc          = mpiicc

Please can you go over the documentation one more time and update the versions and correct the inconsistencies? Please also provide the correct configure flags for the correct PETSc version for compiling and building DAFoam on HPC clusters. I am happy to provide any more information upon request since I have been trying to get DAFoam to work on the cluster for over two months now.

[friedenhe]

Thanks for the feedback. @bernardopacini can you update the doc for the HPC installation?

[bernardopacini]

Yes, I'll go through and update the instructions.

Regarding the second point, can you explain what you mean and what error is given? I am not aware of an incompatibility. We used that version of PETSc on an Intel based cluster until very recently.

In the meantime, for 1) use version 3.0.3 and for 3) use mpiicc.

I am sorry to hear you are having trouble getting it working. Can you further explain at what point of the DAFoam installation it fails and what error is given?

[uttamcadambi07]

Thanks a lot for your response. First off, I want to say that your work on DAFoam is simply outstanding. The support given by @friedenhe is amazing and I am truly indebted to all the work that is done. I was struggling to get a continuous adjoint method working in OpenFOAM simply because deriving the equations for turbulence models is a pain. This has opened up a new area of research for me so I am obliged to thank you.

I will go over the errors one by one since I have spent a lot of time with installation and I have good experience with this.

1. PETSc. So as the documentation says, use version 3.11-nohdf5. We first tried that and got an error (I do not have the exact log with me now but it was clear that hdf5 was required). This was for DAFoam v3.0.3.

Secondly, we tried to install with hdf5, but we encountered an error to install mpi4py (make log -
make.log configure log -
configure.log and mpi4py installation log -
mpi4py.log ). This used mpi4py 3.1.1.

The PETSc developers advised us to use Python2 for installing. So we did that (although also knowing that Python2 compatibility is poor). Then we got this error: one of the packages was missing /petsc/private/garbagecollector.h Bottomline - petsc-3.11-nohdf5 does not have the garbagecollector.h file that is required for compiling some of the other MDO packages. So the developers recommended using petsc-3.18.

But I knew that my workstation uses petsc-3.14.6 and it works fine. So I compiled both petsc-3.14.6 and petsc-3.18 on my workstations. These were my flags:
./configure --configModules=PETSc.Configure --optionsModule=config.compilerOptions PETSC_ARCH=intelmpi --prefix=/scratch/ucp1n22/privatemodules/petsc-3.14.6.2 --with-scalapack-include=/local/software/intel-mkl/2018.1.163/compilers_and_libraries_2018.1.163/linux/mkl/include --with-scalapack-lib="-L/local/software/intel-mkl/2018.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64 -lpthread -lm -ldl" --download-metis=yes --download-mumps=yes --download-parmetis=yes --CC=mpiicc --FC=mpiifort --CXX=mpiicpc --download-petsc4py=yes --download-mpi4py=yes --with-hdf5=1 --download-hdf5=yes --download-superlu_dist=yes --with-shared-libraries=yes --with-fortran-bindings=1 --with-cxx-dialect=C++11
This seems to work fine for me. The modules I load are:

  1) git/2.24.1               3) libfabric/1.7.0   5) boost/1.70.0     7) python/3.7.3           9) intel-mkl/2018.1.163
  2) intel-compilers/18.0.3   4) gcc/6.4.0         6) petsc/3.14.6.2   8) intel-mpi/2018.3.222

2. IPOPT: Now I understand one of the optimizers that is used is IPOPT so it is not fair to expect documentation on the same. But I tried to install IPOPT for which I started with metis, then built mumps and then built IPOPT with mumps. Keep in mind the versions:
   METIS: 5.0
   MUMPS: 3.0.4
   IPOPT: 3.13.3

At this point, I first got an error when I ran a test case with 47000 cells in the mesh and distributed across 2 nodes with 8, 16 and 32 cores per node respectively:

Calculating dFdField using reverse-mode AD for beta
    Adding cells with centre within boxes 1((-30 -30 -1) (30 30 1))
    Adding cells with centre within boxes 1((-30 -30 -1) (30 30 1))
Calculating [dRdField]^T * Psi using reverse-mode AD
Objective Function Sensitivity: 
{'FI': {'beta': array([-0.0000027209566295, -0.0000014715984917, -0.0000000049400267, ...,
        0.0531278466359103,  0.4826668053561129,  0.0002482915520912])}, 'fail': False}
Adjoint Runtime: 39.5267 s
Scaling parameter for objective function = 1.000000e+00
objective scaling factor = 1
No x scaling provided
No c scaling provided
No d scaling provided
Initial values of x sufficiently inside the bounds.
Initial values of s sufficiently inside the bounds.
   Current memory used:         408 bytes
   Maximum memory used:         408 bytes
***Memory allocation failed for OMETIS: cptr. Requested size: 6505260546115368 bytes

So this got me worried.

To overcome this, I used the exact same instructions given in the Ubuntu installation documentation for IPOPT:
Download Ipopt-3.13.2 and set up the relevant environmental variables to loadDAFoam.sh by running:

echo '# Ipopt' >> $HOME/dafoam/loadDAFoam.sh && \
echo 'export IPOPT_DIR=$DAFOAM_ROOT_PATH/packages/Ipopt' >> $HOME/dafoam/loadDAFoam.sh && \
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$IPOPT_DIR/lib' >> $HOME/dafoam/loadDAFoam.sh && \
. $HOME/dafoam/loadDAFoam.sh

Next, compiles the ThirdParty dependencies Metis and Mumps by running:

cd $HOME/dafoam/packages && \
git clone -b stable/3.13 https://github.com/coin-or/Ipopt.git && \
cd $IPOPT_DIR && \
git clone -b stable/2.1 https://github.com/coin-or-tools/ThirdParty-Mumps.git && \
cd ThirdParty-Mumps && \
./get.Mumps && \
./configure --prefix=$IPOPT_DIR && \
make && \
make install

Finally, compile Ipopt by running:

cd $IPOPT_DIR && \
mkdir build && \
cd build && \
../configure --prefix=${IPOPT_DIR} --disable-java --with-mumps --with-mumps-lflags="-L${IPOPT_DIR}/lib -lcoinmumps" --with-mumps-cflags="-I${IPOPT_DIR}/include/coin-or/mumps" && \
make && \
make install

but obviously adapting it to HPC. Now with the same test case, I got this error:

  File "runScript_v2.py", line 174, in <module>
    sol = opt(optProb, sens=optFuncs.calcObjFuncSens, storeHistory=histFile)
  File "/mainfs/scratch/ucp1n22/DAFoam_v3/repos/pyoptsparse-2.8.1/pyoptsparse/pyIPOPT/pyIPOPT.py", line 250, in __call__
    self._waitLoop()
  File "/mainfs/scratch/ucp1n22/DAFoam_v3/repos/pyoptsparse-2.8.1/pyoptsparse/pyOpt_optimizer.py", line 689, in _waitLoop
    info = self.optProb.comm.bcast(info, root=0)
  File "mpi4py/MPI/Comm.pyx", line 1569, in mpi4py.MPI.Comm.bcast
  File "mpi4py/MPI/msgpickle.pxi", line 732, in mpi4py.MPI.PyMPI_bcast
  File "mpi4py/MPI/msgpickle.pxi", line 152, in mpi4py.MPI.pickle_load
  File "mpi4py/MPI/msgpickle.pxi", line 141, in mpi4py.MPI.cloads
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 16: invalid continuation byte

Finally, I got tired and then I managed to search the internet and found these two links useful regarding the working of mpi4py:
https://bitbucket.org/mpi4py/mpi4py/issues/39/mpi-msgpicklepxi-eoferror
https://bitbucket.org/mpi4py/mpi4py/issues/39/mpi-msgpicklepxi-eoferror

I noticed that this error comes up when you use more than one node so I switched to a single node and FINALLY my case runs fine on HPC.

So for the changes to your documentation (for DAFoam v3.0.3)

1. Use Petsc-3.14.6.
2. Change mpicc to mpiicc and mpicpc to mpiicpc
3. When building IPOPT, use the same instructions as that of the Ubuntu installation. For some reason, using metis ordering when building mumps does not seem to sit well with IPOPT. I am configuring Petsc to build metis along with its installation so I am not sure if IPOPT needs this.
4. I have not tested anything for 64 bit. Since the Ubuntu installation will default to 64 bit on 64 bit machines, the same may not be true for an HPC cluster. I have not built IPOPT or metis or mumps or petsc with 64 bit support (or so this is what I feel). In case of SEGV errors I suppose you could just increase the number of processors and it should work? Not sure about that.

I think this is all. I have documented all my steps during installation and have presented all errors on the forum so I think it should be visible to everyone.

[uttamcadambi07]

Hi. Just checking - has the documentation been updated already?

[bernardopacini]

Hi, this is still on our to-do list and I will try to get to it in the next week. I apologize for the delay, I'll get back to you once I have the instructions updated.

[Drawthe]

@bernardopacini
Hello, has the new HPC installation documentation been updated?
In addition, during the installation process under https://dafoam.github.io/mydoc_installation_source_hpc.html, except that mpicxx is not a command of the intel compiler, idwarp and pyhyp are problematic. The command in the URL is the config of the default openmi compiler of cp. mk is actually not advisable. A -r8 parameter is used in config.mk, which is also in intel, but the actual command is openmpi

Finally, it is not recommended to put the final dafoam in the background during the compilation process, because it is easy to ignore the errorLog information.
Looking forward to the new hpc installation process thank you

[Drawthe]

Regarding dependencies, it is best to give compilation parameters uniformly

[uttamcadambi07]

Hi. Is there any update on this? I believe the HPC installation page still has the old instructions. I have made a document for compiling on our HPC system but that is valid only upto dafoam-3.0.3. Now we want to install dafoam-3.0.6 and there is a dependency update in the form of v1.3.0 of OpenFOAM-ADF and ADR. Which means, the intel compilers version suggested on the page (18.0.2) no longer works. I managed to compile the ADR and ADF versions using intel compilers 19.0.3 (took me an entire day) but now I am having issues compiling dafoam 3.0.6 in reverse and forward modes (the original is also not compiling actually). We use intel-mpi/2019.3.199 and python/3.7.3 now we are not sure if this will work. Please can you take this on high priority? A lot of our research work is dependent on using DAFoam on our HPC cluster and this is proving to be our biggest bottleneck in publishing.