Building with spack-stack? #1346

benjamin-cash · 2022-07-23T14:47:48Z

benjamin-cash
Jul 23, 2022

Has anyone tried building ufs-weather-model using spack-stack? @climbfuji has created a spack-stack installation on frontera that I am working with, but attempts to compile the Prototype-8c tag of the code are failing. I am curious if anyone has successfully done this before and might have pointers they could share.

Answered by benjamin-cash

Aug 23, 2022

Update - I was able to successfully build something that is mostly P8 on Frontera this morning using my gnu installation of spack-stack. :) Assuming all of the necessary modules have been loaded via spack, the steps that got me to an executable are:

Check out P8 tag
Update CMakeModules to develop
Change "STATIC" to "SHARED" on line 130 of CMakeLists.txt
Add esmf to DEPENDENCIES on line 9 of GOCART/ESMF/Shared/CMakeLists.txt
mkdir build && cd build
cmake -DAPP=S2SWA -DDEBUG=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DINLINE_POST=ON .. 2>&1 | tee log.make
Delete -lESMF from CMakeFiles/ufs_model.dir/link.txt
make VERBOSE=1 -j4 2>&1 | tee log.make

View full answer

jkbk2004 · 2022-07-23T16:07:17Z

jkbk2004
Jul 23, 2022
Maintainer

@benjamin-cash I have a plan to try build and set an experiment with spack-stack sometime next week. I will keep you posted.

1 reply

benjamin-cash Aug 19, 2022
Author

@jkbk2004 - Did you get a chance to try this?

climbfuji · 2022-07-25T02:48:36Z

climbfuji
Jul 25, 2022
Maintainer

I’ll try that myself on Frontera this week. Also @kgerheiser FYI

…

On Jul 23, 2022, at 10:07 AM, JONG KIM ***@***.***> wrote: @benjamin-cash <https://github.com/benjamin-cash> I have a plan to try build and set an experiment with spack-stack sometime next week. I will keep you posted. — Reply to this email directly, view it on GitHub <#1346 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RJPQNB52IW7YNJN4QLVVQKEDANCNFSM54ODPMHQ>. You are receiving this because you were mentioned.

0 replies

climbfuji · 2022-07-25T15:50:30Z

climbfuji
Jul 25, 2022
Maintainer

@benjamin-cash Can you briefly describe how you build the model, please? I.e. after checking out the code, what steps do you run?

1 reply

benjamin-cash Jul 25, 2022
Author

module purge
module use /work2/06146/tg854455/frontera/spack-stack/modulefiles
module load miniconda/3.9.12
module load ecflow/5.8.4

ulimit -s unlimited
module use /work2/06146/tg854455/frontera/spack-stack/spack-stack-v1/envs/skylab-1.0.0-gnu-9.1.0/install/modulefiles/Core
module load stack-gcc/9.1.0
module load stack-mvapich2/2.3
module load stack-python/3.9.12
module load ufs-weather-model-env

CMAKE_FLAGS="-DAPP=S2SWA" CCPP_SUITES="FV3_GFS_v17_coupled_p8" ./build.sh

When I do so, I hit the following errors:
CMake Error at CMakeLists.txt:120 (add_library):
add_library cannot create ALIAS target "fms" because another target with
the same name already exists.

CMake Error at /work2/06146/tg854455/frontera/spack-stack/spack-stack-v1/envs/skylab-1.0.0-gnu-9.1.0/install/gcc/9.1.0/cmake-3.22.1-mppepwa/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find PIO (missing: C Fortran)
Call Stack (most recent call first):
/work2/06146/tg854455/frontera/spack-stack/spack-stack-v1/envs/skylab-1.0.0-gnu-9.1.0/install/gcc/9.1.0/cmake-3.22.1-mppepwa/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
CMakeModules/Modules/FindPIO.cmake:184 (find_package_handle_standard_args)
CMakeLists.txt:124 (find_package)

The modules loaded at this point are:
login2.frontera(503)$ module list

Currently Loaded Modules:

miniconda/3.9.12 9) bacio/2.4.1 17) parallel-netcdf/1.12.2 25) crtm/2.3.0 33) sp/2.3.3 41) py-numpy/1.22.3 49) py-netcdf4/1.5.3 57) ufs-pyenv/1.0.0
qt5/5.14.2 10) cmake/3.22.1 18) netcdf-c/4.8.1 26) esmf/8.3.0b09 34) ip/3.3.3 42) py-cftime/1.0.3.4 50) py-bottleneck/1.3.2 58) w3nco/2.4.1
ecflow/5.8.4 11) curl/7.29.0 19) nccmp/1.9.0.1 27) fms/release-jcsda 35) ecbuild/3.6.5 43) py-cython/0.29.30 51) py-numexpr/2.7.3 59) ufs-weather-model-env/1.0.0
gcc/9.1.0 12) git/2.24.1 20) netcdf-fortran/4.5.4 28) libjpeg/2.1.0 36) gftl-shared/1.5.0 44) py-f90nml/1.4.2 52) py-six/1.16.0
stack-gcc/9.1.0 13) mvapich2/2.3 21) parallelio/2.5.4 29) jasper/2.0.32 37) yafyaml/0.5.1 45) py-mpi4py/3.1.2 53) py-python-dateutil/2.8.2
mvapich2-x/2.3 14) pkg-config/0.27.1 22) wget/1.14 30) libpng/1.6.37 38) mapl/2.22.0-esmf-8.3.0b09-esmf-8.3.0 46) py-h5py/3.6.0 54) py-pytz/2021.3
stack-mvapich2/2.3 15) zlib/1.2.12 23) base-env/1.0.0 31) g2/3.4.5 39) openblas/0.3.19 47) py-markupsafe/2.0.1 55) py-pandas/1.4.0
stack-python/3.9.12 16) hdf5/1.12.1 24) git-lfs/2.10.0 32) g2tmpl/1.10.0 40) py-setuptools/59.4.0 48) py-jinja2/3.0.3 56) py-pyyaml/6.0
Explicitly loading fms/2022.01 does not solve the problem.

I did not wipe the build directory between attempts though, so it could be that something is left over from a previous attempt? I will try and see if I can get past the FMS problem doing that after lunch.

climbfuji · 2022-07-25T17:14:04Z

climbfuji
Jul 25, 2022
Maintainer

Let's start simple. This works for me:

module purge
module use /work2/06146/tg854455/frontera/spack-stack/modulefiles
module load miniconda/3.9.12
module load ecflow/5.8.4
ulimit -s unlimited
module use /work2/06146/tg854455/frontera/spack-stack/spack-stack-v1/envs/skylab-1.0.0-gnu-9.1.0/install/modulefiles/Core
module load stack-gcc/9.1.0
module load stack-mvapich2/2.3
module load stack-python/3.9.12

module load mapl/2.22.0-esmf-8.3.0b09-esmf-8.3.0
module load fms/2022.01
module load bacio/2.4.1
module load sp/2.3.3
module load w3nco/2.4.1
module load crtm/2.3.0
module load g2/3.4.5
module load g2tmpl/1.10.0
module load ip/3.3.3
module load parallelio/2.5.4

cd /work2/06146/tg854455/frontera/spack-stack/
git clone -b develop --recursive https://github.com/ufs-community/ufs-weather-model ufs-weather-model
cd ufs-weather-model/
mkdir build
cd build

cmake -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16 -DINLINE_POST=ON .. 2>&1 | tee log.make
make VERBOSE=1 -j4 2>&1 | tee log.make

Note that the -DINLINE_POST_ON is mandatory nowadays, but not checked (one of several bugs I came across with the ufs-weather-model build system).

It does NOT work for

cmake -DAPP=S2SWA -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DINLINE_POST=ON .. 2>&1 | tee log.make

because:

Bug in FindPIO.cmake (see Bug fix in Modules/FindPIO.cmake: double periods in shared library names NOAA-EMC/CMakeModules#62)
Some other bug in the aerosol chemistry build system, complains about not finding esmf.mod even though FindESMF.cmake in the cmake step finds esmf.mk. I assume this has to do with mapl/esma using a different way to find ESMF? Didn't spend any time on this yet.

2 replies

benjamin-cash Jul 25, 2022
Author

Running through this now, so far so good.

benjamin-cash Jul 25, 2022
Author

Appears to have run successfully. :)

benjamin-cash · 2022-08-18T18:57:16Z

benjamin-cash
Aug 18, 2022
Author

Update: This no longer works for me, failing with errors like:
CMake Error at /work2/06146/tg854455/frontera/spack-stack/spack-stack-v1/envs/skylab-1.0.0 -gnu-9.1.0/install/gcc/9.1.0/cmake-3.22.1-mppepwa/share/cmake-3.22/Modules/FindPackageHand leStandardArgs.cmake:230 (message):
Could NOT find PIO (missing: C Fortran) (Required is at least version
"2.5.3")

1 reply

benjamin-cash Aug 19, 2022
Author

Turns out this was related to the issue with CMakeModules not being up to date in the repo.

benjamin-cash · 2022-08-19T21:00:24Z

benjamin-cash
Aug 19, 2022
Author

Adding some details to @climbfuji 's note above, when I try to build GOCART as part of the ufs, using spack-stack, I run into the following error:

/work2/02441/bcash/frontera/ufs-weather-model/develop/GOCART/ESMF/Shared/Chem_AeroGeneric.F90:18:7:

18 | use ESMF
| 1
Fatal Error: Cannot open module file ‘esmf.mod’ for reading at (1): No such file or directory

I can build the model in many other configurations without issue, it is specifically when I try to build with GOCART that this issue arises. Do I need to do something beyond loading the esmf model to have it found by GOCART?

0 replies

jkbk2004 · 2022-08-19T22:17:19Z

jkbk2004
Aug 19, 2022
Maintainer

@benjamin-cash PIO issue still there, right? I tried on orion using Kyle's spack-stack installation. I have same issue. With latest hash CMakeModules, it doesn't go thru.

1 reply

benjamin-cash Aug 19, 2022
Author

I got past the PIO issue - in addition to having the updated CMakeModules, you also need to edit develop/CMakeLists.txt (assuming you are using develop as your top directory). On line 130 you need to change STATIC to SHARED, so the line reads:

find_package(PIO 2.5.3 REQUIRED COMPONENTS C Fortran SHARED)

Otherwise it will look for static .a libraries that don't exist. At least, that was the issue I ran into. If that doesn't fix it post the error message here - I'm hardly an expert but I spent quite a while crawling through the code today trying to figure out what was going on and saw a bunch of different errors. :)

jkbk2004 · 2022-08-19T23:20:00Z

jkbk2004
Aug 19, 2022
Maintainer

Great! it moves further. I am building S2SWA. It hits with error: ld: cannot find -lESMF

1 reply

benjamin-cash Aug 19, 2022
Author

That's as far as I have gotten too. If you build ATM, S2S, or S2SW it should complete - the problem appears to be in the interaction with GOCART.

jkbk2004 · 2022-08-19T23:48:04Z

jkbk2004
Aug 19, 2022
Maintainer

sure! at least, good to be on same page. I will check with gocart people monday.

3 replies

benjamin-cash Aug 22, 2022
Author

Just checking to see if you have heard anything. :)-

jkbk2004 Aug 22, 2022
Maintainer

I tossed some information to the gogart side. But orion is connection issue today. I will try to follow once orion is back.

benjamin-cash Aug 22, 2022
Author

Check out the discussion here, I think we're getting close to tracking it down. GEOS-ESM/GOCART#174

benjamin-cash · 2022-08-23T16:30:52Z

benjamin-cash
Aug 23, 2022
Author

Update - I was able to successfully build something that is mostly P8 on Frontera this morning using my gnu installation of spack-stack. :) Assuming all of the necessary modules have been loaded via spack, the steps that got me to an executable are:

Check out P8 tag
Update CMakeModules to develop
Change "STATIC" to "SHARED" on line 130 of CMakeLists.txt
Add esmf to DEPENDENCIES on line 9 of GOCART/ESMF/Shared/CMakeLists.txt
mkdir build && cd build
cmake -DAPP=S2SWA -DDEBUG=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DINLINE_POST=ON .. 2>&1 | tee log.make
Delete -lESMF from CMakeFiles/ufs_model.dir/link.txt
make VERBOSE=1 -j4 2>&1 | tee log.make

4 replies

benjamin-cash Aug 23, 2022
Author

@jkbk2004

climbfuji Aug 23, 2022
Maintainer

Wow, great debugging. Clearly that procedure needs to be simplified ;-)

jkbk2004 Aug 23, 2022
Maintainer

Yeah, I want to see how much the changes can be combined in compile.sh. I will check with @DusanJovic-NOAA when he comes back next: @ChunxiZhang-NOAA.

benjamin-cash Aug 23, 2022
Author

@climbfuji - Yes, I don't know that I would recommend this as a standard procedure. ;)
@jkbk2004 - I know that the need to update CMakeModules is being tracked, so hopefully line 2 can be dropped before too long. @mathomp4 is looking into 4 and 7 and has at least one PR going to address them but we still haven't quite gotten to the bottom of it.

benjamin-cash · 2022-09-07T15:00:23Z

benjamin-cash
Sep 7, 2022
Author

I'm marking this as answered since I do have a path forward, although clearly some of those steps need to be cleaned up. ;)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Building with spack-stack? #1346

{{title}}

Replies: 11 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Building with spack-stack? #1346

benjamin-cash Jul 23, 2022

Replies: 11 comments · 14 replies

jkbk2004 Jul 23, 2022 Maintainer

benjamin-cash Aug 19, 2022 Author

climbfuji Jul 25, 2022 Maintainer

climbfuji Jul 25, 2022 Maintainer

benjamin-cash Jul 25, 2022 Author

climbfuji Jul 25, 2022 Maintainer

benjamin-cash Jul 25, 2022 Author

benjamin-cash Jul 25, 2022 Author

benjamin-cash Aug 18, 2022 Author

benjamin-cash Aug 19, 2022 Author

benjamin-cash Aug 19, 2022 Author

jkbk2004 Aug 19, 2022 Maintainer

benjamin-cash Aug 19, 2022 Author

jkbk2004 Aug 19, 2022 Maintainer

benjamin-cash Aug 19, 2022 Author

jkbk2004 Aug 19, 2022 Maintainer

benjamin-cash Aug 22, 2022 Author

jkbk2004 Aug 22, 2022 Maintainer

benjamin-cash Aug 22, 2022 Author

benjamin-cash Aug 23, 2022 Author

benjamin-cash Aug 23, 2022 Author

climbfuji Aug 23, 2022 Maintainer

jkbk2004 Aug 23, 2022 Maintainer

benjamin-cash Aug 23, 2022 Author

benjamin-cash Sep 7, 2022 Author

benjamin-cash
Jul 23, 2022

Replies: 11 comments 14 replies

jkbk2004
Jul 23, 2022
Maintainer

benjamin-cash Aug 19, 2022
Author

climbfuji
Jul 25, 2022
Maintainer

climbfuji
Jul 25, 2022
Maintainer

benjamin-cash Jul 25, 2022
Author

climbfuji
Jul 25, 2022
Maintainer

benjamin-cash Jul 25, 2022
Author

benjamin-cash Jul 25, 2022
Author

benjamin-cash
Aug 18, 2022
Author

benjamin-cash Aug 19, 2022
Author

benjamin-cash
Aug 19, 2022
Author

jkbk2004
Aug 19, 2022
Maintainer

benjamin-cash Aug 19, 2022
Author

jkbk2004
Aug 19, 2022
Maintainer

benjamin-cash Aug 19, 2022
Author

jkbk2004
Aug 19, 2022
Maintainer

benjamin-cash Aug 22, 2022
Author

jkbk2004 Aug 22, 2022
Maintainer

benjamin-cash Aug 22, 2022
Author

benjamin-cash
Aug 23, 2022
Author

benjamin-cash Aug 23, 2022
Author

climbfuji Aug 23, 2022
Maintainer

jkbk2004 Aug 23, 2022
Maintainer

benjamin-cash Aug 23, 2022
Author

benjamin-cash
Sep 7, 2022
Author