UFS DA cycling on gaea

[jswhit]

What new functionality do you need?

support for DA cycling on gaea c5 (and also c6). After cloning and building on gaea c5, I setup an experiment and started it. The workflow ran successfully through the first forecast, but died in the first da related setup (marinebmat). Looking at the log I see "WARNING: UFSDA NOT SUPPORTED ON THIS PLATFORM". What will it take to this capability working? I've pre-staged the observations and initial conditions I will need.

BTW: The marinebmat error that follows the warning message is

++ jjob_header.sh[79]: cycle=t06z
++ jjob_header.sh[80]: setpdy.sh
/gpfs/f5/nggps_psd/proj-shared/Jeffrey.S.Whitaker/global-workflow/ush/jjob_header.sh: line 80: setpdy.sh: command not found
+ jjob_header.sh[1]: postamble JGLOBAL_MARINE_BMAT 1728083449 127

What are the requirements for the new functionality?

I don't know

Acceptance Criteria

coupled DA cycling working on at least gaea c5, and ultimately c6 also

Suggest a solution (optional)

No response

[jswhit]

@AnilKumar-NOAA

[jswhit]

made some progress by modifying env/GAEA.env to match env/HERCULES.env, and by editing ush/load_ufsda_modules.sh to remove gaea from the 'unsupported' branch. Now failing in gdasmarineanlvar when executing marinevariational/gdas.x.

[AnilKumar-NOAA]

We have not tested DA part yet on gaea c5 system. It's good to know that you did attempt and tested DA on gaea c5. gdas sometime create issues. Will check on this on my side as well. Thanks

[jswhit]

I've gotten past the crash in gdasmarineanlvar with a one line fix to scripts/exglobal_marine_analysis_initialize.py (PR from @guillaumevernieres forthcoming). However, the gdasmarineanlvar fails to finish, even if the wall clock limit is increased to from 15 mins to 1 hour. Can't tell if it's hung or just runs very slow on gaea.

[CoryMartin-NOAA]

FYI I'm working to add C6 support to GDASApp NOAA-EMC/GDASApp#1323

[guillaumevernieres]

Thanks Cory. FYI: NOAA-EMC/GDASApp#1322 I'm not 100% sure that it is compiler related, but we're having memory issues with intel 2021.9.0 , used on hercules and gaea.

…

--- Guillaume Vernieres

On Fri, Oct 11, 2024 at 10:16 AM Cory Martin ***@***.***> wrote: FYI I'm working to add C6 support to GDASApp NOAA-EMC/GDASApp#1323 <NOAA-EMC/GDASApp#1323> — Reply to this email directly, view it on GitHub <#2985 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLBX4ACLNDKNGQ5GOB6OGDZ27MUVAVCNFSM6AAAAABPMYHHCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBXGUYTCMZSGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[CoryMartin-NOAA]

@guillaumevernieres on Gaea C6 it looks like the compiler is intel-classic/2023.2.0, so perhaps we can see if the memory usage is problematic on these nodes

[jswhit2]

PR #3032 enables cycling on gaea c5.