Update GSI analysis jobs to use COMIN/COMOUT #3092

mingshichen-noaa · 2024-11-12T20:19:02Z

Description

NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global jgdas enkf select obs job and relevant .JGLOBAL_ATMOS_ANALYSIS and JGLOBAL_ATMOS_ANALYSIS_CALC jobs.

Refs #2451

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Clone and build on RDHPCS
Cycled tests on Hercules
Forecast-only tests on Hercules

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

…lect_obs_com_in_out

jobs/JGDAS_ENKF_SELECT_OBS

aerorahul

looks good. I am no longer familiar w/ the IN and OUT of EnKF related data, so would appreciate a review from @CatherineThomas-NOAA @RussTreadon-NOAA

KateFriedman-NOAA

Looks good! Please resolve the shellcheck warning in jobs/JGDAS_ENKF_SELECT_OBS and then I can approve.

jobs/JGDAS_ENKF_SELECT_OBS

aerorahul

looks good to me.

KateFriedman-NOAA

Looks good, thanks @mingshichen-noaa !

WalterKolczynski-NOAA · 2024-12-02T21:44:02Z

ush/gaussian_sfcanl.sh

-[[ -d "${COM_ATMOS_ANALYSIS}" ]] || mkdir -p "${COM_ATMOS_ANALYSIS}"
-[[ -d "${COM_ATMOS_RESTART}" ]] || mkdir -p "${COM_ATMOS_RESTART}"
+[[ -d "${COMOUT_ATMOS_ANALYSIS}" ]] || mkdir -p "${COMOUT_ATMOS_ANALYSIS}"
+[[ -d "${COMIN_ATMOS_RESTART}" ]] || mkdir -p "${COMIN_ATMOS_RESTART}"


Shouldn't be creating an input directory.

Suggested change

[[ -d "${COMIN_ATMOS_RESTART}" ]] || mkdir -p "${COMIN_ATMOS_RESTART}"

mingshichen-noaa · 2024-12-03T14:27:11Z

@WalterKolczynski-NOAA
If I change line 137 into [[ -d "${COMOUT_ATMOS_RESTART}" ]] || mkdir -p "${COMOUT_ATMOS_RESTART}", that seems to conflict with line 80:
80 # input data : ${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile*.nc

WalterKolczynski-NOAA · 2024-12-03T14:41:35Z

@WalterKolczynski-NOAA If I change line 137 into [[ -d "${COMOUT_ATMOS_RESTART}" ]] || mkdir -p "${COMOUT_ATMOS_RESTART}", that seems to conflict with line 80: 80 # input data : ${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile*.nc

How so? If it is input data, the directory already exists and does not need to be created.

WalterKolczynski-NOAA · 2024-12-03T17:48:39Z

Changing that one line from COMIN to COMOUT clearly isn't correct:

COMOUT_ATMOS_RESTART isn't defined
You only changed the one instance, so you're making a directory that is never used
The comments are clear about it being an input directory, so you would need to show me where it is being written to in order to have a COMOUT version

mingshichen-noaa · 2024-12-03T18:59:42Z

@WalterKolczynski-NOAA
you are right. I should remove the line 137 because COMIN_ATMOS_RESTART is an input directory declared at line 36 of jobs/JGLOBAL_ATMOS_ANALYSIS_CALC file.

WalterKolczynski-NOAA · 2024-12-03T20:01:58Z

@mingshichen-noaa can you point me to your experiment directories where you tested this PR?

mingshichen-noaa · 2024-12-03T20:44:50Z

@WalterKolczynski-NOAA
Today I have made some modification on usr/gaussian_sfcanl.sh. Just now I cd sorc and ran ./build_all.sh -gu again. The recent test results are at
/work2/noaa/global/mchen/test/jobs/JGDAS_ENKF_SELECT_OBS/EXPDIR/C96C48_hybatmDA
platform is Hercules.

WalterKolczynski-NOAA · 2024-12-04T19:54:25Z

@WalterKolczynski-NOAA Today I have made some modification on usr/gaussian_sfcanl.sh. Just now I cd sorc and ran ./build_all.sh -gu again. The recent test results are at /work2/noaa/global/mchen/test/jobs/JGDAS_ENKF_SELECT_OBS/EXPDIR/C96C48_hybatmDA platform is Hercules.

This experiment is not progressing. Did you put rocotorun into your crontab?

mingshichen-noaa · 2024-12-04T20:27:56Z

@WalterKolczynski-NOAA
Yesterday I made manually rocotorun and got results. Today I did crontab -e as follows:
MAILTO="[email protected]"
*/5 * * * * /apps/contrib/rocoto/1.3.7/bin/rocotorun -d /work2/noaa/global/mchen/test/jobs/JGDAS_ENKF_SELECT_OBS/EXPDIR/C96C48_hybatmDA/C96C48_hybatmDA.db -w /work2/noaa/global/mchen/test/jobs/JGDAS_ENKF_SELECT_OBS/EXPDIR/C96C48_hybatmDA/C96C48_hybatmDA.xml

WalterKolczynski-NOAA · 2024-12-04T20:40:33Z

@WalterKolczynski-NOAA Yesterday I made manually rocotorun and got results. Today I did crontab -e as follows: MAILTO="[email protected]" */5 * * * * /apps/contrib/rocoto/1.3.7/bin/rocotorun -d /work2/noaa/global/mchen/test/jobs/JGDAS_ENKF_SELECT_OBS/EXPDIR/C96C48_hybatmDA/C96C48_hybatmDA.db -w /work2/noaa/global/mchen/test/jobs/JGDAS_ENKF_SELECT_OBS/EXPDIR/C96C48_hybatmDA/C96C48_hybatmDA.xml

Looks like it is running now.

aerorahul · 2024-12-09T21:30:14Z

jobs/JGLOBAL_ATMOS_ANALYSIS_CALC


-YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_ANALYSIS COM_ATMOS_RESTART
+YMD=${PDY} HH=${cyc} declare_from_tmpl -rx 


@mingshichen-noaa
This line is causing the failure. Please fix

@aerorahul

After I checked codes in jobs/JGLOBAL_ATMOS_ANALYSIS_CALC from mingshichen-noaa:feature/jgdas_enkf_select_obs_com_in_out, the codes had been modified as follows:
30
31 RUN=${rCDUMP} YMD=${PDY} HH=${cyc} declare_from_tmpl -rx
32 COMIN_OBS:COM_OBS_TMPL
33
34 YMD=${PDY} HH=${cyc} declare_from_tmpl -rx
35 COMOUT_ATMOS_ANALYSIS:COM_ATMOS_ANALYSIS_TMPL
36 COMIN_ATMOS_RESTART:COM_ATMOS_RESTART_TMPL
37
38 RUN=${GDUMP} YMD=${gPDY} HH=${gcyc} declare_from_tmpl -rx
39 COMIN_OBS_PREV:COM_OBS_TMPL
40 COMIN_ATMOS_HISTORY_PREV:COM_ATMOS_HISTORY_TMPL
41

mingshichen-noaa · 2024-12-10T14:39:49Z

@aerorahul
Do you think whether I should make a change on line 39 in jobs/JGDAS_ENKF_SELECT_OBS
from
39 declare -rx COMOUT_ATMOS_ANALYSIS_ENS="${COMOUT_ATMOS_ANALYSIS}"
into
39 declare -rx COMOUT_ATMOS_ANALYSIS_ENS:COM_ATMOS_ANALYSIS_TMPL

aerorahul · 2024-12-10T15:13:32Z

jobs/JGLOBAL_ATMOS_ANALYSIS_CALC


-YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_ANALYSIS COM_ATMOS_RESTART
+YMD=${PDY} HH=${cyc} declare_from_tmpl -rx 


Suggested change

YMD=${PDY} HH=${cyc} declare_from_tmpl -rx

YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \

This is the fix needed.

@aerorahul
I have changed line 34 as follows:
34 YMD=${PDY} HH=${cyc} declare_from_tmpl -rx
35 COMOUT_ATMOS_ANALYSIS:COM_ATMOS_ANALYSIS_TMPL
36 COMIN_ATMOS_RESTART:COM_ATMOS_RESTART_TMPL

@aerorahul
My changes are located the branch: mingshichen-noaa:feature/jgdas_enkf_select_obs_com_in_out

please push them

WalterKolczynski-NOAA · 2024-12-10T17:28:12Z

@mingshichen-noaa Please rewind the dead jobs in your experiment so it can continue with the fix.

Mingshi Chen added 2 commits November 8, 2024 15:25

changing COM_ into COMIN_ or COMOUT_

fcd075e

Merge remote-tracking branch 'emc/develop' into feature/jgdas_enkf_se…

c1d9f8c

…lect_obs_com_in_out

mingshichen-noaa requested review from RussTreadon-NOAA, CoryMartin-NOAA and CatherineThomas-NOAA as code owners November 12, 2024 20:19

mingshichen-noaa removed request for CoryMartin-NOAA, RussTreadon-NOAA and CatherineThomas-NOAA November 12, 2024 20:19

github-advanced-security bot found potential problems Nov 12, 2024

View reviewed changes

jobs/JGDAS_ENKF_SELECT_OBS Fixed Show resolved Hide resolved

mingshichen-noaa requested review from WalterKolczynski-NOAA, KateFriedman-NOAA and DavidHuber-NOAA November 12, 2024 20:20

mingshichen-noaa self-assigned this Nov 12, 2024

mingshichen-noaa added the CI/CD Issue related to CI/CD label Nov 12, 2024

aerorahul reviewed Nov 12, 2024

View reviewed changes

jobs/JGDAS_ENKF_SELECT_OBS Outdated Show resolved Hide resolved

aerorahul reviewed Nov 12, 2024

View reviewed changes

aerorahul requested review from RussTreadon-NOAA, CatherineThomas-NOAA and DavidGrumm-NOAA November 12, 2024 20:30

don't need -m 755 in mkdir

1ff5591

KateFriedman-NOAA reviewed Nov 13, 2024

View reviewed changes

DavidHuber-NOAA reviewed Nov 13, 2024

View reviewed changes

jobs/JGDAS_ENKF_SELECT_OBS Outdated Show resolved Hide resolved

TerrenceMcGuinness-NOAA removed the CI/CD Issue related to CI/CD label Nov 13, 2024

remove trailing space on line 37

677b8ca

aerorahul previously approved these changes Nov 13, 2024

View reviewed changes

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

01216a5

KateFriedman-NOAA previously approved these changes Nov 14, 2024

View reviewed changes

WalterKolczynski-NOAA previously approved these changes Nov 15, 2024

View reviewed changes

WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Nov 15, 2024

emcbot removed the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Nov 15, 2024

mingshichen-noaa added 3 commits November 26, 2024 15:10

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

594f983

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

0303a9c

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

4d94381

WalterKolczynski-NOAA requested changes Dec 2, 2024

View reviewed changes

changing COMIN_ATMOS_RESTART into COMOUT_ATMOS_RESTART

0826c3e

mingshichen-noaa dismissed KateFriedman-NOAA’s stale review via 0826c3e December 3, 2024 14:46

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

67acf61

remove line 137 because COMIN_ATMOS_RESTART is an input directory

fa937ef

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

18eb1f4

mingshichen-noaa added 3 commits December 5, 2024 11:22

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

4c05e20

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

4329a19

Merge branch 'develop' into feature/jgdas_enkf_select_obs_com_in_out

bc771e9

aerorahul reviewed Dec 9, 2024

View reviewed changes

aerorahul reviewed Dec 10, 2024

View reviewed changes

missing \

df83c7b

WalterKolczynski-NOAA changed the title ~~Update JGDAS ENKF SELECT OBS JGLOBAL_ATMOS_ANALYSIS and JGLOBAL_ATMOS_ANALYSIS_CALC jobs~~ Update GSI analysis jobs to use COMIN/COMOUT Dec 10, 2024

WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Dec 11, 2024

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Dec 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update GSI analysis jobs to use COMIN/COMOUT #3092

Update GSI analysis jobs to use COMIN/COMOUT #3092

mingshichen-noaa commented Nov 12, 2024 •

edited

Loading

aerorahul left a comment

KateFriedman-NOAA left a comment

aerorahul left a comment

KateFriedman-NOAA left a comment

WalterKolczynski-NOAA Dec 2, 2024

mingshichen-noaa commented Dec 3, 2024

WalterKolczynski-NOAA commented Dec 3, 2024

WalterKolczynski-NOAA commented Dec 3, 2024

mingshichen-noaa commented Dec 3, 2024

WalterKolczynski-NOAA commented Dec 3, 2024

mingshichen-noaa commented Dec 3, 2024

WalterKolczynski-NOAA commented Dec 4, 2024

mingshichen-noaa commented Dec 4, 2024

WalterKolczynski-NOAA commented Dec 4, 2024

aerorahul Dec 9, 2024

mingshichen-noaa Dec 10, 2024

mingshichen-noaa commented Dec 10, 2024

aerorahul Dec 10, 2024

mingshichen-noaa Dec 10, 2024

mingshichen-noaa Dec 10, 2024

aerorahul Dec 10, 2024

WalterKolczynski-NOAA commented Dec 10, 2024


		YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_ANALYSIS COM_ATMOS_RESTART
		YMD=${PDY} HH=${cyc} declare_from_tmpl -rx

Update GSI analysis jobs to use COMIN/COMOUT #3092

Are you sure you want to change the base?

Update GSI analysis jobs to use COMIN/COMOUT #3092

Conversation

mingshichen-noaa commented Nov 12, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

aerorahul left a comment

Choose a reason for hiding this comment

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA Dec 2, 2024

Choose a reason for hiding this comment

mingshichen-noaa commented Dec 3, 2024

WalterKolczynski-NOAA commented Dec 3, 2024

WalterKolczynski-NOAA commented Dec 3, 2024

mingshichen-noaa commented Dec 3, 2024

WalterKolczynski-NOAA commented Dec 3, 2024

mingshichen-noaa commented Dec 3, 2024

WalterKolczynski-NOAA commented Dec 4, 2024

mingshichen-noaa commented Dec 4, 2024

WalterKolczynski-NOAA commented Dec 4, 2024

aerorahul Dec 9, 2024

Choose a reason for hiding this comment

mingshichen-noaa Dec 10, 2024

Choose a reason for hiding this comment

mingshichen-noaa commented Dec 10, 2024

aerorahul Dec 10, 2024

Choose a reason for hiding this comment

mingshichen-noaa Dec 10, 2024

Choose a reason for hiding this comment

mingshichen-noaa Dec 10, 2024

Choose a reason for hiding this comment

aerorahul Dec 10, 2024

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Dec 10, 2024

mingshichen-noaa commented Nov 12, 2024 •

edited

Loading