merged_ptype issues in restart

[PerryShafran-NOAA]

Hello, everyone,

Restart is by and large working as it should for restart except for ptype, which I don't understand how this can be fixed.

In a full interrupted run, there is no error for ptype. Only in restart cases do I see this error. It begins with a WARNING that certain files are missing:

BEGIN: mesoscale_stats_grid2obs_create_merged_ptype.py

WARNING: Missing input files (/lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_mesoscale_nam_grid2obs_stats_00.178317615.cbqs01/gri
d2obs/METplus_output/workdirs/generate/job361/metar/regrid_data_plane/nam.20250128/regrid_data_plane_nam_t00z_metar_conusp_job361_fhr00.
nc) cannot make output file /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_mesoscale_nam_grid2obs_stats_00.178317615.cbqs01/grid2
obs/data/nam/merged_ptype/merged_ptype_metar_conusp_job361_init2025012800_fhr000.nc
END: mesoscale_stats_grid2obs_create_merged_ptype.py

Later, we see an error in the METplus run:

01/29 14:04:04.733 metplus.bd12844c (command_builder.py:264) ERROR: (command_builder.py:619) Could not find FCST_INPUT file /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_mesoscale_nam_grid2obs_stats_00.178317615.cbqs01/grid2obs/data/nam/merged_ptype/merged_ptype_metar_conusp_job361_init2025012800_fhr000.nc using template /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_mesoscale_nam_grid2obs_stats_00.178317615.cbqs01/grid2obs/data/nam/merged_ptype/merged_ptype_metar_conusp_job361_init{init?fmt=%Y%m%d%H}_fhr{lead?fmt=%3H}.nc

Why is METplus being run in this case? I'm not sure. The way the code is written now, if the job is written in the completed_jobs.txt file, then everything should be commented out, including any calls to METplus.

But when I look at the job361 job file in the restart run, I see this:

    for FHR in `seq ${FHR_START} ${FHR_INCR} ${FHR_END}`; do
        export FHR=$(printf "%02d" $FHR)
        #jobs were restarted, and the following has already run successfully
        #run_metplus.py -c /lfs/h2/emc/vpppg/noscrub/perry.shafran/EVS_mesoscale_restart/EVS/parm/metplus_config/machine.conf -c /lfs/h2/emc/vpppg/noscrub/perry.shafran/EVS_mesoscale_restart/EVS/parm/metplus_config/stats/mesoscale/grid2obs/RegridDataPlane_fcstMESOSCALE_PTYPE.conf
        #python -c "import mesoscale_util as cutil; cutil.copy_data_to_restart(\"${DATA}\", \"${RESTART_DIR}\", njob=\"361\", verif_case=\"${VERIF_CASE}\", verif_type=\"${VERIF_TYPE}\", vx_mask=\"${NEST}\", met_tool=\"regrid_data_plane\", vdate=\"${VDATE}\", vhour=\"${VHOUR}\", fhr_start=\"${FHR_START}\", fhr_end=\"${FHR_END}\", fhr_incr=\"${FHR_INCR}\", model=\"${MODELNAME}\", )"
        #python /lfs/h2/emc/vpppg/noscrub/perry.shafran/EVS_mesoscale_restart/EVS/ush/mesoscale/mesoscale_stats_grid2obs_create_merged_ptype.py
        #python -c "import mesoscale_util as cutil; cutil.copy_data_to_restart(\"${DATA}\", \"${RESTART_DIR}\", njob=\"361\", verif_case=\"${VERIF_CASE}\", verif_type=\"${VERIF_TYPE}\", vx_mask=\"${NEST}\", met_tool=\"merged_ptype\", vdate=\"${VDATE}\", vhour=\"${VHOUR}\", fhr_start=\"${FHR_START}\", fhr_end=\"${FHR_END}\", fhr_incr=\"${FHR_INCR}\", model=\"${MODELNAME}\", )"
        #run_metplus.py -c /lfs/h2/emc/vpppg/noscrub/perry.shafran/EVS_mesoscale_restart/EVS/parm/metplus_config/machine.conf -c /lfs/h2/emc/vpppg/noscrub/perry.shafran/EVS_mesoscale_restart/EVS/parm/metplus_config/stats/mesoscale/grid2obs/PointStat_fcstMESOSCALE_obsMETAR_PTYPE.conf
        #python -c "import mesoscale_util as cutil; cutil.copy_data_to_restart(\"${DATA}\", \"${RESTART_DIR}\", njob=\"361\", verif_case=\"${VERIF_CASE}\", verif_type=\"${VERIF_TYPE}\", vx_mask=\"${NEST}\", met_tool=\"point_stat\", vdate=\"${VDATE}\", vhour=\"${VHOUR}\", fhr_start=\"${FHR_START}\", fhr_end=\"${FHR_END}\", fhr_incr=\"${FHR_INCR}\", model=\"${MODELNAME}\", var_name=\"${VAR_NAME}\")"
        python /lfs/h2/emc/vpppg/noscrub/perry.shafran/EVS_mesoscale_restart/EVS/ush/mesoscale/mesoscale_stats_grid2obs_create_merged_ptype.py
        python -c "import mesoscale_util as cutil; cutil.copy_data_to_restart(\"${DATA}\", \"${RESTART_DIR}\", njob=\"361\", verif_case=\"${VERIF_CASE}\", verif_type=\"${VERIF_TYPE}\", vx_mask=\"${NEST}\", met_tool=\"merged_ptype\", vdate=\"${VDATE}\", vhour=\"${VHOUR}\", fhr_start=\"${FHR_START}\", fhr_end=\"${FHR_END}\", fhr_incr=\"${FHR_INCR}\", model=\"${MODELNAME}\", )"
        run_metplus.py -c /lfs/h2/emc/vpppg/noscrub/perry.shafran/EVS_mesoscale_restart/EVS/parm/metplus_config/machine.conf -c /lfs/h2/emc/vpppg/noscrub/perry.shafran/EVS_mesoscale_restart/EVS/parm/metplus_config/stats/mesoscale/grid2obs/PointStat_fcstMESOSCALE_obsMETAR_PTYPE.conf
        python -c "import mesoscale_util as cutil; cutil.copy_data_to_restart(\"${DATA}\", \"${RESTART_DIR}\", njob=\"361\", verif_case=\"${VERIF_CASE}\", verif_type=\"${VERIF_TYPE}\", vx_mask=\"${NEST}\", met_tool=\"point_stat\", vdate=\"${VDATE}\", vhour=\"${VHOUR}\", fhr_start=\"${FHR_START}\", fhr_end=\"${FHR_END}\", fhr_incr=\"${FHR_INCR}\", model=\"${MODELNAME}\", var_name=\"${VAR_NAME}\")"
        python -c 'import mesoscale_util; mesoscale_util.mark_job_completed("/lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_mesoscale_nam_grid2obs_stats_00.178317615.cbqs01/grid2obs/completed_jobs.txt", "job361", job_type="generate")'
        if [ -f /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_mesoscale_nam_grid2obs_stats_00.178317615.cbqs01/grid2obs/completed_jobs.txt ] && [ $SENDCOM == YES ]; then cp -rpfv /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_mesoscale_nam_grid2obs_stats_00.178317615.cbqs01/grid2obs/completed_jobs.txt /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/stats/mesoscale/atmos.20250128/nam/grid2obs/restart/c07; fi

You can see that there are some jobs that are not commented out. Jobs are supposed to be commented out (thus not run) when the item exists in the completed_jobs.txt file (and the job is in the completed text file - 6 times actually).

It should also be noted that at the start of the code, a whole bunch of PTYPE stat files from the restart directory are copied over to the working directory, so it's clear that ptype data had been done in the interrupted run and copied over, thus no need to do any more ptype processing. The final stat file for the restart job is identical to the final stat file of the full, uninterrupted job. The question is why any of the uncommented jobs are listed in the job file - that would only be the case if job361 didn't exist in the completed_jobs.txt file.

@MarcelCaron-NOAA @malloryprow

[malloryprow]

I won't be able to help until after the ecFlow training, but please update here on any progress you make.

[PerryShafran-NOAA]

Thank you Mallory. Unfortunately, I didn't make much progress on this and am still confused how and why some of these jobs were allowed to run even though the job was in completed status before restart and thus should have not run. We can circle around to this when you are next available.

[malloryprow]

Okay! Please keep looking at it today. There is another ecFlow training today.

[PerryShafran-NOAA]

I think I may have found the problem. Indentations were all messed up, so I think the items that don't have the # were uncommented and ran because they were outside of the if block. I'm trying to see if I can fix the indentations in the ptype section.

[PerryShafran-NOAA]

We can close this issue.