The MET_ensemble_verification_only_vx_time_lag WE2E test runs long in run_MET_GenEnsProd_vx_#VAR# task for GNU

[MichaelLueken]

Expected behavior

For the MET_ensemble_verification_only_vx_time_lag WE2E test, the run_MET_GenEnsProd_vx_#VAR# tasks should complete within two and a half hours (or even quicker).

Current behavior

In PR #1137, the run_MET_GenEnsProd_vx_UPA task is spending an additional 15 minutes per forecast lead hour in gen_ens_prod. This is causing the MET_ensemble_verification_only_vx_time_lag WE2E test to fail on Hera GNU. It should also be noted that there is an increase in the run_MET_GenEnsProd_vx_SFC task as well. Originally, it took just shy of an hour to complete, but now, it is taking two and a half hours.

Machines affected

Hera GNU (but it would likely occur while running on any machine using GNU compilers)

Steps To Reproduce

1. Clone the feature/daily_obs_tasks branch from Gerard's fork on Hera - git clone -b feature/daily_obs_tasks [email protected]:gsketefian/ufs-srweather-app.git
2. Build the SRW App using the GNU compiler - ./devbuild.sh -p=hera -c=gnu
3. Run the MET_ensemble_verification_only_vx_time_lag WE2E test - ./run_WE2E_tests.py -t MET_ensemble_verification_only_vx_time_lag -m hera -a epic -c gnu
4. See that the run_MET_GenEnsProd_vx_UPA task runs for four hours and ten minutes

Detailed Description of Fix (optional)

To move forward with merging PR #1137 to develop, the wallclock will be increased from 02:30:00 to 04:15:00 for run_MET_GenEnsProd_vx_#VAR# tasks.

Additional Information (optional)

It is currently suspected that something specific to time-lagging is occurring. For example, overlooking a time shift in the new code that calculates forecast output times or vx compare times.