FV3 nesting incompatible with stochastic physics due to tile size equality assumption

[SamuelTrahanNOAA]

When using FV3 nesting, the tiles can be different sizes. It appears the cellular automata code makes the assumption that all tiles have the same size. That is rarely true for nested configurations. The configuration I'm testing is a global model with a static nest. That's seven tiles: six global tiles of equal size, and a seventh tile that has far more gridpoints.

Presently, I don't have a test case you can easily run to reproduce this problem because the model will fail for a great many other reasons first (such as NOAA-EMC/fv3atm#797 and NOAA-GFDL/GFDL_atmos_cubed_sphere#328). I'm hoping to have a test case soon for debugging with a branch that has fixes and workarounds for all other problems.

This is the failure. It happens only on the nest:

 977: forrtl: severe (408): fort: (2): Subscript #1 of the array CONDITIONGRID has value 43 which is greater than the upper bound of 42
 977:
 977: Image              PC                Routine            Line        Source
 977: ufs_model.x        00000000104432EF  Unknown               Unknown  Unknown
 977: ufs_model.x        000000000D1D6D92  cellular_automata         232  cellular_automata_sgs.F90
 977: ufs_model.x        00000000045FDB33  stochastic_physic         400  stochastic_physics_wrapper.F90
 977: ufs_model.x        00000000044812D5  atmos_model_mod_m         297  atmos_model.F90
 977: ufs_model.x        0000000003F72F14  module_fcst_grid_        1308  module_fcst_grid_comp.F90

Line 232 of cellular_automata_sgs.F90 is here:

!Initialize the CA when the condition field is populated
  do j=1,nyc
   do i=1,nxc
     condition(i,j)=conditiongrid(inci/ncells,incj/ncells)  ! <------ FAILS ON THIS LINE
     uhigh(i,j)=uwindi(inci/ncells,incj/ncells)
     vhigh(i,j)=vwindi(inci/ncells,incj/ncells)
     dxhigh(i,j)=dxi(inci/ncells,incj/ncells)/ncells !dx on the finer grid
     if(i.eq.inci)then
     inci=inci+ncells
     endif
   enddo
   inci=ncells
   if(j.eq.incj)then
   incj=incj+ncells
   endif
  enddo

The stochastic_physics_wrapper.F90 line 400 is calling stochastic_physics_wrapper.F90 and sending the information for the current tile Atm(mygrid)

         call cellular_automata_sgs(GFS_Control%kdt,GFS_control%dtp,GFS_control%restart,GFS_Control%first_time_step,              &
            sst,lmsk,lake,uwind,vwind,height,dx,condition,ca_deep_cpl,ca_turb_cpl,ca_shal_cpl, Atm(mygrid)%domain_for_coupler,nblks,      &
            Atm_block%isc,Atm_block%iec,Atm_block%jsc,Atm_block%jec,Atm(mygrid)%npx,Atm(mygrid)%npy, levs,                        &
            GFS_Control%nthresh,GFS_Control%tile_num,GFS_Control%nca,GFS_Control%ncells,GFS_Control%nlives,                       &
            GFS_Control%nfracseed, GFS_Control%nseed,GFS_Control%iseed_ca,GFS_Control%ca_advect,                                  &
            GFS_Control%nspinup,GFS_Control%ca_trigger,Atm_block%blksz(1),GFS_Control%master,GFS_Control%communicator)

[pjpegion]

It seems to be that the CA should only run on the global domain, and the nest should get the same values as the global domain. Lisa is on vacation right now, but I will talk to her to get her thoughts.

[SamuelTrahanNOAA]

What do you mean by this?

nest should get the same values are the global domain

[pjpegion]

I had a typo. I am saying that I don't think a separate CA should not be running inside the nest. But @lisa-bengtsson would know better.

[lisa-bengtsson]

@SamuelTrahanNOAA it would require some development. 7 tiles is currently not supported, neither is running tiles with different size, with the CA. Could you turn it off at a high level if do_ca = .true. and a nested configuration is running? I would need some time to develop this capability.

[SamuelTrahanNOAA]

At present, in the nested global workflows I'm using (global-workflow and HAFS), turning on nesting automatically turns off CA.