Cannot aggregate ramip noresm data

[bnlawrence]

I get the expected results for Amon from CanESM:

$python cfamip.py CanESM5-1_proc/CanESM5-1_r4i1p1f1_Amon_input_files.txt 
[<CF Field: cloud_area_fraction_in_atmosphere_layer(time(444), atmosphere_hybrid_sigma_pressure_coordinate(49), latitude(64), longitude(128)) %>,
 <CF Field: atmosphere_mass_content_of_cloud_ice(time(444), latitude(64), longitude(128)) kg m-2>,
 <CF Field: cloud_area_fraction(time(444), latitude(64), longitude(128)) %>,
 <CF Field: atmosphere_mass_content_of_cloud_condensed_water(time(444), latitude(64), longitude(128)) kg m-2>,
 <CF Field: water_evapotranspiration_flux(time(444), latitude(64), longitude(128)) kg m-2 s-1>,
 <CF Field: surface_upward_latent_heat_flux(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: surface_upward_sensible_heat_flux(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: relative_humidity(time(444), latitude(64), longitude(128)) %>,
 <CF Field: specific_humidity(time(444), air_pressure(19), latitude(64), longitude(128)) 1>,
 <CF Field: precipitation_flux(time(444), latitude(64), longitude(128)) kg m-2 s-1>,
 <CF Field: snowfall_flux(time(444), latitude(64), longitude(128)) kg m-2 s-1>,
 <CF Field: surface_air_pressure(time(444), latitude(64), longitude(128)) Pa>,
 <CF Field: air_pressure_at_mean_sea_level(time(444), latitude(64), longitude(128)) Pa>,
 <CF Field: surface_downwelling_longwave_flux_in_air(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: surface_downwelling_longwave_flux_in_air_assuming_clear_sky(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: surface_upwelling_longwave_flux_in_air(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: toa_outgoing_longwave_flux(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: toa_outgoing_longwave_flux_assuming_clear_sky(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: surface_downwelling_shortwave_flux_in_air(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: surface_downwelling_shortwave_flux_in_air_assuming_clear_sky(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: toa_incoming_shortwave_flux(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: surface_upwelling_shortwave_flux_in_air(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: surface_upwelling_shortwave_flux_in_air_assuming_clear_sky(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: toa_outgoing_shortwave_flux(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: toa_outgoing_shortwave_flux_assuming_clear_sky(time(444), latitude(64), longitude(128)) W m-2>,
 <CF Field: air_temperature(time(444), air_pressure(19), latitude(64), longitude(128)) K>,
 <CF Field: air_temperature(time(444), latitude(64), longitude(128)) K>,
 <CF Field: air_temperature(time(444), latitude(64), longitude(128)) K>,
 <CF Field: air_temperature(time(444), latitude(64), longitude(128)) K>,
 <CF Field: surface_temperature(time(444), latitude(64), longitude(128)) K>,
 <CF Field: eastward_wind(time(444), air_pressure(19), latitude(64), longitude(128)) m s-1>,
 <CF Field: eastward_wind(time(444), latitude(64), longitude(128)) m s-1>,
 <CF Field: northward_wind(time(444), air_pressure(19), latitude(64), longitude(128)) m s-1>,
 <CF Field: northward_wind(time(444), latitude(64), longitude(128)) m s-1>,
 <CF Field: lagrangian_tendency_of_air_pressure(time(444), air_pressure(19), latitude(64), longitude(128)) Pa s-1>,
 <CF Field: geopotential_height(time(444), air_pressure(19), latitude(64), longitude(128)) m>]
Read 36 fields in 373.52s
Aggregation to 36 in 0.07s
Writing cfa file took 173.69s

but we get a crash with NorESM ...

python cfamip.py NorESM2-LM_proc/NorESM2-LM_r4i1p1f1_Amon_input_files.txt 
[<CF Field: cloud_area_fraction_in_atmosphere_layer(time(540), atmosphere_hybrid_sigma_pressure_coordinate(32), latitude(96), longitude(144)) %>,
 <CF Field: atmosphere_mass_content_of_cloud_ice(time(540), latitude(96), longitude(144)) kg m-2>,
 <CF Field: cloud_area_fraction(time(540), latitude(96), longitude(144)) %>,
 <CF Field: atmosphere_mass_content_of_cloud_condensed_water(time(540), latitude(96), longitude(144)) kg m-2>,
 <CF Field: water_evapotranspiration_flux(time(540), latitude(96), longitude(144)) kg m-2 s-1>,
 <CF Field: surface_upward_latent_heat_flux(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: surface_upward_sensible_heat_flux(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: relative_humidity(time(540), latitude(96), longitude(144)) %>,
 <CF Field: specific_humidity(time(540), air_pressure(19), latitude(96), longitude(144)) 1>,
 <CF Field: atmosphere_net_upward_convective_mass_flux(time(540), atmosphere_hybrid_sigma_pressure_coordinate(33), latitude(96), longitude(144)) kg m-2 s-1>,
 <CF Field: precipitation_flux(time(540), latitude(96), longitude(144)) kg m-2 s-1>,
 <CF Field: snowfall_flux(time(540), latitude(96), longitude(144)) kg m-2 s-1>,
 <CF Field: surface_air_pressure(time(540), latitude(96), longitude(144)) Pa>,
 <CF Field: air_pressure_at_mean_sea_level(time(540), latitude(96), longitude(144)) Pa>,
 <CF Field: surface_downwelling_longwave_flux_in_air(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: surface_downwelling_longwave_flux_in_air_assuming_clear_sky(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: toa_outgoing_longwave_flux(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: toa_outgoing_longwave_flux_assuming_clear_sky(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: surface_downwelling_shortwave_flux_in_air(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: surface_downwelling_shortwave_flux_in_air_assuming_clear_sky(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: toa_incoming_shortwave_flux(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: surface_upwelling_shortwave_flux_in_air(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: toa_outgoing_shortwave_flux(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: toa_outgoing_shortwave_flux_assuming_clear_sky(time(540), latitude(96), longitude(144)) W m-2>,
 <CF Field: air_temperature(time(540), air_pressure(19), latitude(96), longitude(144)) K>,
 <CF Field: air_temperature(time(540), latitude(96), longitude(144)) K>,
 <CF Field: air_temperature(time(540), latitude(96), longitude(144)) K>,
 <CF Field: air_temperature(time(540), latitude(96), longitude(144)) K>,
 <CF Field: surface_temperature(time(540), latitude(96), longitude(144)) K>,
 <CF Field: eastward_wind(time(540), air_pressure(19), latitude(96), longitude(144)) m s-1>,
 <CF Field: eastward_wind(time(540), latitude(96), longitude(144)) m s-1>,
 <CF Field: northward_wind(time(540), air_pressure(19), latitude(96), longitude(144)) m s-1>,
 <CF Field: northward_wind(time(540), latitude(96), longitude(144)) m s-1>,
 <CF Field: lagrangian_tendency_of_air_pressure(time(540), air_pressure(19), latitude(96), longitude(144)) Pa s-1>,
 <CF Field: geopotential_height(time(540), air_pressure(19), latitude(96), longitude(144)) m>]
Read 35 fields in 117.31s
Aggregation to 35 in 0.07s
Traceback (most recent call last):
  File "/gws/nopw/j04/hiresgw/ramip/cfamip.py", line 37, in <module>
    filelist2cfa(flist)
  File "/gws/nopw/j04/hiresgw/ramip/cfamip.py", line 29, in filelist2cfa
    cf.write(fewer_fields,cfa_file, cfa={'absolute_paths':False,
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cf/read_write/write.py", line 808, in write
    netcdf.write(
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4919, in write
    self._file_io_iteration(
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5192, in _file_io_iteration
    self._write_field_or_domain(f)
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4037, in _write_field_or_domain
    self._write_netcdf_variable(
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 2755, in _write_netcdf_variable
    kwargs = self._customise_createVariable(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cf/read_write/netcdf/netcdfwrite.py", line 155, in _customise_createVariable
    if self._write_as_cfa(cfvar, construct_type, domain_axes):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/users/lawrence/.conda/envs/mampy24b/lib/python3.12/site-packages/cf/read_write/netcdf/netcdfwrite.py", line 105, in _write_as_cfa
    raise ValueError(
ValueError: Can't write <CF Field: cloud_area_fraction_in_atmosphere_layer(time(540), atmosphere_hybrid_sigma_pressure_coordinate(32), latitude(96), longitude(144)) %> as a CFA-netCDF aggregation variable. Possible reasons for this include 1) there is more than one Dask chunk per fragment, and 2) data values have been changed relative to those in the fragments.

Obviously the crash is an issue, but why are there 36 in one and 35 in the other? What is the difference? What does RAMIP want?

[bnlawrence]

Looks like NorESM doesn' t have atmosphere_net_upward_convective_mass_flux? That might just be what they can write out.

[bnlawrence]

To bypass the problem, a work around is to read the file with chunks=None, and then write it out, but surely this is a bug (I know @davidhassell and I have talked about this before, but I can't remember what we decided.)