Implementation of UGRID (part 1 of 2)

[davidhassell]

Fixes #270

Changes

New data model constructs that describe meshes

• core/cellconnectivity.py
• core/domaintopology.py
• cellconnectivity.py
• domaintopology.py

Misc.

• auxiliarycoordinate.py Storage of netCDF node coordinate variable name
• cfdmimplementation.py Support for the new constructs and their (compressed) data
• constructs.py Support for the new constructs. Includes a refactoring to share code between four similar filter_by_* methods.
• core/cellmeasure.py Typo
• core/constructs.py Support for the new constructs
• core/domain.py Support for the new constructs
• [core/field.py]((https://github.com/NCAS-CMS/cfdm/pull/272/files#diff-f1a9cb533d24b549f3cdb562875ace5856147c5e55177e476513d6d88e282eaf) Support for the new constructs
• domain.py Support for the new constructs: print, creation_commands
• examplefields.py Three new UGRID example fields
• field.py Support for the new constructs: mesh_id
• functions.py new integer_dtype function (return the smallest data type that can store the given integer)

Data

A whole bunch of changes for representing mesh arrays found in UGRID netCDF files, following the compressed array framework

mesh_id

The new recording of a "mesh_id" (netcdfready.py, fielddomain.py, field.py) is forward looking to the time when we write these things out (see #271). However, it's possible that when it comes to it we find that we don't need it.

As a result it is deliberate that, for now, the various *_mesh_id methods remain undocumented in cfdm/docs/

[davidhassell]

A big ol' PR this one - I shall add guidance before requesting a review!

[sadielbartholomew]

I'm mid-review, though probably not even half-way yet in terms of eyeballing the code and functionality checks. Only minor comments (mostly typos) so far. Possibly #272 (comment) is slightly more important as the logic might be wrong on one line.

(Submitting this intermediate review since I am in the office and might be WFH early next week and even though GitHub might be clever preserve the review state in terms of notably comments raised, I don't want to risk losing the comments.)

[sadielbartholomew]

Some of my new, and (for some strange reason) old, comments are marked as 'pending' so just submitting a intermediate review to get those to appear. I'll submit my full final review later today.

[sadielbartholomew]

Just doing final high-level checks, but this is looking all very good. For now I will submit more minor comments, raised in-line. Feel free to address these whist I finish my reviewing, though I won't take long (by tomorrow morning I will be done).

Also:

• a few of my comments pertain to packaging and how scipy should be dealt with in cfdm and downstream in cf-python;
• I've noticed there are some source class .rst issues meaning that the Sphinx build fails early. Some changes that need to be made include:

1. Typos:

diff --git a/docs/source/class/cfdm.GatheredArray.rst b/docs/source/class/cfdm.GatheredArray.rst
index 0f6d60aa9..96cd5cb96 100644
--- a/docs/source/class/cfdm.GatheredArray.rst
+++ b/docs/source/class/cfdm.GatheredArray.rst
@@ -101,7 +101,7 @@ Docstring substitutions
    :toctree: ../method/                   
    :template: method.rst                  
                                           
-   ~cfdm.GatherArray._docstring_special_substitutions
-   ~cfdm.GatherArray._docstring_substitutions        
-   ~cfdm.GatherArray._docstring_package_depth        
-   ~cfdm.GatherArray._docstring_method_exclusions    
+   ~cfdm.GatheredArray._docstring_special_substitutions
+   ~cfdm.GatheredArray._docstring_substitutions        
+   ~cfdm.GatheredArray._docstring_package_depth        
+   ~cfdm.GatheredArray._docstring_method_exclusions    

2. More typos:

diff --git a/docs/source/class/cfdm.core.Bounds.rst b/docs/source/class/cfdm.core.Bounds.rst
index a827601e2..2ec0e463b 100644
--- a/docs/source/class/cfdm.core.Bounds.rst
+++ b/docs/source/class/cfdm.core.Bounds.rst
@@ -92,7 +92,7 @@ Docstring substitutions
    :toctree: ../method/                        
    :template: method.rst                       
                                                
-   ~cfdm.core.Bounds_special_substitutions
-   ~cfdm.core.Bounds_substitutions        
-   ~cfdm.core.Bounds_package_depth        
-   ~cfdm.core.Bounds_method_exclusions    
+   ~cfdm.core.Bounds._special_substitutions
+   ~cfdm.core.Bounds._substitutions        
+   ~cfdm.core.Bounds._package_depth        
+   ~cfdm.core.Bounds._method_exclusions    

3. Removing methods which aren't valid as referenced in the following Sphinx traceback I see:

WARNING: [autosummary] failed to import cfdm.Array.get_filename.
Possible hints:
* PycodeError: no source found for module 'builtins'
* ModuleNotFoundError: No module named 'cfdm.Array'
* AttributeError: type object 'Array' has no attribute 'get_filename'
WARNING: [autosummary] failed to import cfdm.Array.get_filenames.
Possible hints:
* PycodeError: no source found for module 'builtins'
* ModuleNotFoundError: No module named 'cfdm.Array'
* AttributeError: type object 'Array' has no attribute 'get_filenames'
WARNING: [autosummary] failed to import cfdm.CompressedArray.get_filename.
Possible hints:
* AttributeError: type object 'CompressedArray' has no attribute 'get_filename'
* ModuleNotFoundError: No module named 'cfdm.CompressedArray'
* PycodeError: no source found for module 'builtins'
WARNING: [autosummary] failed to import cfdm.CoordinateConversion.__nonzero__.
Possible hints:
* AttributeError: type object 'CoordinateConversion' has no attribute '__nonzero__'
* ModuleNotFoundError: No module named 'cfdm.CoordinateConversion'
* PycodeError: no source found for module 'builtins'
WARNING: [autosummary] failed to import cfdm.Datum.__nonzero__.
Possible hints:
* PycodeError: no source found for module 'builtins'
* ModuleNotFoundError: No module named 'cfdm.Datum'
* AttributeError: type object 'Datum' has no attribute '__nonzero__'

So it would be a good idea to attempt to build the dev-scrub version of the docs to find some issues that need fixing on the docs side.

[davidhassell]

Hi Sadie - 1., 2., and 3. fixed in db3a0a4

[sadielbartholomew]

Hi @davidhassell, thanks for your comments and clarifications. Very useful. The feedback responses all look good (bar the odd inevitable copy-paste typo which I'll raise as part of my final review submission). I'll submit the final review by the early afternoon (I forgot I had a meeting 10-12 I am in right now so slightly later than I said yesterday).

[sadielbartholomew]

Briefly, it looks like the latest SciPy, scipy>=1.11.3, which we request here now, only works with Python 3.9+:

...
Could not solve for environment specs
The following packages are incompatible
└─ scipy 1.11.3  is installable with the potential options
   ├─ scipy 1.11.3 would require
   │  └─ python >=3.10,<3.11.0a0 , which can be installed;
   ├─ scipy 1.11.3 would require
   │  └─ python >=3.11,<3.12.0a0 , which can be installed;
   ├─ scipy 1.11.3 would require
   │  └─ python >=3.12,<3.13.0a0 , which can be installed;
   └─ scipy 1.11.3 would require
      └─ python >=3.9,<3.10.0a0 , which can be installed.
...

so we might have to stop supporting Python 3.8... To avoid that complication is an earlier version of SciPy sufficient for the new logic?

[davidhassell]

Good catch. Let's just change it to the newest scipy version that supports python 3.8: a086e8a

[sadielbartholomew]

I am happy! The approach to incorporating UGRID is very well considered, with the data model at its heart which is a major advantage and will be going forward. The functionality all works as best as I can determine, without being a UGRID expert.

I've raised a final batch of minor comments for your consideration. Once those are addressed, please merge when you feel ready.

[sadielbartholomew]

Comments on responses to some comments I previously raised...

[sadielbartholomew]

For future reference, we are now force-pushing the squashed branch referenced above onto the current un-squashed branch to effectively squash down all of the development commits before the post-review commits, to tidy...

[sadielbartholomew]

Squash complete. Just doing a final sanity check that the code is as it was (running unit tests locally etc.) and then I will merge...

[sadielbartholomew]

All good! Merging...

[davidhassell]

Hooray!