Handling of arrays of lists/arrays as field values

[jsayc]

I plan to use fipy as a prototyping tool for radiation simulation. As a basic functionality
the need to store an array of arrays or lists in each cell arises. The length
of that array of arrays/lists should be allowed to differ from cell to cell.

To test this functionality the code below is used as a minimum working example.
valueNonUniform is the array populated with cell values. Since the grid has four cells,
the array has four entries.
The first entry "arrShort" differs from the following other entries "arrLong".
The corresponding CellVariable VarNonUniform behaves as expected:

VarNonUniform.value
Out[102]: 
array([array([[1, 2],
              [2, 3]]), array([[1, 2],
                               [2, 3],
                               [4, 5]]), array([[1, 2],
                                                [2, 3],
                                                [4, 5]]), array([[1, 2],
                                                                 [2, 3],
                                                                 [4, 5]])],
      dtype=object)
 

But in case the array is populated with all entries being the same, which is demonstrated with the array
valueUniform, which is an array of arrays of kind arrLong, one gets some strange behavior:

VarUniform.value
Out[104]: 
array([[[[1, 1, 1, 1],
         [2, 2, 2, 2]],
        [[2, 2, 2, 2],
         [3, 3, 3, 3]],
        [[4, 4, 4, 4],
         [5, 5, 5, 5]]],
       [[[1, 1, 1, 1],
         [2, 2, 2, 2]],
        [[2, 2, 2, 2],
         [3, 3, 3, 3]],
        [[4, 4, 4, 4],
         [5, 5, 5, 5]]],
       [[[1, 1, 1, 1],
         [2, 2, 2, 2]],
        [[2, 2, 2, 2],
         [3, 3, 3, 3]],
        [[4, 4, 4, 4],
         [5, 5, 5, 5]]],
       [[[1, 1, 1, 1],
         [2, 2, 2, 2]],
        [[2, 2, 2, 2],
         [3, 3, 3, 3]],
        [[4, 4, 4, 4],
         [5, 5, 5, 5]]]])
       [[[1, 1, 1, 1],
         [2, 2, 2, 2]],
        [[2, 2, 2, 2],
         [3, 3, 3, 3]],
        [[4, 4, 4, 4],
         [5, 5, 5, 5]]],
       [[[1, 1, 1, 1],
         [2, 2, 2, 2]],
        [[2, 2, 2, 2],
         [3, 3, 3, 3]],
        [[4, 4, 4, 4],
         [5, 5, 5, 5]]]])    

VarNonUniform.value is of shape (4,)

fipy takes each entry of valueNonUniform and attributes it to one cell, but in case
of valueUniform, where all entries are the same, fipy seems to attribute the whole array valueUniform to each cell
and perform some transpose operations...

VarUniform.value is of shape (4,3,2,4)

Why is that and how to tell fipy to handle the array valueUniform in the exact same way as the array valueNonUniform?

It would be quite helpful to get a hint on how to avoid this behavior or understand the potential wisdom behind.

from fipy import CellVariable
from fipy.meshes import Grid2D
from fipy.tools import numerix
import numpy as np

arrShort = numerix.array( [[1,2],[2,3]] )
arrLong  = numerix.array( [[1,2],[2,3],[4,5]] )

valueNonUniform =  numerix.array( [ arrShort, arrLong, arrLong, arrLong ] )
valueUniform =     numerix.array( [ arrLong,  arrLong, arrLong, arrLong ] )

mesh = Grid2D(dx=1, dy=1, nx=2, ny=2)

VarNonUniform = CellVariable(mesh,value=valueNonUniform)
VarUniform    = CellVariable(mesh,value=valueUniform)

[guyer]

Why is that and how to tell fipy to handle the array valueUniform in the exact same way as the array valueNonUniform?

It would be quite helpful to get a hint on how to avoid this behavior or understand the potential wisdom behind.

FiPy CellVariable and FaceVariable objects are designed to hold fields of values. Those fields can be different ranks, or arbitrary shapes unassociated with rank, but they must be the same for every element[*].

I would say that the behavior of valueNonUniform is a bug, even for your purposes, and certainly for what FiPy is expecting. It's not really a bug on FiPy's part, though. You're creating a numpy array out of a list of numpy arrays. Numpy lets you do that, but note that it assigns dtype=object, which means that it has no idea what you're doing and it's not going to try to figure it out.

[*] Internally, FiPy does have need to store irregular information for, e.g., the distances to neighboring cells for each cell of a mesh comprising arbitrary polygons or polyhedra. FiPy uses masked arrays for this. We keep all elements of the shape the same, except for the last one, so the number of values associated with a cell can change, but each of those values has the same shape. How these get created depends on what they are.

[jsayc]

Thank you very much for the fast reply and the hint to use masked arrays.
If I write a masked array to a CellVariable, it stores only the values but not the mask itself.
Is there a custom way to use CellVariable with masked arrays or irregular information?

Thanks in advance!

The example I used for testing:

from fipy import CellVariable
from fipy.meshes import Grid2D
import numpy.ma as ma
from fipy.tools import numerix

mesh = Grid2D(dx=1, dy=1, nx=2, ny=1)
arrShort = np.array([[1,2],[3,4],[5,6]]).transpose()
arrLong  = np.array([[1,2],[3,4],[5,6],[7,8]]).transpose()
value = np.ones([2,4,int(mesh.globalNumberOfCells)],dtype=float)*-1
value[:,0:3,0] = arrShort 
value[:,:,1] = arrLong
value_ma = numerix.ma.masked_array(value, mask =(value <= 0))
cellVar = CellVariable( mesh, value = value_ma )
print(cellVar.value)

[guyer]

If I write a masked array to a CellVariable, it stores only the values but not the mask itself.

You're absolutely right. There's a comment in the code that .setValue() "Can take a masked array", but it's not true, or at least it's not robust (#888).

Is there a custom way to use CellVariable with masked arrays or irregular information?

FiPy's internal use of masked_array does not work the way it works in my head (in my defense, I have an experimental branch where it does, but that branch is a long way off from being merged).

All of the uses of masked_array are just as they are. They're the same shape as if they were a CellVariable, but they're not a CellVariable. Getting this working is a significant bootstrapping issue and so we didn't originally bother. The branch where I did get it working had significant performance issues the last time I looked at it.

I think you're going to need to just manually work with masked arrays that have the same underlying shape as a CellVariable.

[jsayc]

Ok. Thank you very much!
Maybe the performance issues are related to the masked array itself.
I started to use masked arrays and eventually try to mimic some behavior of CellVariable later.
At the moment its hard to see for me if irregular data structures are faster compared to masked arrays, so I am going to start with the latter ones.

[guyer]

Maybe the performance issues are related to the masked array itself.

I don't think so. It's more an issue with FiPy's lazy evaluation, which gets even more Byzantine when meshes are defined in terms of CellVariable and FaceVariable objects, which are in turn defined on a mesh. Masked arrays probably don't help, but I haven't seen signs that they particularly hurt.

One complication for you may be whether the masking pattern changes during the course of a simulation.