For multi-tile I/O, allow concurrent reads/writes of the different tiles

[billsacks]

There are performance issues with multi-tile I/O since the tiles are written sequentially (see recent notes in https://github.com/esmf-org/esmf-support/issues/489).

We'd like to investigate being able to write tiles concurrently - either by having multiple PIO instances or by changing something else about the PIO setup to allow concurrency or at least get similar performance to what we'd get from this concurrency.

See notes on this in the 2025-01-15 Core Team Meeting Notes.

[anntsay]

from Core team;

Option 1: if the right way, we need to rework how PIO is initialized in ESMF entirely. Feel like this is the right thing is to initiate once and cleaner. But effort is huge
Option 2: reinitialize PIO as many times as we want. But it will make the rework in the future harder. It would be faster development but still tricky. Maybe a week work? But will need to plan various gotchas.

Bill’s inclination is to go with the quicker and dirtier approach of Option 2: keeping the PIO initialization happening where it currently is, but creating a separate PIO instance for each tile.
Bill: something that might make this hard is: for concurrency, it might be necessary for PIO instance 1 to use a subset of PEs that appear on tile 1, instance 2 to use a subset of PEs that appear on tile 2, etc. Bill isn’t sure if this would be needed for concurrency

Action:
Talk to UFS team on the urgency timeline -see if we could sync this up with 8.9 release in 3 months or so.
Figure out who may be able to take on this
(Bill / Jim / Ann ) Ask Jim E. to see if we could do this at PIO level where it can internally create multiple instances rather than ESMF level. Or could PIO have some other mechanism for concurrently writing these different tiles even given a single PIO instance?

[anntsay]

Jim responded that probably should not implement at the PIO level.
UFS Team currently has a workaround.

UFS team response
from Dusan 1/21/2025
I do not think this is critical because we can continue to use current code which does not have performance issues since it's not using updated multi-tile FieldBundleWrite, but rather Gerhard's implementation which creates 6 sub components.

I am also experimenting with using our own write_netcdf subroutines which we currently use for writing bundles to also support cubed_sphere grids. Gerhard help me sometime ago with explanation on how to use FieldGather with multi-tiles fields/grids. I just need to find out if we can use write grid component with number of tasks per group that are not divisible by 6. If that works, that's also a possible alternative.

from Jun 1/22/2025:
I agree. This work is to update the model code to use the standard ESMF_FieldBundleWrite(), the UFS weather model will catch up with more recent ESMF capabilities and test them. But the current code (Gerhard's implementation which creates 6 sub components) is still working. I don't think it is urgent, it would be good if it is planned to be included in ESMF's future release.

---

Team: we want to encourage others to move to correct solution (and not the experimental code that we do not have the capacity to maintain).

[billsacks]

One issue that may need some thought here is how to set up each PIO instance so that it has a subset of the PEs just for a given tile. I'm not sure this is essential, but it seems like it might be needed to actually get the performance boost we're trying to get: If (for example) the PIO instance for tile 1 was using some PEs that overlap with the PEs used for the data used for other tiles, then we might not get the concurrency we're looking for (though I'm not sure about this).

We should at least make sure this gives us the concurrency we want in simple / standard cases: it would probably be okay (at least for an initial implementation) if we don't get the ideal concurrency in unusual cases, like having multiple DEs per PET, if it's harder to support those cases.