Investigate performance optimizations for single locale pdarrays when compiled as multi-dimensional

[ajpotts]

Investigate performance optimizations for single locale pdarrays when compiled as multi-dimensional. See user comment below.

[ajpotts]

User comment : Hi all! I wanted to bring up something concerning performance when Arkouda (and Arachne) are running on a single locale, either when CHPL_COMM is none or it is turned on but the number of locales is only 1. In Arachne we've experienced situations where our code is very performant when it is compiled without the support for multilocale mode, but as soon as we turn multilocale mode on, the performance drops. We know this has something to do with using block-distributed arrays when there isn't a need for them, and we have been able to fix this by just relying on Arkouda's makeDistDom and makeDistArray to handle picking the domain maps as it does here.
However, it is a worry of mine that users will gloss over this explanation in our documentation, and build Chapel and Arkouda/Arachne in multilocale mode but only launch the server with one locale and then experience bad performance. Not sure if Arkouda also suffers from this, I have honestly not tested it, but I was wondering if we could have a conversation on adding more automatic ways to toggle which type of map to use. For example, if it is possible, we could also check how many locales the server is currently being launched on, and if it is 1, without co-locales, it could also default to Dmap.defaultRectangular. Thanks! I can also add this as an issue to the GitHub if needed.