Changes supporting GEOSX Trilinos/Tpetra wrapper

[klevzoff]

This PR adds a simple function in ArrayView (and NewChaiBuffer) to get the data pointer in a specific MemorySpace, whether most current or not.

While this should not be needed/used by most developers, Tpetra implements dual memory space semantics, meaning that Tpetra::Vector for example contains pointers in both host and device memory spaces, and can potentially move data between them as needed. Therefore, in order to support the zero-copy creation of parallel vectors from ArrayView of local values, we have to be able to deliver pointers in both memory spaces.

Related to GEOS-DEV/GEOS#1086

[klevzoff]

Note: code in GEOSX that uses this is currently disabled behind an #if 0, because I found out some Tpetra-based solvers only work reliably with Unified Memory, and so the zero-copy dual-view semantics cannot be implemented. So this PR could in principle be abandoned, however I though it might be useful in future.

[rrsettgast]

Is this sort of error prone? by this I mean that it seems likely that we can get a pointer that is not current? It there an underlying check/move?

[klevzoff]

The code that uses this in GEOSX calls a move before. Would you rather have a call to move embedded in here? This sort of makes synchronization unavoidable, and I'm not sure we always want it.

[corbett5]

I'm fine with this for the use case you describe above you don't want to move the data before getting the pointers our. Obviously you could really shoot yourself but this should only be used in a few places.

Could you

1. Add a check to error out if the array wasn't allocated in that space m_pointer_record->m_pointers[ space ] == nullptr?
2. Add the following the MallocBuffer and StackBuffer?

T * data( MemorySpace const space ) const
{
  LVARRAY_ERROR_IF_NE( space, MemorySpace::CPU );
  return data();
}

3. Add a simple test to testArrayView.

[klevzoff]

@corbett5 done

[rrsettgast]

Note: code in GEOSX that uses this is currently disabled behind an #if 0, because I found out some Tpetra-based solvers only work reliably with Unified Memory, and so the zero-copy dual-view semantics cannot be implemented. So this PR could in principle be abandoned, however I though it might be useful in future.

Do we need to update Trilinos?

[klevzoff]

Do we need to update Trilinos?

No, there's no new version available. And it's not something they seem to have fixed yet. All Tpetra build instructions, FAQs, etc. explicitly require configuring Kokkos to use UM as default memory space for CUDA.

[rrsettgast]

Do we need to update Trilinos?

No, there's no new version available. And it's not something they seem to have fixed yet. All Tpetra build instructions, FAQs, etc. explicitly require configuring Kokkos to use UM as default memory space for CUDA.

🤦

[corbett5]

No, there's no new version available. And it's not something they seem to have fixed yet. All Tpetra build instructions, FAQs, etc. explicitly require configuring Kokkos to use UM as default memory space for CUDA.

🤦

We can always just allocate the matrix with unified memory, it would be pretty easy to add that support.

[corbett5]

Thanks!