Basically there is a very simple extension to memkind which handles distributed memory allocation (for instance distributed malloc) and simple directory of what is where which can be extended. Obviously the API will change as we mature exactly what we want to use this stuff for, but it at least gives us a starting point for discussion.

In the examples directory, the GASPI example is more interesting, and the second thing we do in the main function there is to create a GASPI segment and then create a distributed memkind based upon this. This distributed memkind allows us to both use memkind/jemalloc in the traditional way for local memory allocation in that segment, and also the distributed malloc call for distributed memory allocations in that segment across processes. I think this illustrates how we can interface GASPI segments to the allocator and we could do more complex stuff such as distributed allocations in combinations of these segments if that is needed

We have extended the memkind structure to hold distributed information and distkinds are registered via a hashmap. To use a GASPI segment with memkind need to point to pmem_mmap - this is like the pmem example and we are using anonymous (i.e. not file backed) memory. The offsets are quite important here, you can see in distmem_gaspi.c how this is is done (my_pmem_mmap fn.) This ends up backing what jemalloc allocates into, so effectively we have an efficient heap allocator for these segments of memory. We are assuming a specific chunksize here, which is the same as the memkind examples. I think this is sensible. There is a minimum segement size imposed by this, which is memkind/jemalloc internal.