Memory management when using PythonCall

[anushasekar1]

What is the best way to manage memory when working with PythonCall? I would like to allocate memory in Julia, then make a call to a Python function that modifies what I have passed in, without allocating any new memory. Can you post an example of how to do this using PythonCall? I have been trying, but running into issues.

[cjdoris]

Maybe you could post what you have tried. If you create an array in Julia and pass it to Python via PythonCall then by default the underlying array is wrapped so that modifying the array from Python modifies the original.

[anushasekar1]

I think the problem might be that the Python code I am trying to wrap requires numpy arrays, and PythonCall only creates Arrays. When I convert to a numpy array, this allocates new memory. Is there a way to accomplish what I am trying to do without memory allocation within Python?

def multiply_by_two_inplace_try1(x, y):
    print("py incoming: id(x): ", id(x), "; id(y): ",id(y))
    y[:,:] = 2*x
    return None

def multiply_by_two_inplace_try2(x, y):
    print("py incoming: id(x): ", id(x), "; id(y): ",id(y))
    x = np.array(x)
    y = np.array(y)
    print("after numpy: id(x): ", id(x), "; id(y): ",id(y))
    np.multiply(2,x,out=y)
    return None

[cjdoris]

If you use np.asarray instead of np.array you get a numpy array which wraps the underlying Julia array, so no new memory is allocated (whereas np.array always takes a copy).

[anushasekar1]

Unfortunately, that seems to only be true if the incoming type is ndarray of the same dtype? What type does PythonCall pass in arrays as? https://numpy.org/doc/stable/reference/generated/numpy.asarray.html

The interesting thing is now the underlying Julia array is indeed changed, but there is memory allocated! Or am I interpreting the change in id(x) and id(y) incorrectly?

def multiply_by_two_inplace_try2(x, y):
    print("py incoming: id(x): ", id(x), "; id(y): ",id(y))
    x = np.array(x)
    y = np.array(y)
    print("after numpy: id(x): ", id(x), "; id(y): ",id(y))
    np.multiply(2,x,out=y)
    return None

def multiply_by_two_inplace_try3(x, y):
    print("py incoming: id(x): ", id(x), "; id(y): ",id(y))
    x = np.asarray(x)
    y = np.asarray(y)
    print("after numpy: id(x): ", id(x), "; id(y): ",id(y))
    np.multiply(2,x,out=y)
    return None

[cjdoris]

I think you're focussing to much on allocations here. There is indeed an allocation, but that's because creating a new object in Python always does a small allocation - but it's fast because Python has an allocator optimised for lots of small short-lived allocations. What's important is that doing np.asarray(x) doesn't copy the underlying array.

As you mentioned, if you change the dtype of the array (np.asarray(x, dtype=...)) then this has to allocate a new array of the given type. If you want to mutate an existing array, don't specify the dtype.