Memory leak when opening a DataArray and closing it again with the threading scheduler

[lumbric]

What happened?

I am using xr.open_dataarray() with chunks and do some simple computation. After that 800mb of RAM is used, no matter whether I close the file explicitly, delete the xarray objects or invoke the Python garbage collector.

What seems to work: do not use the threading Dask scheduler. The issue does not seem to occur with the single-threaded or processes scheduler. Also setting MALLOC_MMAP_MAX_=40960 seems to solve the issue as suggested above (disclaimer: I don't fully understand the details here).

If I understand things correctly, this indicates that the issue is a consequence of dask/dask#3530. Not sure if there is anything to be fixed on the xarray side or what would be the best work around. I will try to use the processes scheduler.

See also #2186, which has been closed without fix and my comment there.

What did you expect to happen?

Not consuming significantly more memory than before opening the NetCDF file.

Minimal Complete Verifiable Example

import gc
import dask
import psutil
import os.path
import numpy as np
import xarray as xr


# a value of 1_000_000 would make much more sense here, but there seems to be a larger memory leak
# with small chunk size for some reason
CHUNK_SIZE = 1_000_000


def print_used_mem():
    process = psutil.Process()
    print("Used RAM in GB:", process.memory_info().rss / 1024**3)


def read_test_data():
    print("Opening DataArray...")
    print_used_mem()

    data = xr.open_dataarray('tempdata.nc', chunks=CHUNK_SIZE, cache=False)
    print_used_mem()

    print("Compute sum...")
    result = data.sum()
    print_used_mem()

    print("Print result...")
    print("Result", float(result))

    print_used_mem()
    data.close()
    del result
    del data
    print_used_mem()


def main():
    # preparation:
    # create about 7.5GB of data  (8 * 10**9 / 1024**3)
    if not os.path.exists('tempdata.nc'):
        print("Creating 7.5GB file tempdata.nc...")
        data = xr.DataArray(np.zeros(10**9))
        data.to_netcdf('tempdata.nc')
        print("Test file created!")

    with dask.config.set(scheduler='threading'):
        print("Starting read test...")
        print_used_mem()
        read_test_data()

        print("not inside any function any longer")
        print_used_mem()

        print("Garbage collect:", gc.collect())
        print_used_mem()


if __name__ == '__main__':
    print("Used memory before test:")
    print_used_mem()
    print("")

    main()

    print("\nUsed memory after test:")
    print_used_mem()

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

# output running the minimal example above:

Used memory before test:
Used RAM in GB: 0.11433029174804688

Creating 7.5GB file tempdata.nc...
Test file created!
Starting read test...
Used RAM in GB: 0.13946533203125
Opening DataArray...
Used RAM in GB: 0.13946533203125
Used RAM in GB: 0.14622879028320312
Compute sum...
Used RAM in GB: 0.14670944213867188
Print result...
Result 0.0
Used RAM in GB: 0.6771659851074219
Used RAM in GB: 0.6771659851074219
not inside any function any longer
Used RAM in GB: 0.6771659851074219
Garbage collect: 1113
Used RAM in GB: 0.6744232177734375

Used memory after test:
Used RAM in GB: 0.6744194030761719

Anything else we need to know?

No response

Environment

I did the tests in a new conda environment installing only relevant packages: micromamba install -c conda-forge xarray dask netcdf4.

xr.show_versions():

INSTALLED VERSIONS ------------------ commit: None python: 3.13.0 | packaged by conda-forge | (main, Oct 8 2024, 20:04:32) [GCC 13.3.0] python-bits: 64 OS: Linux OS-release: 5.4.0-190-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.14.4 libnetcdf: 4.9.2

xarray: 2024.10.0
pandas: 2.2.3
numpy: 2.1.3
scipy: None
netCDF4: 1.7.2
pydap: None
h5netcdf: None
h5py: None
zarr: None
cftime: 1.6.4
nc_time_axis: None
iris: None
bottleneck: None
dask: 2024.11.2
distributed: 2024.11.2
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2024.10.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: None
pip: 24.3.1
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None

conda list:

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge aws-c-auth 0.8.0 hb88c0a9_10 conda-forge aws-c-cal 0.8.0 hecf86a2_2 conda-forge aws-c-common 0.10.3 hb9d3cd8_0 conda-forge aws-c-compression 0.3.0 hf42f96a_2 conda-forge aws-c-event-stream 0.5.0 h1ffe551_7 conda-forge aws-c-http 0.9.1 hab05fe4_2 conda-forge aws-c-io 0.15.2 hdeadb07_2 conda-forge aws-c-mqtt 0.11.0 h7bd072d_8 conda-forge aws-c-s3 0.7.1 h3a84f74_3 conda-forge aws-c-sdkutils 0.2.1 hf42f96a_1 conda-forge aws-checksums 0.2.2 hf42f96a_1 conda-forge aws-crt-cpp 0.29.4 h21d7256_1 conda-forge aws-sdk-cpp 1.11.449 h1a02111_2 conda-forge azure-core-cpp 1.14.0 h5cfcd09_0 conda-forge azure-identity-cpp 1.10.0 h113e628_0 conda-forge azure-storage-blobs-cpp 12.13.0 h3cf044e_1 conda-forge azure-storage-common-cpp 12.8.0 h736e048_1 conda-forge azure-storage-files-datalake-cpp 12.12.0 ha633028_1 conda-forge blosc 1.21.6 hef167b5_0 conda-forge bokeh 3.6.1 pyhd8ed1ab_0 conda-forge brotli-python 1.1.0 py313h46c70d0_2 conda-forge bzip2 1.0.8 h4bc722e_7 conda-forge c-ares 1.34.3 heb4867d_0 conda-forge ca-certificates 2024.8.30 hbcca054_0 conda-forge certifi 2024.8.30 pyhd8ed1ab_0 conda-forge cffi 1.17.1 py313hfab6e84_0 conda-forge cftime 1.6.4 py313ha014f3b_1 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge cloudpickle 3.1.0 pyhd8ed1ab_1 conda-forge contourpy 1.3.1 py313h33d0bda_0 conda-forge cytoolz 1.0.0 py313h536fd9c_1 conda-forge dask 2024.11.2 pyhff2d567_1 conda-forge dask-core 2024.11.2 pyhff2d567_1 conda-forge dask-expr 1.1.19 pyhd8ed1ab_0 conda-forge distributed 2024.11.2 pyhff2d567_1 conda-forge freetype 2.12.1 h267a509_2 conda-forge fsspec 2024.10.0 pyhff2d567_0 conda-forge gflags 2.2.2 h5888daf_1005 conda-forge glog 0.7.1 hbabe93e_0 conda-forge h2 4.1.0 pyhd8ed1ab_0 conda-forge hdf4 4.2.15 h2a13503_7 conda-forge hdf5 1.14.4 nompi_h2d575fe_103 conda-forge hpack 4.0.0 pyh9f0ad1d_0 conda-forge hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge importlib-metadata 8.5.0 pyha770c72_0 conda-forge jinja2 3.1.4 pyhd8ed1ab_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge krb5 1.21.3 h659f571_0 conda-forge lcms2 2.16 hb7c19ff_0 conda-forge ld_impl_linux-64 2.43 h712a8e2_2 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libabseil 20240722.0 cxx17_h5888daf_1 conda-forge libaec 1.1.3 h59595ed_0 conda-forge libarrow 18.0.0 h3b997a5_7_cpu conda-forge libarrow-acero 18.0.0 h5888daf_7_cpu conda-forge libarrow-dataset 18.0.0 h5888daf_7_cpu conda-forge libarrow-substrait 18.0.0 h5c8f2c3_7_cpu conda-forge libblas 3.9.0 25_linux64_openblas conda-forge libbrotlicommon 1.1.0 hb9d3cd8_2 conda-forge libbrotlidec 1.1.0 hb9d3cd8_2 conda-forge libbrotlienc 1.1.0 hb9d3cd8_2 conda-forge libcblas 3.9.0 25_linux64_openblas conda-forge libcrc32c 1.1.2 h9c3ff4c_0 conda-forge libcurl 8.10.1 hbbe4b11_0 conda-forge libdeflate 1.22 hb9d3cd8_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 hd590300_2 conda-forge libevent 2.1.12 hf998b51_1 conda-forge libexpat 2.6.4 h5888daf_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc 14.2.0 h77fa898_1 conda-forge libgcc-ng 14.2.0 h69a702a_1 conda-forge libgfortran 14.2.0 h69a702a_1 conda-forge libgfortran5 14.2.0 hd5240d6_1 conda-forge libgomp 14.2.0 h77fa898_1 conda-forge libgoogle-cloud 2.31.0 h804f50b_0 conda-forge libgoogle-cloud-storage 2.31.0 h0121fbd_0 conda-forge libgrpc 1.67.1 hc2c308b_0 conda-forge libiconv 1.17 hd590300_2 conda-forge libjpeg-turbo 3.0.0 hd590300_1 conda-forge liblapack 3.9.0 25_linux64_openblas conda-forge libmpdec 4.0.0 h4bc722e_0 conda-forge libnetcdf 4.9.2 nompi_h2564987_115 conda-forge libnghttp2 1.64.0 h161d5f1_0 conda-forge libopenblas 0.3.28 pthreads_h94d23a6_1 conda-forge libparquet 18.0.0 h6bd9018_7_cpu conda-forge libpng 1.6.44 hadc24fc_0 conda-forge libprotobuf 5.28.2 h5b01275_0 conda-forge libre2-11 2024.07.02 hbbce691_1 conda-forge libsqlite 3.47.0 hadc24fc_1 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx 14.2.0 hc0a3c3a_1 conda-forge libstdcxx-ng 14.2.0 h4852527_1 conda-forge libthrift 0.21.0 h0e7cc3e_0 conda-forge libtiff 4.7.0 he137b08_1 conda-forge libutf8proc 2.8.0 h166bdaf_0 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libwebp-base 1.4.0 hd590300_0 conda-forge libxcb 1.17.0 h8a09558_0 conda-forge libxml2 2.13.5 h064dc61_0 conda-forge libzip 1.11.2 h6991a6a_0 conda-forge libzlib 1.3.1 hb9d3cd8_2 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4 4.3.3 py313h010b13d_1 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge markupsafe 3.0.2 py313h8060acc_0 conda-forge msgpack-python 1.1.0 py313h33d0bda_0 conda-forge ncurses 6.5 he02047a_1 conda-forge netcdf4 1.7.2 nompi_py313h1dd084c_100 conda-forge numpy 2.1.3 py313h4bf6692_0 conda-forge openjpeg 2.5.2 h488ebb8_0 conda-forge openssl 3.4.0 hb9d3cd8_0 conda-forge orc 2.0.3 he039a57_0 conda-forge packaging 24.2 pyhff2d567_1 conda-forge pandas 2.2.3 py313ha87cce1_1 conda-forge partd 1.4.2 pyhd8ed1ab_0 conda-forge pillow 11.0.0 py313h2d7ed13_0 conda-forge pip 24.3.1 pyh145f28c_0 conda-forge psutil 6.1.0 py313h536fd9c_0 conda-forge pthread-stubs 0.4 hb9d3cd8_1002 conda-forge pyarrow 18.0.0 py313h78bf25f_1 conda-forge pyarrow-core 18.0.0 py313he5f92c8_1_cpu conda-forge pycparser 2.22 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.13.0 h9ebbce0_100_cp313 conda-forge python-dateutil 2.9.0.post0 pyhff2d567_0 conda-forge python-tzdata 2024.2 pyhd8ed1ab_0 conda-forge python_abi 3.13 5_cp313 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pyyaml 6.0.2 py313h536fd9c_1 conda-forge re2 2024.07.02 h77b4e00_1 conda-forge readline 8.2 h8228510_1 conda-forge s2n 1.5.9 h0fd0ee4_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.2.1 ha2e4443_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge tblib 3.0.0 pyhd8ed1ab_0 conda-forge tk 8.6.13 noxft_h4845f30_101 conda-forge toolz 1.0.0 pyhd8ed1ab_0 conda-forge tornado 6.4.1 py313h536fd9c_1 conda-forge tzdata 2024b hc8b5060_0 conda-forge urllib3 2.2.3 pyhd8ed1ab_0 conda-forge xarray 2024.10.0 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.11 hb9d3cd8_1 conda-forge xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge xyzservices 2024.9.0 pyhd8ed1ab_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge zict 3.0.0 pyhd8ed1ab_0 conda-forge zipp 3.21.0 pyhd8ed1ab_0 conda-forge zlib 1.3.1 hb9d3cd8_2 conda-forge zstandard 0.23.0 py313h80202fe_1 conda-forge zstd 1.5.6 ha6fb4c9_0 conda-forge