accepted change from santosh on github

[cjac]

No description provided.

[cjac]

@vdechand - can you help us with this one? I see in jcrist/skein#165 that you have some experience with dask-yarn

I must use dask-yarn<2022.2 in order for my tests to pass

https://github.com/GoogleCloudDataproc/initialization-actions/blob/master/dask/verify_dask_yarn.py

[vdechand]

Hi, I wish I could help you with that, sorry. The reality is that I haven't used dask since 2019.

[jacobtomlinson]

Happy to try and help out here. Let me know what you are trying to do and I'll see if I can help.

[cjac]

Hello Jacob,

dask-yarn seems to be pretty out of date and depends on an implementation that's contrary to the latest libraries. Can you help to get the yarn integration back up to snuff?

[jacobtomlinson]

@cjac unfortunately I don't have time to work on dask-yarn at the moment. I'm happy to help review PRs and merge things in, but I'm not in a position to actively develop here. To be able to justify time here I would need to see some RAPIDS users making use of this library and be blocked by the current version.

[cjac]

I have a case from June where the user ragequit over lack of dask. I'll look at that one. Not sure whether it was just dask or dask plus rapids.

I found that falling back to a dask version from 2022 got my tests working.

https://github.com/cjac/initialization-actions/blob/dask-20241026/dask/verify_dask_yarn.py

https://github.com/cjac/initialization-actions/blob/70aae930d74e3712bb2dc011e1bebd5e9ec21adb/dask/dask.sh#L433

if [[ "${DASK_RUNTIME}" == 'yarn' ]]; then
    # Pin `distributed` and `dask` package versions to old release
    # because `dask-yarn` 0.9 uses skein in a way which
    # is not compatible with `distributed` package 2022.2 and newer:
    # https://github.com/dask/dask-yarn/issues/155

    dask_spec="dask<2022.2"
    python_spec="python>=3.7,<3.8.0a0"
    if is_ubuntu18 ; then
      # the libuuid.so.1 distributed with fiona 1.8.22 dumps core when calling uuid_generate_time_generic
      CONDA_PACKAGES+=("fiona<1.8.22")
    fi
    CONDA_PACKAGES+=('dask-yarn=0.9' "distributed<2022.2")
  fi