Support for Dask-Yarn

[bschreck]

I'm not sure how hard it would be to allow Dask Yarn clusters. I tried it, and it fails because there is no asynchronous keyword in YarnCluster.__init__. I'm not sure if launching the cluster asynchronously would be necessary for this functionality.

[mrocklin]

I think that this would be resolved by dask/dask-yarn#95 , but I'm not sure. It would require some experimentation. cc @jcrist

…

On Wed, Aug 7, 2019 at 11:52 AM Ben Schreck ***@***.***> wrote: I'm not sure how hard it would be to allow Dask Yarn clusters. I tried it, and it fails because there is no asynchronous keyword in YarnCluster.__init__. I'm not sure if launching the cluster asynchronously would be necessary for this functionality. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#77?email_source=notifications&email_token=AACKZTAQTT5XCZONAPYZ57DQDMKVHA5CNFSM4IKDJJSKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HD7AQPA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTANM6XDI4ORRFAHJB3QDMKVHANCNFSM4IKDJJSA> .

[ian-r-rose]

While the async story settles, you might want to try using this branch, which removes the need for async cluster starting.

[bschreck]

@mrocklin That looks like it would probably resolve the issue.

In the meantime I think I can easily create my own branch. @ian-r-rose that doesn't quite work because there's no cluster.scheduler attribute for the YarnCluster.

[bschreck]

This seems to be working: https://github.com/bschreck/dask-labextension

[AlJohri]

hi, just wanted to check in on this issue- what is required to support Dask Yarn with the lab extension? I mostly just want to see the dask dashboard within jupyter lab, I can start the dask YarnCluster manually. Does this already work?

[mrocklin]

I recommend that people ask these questions at github.com/dask/dask-yarn . The maintainer of Dask-Yarn sometimes doesn't check issues here.

[jcrist]

Apologies for the late response here. As of dask-yarn 0.8.0, asynchronous support works fine. Tested and verified it works with dask-labextension. This can be closed.