Add Windows support to SSHLauncher #842
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allow manual starting of tess
ssh cluster support for windows
# Conflicts: # .github/workflows/test.yml
old docker-compose is removed
back to skipping
at least while Windows setup takes forever
|
Finally landing this one. Thank you so much, @ottointhesky! |
minrk
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed in the issue 837 I have implemented an abstraction layer (
ipyparallel.cluster.shellcmd.ShellCommandSendand...ShellCommandReceive) for sending shell commands in a platform independent manner. The API provides a variety of different standard shell commands (start, running, kill, mkdir, rmdir, exists and remove) and some generic shell/python code command execution features which encapsulates necessary quoting and different handling of different platforms and shells. Currently, Linux (bash, ssh) and Windows (cmd, powershell, ssh) are supported but it should be easy to support MacOS as well (I will have a look after sending the pull request).The abstraction layer was mainly developed to send shell command through an ssh connection but it works equally for sending commands to a local shell. All possible types of shell connections are test in the corresponding pytest script (test_shellcmd.py)
I have extended the ssh-cluster-test that it also runs under Windows, BUT unfortunately there is an issue that prevents the ssh-cluster-test (and test_shellcmd) to run successfully in Github Actions. Everything works perfectly in a local environment, but with Github Runners it is not possible to start a process with the CREATE_BREAKAWAY_FROM_JOB flag set (the flag is absolute necessary, otherwise the started process gets kill when the ssh connection is closed). I’m not the first one that came across this issue, but obviously there is no easy work-a-round or solution to this (Have a look at win-start_issue.py which shows the Github behaviour in the corresponding test )
I will keep trying to find a trick or work-a-round to that issue, but I fear there isn’t. Not testing ssh-cluster under Windows isn’t an option (from my point of view), so maybe a self-hosted Windows runner could be the solution. Do you have experience with that? How should we proceed?
By the way, shellcmd implementation also has a cli
ipyparallel.shellcmdwhich is NOT used. Hence, it could be safely dropped…