vagrant ssh does not work on Git Bash/MSYS2 environments after 2.4.0 update

[guillermo-jano]

After #13219, looks like Vagrant now defaults to generating ed25519 key pairs when replacing the insecure key upon new box creation. On Windows, this is generated with CRLF line endings and apparently the OpenSSL version shipping with Git Bash and MSYS2 (probably also Cygwin, not tested) does not like that (this seemingly wasn't a problem with RSA keys though). vagrant ssh works as expected on Windows PowerShell and traditional Command Prompt (CMD.EXE).

Debug output

$ vagrant ssh
[email protected]'s password:

If I try vagrant ssh -- -v instead I can see

Load key "C:/MYBOX/.vagrant/machines/default/virtualbox/private_key": error in libcrypto

Also if I run dos2unix .vagrant/machines/default/virtualbox/private_key and then run vagrant ssh again connection works as expected. So apparently the problem is the OpenSSL version in Git Bash/MSYS2 does not like ed25519 keys with CRLF line endings.

Expected behavior

vagrant ssh should open a session with the newly created box without asking for a password

Actual behavior

vagrant ssh asks for a password

Reproduction information

Vagrant version

Vagrant 2.4.0

Host operating system

Microsoft Windows 10 Enterprise 10.0.19044 Build 19044

Guest operating system

Ubuntu 18.04.3 LTS

Steps to reproduce

1. Create a new Vagrant box with vagrant up
2. Run vagrant ssh from a Git Bash / MSYS2 environment (probably also Cygwin, not tested)

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/bionic64"
end

[iliareshetov]

I experienced a similar problem, which was resolved by switching CRLF to LF.

[guillermo-jano]

Yes the dos2unix command line I mentioned in the issue body fixes the problem, but IMO it would be better if this worked out of the box as it used to.

[iliareshetov]

Hey, I’m with you on this and I just want to confirm that this issue is there and it’s rather hard to figure out why it doesn’t work.

The command vagrant ssh --debug doesn’t help much and only when I used the ssh command and only then got error in libcrypto, which brought me here.

Definitely should work out of the box.

[OIOIOOOI]

Same issue but different error.
Having keys generated with CRLF and trying vagrant ssh yields Error: [email protected]: Permission denied (publickey).
This is solved by changing the line endings, but it's truly an annoying thing to do every time the VM is created.

[guillermo-jano]

Yes, hopefully this gets some attention from the developers and can be fixed in a future release.

[DanRagle]

Can confirm this happens in cygwin environments, as well. So new box provisions are broken since the first rsync (happens after the insecure key is replaced with the borked CR/LF version) fails.

[mkot02]

I came across the same issue. I have setup with multiple hosts and ansible running in one of them. Ansible fails to do configuration on other hosts with message: invalid format.

As a workaround I'm converting private_keys using this command:

find .vagrant -type f -name private_key | xargs dos2unix

I have this problem after upgrading to 2.4.0

[EyalDagiAvdor]

Can confirm this happens in git bash on vagrant 2.4.0.
vagrant ssh
[email protected]'s password:

The workaround, find .vagrant -type f -name private_key | xargs dos2unix, fixes the issue, but this should work out of the box.

[ghalse]

I believe this problem stems from how the private key is written here:
https://github.com/hashicorp/vagrant/blob/v2.4.0/plugins/communicators/ssh/communicator.rb#L221-L223

Note the \n generated here is converted to \r\n in the resulting file, suggesting that a conversion is happening.

The open("w+") when the private key is written is in text file mode, and so Ruby does line-ending conversions depending on operating system.

The problem can be resolved by changing the open() to use binary mode to suppress the EOL <-> CRLF conversion on Windows, preserving the line endings generated in keypair.rb.

--- a/plugins/communicators/ssh/communicator.rb        2024-01-11 16:06:33.948133477 +0200
+++ b/plugins/communicators/ssh/communicator.rb     2024-01-11 16:06:53.192133293 +0200
@@ -218,7 +218,7 @@

           # Write out the private key in the data dir so that the
           # machine automatically picks it up.
-          @machine.data_dir.join("private_key").open("w+") do |f|
+          @machine.data_dir.join("private_key").open("wb+") do |f|
             f.write(priv)
           end

As of writing this, the problem still exists in the main branch too, even after #13327.

[guillermo-jano]

I think this has been the case forever, however the SSH client in Git Bash / MSYS2 does not seem to have an issue with RSA private keys but it does not like ed25519 keys in CRLF format. Anyway since the built-in Windows SSH client is fine with LF keys, I'd say your solution should harm no one, and vagrant ssh would work in every kind of environment.

[ghalse]

I think this has been the case forever, however the SSH client in Git Bash / MSYS2 does not seem to have an issue with RSA private keys but it does not like ed25519 keys in CRLF format.

It is a bit more subtle than that. The old RSA private keys were PEM encoded using OpenSSL; the ED25519 keys are encoded using OpenSSH's more lightweight format. Having looked at OpenSSH's source code for this, I think you'd have the same problem with an RSA key saved using the OpenSSH file format. (i.e. the changes in the way vagrant generates the keys might make this a problem for new RSA keys too).

[guillermo-jano]

Ahh, that might be, not sure how to ask Vagrant to generate a RSA key instead of ed25519 now anyway...

[chrisroberts]

I had not forgotten about this issue! The fix will be included in 2.4.1 🙂