Skip to content
This repository was archived by the owner on Jun 1, 2020. It is now read-only.

can't ssh to container #27

Open
edbond opened this issue Aug 28, 2014 · 11 comments
Open

can't ssh to container #27

edbond opened this issue Aug 28, 2014 · 11 comments
Labels
Defect

Comments

@edbond
Copy link

edbond commented Aug 28, 2014 

→  docker ps
.....
099dc3966110        hectcastro/riak:latest   /sbin/my_init --quie   8 minutes ago       Up 8 minutes        0.0.0.0:49163->8087/tcp, 0.0.0.0:49164->8098/tcp   riak01,riak02/seed,riak03/seed,riak04/seed,riak05/seed   

→  docker inspect 099dc3966110 | grep IP
        "IPAddress": "172.17.0.7",
        "IPPrefixLen": 16,
→  ssh -i insecure_key [email protected]
The authenticity of host '172.17.0.7 (172.17.0.7)' can't be established.
ECDSA key fingerprint is f1:56:8c:fc:8b:95:75:ff:5f:bc:4a:53:72:3d:da:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.7' (ECDSA) to the list of known hosts.
Connection to 172.17.0.7 closed by remote host.
Connection to 172.17.0.7 closed.
@edbond edbond changed the title ssh is not working can't ssh to container Aug 28, 2014
@hectcastro
Copy link
Owner

Attempting to SSH with -v would be helpful to determine what went wrong. I just attempted to SSH into a container and seemed to work:

$ docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                                              NAMES
0ce77a80a29c        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49161->8087/tcp, 0.0.0.0:49162->8098/tcp   riak05
d1c442962191        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49159->8087/tcp, 0.0.0.0:49160->8098/tcp   riak04
b0e5e154e7d5        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49157->8087/tcp, 0.0.0.0:49158->8098/tcp   riak03
fc5a28476d5d        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49155->8087/tcp, 0.0.0.0:49156->8098/tcp   riak02
4eac9f286dca        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49153->8098/tcp, 0.0.0.0:49154->8087/tcp   riak01,riak02/seed,riak03/seed,riak04/seed,riak05/seed
$ docker inspect b0e5e154e7d5 | grep IPAddress
        "IPAddress": "172.17.0.10",
$ ssh -i insecure_key [email protected]
The authenticity of host '172.17.0.10 (172.17.0.10)' can't be established.
ECDSA key fingerprint is a3:0e:aa:96:22:9d:55:48:44:8f:dc:26:c0:90:b9:ba.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.10' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-35-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
root@b0e5e154e7d5:~#

@edbond
Copy link
Author

edbond commented Aug 29, 2014

here is ssh -v output:

→  ssh -v -i insecure_key [email protected]
OpenSSH_6.4, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /home/eduard/.ssh/config
debug1: /home/eduard/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 51: Applying options for *
debug1: Connecting to 172.17.0.3 [172.17.0.3] port 22.
debug1: Connection established.
debug1: identity file insecure_key type -1
debug1: identity file insecure_key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.2
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.2 pat OpenSSH_5*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 [email protected]
debug1: kex: client->server aes128-ctr hmac-md5 [email protected]
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA sn:ip
debug1: Host '172.17.0.3' is known and matches the ECDSA host key.
debug1: Found key in /home/eduard/.ssh/known_hosts:1681
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: eduard@volcano
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: insecure_key
debug1: read PEM private key done: type RSA
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to 172.17.0.3 ([172.17.0.3]:22).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: Requesting authentication agent forwarding.
debug1: Sending environment.
debug1: Sending env LC_PAPER = uk_UA.utf8
debug1: Sending env LC_MONETARY = uk_UA.utf8
debug1: Sending env LC_NUMERIC = uk_UA.utf8
debug1: Sending env XMODIFIERS = @im=ibus
debug1: Sending env LANG = en_US.utf8
debug1: Sending env LC_MEASUREMENT = uk_UA.utf8
debug1: Sending env LC_TIME = uk_UA.utf8
debug1: channel 0: free: client-session, nchannels 1
Connection to 172.17.0.3 closed by remote host.
Connection to 172.17.0.3 closed.
Transferred: sent 3624, received 1472 bytes, in 0.0 seconds
Bytes per second: sent 2425815.1, received 985320.1
debug1: Exit status -1
debug1: compress outgoing: raw data 732, compressed 438, factor 0.60
debug1: compress incoming: raw data 17, compressed 10, factor 0.59

I ssh every day to remote servers without problems. Maybe locale problem?

@hectcastro
Copy link
Owner

Are you on RHEL? Some similarities in this issue regarding SELinux on RHEL: moby/moby#5032

@edbond
Copy link
Author

edbond commented Sep 1, 2014

yes, I'm using Fedora 20. Disabling SElinux didn't help.

snippets from ssh -vvvv

debug1: Trying private key: insecure_key
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA e9:5b:00:88:1f:53:ec:4b:b8:68:ce:50:f6:b4:27:66
debug2: we sent a publickey packet, wait for reply
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to 172.17.0.6 ([172.17.0.6]:22).

.......

debug3: Ignored env _system_name
debug3: Ignored env COLORTERM
debug3: Ignored env XAUTHORITY
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i0/0 o0/0 fd 4/5 cc -1)

Connection to 172.17.0.6 closed by remote host.
Connection to 172.17.0.6 closed.
Transferred: sent 3624, received 1472 bytes, in 0.0 seconds
Bytes per second: sent 1473740.3, received 598605.3
debug1: Exit status -1
debug1: compress outgoing: raw data 732, compressed 437, factor 0.60
debug1: compress incoming: raw data 17, compressed 10, factor 0.59

@hectcastro
Copy link
Owner

Can you please provide the output of docker version? I'll try to reproduce on Fedora 20.

@edbond
Copy link
Author

edbond commented Sep 1, 2014 

→  docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.2
Git commit (client): d84a070/1.1.2
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.2
Git commit (server): d84a070/1.1.2

Thank you

@edbond
Copy link
Author

edbond commented Sep 1, 2014 

→  yum info docker-io
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
Name        : docker-io
Arch        : x86_64
Version     : 1.1.2
Release     : 2.fc20

@hectcastro
Copy link
Owner

I just tried reproducing on a Fedora 20 Vagrant box. The main modification, other than installing the docker-io package, was to edit /etc/sysconfig/docker:

[vagrant@localhost ~]$ cat /etc/sysconfig/docker
# /etc/sysconfig/docker
OPTIONS=--selinux-enabled -H tcp://127.0.0.1:2375

From there, I downloaded the insecure_key and set the DOCKER_HOST environmental variable:

[vagrant@localhost ~]$ export DOCKER_HOST="tcp://127.0.0.1:2375"

Creating the Riak cluster succeeded:

[vagrant@localhost vagrant]$ DOCKER_RIAK_AUTOMATIC_CLUSTERING=1 DOCKER_RIAK_CLUSTER_SIZE=5 make start-cluster
./bin/start-cluster.sh

Bringing up cluster nodes:

  Successfully brought up [riak01]
  Successfully brought up [riak02]
  Successfully brought up [riak03]
  Successfully brought up [riak04]
  Successfully brought up [riak05]

Please wait approximately 30 seconds for the cluster to stabilize.

Then, I got the internal IP of one of the containers:

[vagrant@localhost vagrant]$ docker inspect a4737230156a | grep IPAddress
        "IPAddress": "172.17.0.4",

And successfully connected via SSH:

[vagrant@localhost ~]$ ssh -vvvv -i insecure_key [email protected]
OpenSSH_6.4, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 51: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 172.17.0.4 [172.17.0.4] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "insecure_key" as a RSA1 public key
debug1: identity file insecure_key type -1
debug1: identity file insecure_key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.2
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.2 pat OpenSSH_5*
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "172.17.0.4" from file "/home/vagrant/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/vagrant/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 1f:d6:c5:e4:1c:33:cd:5e:62:66:8a:2c:ab:3b:9f:12
debug3: load_hostkeys: loading entries for host "172.17.0.4" from file "/home/vagrant/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/vagrant/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host '172.17.0.4' is known and matches the ECDSA host key.
debug1: Found key in /home/vagrant/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: insecure_key ((nil)), explicit
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: insecure_key
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA e9:5b:00:88:1f:53:ec:4b:b8:68:ce:50:f6:b4:27:66
debug2: we sent a publickey packet, wait for reply
debug1: Authentication succeeded (publickey).
Authenticated to 172.17.0.4 ([172.17.0.4]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env HOSTNAME
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env HISTSIZE
debug3: Ignored env SSH_CLIENT
debug3: Ignored env DOCKER_HOST
debug3: Ignored env OLDPWD
debug3: Ignored env SSH_TTY
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env HISTCONTROL
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env LESSOPEN
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Mon Sep  1 17:06:04 2014 from 172.17.42.1
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-35-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
root@a4737230156a:~#

Any noticeable differences between those steps and what you're doing?

@edbond
Copy link
Author

edbond commented Sep 1, 2014

I run the same commands, here is a patch comparing ssh -vvvv output: https://gist.github.com/edbond/fdaf0f82dbec799f3876

@hectcastro
Copy link
Owner

You may want to try using this method of running commands within a Docker container. Perhaps then, you could restart SSH on another port without daemonizing to see if that provides any more information.

@cosmin-marginean
Copy link

cosmin-marginean commented Sep 1, 2016 •
edited
Loading

Having similar issues on OSX

$ ssh -vvvv -i insecure_key [email protected]
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /Users/cosmin/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 172.17.0.2 [172.17.0.2] port 22.

Stuck here ^

Cluster seems alright though:

$ make test-cluster | egrep -A6 "ring_members"
    "ring_members": [
        "[email protected]",
        "[email protected]",
        "[email protected]",
        "[email protected]",
        "[email protected]"
    ],

It's essential for us to ssh though as we need to perform some riak-admin tasks. Any suggestions are welcome.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Defect
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@edbond @hectcastro @cosmin-marginean