Skip to content
This repository has been archived by the owner on Feb 2, 2023. It is now read-only.

getsockopt: connection timed out #24

Open
joonmoonlee opened this issue Jan 2, 2020 · 18 comments
Open

getsockopt: connection timed out #24

joonmoonlee opened this issue Jan 2, 2020 · 18 comments

Comments

@joonmoonlee
Copy link

Dear Team,

I completed successfully, but I couldn't add an array of the following message.
"Unable to connect. Error: Get https://192.168.0.110/api/api_version: dial tcp 192.168.0.110:443: getsockopt: connection timed out"

  • Need your help *
  1. Would you hint how to figure this problem out?

When I click "the array name" at "Add New Array" page during connecting the array, I was able to access the array, so I thought that the network was no problem.
A ping test from array to Pure1-Unplugged VM host was fine.

Thank you and regards,
Joonmoon Lee
@Pure-AdamuKaapan
Copy link
Collaborator

Hello! Thanks for filing your report.

To verify a couple things:

  1. Can you try pinging the opposite direction (from VM to array), just as a sanity check?
  2. Can you try doing curl https://192.168.0.110/api/api_version (or using Postman or your preferred request mechanism of choice) from another machine, to validate that the FA/FB's (whichever one you have) REST server is working as expected?

Let me know once you've tried those two things and we'll continue from there. Thanks!

@joonmoonlee
Copy link
Author

joonmoonlee commented Jan 5, 2020 •
edited
Loading

Dear Adam,

I did what you commented, I think that it didn't work out well.
I'm tried to but I don't know exactly such as using directory service.
If you have any comments, please let me know.

  1. The ping test was okay.
  2. The output of "curl https://192.168.0.110/api/api_version" the following messages
  • [root@localhost ~]# curl https://192.168.0.110/api/api_version
    curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
    More details here: http://curl.haxx.se/docs/sslcerts.html
    curl performs SSL certificate verification by default, using a "bundle"
    of Certificate Authority (CA) public keys (CA certs). If the default
    bundle file isn't adequate, you can specify an alternate file
    using the --cacert option.
    If this HTTPS server uses a certificate signed by a CA represented in
    the bundle, the certificate verification probably failed due to a
    problem with the certificate (it might be expired, or the name might
    not match the domain name in the URL).
    If you'd like to turn off curl's verification of the certificate, use
    the -k (or --insecure) option.

Thank you and best regards,
Joonmoon Lee

@joonmoonlee
Copy link
Author

Dear Adam,

WhenI run the "curl -k option", the output was "{"version": ["1.0", "1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "1.7", "1.8", "1.9", "1.10", "1.11", "1.12", "1.13", "1.14", "1.15", "1.16", "1.17", "2.0"]}"
Also my concern is that I didn't set up the directory service in config.yaml.

Thank you and regards,
Joonmoon Lee

@jinnam-lee
Copy link

@Pure-AdamuKaapan , do you have any further advice on what @joonmoonlee responded?

Thanks in advance

@ibernaldo
Copy link

Hi, I am suffering the same issue as described before, also I have tried the same steps to troubleshoot with ping command and curl and I obtained the same results: ping works perfectly and with curl I have to use -k option to avoid certificate verification checks. Seems that there is a problem with certificates, but I don't know how to troubleshoot this issue. May someone assists us? thanks in advance.

@Pure-AdamuKaapan
Copy link
Collaborator

Sorry, my apologies for the delay; it would indeed seem to be a certificate-related issue. We have some self-signed certs in use on our testing server today, so it's odd that they aren't working here.

What may be useful for debugging on my end is a log dump. Could both of you (@joonmoonlee , @ibernaldo ) run puctl infra packlogs and then send me the .tar.gz file that it creates? (It should look something like pure1-unplugged-logs-2020-01-08-18:36:36.tar.gz). Once that's done, please send it to me at [email protected] and I can take a closer look.

If you feel the need to go through and censor any of the files produced here, feel free to do so.
Thank you!

@ibernaldo
Copy link

ibernaldo commented Jan 8, 2020 via email

@Pure-AdamuKaapan
Copy link
Collaborator

Thanks @ibernaldo , I don't see much different in the logs unfortunately. I'm currently wondering if there's a firewall rule or something outside of the VM. Could you try the same curl command but inside the Unplugged VM? I found through experimenting with iptables that blocking port 443 produces the same "connection timed out" error.

Basically, the conclusion I'm coming to so far is that everything else about this situation is essentially identical to our testing server (self-signed certs, device REST server working correctly, ping works, etc.), so my suspicion is now on some external factors.

@ibernaldo
Copy link

ibernaldo commented Jan 8, 2020 via email

@jinnam-lee
Copy link

jinnam-lee commented Jan 8, 2020 •
edited
Loading

Hi @Pure-AdamuKaapan , I have sent you a log file directly on behalf of @joonmoonlee who will come online shortly.

Let us know if there are anything you want from the VM

By the way, here is the output of curl output in the VM
[root@localhost ~]# curl https://192.168.0.110/api/api_version
curl: (60) Peer's certificate issuer has been marked as not trusted by the user. More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.

[root@localhost ~]# curl -k https://192.168.0.110/api/api_version
{"version": ["1.0", "1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "1.7", "1.8", "1.9", "1.10", "1.11", "1.12", "1.13", "1.14", "1.15", "1.16", "1.17", "2.0"]}

@Pure-AdamuKaapan
Copy link
Collaborator

@ibernaldo the OS version should be the same no matter what, since the iso is an OS install disk. Adding -k is normal and expected, we have the same setup here, and you don't need to add the cert anywhere.

Honestly, since the VM can reach the API server just fine, I'll need to start doing some digging to figure out what in Kubernetes specifically might be breaking this, which could take a while (especially because I'm spread pretty thin right now). If you're so inclined, a reinstall can't hurt, and could help ensure it's a reproducible issue.

@joonmoonlee
Copy link
Author

Dear Adam, Jinnam, @ibernaldo

If I commented about firewalls about Pure Storage Korea Lab, there is "No FireWall" and it is only connecting to a dummy L2 network switch.
Also, I succeeded in the PureELK in the same environments two weeks ago.
I will check the IP Tables at CLI, but I think that I was inactive once I was testing.
By the way, if you want to access remotely, I'm willing to open Pure Storage Korea Lab.

Thank you and best regards,
Joonmoon Lee

@ibernaldo
Copy link

ibernaldo commented Jan 9, 2020 via email

@joonmoonlee
Copy link
Author

Hello Adam,

I double-checked the iptabes and firewalld, and then I stopped again and disable the services permanently.
But, the error displayed the same errors, so my conclusion was that environments were fine.
Would you help me to figure out?

[root@localhost ~]# systemctl stop iptables
Failed to stop iptables.service: Unit iptables.service not loaded.
[root@localhost ~]# systemctl stop ip6tables
Failed to stop ip6tables.service: Unit ip6tables.service not loaded.
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable ip6tables
Failed to execute operation: No such file or directory
[root@localhost ~]# systemctl disable iptables
Failed to execute operation: No such file or directory
[root@localhost ~]# systemctl disable firewalld

Thanks,
Joonmoon Lee

@joonmoonlee
Copy link
Author

Hello Adam,

I can open the lab's gears to you in the Korea office whenever you want.

Thank you and best regards,
Joonmoon Lee

@Pure-AdamuKaapan
Copy link
Collaborator

Hello again @joonmoonlee and @ibernaldo , thanks for all your information. I have some debugging ideas that I can do but will probably have to wait until tomorrow. If you wanted to get the connection available to me at least I can start poking around tomorrow morning. Feel free to either send an email or Slack message to me with instructions on how to connect and we can go from there. Thanks!

@joonmoonlee
Copy link
Author

Hello Team,

I tested Unplugged Pure1 on VMware 6.7 included the following detail environment.
But the error was the same, and I compared the Singapore lab's config.yaml but there was no difference except IP address.

  • If you have any additional guidance, would you share it?

  1. VMware ESXi

    1. Version: 6.7.0
    2. Build: Releasebuild-15160138
    3. Update: 3
    4. Patch: 89

  2. Purity FA: 5.3.4

  3. 443 ports open and IP Tables disable.

Thank you and best regards,
Joonmoon Lee

@jinnam-lee
Copy link

jinnam-lee commented Mar 13, 2020 •
edited
Loading

@joonmoonlee The issue was due to similar 198.168.x.x network used by podCIDR.
So, it was solved by changing an IP address in config.yaml as follows;
podCIDR: "192.168.0.0/16" to podCIDR: "10.112.0.12/12"

I remotely access your lab, then fixed the problem accordingly.

This change should be added into up-to-date installation guide for future Pure1_Unplugged installation.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jinnam-lee @Pure-AdamuKaapan @joonmoonlee @ibernaldo