Timeouts because of mutual tls?

[EhsanLunar]

Hey,
We are seeing this weird problem tls=no_tls_from_remote:

req id=5:0 proxy=in  src=100.127.0.0:32780 dst=100.127.0.24:9995 tls=no_tls_from_remote :method=GET :authority=100.127.0.24:9995 :path=/ping
req id=5:1 proxy=in  src=100.127.0.0:32782 dst=100.127.0.24:9995 tls=no_tls_from_remote :method=GET :authority=100.127.0.24:9995 :path=/ready
rsp id=5:0 proxy=in  src=100.127.0.0:32780 dst=100.127.0.24:9995 tls=no_tls_from_remote :status=200 latency=370µs
end id=5:0 proxy=in  src=100.127.0.0:32780 dst=100.127.0.24:9995 tls=no_tls_from_remote duration=24µs response-length=5B
rsp id=5:1 proxy=in  src=100.127.0.0:32782 dst=100.127.0.24:9995 tls=no_tls_from_remote :status=200 latency=506µs
end id=5:1 proxy=in  src=100.127.0.0:32782 dst=100.127.0.24:9995 tls=no_tls_from_remote duration=10µs response-length=3B

and direct connections must be mutually authenticated but it seems like it's complaining about the communication between the pods gateway and the container:

/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         100.127.0.0     0.0.0.0         UG    0      0        0 eth0
100.96.0.0      0.0.0.0         255.224.0.0     U     0      0        0 eth0

As you can see the 100.127.0.0 is the gateway for 100.127.0.24 and the communication between is saying tls=no_tls_from_remote and direct connections must be mutually authenticated.
In this example, I just took linkerd-proxy-injector from the linkerd namespace.
When we take a look at services deployed in our cluster we see the same pattern but with different error. So for example:

{"timestamp":"[ 99288.816774s]","level":"INFO","fields":{"message":"Request failed","error":"error trying to connect: connect timed out after 1s"},"target":"linkerd_app_core::errors::respond","spans":[{"name":"inbound"},{"port":3000,"name":"server"},{"client.addr":"100.124.0.0:42824","name":"rescue"}],"threadId":"ThreadId(1)"}

This example pod has the IP of 100.124.0.29 and the client.addr value (100.124.0.0:42824) is just the gateway IP of that pod.
Have we configured something badly or are we looking at the wrong things here? Do you have any idea why we see this?

[olix0r]

Let's unpack this a bit:

It looks like you're seeing tls=no_tls_from_remote on kubelet probes (/ready and /ping)? This is expected, since kubelet doesn't run in the mesh.

Regarding errors like direct connections must be mutually authenticated: where do you see these? We'd only expect to see this error message on connections that target port 4143 (i.e. connections that are "direct" to the outbound proxy and don't target an application port). Do you have port scanners or something that may be initiating unexpected connections to the proxy on 4143?

When we take a look at services deployed in our cluster we see the same pattern but with different error.

The connect timed out after 1s error indicates that the proxy was unable to get a connection on port 3000. What is connecting to the proxy injector on port 3000? This isn't a port exposed by the proxy injector pod.

Moreover, are you seeing any behavioral problems in your application? Or are you just trying to diagnose unexpected log messages?