etcd3 seems te have a grpc-gateway issue wenn the gateway is connecting to etcd via https ( --listen-client-urls https://172.20.0.2:2379,https://127.0.0.1:2379)

[ntap-jbo]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

I get this error:
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.412619Z","caller":"grpclog/grpclog.go:37","msg":"[core][Channel #4 SubChannel #6] Subchannel Connectivity change to IDLE, last error: connection error: desc = "error reading server preface: remote error: tls: bad certificate""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.412821Z","caller":"grpclog/grpclog.go:37","msg":"[core][pick-first-lb 0xc001414d80] Received SubConn state update: 0xc001414f60, {ConnectivityState:IDLE ConnectionError:connection error: desc = "error reading server preface: remote error: tls: bad certificate"}"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.41287Z","caller":"grpclog/grpclog.go:37","msg":"[core][Channel #4 SubChannel #6] Subchannel Connectivity change to CONNECTING"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.412931Z","caller":"grpclog/grpclog.go:37","msg":"[core][Channel #4 SubChannel #6] Subchannel picks a new address "127.0.0.1:2379" to connect"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.41311Z","caller":"grpclog/grpclog.go:37","msg":"[core][pick-first-lb 0xc001414d80] Received SubConn state update: 0xc001414f60, {ConnectivityState:CONNECTING ConnectionError:}"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"warn","ts":"2024-05-08T10:30:19.415787Z","caller":"embed/config_logging.go:287","msg":"rejected connection","remote-addr":"127.0.0.1:40384","server-name":"","error":"tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.415902Z","caller":"grpclog/grpclog.go:37","msg":"[transport][client-transport 0xc000153b00] Closing: connection error: desc = "error reading server preface: remote error: tls: bad certificate""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.416043Z","caller":"grpclog/grpclog.go:37","msg":"[core]Creating new client transport to "{Addr: \"127.0.0.1:2379\", ServerName: \"127.0.0.1:2379\", }": connection error: desc = "error reading server preface: remote error: tls: bad certificate""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"warn","ts":"2024-05-08T10:30:19.416078Z","caller":"grpclog/grpclog.go:46","msg":"[core][Channel #4 SubChannel #6] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "error reading server preface: remote error: tls: bad certificate""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.416107Z","caller":"grpclog/grpclog.go:37","msg":"[core][Channel #4 SubChannel #6] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "error reading server preface: remote error: tls: bad certificate""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.416152Z","caller":"grpclog/grpclog.go:37","msg":"[core][pick-first-lb 0xc001414d80] Received SubConn state update: 0xc001414f60, {ConnectivityState:TRANSIENT_FAILURE ConnectionError:connection error: desc = "error reading server preface: remote error: tls: bad certificate"}"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.416204Z","caller":"grpclog/grpclog.go:37","msg":"[transport][client-transport 0xc000153b00] loopyWriter exiting with error: transport closed by client"}

What did you expect to happen?

That grpc-gateway can connect via https to etcd

How can we reproduce it (as minimally and precisely as possible)?

I've created the certificates like:
https://medium.com/nirman-tech-blog/setting-up-etcd-cluster-with-tls-authentication-enabled-49c44e4151bb
My instances are called etcd1 etc in stead of member-1 etc

My configuration /etc/systemd/system/etcd.service is:
[Unit]
Description=etcd service
Documentation=https://github.com/coreos/etcd

[Service]
User=root
Type=notify
ExecStart=/usr/bin/etcd
--name etcd1
--data-dir /var/lib/etcd
--initial-advertise-peer-urls https://172.20.0.2:2380
--listen-peer-urls https://172.20.0.2:2380
--listen-client-urls https://172.20.0.2:2379,https://127.0.0.1:2379
--advertise-client-urls https://172.20.0.2:2379
--initial-cluster-token etcd-cluster-1
--initial-cluster etcd1=https://172.20.0.2:2380,etcd2=https://172.20.0.3:2380,etcd3=https://172.20.0.4:2380
--client-cert-auth --trusted-ca-file=/etc/etcd/ca.pem
--cert-file=/etc/etcd/server.pem --key-file=/etc/etcd/server-key.pem
--peer-client-cert-auth --peer-trusted-ca-file=/etc/etcd/ca.pem
--peer-cert-file=/etc/etcd/etcd1.pem --peer-key-file=/etc/etcd/etcd1-key.pem
--initial-cluster-state new
--log-outputs="systemd/journal"
--logger="zap"
--log-level=debug
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Anything else we need to know?

When I change this line:
--listen-client-urls https://172.20.0.2:2379,https://127.0.0.1:2379
to
--listen-client-urls https://172.20.0.2:2379,http://127.0.0.1:2379 \

I don't have an issue anymore. (The set-up is a etcd3 Patroni setup.) with above setting (http) it works flawlessly.

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

etcdctl version: 3.4.32
API version: 3.4

and

etcdctl --cacert /etc/etcd/ca.pem --cert /etc/etcd/client.pem --key /etc/etcd/client-key.pem --endpoints=https://127.0.0.1:2379 version
etcdctl version: 3.5.13
API version: 3.5

Etcd configuration (command line flags or environment variables)

paste your configuration here

[Unit]
Description=etcd service
Documentation=https://github.com/coreos/etcd

[Service]
User=root
Type=notify
ExecStart=/usr/bin/etcd
--name etcd1
--data-dir /var/lib/etcd
--initial-advertise-peer-urls https://172.20.0.2:2380
--listen-peer-urls https://172.20.0.2:2380
--listen-client-urls https://172.20.0.2:2379,https://127.0.0.1:2379
--advertise-client-urls https://172.20.0.2:2379
--initial-cluster-token etcd-cluster-1
--initial-cluster etcd1=https://172.20.0.2:2380,etcd2=https://172.20.0.3:2380,etcd3=https://172.20.0.4:2380
--client-cert-auth --trusted-ca-file=/etc/etcd/ca.pem
--cert-file=/etc/etcd/server.pem --key-file=/etc/etcd/server-key.pem
--peer-client-cert-auth --peer-trusted-ca-file=/etc/etcd/ca.pem
--peer-cert-file=/etc/etcd/etcd1.pem --peer-key-file=/etc/etcd/etcd1-key.pem
--initial-cluster-state new
--log-outputs="systemd/journal"
--logger="zap"
--log-level=debug
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

root@etcd1:# etcdctl --cacert /etc/etcd/ca.pem --cert /etc/etcd/client.pem --key /etc/etcd/client-key.pem --endpoints=https://127.0.0.1:2379 member list -w table
+------------------+---------+-------+-------------------------+-------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-------+-------------------------+-------------------------+------------+
| 1ece3cda8fd346db | started | etcd1 | https://172.20.0.2:2380 | https://172.20.0.2:2379 | false |
| 4f614e32162d6bdb | started | etcd3 | https://172.20.0.4:2380 | https://172.20.0.4:2379 | false |
| d2878f3fa0c02172 | started | etcd2 | https://172.20.0.3:2380 | https://172.20.0.3:2379 | false |
+------------------+---------+-------+-------------------------+-------------------------+------------+
root@etcd1:#

Relevant log output

3.4.32

May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.412619Z","caller":"grpclog/grpclog.go:37","msg":"[core][Channel #4 SubChannel #6] Subchannel Connectivity change to IDLE, last error: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.412821Z","caller":"grpclog/grpclog.go:37","msg":"[core][pick-first-lb 0xc001414d80] Received SubConn state update: 0xc001414f60, {ConnectivityState:IDLE ConnectionError:connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"}"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.41287Z","caller":"grpclog/grpclog.go:37","msg":"[core][Channel #4 SubChannel #6] Subchannel Connectivity change to CONNECTING"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.412931Z","caller":"grpclog/grpclog.go:37","msg":"[core][Channel #4 SubChannel #6] Subchannel picks a new address \"127.0.0.1:2379\" to connect"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.41311Z","caller":"grpclog/grpclog.go:37","msg":"[core][pick-first-lb 0xc001414d80] Received SubConn state update: 0xc001414f60, {ConnectivityState:CONNECTING ConnectionError:<nil>}"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"warn","ts":"2024-05-08T10:30:19.415787Z","caller":"embed/config_logging.go:287","msg":"rejected connection","remote-addr":"127.0.0.1:40384","server-name":"","error":"tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.415902Z","caller":"grpclog/grpclog.go:37","msg":"[transport][client-transport 0xc000153b00] Closing: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.416043Z","caller":"grpclog/grpclog.go:37","msg":"[core]Creating new client transport to \"{Addr: \\\"127.0.0.1:2379\\\", ServerName: \\\"127.0.0.1:2379\\\", }\": connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"warn","ts":"2024-05-08T10:30:19.416078Z","caller":"grpclog/grpclog.go:46","msg":"[core][Channel #4 SubChannel #6] grpc: addrConn.createTransport failed to connect to {Addr: \"127.0.0.1:2379\", ServerName: \"127.0.0.1:2379\", }. Err: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.416107Z","caller":"grpclog/grpclog.go:37","msg":"[core][Channel #4 SubChannel #6] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.416152Z","caller":"grpclog/grpclog.go:37","msg":"[core][pick-first-lb 0xc001414d80] Received SubConn state update: 0xc001414f60, {ConnectivityState:TRANSIENT_FAILURE ConnectionError:connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"}"}
May 08 10:30:19 etcd1 etcd[602]: {"level":"info","ts":"2024-05-08T10:30:19.416204Z","caller":"grpclog/grpclog.go:37","msg":"[transport][client-transport 0xc000153b00] loopyWriter exiting with error: transport closed by client"}



3.5.13

May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.979952Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE, last error: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.980293Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] [pick-first-lb 0xc000fc0ba0] Received SubConn state update: 0xc000fc0e70, {ConnectivityState:IDLE ConnectionError:connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"}"}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.980383Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING"}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.980487Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] [Channel #3 SubChannel #4] Subchannel picks a new address \"127.0.0.1:2379\" to connect"}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.980927Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] [pick-first-lb 0xc000fc0ba0] Received SubConn state update: 0xc000fc0e70, {ConnectivityState:CONNECTING ConnectionError:<nil>}"}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.985105Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[transport] [client-transport 0xc000169d40] Closing: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.985325Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] Creating new client transport to \"{Addr: \\\"127.0.0.1:2379\\\", ServerName: \\\"127.0.0.1:2379\\\", }\": connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"warn","ts":"2024-05-08T10:45:57.985374Z","caller":"zapgrpc/zapgrpc.go:191","msg":"[core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: \"127.0.0.1:2379\", ServerName: \"127.0.0.1:2379\", }. Err: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.98541Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\""}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.985451Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] [pick-first-lb 0xc000fc0ba0] Received SubConn state update: 0xc000fc0e70, {ConnectivityState:TRANSIENT_FAILURE ConnectionError:connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"}"}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"info","ts":"2024-05-08T10:45:57.985483Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[transport] [client-transport 0xc000169d40] loopyWriter exiting with error: transport closed by client"}
May 08 10:45:57 etcd1 etcd[1213]: {"level":"warn","ts":"2024-05-08T10:45:57.98556Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:43624","server-name":"","error":"tls: failed to verify certificate: x509: certificate specifies an incompatible key usage"}

[jmhbnz]

Hey @ntap-jbo - Thanks for raising this. Based on your configuration provided you are using client-cert-auth. To the best of my knowledge this is not supported for grpc-gateway. Refer:

• Support for CN based authentication on grpc gateway #17025
• *: disable CommonName auth for gRPC-gateway #10366
• https://etcd.io/docs/v3.5/op-guide/authentication/rbac/#using-tls-common-name

As of version v3.2 if an etcd server is launched with the option --client-cert-auth=true, the field of Common Name (CN) in the client’s TLS cert will be used as an etcd user. In this case, the common name authenticates the user and the client does not need a password. Note that if both of 1. --client-cert-auth=true is passed and CN is provided by the client, and 2. username and password are provided by the client, the username and password based authentication is prioritized. Note that this feature cannot be used with gRPC-proxy and gRPC-gateway. This is because gRPC-proxy terminates TLS from its client so all the clients share a cert of the proxy. gRPC-gateway uses a TLS connection internally for transforming HTTP request to gRPC request so it shares the same limitation. Therefore the clients cannot provide their CN to the server correctly. gRPC-proxy will cause an error and stop if a given cert has non empty CN. gRPC-proxy returns an error which indicates that the client has an non empty CN in its cert.