Timeout error when using NATS in a linkerd environment

[guhuajun]

Greetings,

I have two K3D clusters with linkerd enabled in my lab. One is for application, the other one is for middleware (like NATS) and database.

When playing latest technologies, sometimes you will encounter gotchars. Due to Linkerd's TCP Proxying and Protocol Detection feature, NATS traffic will be blocked by default. (A simple timeout error from appliction perspective.)

Linkerd maintains a default list of opaque ports that corresponds to the standard ports used by protocols that interact poorly with protocol detection. As of the 2.11 release, that list is: 25 (SMTP), 587 (SMTP), 3306 (MySQL), 4444 (Galera), 5432 (Postgres), 6379 (Redis), 9300 (ElasticSearch), and 11211 (Memcache).

Investigation:
I have a utility container with netcat equiped, also I have a notebook file with NATS test code. This utility container is also deployed to the same cluster and same namespace. It works fine because I don't need the auto injection feature from Linkerd for this utility container, so linkerd.io/inject: enabled is NOT added to the manifest file.

Then I have to wonder when two CNCF projects mixed up, what will be happend. Luckily, Bing directs me to the right Linkerd's documentation.

Workaround:
See above Linkerd's documentation, for my lab, i just add config.linkerd.io/skip-outbound-ports: "4222" annotation to my application manifest.

Future:
Maybe NATS team can team up with Linkerd team to add NATS ports to support protocol detetion or simplly add ports to Linkerd's opaque ports list?

Please feel free to close this issue without confirmation.

Thanks,
Greg

[guhuajun]

Discussion in Linkerd project, linkerd/linkerd2#8438

[wallyqs]

Hi @guhuajun thanks for sharing the solution/workaround for this. Not familiar with contributing to linkerd but it would be great for linkerd2 to have built-in support.