ScyllaDB JMX opens random port on all interfaces

[tnozicka]

This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at [email protected] or in our slack channel.

• I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): 5.4.3
Cluster size: 1
OS (RHEL/CentOS/Ubuntu/AWS AMI): container image

Using ScyllaDB container image, the JMX service opens a random port on all interfaces. This is bad for security.

1. The JMX itself listens only on localhost (7199), but the random port listens for remote connections as well.
2. The port is random that makes it hard to configure networking or validate allowed exposed ports

$ podman run -it --rm --entrypoint=/usr/bin/bash docker.io/scylladb/scylla:5.4.3 '-c' 'supervisord -c /etc/supervisord.conf& apt update && apt install -y net-tools && netstat -tulpn | grep java && ps faux | grep jmx'
...
tcp6       0      0 127.0.0.1:7199          :::*                    LISTEN      207/java            
tcp6       0      0 :::46055                :::*                    LISTEN      207/java            
...
root         207 50.7  0.1 3080520 91468 pts/0   Sl   12:57   0:02  \_ /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java -Xmx256m -XX:+UseSerialGC -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.host=localhost -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7199 -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.rmi.port=7199 -Djavax.management.builder.initial=com.scylladb.jmx.utils.APIBuilder -jar /opt/scylladb/jmx/scylla-jmx-1.0.jar
...

[tnozicka]

@mykaul the random port blocks security validation for the operator (scylladb/scylla-operator#1759) do we have anyone to assign this to?
(I can't even allow the port because it's random.)

[denesb]

If all goes well, JMX is going to be retired in 6.0 (at least from base installations), as well as java in general.
Will that solve your problem? Or will you need this fixed in older versions too?

[tnozicka]

As much as I like JMX going away, we'll have to live with the older versions + enterprise for quite a bit :(

[denesb]

You will need @mykaul to find you somebody who can actually Java, because I can't.

[mykaul]

I wonder if it happens only with IPv6.

[tnozicka]

I wonder if it happens only with IPv6.

the tcp6 in the output here comes from kernel and means IPv4 + IPv6 (listens on random port on both stacks)

[tarzanek]

so this bug should be in https://github.com/scylladb/scylla-jmx

[tarzanek]

@elcallio @amnonh do you guys remember why there is extra port open and who allocates it after registermbean in https://github.com/scylladb/scylla-jmx/blob/master/src/main/java/com/scylladb/jmx/main/Main.java ?

could it be some debug leftover?

[tarzanek]

I think I might have an answer - it's about local jconsole connection it seems - https://www.baeldung.com/jmx-ports
so

  -Dcom.sun.management.jmxremote.port=1234 
  -Dcom.sun.management.jmxremote.rmi.port=1234 
  -Dcom.sun.management.jmxremote.local.port=1235 

might fix this and bind the port to static
wondering what -XX:+DisableAttachMechanism will do

[tarzanek]

so try to fix here https://github.com/scylladb/scylla-jmx/blob/master/scripts/scylla-jmx#L134 ?
(check service for install location of this script, on non container changing this and restarting scylla-jmx will show the effect asap, so it is possible it's a tiny config change/fix)

[tarzanek]

huh ... and I didn't realize but it says that .local.port is there since JDK 16 only ... d'oh :-(

[tnozicka]

@mykaul can you please move this to https://github.com/scylladb/scylla-jmx? How do we ensure this gets an assignee and not linger around?

[mykaul]

We are deprecating jmx in 6.0 or 6.1 the latest.