scylla-jmx OOM when running 5k tables backup tasks

[fgelcer]

it also failed to dump when crashing due to "Permission denied" error:

Dec 25 09:48:05 ip-10-0-244-119.eu-west-1.compute.internal scylla-jmx[3305]: Starting the JMX server
Dec 25 09:49:32 ip-10-0-244-119.eu-west-1.compute.internal scylla-jmx[3305]: java.lang.OutOfMemoryError: Java heap space
Dec 25 09:49:32 ip-10-0-244-119.eu-west-1.compute.internal scylla-jmx[3305]: Dumping heap to java_pid3305.hprof ...
Dec 25 09:49:32 ip-10-0-244-119.eu-west-1.compute.internal scylla-jmx[3305]: Unable to create java_pid3305.hprof: Permission denied

logs will be uploaded and posted in next comment.

[fgelcer]

logs:
https://drive.google.com/file/d/1iw_do7gcc08uLY4ukYSwojiJtjCwp_af/view?usp=sharing
https://drive.google.com/file/d/11GbJaTydOK8KDCdZ2s0v3y-b4xQQ4Jo1/view?usp=sharing

[slivne]

you need to change scylla-jmx memory .... /usr/lib/scylla/jmx/scylla-jmx change memory to be
-Xmx1024m

[fgelcer]

you need to change scylla-jmx memory .... /usr/lib/scylla/jmx/scylla-jmx change memory to be
-Xmx1024m

yes, this did the trick... jmx is now up and running.

[tzach]

you need to change scylla-jmx memory .... /usr/lib/scylla/jmx/scylla-jmx change memory to be
-Xmx1024m

@slivne why not making this the default? (at least for anything bigger than t2)

[avikivity]

Many machines don't have a gigabyte to spare.

Looks like we're consuming 100-200kB per table, that's crazy.

[tzach]

Many machines don't have a gigabyte to spare.

Sure, this is why it can be a per machine (or percent) parameter.
It does not make sense to let users fail at this time after time.