Smooth migration story: mapping or supporting critical OpenJDK options

[zl-wang]

While it needs to be investigated if there is any non -XX options still to be recognized (these are functional options in some senses ... I am aware of none), the priority is on mapping or supporting performance-sensitive critical OpenJDK options ... for migration or transparent drop-in.

I understood this is necessarily on-going, we have to collect this important set of -XX options. To start off, I listed the GC thread options (which are repetitively mentioned to be performance-sensitive when engaging clients), and other important options as they come up:

1. -XX:ParallelGCThreads=N (this just is a simple mapping to -XgcthreadsN)
2. -XX:ConcGCThreads=N (I hoped this is a simple mapping as well: -XconcurrentlevelN or the corresponding option in parallel scavenger)
3. -XX:ParallelCMSThreads=N (I don't know if we have something similar)
4. -XX:+UseLargePages (-Xlp seemed applicable)
5. -XX:LargePageSizeInBytes=2m (-Xlp2m seemed matching)

[zl-wang]

@gita-omr @vijaysun-omr @mstoodle @pshipton @DanHeidinga

[pshipton]

@charliegracie @dmitripivkine @amicic

[AlenBadel]

Table of Contents

Hotspot Option	Issue	Implementation	Docs
-XX:ParallelCMSThreads=N	#8655	#7723	eclipse-openj9/openj9-docs#464
-XX:ConcGCThreads=N	#8655	#7723	eclipse-openj9/openj9-docs#464
-XX:ParallelGCThreads=N	#8655	#7723	eclipse-openj9/openj9-docs#464
-XX:+UseLargePages	#8671	#7476 eclipse-omr/omr#4762	TBD
-XX:LargePageSizeInBytes=<size>	#8671	#7476 eclipse-omr/omr#4762	TBD

Related Changes

Description	PR	Issues
-Xlp<size>, -Xlp:codecache:pagesize<size> order priority	#7769
-Xlp<Size> Fix broken functionality	#7722	#7667

[dmitripivkine]

Just note that -Xlp format for object heap is -Xlp:objectheap:pagesize=<size>[,[non]pageble]]
where [non]pageble] is applicable for ZOS only.
Default behaviour might depend on platform. xLinux for instance would use 2M (large page) by default however pLinux default is 64k (small page) not 1g (large page)

[AlenBadel]

@dmitripivkine

Default behaviour might depend on platform. xLinux for instance would use 2M (large page) by default however pLinux default is 64k (small page) not 1g (large page)

Does the JVM pull this from the OS defined large pages? Eg On Linux this is called hugepages which is set to 2M by default.

# cat /proc/meminfo | grep Huge
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

If the user doubles Hugepagesize will the default value used by -Xlp (both objectheap and codecache) use that new value, or is the default value statically defined?
Only asking since this is the behaviour of XX:+UseLargePages

[DanHeidinga]

@AlenBadel Do you know whether the -XX:LargePageSizeInBytes=<size> effects both the heap and the code cache for Hotspot VMs? What about the failure cases where the sizes don't match?

We need to be careful that mapped options do the "right" thing on OpenJ9 that maps as closely as possible to the expected behaviour.

[AlenBadel]

Do you know whether the -XX:LargePageSizeInBytes=<size> effects both the heap and the code cache for Hotspot VMs?

Yes, on Hotspot VMs -XX:LargePageSizeInBytes=<size> effects both the heap and the code cache.

See

-XX:LargePageSizeInBytes=256m
Causes the Java heap, including the permanent generation, and the compiled code cache to use as a minimum size one 256 MB page (for those platforms which support it).
https://www.oracle.com/technetwork/java/tuning-139912.html

[LinHu2016]

XX:+UseLargePages can be easily mapped to -xlp.
However that changes in Java 9+, where -xlp is deprecated in exchange we need to enable both -Xlp:codecache and -Xlp:objectheap.

Here I'm assuming that -Xlp:codecache and -Xlp:objectheap will use OS defined large page sizes by default if no value is passed. That's what XX:+UseLargePages does.

-XX:LargePageSizeInBytes=<size> similarly can be mapped to -Xlp:codecache=<size> and
-Xlp:objectheap=<size>. Only issue here is that we don't have the infrastructure to map one to many, and that needs to be implemented.

-XX:ParallelCMSThreads=N was deprecated in JDK9+. -XconcurrentlevelN seems like the best candidate. Since it controls the number of concurrent GC threads for global marking in gencon.
Gencon is probably the best candidate GC to match CMS, along with another option to enable concurrent sweep.

@LinHu2016 Would you have any background on XconcurrentbackgroundN? I wasn't able to find any documentation on that option.

-XX:ConcGCThreads=N and -XX:ParallelGCThreads=N are closely related on Hotspot and are used to control their G1 GC. The relation by default is something like ConcGcThreads = (ParallelGCThreads + 3)/4.
I could only guess to use XconcurrentlevelN again here, but it's not obvious since it's not clear since these are G1 specific options.

Hi @AlenBadel, here is document link for -Xconcurrentbackground (https://www.eclipse.org/openj9/docs/xconcurrentbackground/), I still think -Xconcurrentbackground option is the most close to -XX:ConcGCThreads=N on hotspot(-XconcurrentlevelN is Allocation “tax” rate, in OpenJ9 we do have different implementation, which ask mutator threads to pay “tax” during concurrent mark)

[dmitripivkine]

Default behaviour might depend on platform. xLinux for instance would use 2M (large page) by default however pLinux default is 64k (small page) not 1g (large page)

Does the JVM pull this from the OS defined large pages? Eg On Linux this is called hugepages which is set to 2M by default.

# cat /proc/meminfo | grep Huge
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

If the user doubles Hugepagesize will the default value used by -Xlp (both objectheap and codecache) use that new value, or is the default value statically defined?
Only asking since this is the behaviour of XX:+UseLargePages

No. Preferred page sizes for objectheap are hardcoded https://github.com/eclipse/omr/blob/master/gc/base/GCExtensionsBase.cpp#L133
If preferred page size is not provisioned (known) in OS a smallest page size available would be selected (in case if there is no -Xlp request).
Preferred page sizes in many cases are selected because of support in OS/HW and could provide best performance like 64k on AIX or 1m pageable on ZOS.
At startup time Port Library structure (see omrvmem_supported_page_sizes()) is initialized with provisioned (known) page sizes in OS. Most of platforms would have one or two available sizes however there can be more than two. For example AIX can support [4k, 64k, 16m, 16g] or ZOS [4k_nonPageable, 1m_Pageable, 2g_nonpageable].
In most of the cases Large Pages are shared pages so using of them required extra provisioning (reservation) in OS. In case if memory allocation in LPs is requested but can not be fulfilled die lack of them in OS the warning would be issued an allocation would be attempted with smaller pages.
An importance of -Xlp options in any form on Linux platforms has been reduced significantly last years due Anonymous Huge Pages support. The customer can get Huge Pages benefits without any extra provisioning

[AlenBadel]

I still think -Xconcurrentbackground option is the most close to -XX:ConcGCThreads=N on hotspot(-XconcurrentlevelN is Allocation “tax” rate, in OpenJ9 we do have different implementation, which ask mutator threads to pay “tax” during concurrent mark)

I agree. -Xconcurrentbackground should be used for both.

The objectheap then won't use the OS defined "Huge Pages"(Linux) unless we specify it using -Xlp:objectheap=< size of huge pages >.
To effectively map X:+UseLargePages we would need to immediately extract the size of the OS defined Huge Pages, or AIX, ZOS equivalent and map it to xlp:objectheap= < size of extracted huge pages>.

Need to figure out if this is the same case for the codecache.

[dmitripivkine]

The objectheap then won't use the OS defined "Huge Pages"(Linux) unless we specify it using -Xlp:objectheap=< size of huge pages >.

except it is hardcoded 2m for xLinux or 1m for zLinux

[AlenBadel]

except it is hardcoded 2m for xLinux or 1m for zLinux

@dmitripivkine
Even if we specify the size of the maximum page using -Xlp:objectheap=< size > ? I thought it was hard coded only the case that no size is specified.

[dmitripivkine]

except it is hardcoded 2m for xLinux or 1m for zLinux

@dmitripivkine
Even if we specify the size of the maximum page using -Xlp:objectheap=< size > ? I thought it was hard coded only the case that no size is specified.

Default value would be overwritten by option if requested

[AlenBadel]

Default value would be overwritten by option if requested

That was what I meant. My apologies if that wasn't clear.

[AlenBadel]

I've updated the changes and mapping required #4930 (comment).

[gita-omr]

Would it be possible to summarize the default HotSpot and OpenJ9 behaviour for the pages sizes? For example:

1. How are the page sizes chosen when no XX:+UseLargePages or -Xlp is specified on the command line for HS and OpenJ9 correspondingly?
2. How are the page sizes selected when those options are specified in each VM correspondingly?
3. Are we going to change anything in terms of the actual page sizes chosen when XX:+UseLargePages is specified with OpenJ9?
4. And just to confirm: specifying page size using -XX:LargePageSizeInBytes=<size> or -Xlp:<size> on the command line overrides all of the above.

[AlenBadel]

• How are the page sizes chosen when no XX:+UseLargePages or -Xlp is specified on the command line for HS and OpenJ9 correspondingly?

On Hotspot I don't have enough knowledge on how page sizes are chosen without specifying an option.
On OpenJ9 these values are hard coded and differ based on architecture.

As Dmitri commented:

Preferred page sizes for objectheap are hardcoded https://github.com/eclipse/omr/blob/master/gc/base/GCExtensionsBase.cpp#L133

How are the page sizes selected when those options are specified in each VM correspondingly?

As I've noted

XX:+UseLargePages
Sets the largest page file size used to be of what is set in the OS (or hypervisor, based on platform).

Xlp does exactly the same.

• Are we going to change anything in terms of the actual page sizes chosen when XX:+UseLargePages is specified with OpenJ9?

No, we're simply mapping XX:+UseLargePages to (Xlp: Java8) (Xlp:objectheap and Xlp:codecache Java 9+)

• And just to confirm: specifying page size using -XX:LargePageSizeInBytes= or -Xlp: on the command line overrides all of the above.

-XX:LargePageSizeInBytes= overrides any hardcoded value, or default value with the size of the page passed in by the user.

[dmitripivkine]

• Are we going to change anything in terms of the actual page sizes chosen when XX:+UseLargePages is specified with OpenJ9?

No, we're simply mapping XX:+UseLargePages to (Xlp: Java8) (Xlp:objectheap and Xlp:codecache Java 9+)

Why do you need to do something for Java 8 differently? Java 8 and higher should have identical implementations

[AlenBadel]

Why do you need to do something for Java 8 differently? Java 8 and higher should have identical implementations

This was a consideration due to the deprecation of -Xlp in Java9+.
https://www.eclipse.org/openj9/docs/xlp/

It would depend on the availability of -Xlp:codecache and -Xlp:objectheap. If they are available in Java 8, and the behaviour is the same then these options will be mapped instead of -Xlp.

[AlenBadel]

From my understanding -Xlp is deprecated, but is still accepted and is functionally sane in Java9+. Could anyone confirm this?

To avoid making changes to the mapping infrastructure, and implementing one to many mapping we could do one of two things.

If using -Xlp is functionally sane, then we can map -XX:+UseLargePages and -XX:LargePageSizeInBytes=<size> to -Xlp instead of both -Xlp:codecache, and -Xlp:opbjectheap. Which should be equivalent mappings.
This of course would need to be changed later on if we choose to remove -Xlp handling.

Keeping in mind that the codecache is much smaller than the objectheap. The performance yields would be higher using larger pages on an objectheap versus codecache. We can consider simply mapping -XX:+UseLargePages, and -XX:LargePageSizeInBytes=<size> to -xlp:objectheap. I don't see any potential issues with codecache using different page sizes as the objectheap.

[dmitripivkine]

I believe this is movement in wrong direction. -Xlp functionality is not exactly the same as -Xlp:objectheap:* -Xlp:codecache:* for obsolete backwards compatibility issues. Also an implementation should not limit required functionality. So you can not say we would set -Xlp:objectheap: but not codecache because of limitations of implementation.

[pshipton]

Doesn't seem anything is happening here for the 0.17 release, moving to the next one.

[AlenBadel]

Is this confirmed?

Not yet, as designed that's the intention. I'll link the issue you've created #8562. If that's broken, and that it's the case that 64K pages are not being used by default then it'll be fixed to make sure that 64K pages are used by default.

[gita-omr]

I think this discussion is moving in a wrong direction again. I don't think the purpose of this issue is to fix/document everything about the current -Xlp behavior. I think all the design at #4930 (comment) should say is that -XX:UseLargePages will select page size and type in exactly the same way as -Xlp, except for zOS. Then, describe the zOS behavior. I think it was written that way at some point but then got lost after editing. Also I believe zOS is not supported by OpenJ9 so it's a bit sad that this issue is being delayed so much due to it.

I also recommend to remove all the text in #4930 (comment) after the table since a lot of it is a duplication and only causes more confusion.

And, of course, all the discovered bugs and potential improvements should be addressed in separate issues.

[AlenBadel]

@dmitripivkine Does #4930 (comment) answer your remaining concerns?

@joransiu Just a reminder that I need your OK on Z/OS relevant proposals here.

[AlenBadel]

I've went ahead and cleaned this issue up a bit. There was a lot going on.

I've created two children issues to address the options in two different groups. They're separated as general GC options, and large page related options.
The table of contents can be seen here: #4930 (comment)

[pshipton]

Seems this won't be resolved in the 0.20 release, moving it forward.

[pshipton]

As this wasn't resolved before branching for the 0.21 release, moving to the next.

[amicic]

We are trying to add similar (possibly identical) functionality to OpenJDK's -XX:+AlwaysPreTouch here: #10386. Anything else we should do, if we match the option name?

[DanHeidinga]

We are trying to add similar (possibly identical) functionality to OpenJDK's -XX:+AlwaysPreTouch here: #10386. Anything else we should do, if we match the option name?

We should reuse the existing OpenJDK option and avoid introducing a similar feature under a different name. In general, all new options should be of the form -XX:[+-]Something or -XX:Something.

[dmitripivkine]

moved to next milestone

[dmitripivkine]

There is no activity or resources to work on this, put it to backlog