8347406: [REDO] C1/C2 don't handle allocation failure properly during initialization (RuntimeStub::new_runtime_stub fatal crash)

[dafedafe]

Issue

The test src/hotspot/share/opto/c2compiler.cpp fails intermittently due to a crash that happens when trying to allocate code cache space for C1 and C2 in RuntimeStub::new_runtime_stub and SingletonBlob::operator new.

Causes

There are a few call paths during the initialization of C1 and C2 that can lead to the code cache allocations in RuntimeStub::new_runtime_stub (through RuntimeStub::operator new) and SingletonBlob::operator new triggering a fatal error if there is no more space. The paths in question are:

1. Compiler::init_c1_runtime -> Runtime1::initialize -> Runtime1::generate_blob_for -> Runtime1::generate_blob -> RuntimeStub::new_runtime_stub
2. C2Compiler::initialize -> C2Compiler::init_c2_runtime -> OptoRuntime::generate -> OptoRuntime::generate_stub -> Compile::Compile -> Compile::Code_Gen -> PhaseOutput::install -> PhaseOutput::install_stub -> RuntimeStub::new_runtime_stub
3. C2Compiler::initialize -> C2Compiler::init_c2_runtime -> OptoRuntime::generate -> OptoRuntime::generate_uncommon_trap_blob -> UncommonTrapBlob::create -> new UncommonTrapBlob
4. C2Compiler::initialize -> C2Compiler::init_c2_runtime -> OptoRuntime::generate -> OptoRuntime::generate_exception_blob -> ExceptionBlob::create -> new ExceptionBlob

Solution

Instead of fatally crashing the we can use the alloc_fail_is_fatal flag of RuntimeStub::new_runtime_stub to avoid crashing in cases 1 and 2 and add a similar flag to SingletonBlob::operator new for cases 3 and 4. In the latter case we need to adjust all calls accordingly.

Note: In JDK-8326615 it was argued that increasing the minimum code cache size would solve the issue but that wasn't entirely accurate: doing so possibly decreases the chances of a failed allocation in these 4 places but doesn't totally avoid it.

Testing

The original failing regression test in test/hotspot/jtreg/compiler/startup/StartupOutput.java has been modified to run multiple times with randomized values (within the original failing range) to increase the chances of hitting the fatal assertion.

Tests: Tier 1-4 (windows-x64, linux-x64, linux-aarch64, and macosx-x64; release and debug mode)

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8347406: [REDO] C1/C2 don't handle allocation failure properly during initialization (RuntimeStub::new_runtime_stub fatal crash) (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/23630/head:pull/23630
$ git checkout pull/23630

Update a local copy of the PR:
$ git checkout pull/23630
$ git pull https://git.openjdk.org/jdk.git pull/23630/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 23630

View PR using the GUI difftool:
$ git pr show -t 23630

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/23630.diff

[bridgekeeper]

👋 Welcome back dfenacci! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@dafedafe this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout JDK-8347406
git fetch https://git.openjdk.org/jdk.git master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push

[openjdk]

@dafedafe The following labels will be automatically applied to this pull request:

• hotspot
• shenandoah

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.