8324345: Stack overflow during C2 compilation when splitting memory phi

[satyenme]

Backporting JDK-8324345: Stack overflow during C2 compilation when splitting memory phi. In some cases ConnectionGraph::find_inst_mem contains recursive calls that can lead to a native C++ stack overflow - this changeset takes the following action to address this. Adds a simple recursion depth limit to ConnectionGraph::find_inst_mem. When find_inst_mem reaches the limit, we bail out from the current compilation and retry without escape analysis. The depth limit is currently set to a fixed value of 1000, and a regression test TestFindInstMemRecursion.java. Also removes the implicit memory limit (due to JDK-8331185) from VectorReplicateLongSpecialImmTest.java: The memory limit triggers after this changeset, but the (potential) problem is most likely unrelated to this issue, addressed in JDK-8337821. Ran GHA Sanity Checks, local Tier 1 and 2, and new test/hotspot/jtreg/compiler/escapeAnalysis/TestFindInstMemRecursion.java tests. Patch is clean.

---

Progress

• Change must not contain extraneous whitespace
• JDK-8324345 needs maintainer approval
• Commit message must refer to an issue

Issue

• JDK-8324345: Stack overflow during C2 compilation when splitting memory phi (Bug - P3 - Requested)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk21u-dev.git pull/1692/head:pull/1692
$ git checkout pull/1692

Update a local copy of the PR:
$ git checkout pull/1692
$ git pull https://git.openjdk.org/jdk21u-dev.git pull/1692/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 1692

View PR using the GUI difftool:
$ git pr show -t 1692

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk21u-dev/pull/1692.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back ssubramaniam! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[openjdk]

⚠️ @satyenme This change is now ready for you to apply for maintainer approval. This can be done directly in each associated issue or by using the /approval command.

[mlbridge]

Webrevs

• 00: Full (012db9d6)

[satyenme]

/approval request for backport. Fix addresses cases where ConnectionGraph::find_inst_mem contains recursive calls that leads to a native C++ stack overflow.

Risk: Medium. Tip change was merged in July 2024, but touches compilation logic. Adds test to help mitigate risk.

Testing:

• GHA Sanity Checks
• Tier 1 and 2 tests locally

[openjdk]

@satyenme
8324345: The approval request has been created successfully.