Unable to generate core dump when native library crashes

[lokesh-penukuduru]

How to generate core dump when a native library crash in a java service

I have a java service which is running in a Kubernetes cluster, uses a cpp native library via JNI and when there occurs a segmentation fault in native library, it was unable to generate the core dump, the logs are as follows,

> > #
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007efde450e688, pid=1, tid=103
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.13+11 (17.0.13+11) (build 17.0.13+11)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.13+11 (17.0.13+11, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc,
linux-amd64)
# Problematic frame:
# C  [lib.Converter.so.1.0+0xee688]  SetConverterMode+0xc8
#
# Core dump will be written. Default location: //core.1
#
# JFR recording file will be written. Location: //hs_err_pid1.jfr
#
# An error report file with more information is saved as:
# /nfs/my-folder/core_dump_%t.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at
pc=0x00007eff17d249a2]

Java version:

Picked up JAVA_TOOL_OPTIONS: -Xms1024m -Xmx4096m
openjdk version "17.0.13" 2024-10-15
OpenJDK Runtime Environment Temurin-17.0.13+11 (build 17.0.13+11)
OpenJDK 64-Bit Server VM Temurin-17.0.13+11 (build 17.0.13+11, mixed mode, sharing)

Your operating system and platform:

OS: linux (amd64)
OS Image: Amazon Linux 2
Kernel version: 5.10.230-223.885.amzn2.x86_64

[karianna]

@lokesh-penukuduru please try with 17.0.14 and report back.

[lokesh-penukuduru]

@karianna, but I see the latest tag of eclipse-temurin in docker hub was 17.0.13 at the moment. But I could see the specified version released here

[karianna]

Yeah it might be a few days before our official images get updated

[lokesh-penukuduru]

@karianna, here I have tried with 17.0.14 and the issue still persists

> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00007f1bd8001189, pid=1, tid=102
> #
> # JRE version: OpenJDK Runtime Environment Temurin-17.0.14+7 (17.0.14+7) (build 17.0.14+7)
> # Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.14+7 (17.0.14+7, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
> # Problematic frame:
> # C  [lib.Converter.so.1.0+0xee688]  SetConverterMode+0xc8
> #
> # Core dump will be written. Default location: /nfs/core.%t.1
> #
> # JFR recording file will be written. Location: //hs_err_pid1.jfr
> #
> # An error report file with more information is saved as:
> # /nfs/hs_err_pid.1_%t_%u.log
> #
> # If you would like to submit a bug report, please visit:
> #   https://github.com/adoptium/adoptium-support/issues
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> 
> [error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at
> pc=0x00007f1bdc4299a2]

It was generating the core dump at the specified nfs location when I kill using this command, 'kill -s SIGSEGV $$', but it was unable to create core dump when a crash is occurring in native library

[karianna]

What program does lib.Converter.so come from?

[lokesh-penukuduru]

This is a native C++ library used by our Java application via JNI.

[karianna]

OK, so the crash/bug is actually in that native library, Java is just propagating back the error message. You'll want to attach a C++ debugger to that lib in the SetConverterMode method

[lokesh-penukuduru]

Isn't the core dumped when a crash occurs in native library in my case?
My plan is to analyse that dump with the debuggers

[karianna]

It seems to think it's dumping the core at /nfs/core.%t.1

[lokesh-penukuduru]

@karianna, actually it is not dumping there, I can confirm this as there is no such file was present under the nfs, which is persistent storage. But I could see the error report file, "hs_err_pid.1_%t_%u.log" under the nfs.