Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] [python-package] temporarily stop testing against scikit-learn nightlies, load lib_lightgbm earlier #6654

Merged
merged 10 commits into from
Sep 24, 2024
Prev Previous commit
Next Next commit
load libgomp.so.1 earlier
  • Loading branch information
jameslamb committed Sep 22, 2024
commit 7a8c6d0b8fbbac1c4288c38c4502ade5319b9643
43 changes: 43 additions & 0 deletions docs/FAQ.rst
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,49 @@ Detailed description of conflicts between multiple OpenMP instances is provided

If this is not your case, then you should find conflicting OpenMP library installations on your own and leave only one of them.

17. Loading LightGBM fails like: ``cannot allocate memory in static TLS block``
-------------------------------------------------------------------------------

When loading LightGBM, you may encounter errors like the following.

.. code-block:: console

lib/libgomp.so.1: cannot allocate memory in static TLS block

This most commonly happens on aarch64 Linux systems.

``gcc``'s OpenMP library (``libgomp.so``) tries to allocate a small amount of static thread-local storage ("TLS")
when it's dynamically loaded.

That error can happen when the loader isn't able to find a large enough block of memory.

On aarch64 Linux, processes and loaded libraries share the same pool of static TLS,
which makes such failures more likely. See these discussions:

* https://bugzilla.redhat.com/show_bug.cgi?id=1722181#c6
* https://gcc.gcc.gnu.narkive.com/vOXMQqLA/failure-to-dlopen-libgomp-due-to-static-tls-data

If you are experiencing this issue when using the ``lightgbm`` Python package, try upgrading
to at least ``v4.6.0``.

For older versions of the Python package, or for other LightGBM APIs, this issue can
often be avoided by loading ``libgomp.so.1``. That can be done directly by setting environment
variable ``LD_PRELOAD``, like this:

.. code-block:: console

export LD_PRELOAD=/root/miniconda3/envs/test-env/lib/libgomp.so.1

It can also be done indirectly by changing the order that other libraries are loaded
into processes, which varies by programming language and application type.

For more details, see these discussions:

* https://github.com/microsoft/LightGBM/pull/6654#issuecomment-2352014275
* https://github.com/microsoft/LightGBM/issues/6509
* https://maskray.me/blog/2021-02-14-all-about-thread-local-storage
* https://bugzilla.redhat.com/show_bug.cgi?id=1722181#c6

------

R-package
Expand Down
47 changes: 2 additions & 45 deletions python-package/lightgbm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,53 +4,10 @@
Contributors: https://github.com/microsoft/LightGBM/graphs/contributors.
"""

import platform

# gcc's libgomp tries to allocate a small amount of aligned static thread-local storage ("TLS")
# when it's dynamically loaded.
#
# If it's not able to find a block of aligned memory large enough, loading fails like this:
#
# > ../lib/libgomp.so.1: cannot allocate memory in static TLS block
#
# On aarch64 Linux, processes and loaded libraries share the same pool of static TLS,
# which makes such failures more likely on that architecture.
# (ref: https://bugzilla.redhat.com/show_bug.cgi?id=1722181#c6)
#
# Therefore, the later in a process libgomp.so is loaded, the greater the risk that loading
# it will fail in this way... so lightgbm tries to dlopen() it immediately, before any
# other imports or computation.
#
# This should generally be safe to do ... many other dynamically-loaded libraries have fallbacks
# that allow successful loading if there isn't sufficient static TLS available.
#
# libgomp.so absolutely needing it, by design, makes it a special case
# (ref: https://gcc.gcc.gnu.narkive.com/vOXMQqLA/failure-to-dlopen-libgomp-due-to-static-tls-data).
#
# other references:
#
# * https://github.com/microsoft/LightGBM/pull/6654#issuecomment-2352014275
# * https://github.com/microsoft/LightGBM/issues/6509
# * https://maskray.me/blog/2021-02-14-all-about-thread-local-storage
# * https://bugzilla.redhat.com/show_bug.cgi?id=1722181#c6
#
if platform.system().lower() == "linux" and platform.processor().lower() == "aarch64":
import ctypes

try:
# this issue seems specific to libgomp, so no need to attempt e.g. libomp or libiomp
_ = ctypes.CDLL("libgomp.so.1", ctypes.RTLD_GLOBAL)
except: # noqa: E722
# this needs to be try-catched, to handle these situations:
#
# * LightGBM built without OpenMP (-DUSE_OPENMP=OFF)
# * non-gcc OpenMP used (e.g. clang/libomp, icc/libiomp)
# * no file "libgomp.so" available to the linker (e.g. maybe only "libgomp.so.1")
#
pass

from pathlib import Path

# .basic is intentionally loaded as early as possible, to dlopen() lib_lightgbm.{dll,dylib,so}
# and its dependencies as early as possible
from .basic import Booster, Dataset, Sequence, register_logger
from .callback import EarlyStopException, early_stopping, log_evaluation, record_evaluation, reset_parameter
from .engine import CVBooster, cv, train
Expand Down
45 changes: 19 additions & 26 deletions python-package/lightgbm/basic.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
# coding: utf-8
"""Wrapper for C API of LightGBM."""

# This import causes lib_lightgbm.{dll,dylib,so} to be loaded.
# It's intentionally done here, as early as possible, to avoid issues like
# "libgomp.so.1: cannot allocate memory in static TLS block" on aarch64 Linux.
#
# For details, see the "cannot allocate memory in static TLS block" entry in docs/FAQ.rst.
from .libpath import _LIB # isort: skip

import abc
import ctypes
import inspect
Expand Down Expand Up @@ -37,7 +44,6 @@
pd_DataFrame,
pd_Series,
)
from .libpath import find_lib_path

if TYPE_CHECKING:
from typing import Literal
Expand Down Expand Up @@ -160,6 +166,12 @@
_MULTICLASS_OBJECTIVES = {"multiclass", "multiclassova", "multiclass_ova", "ova", "ovr", "softmax"}


class LightGBMError(Exception):
"""Error thrown by LightGBM."""

pass


def _is_zero(x: float) -> bool:
return -ZERO_THRESHOLD <= x <= ZERO_THRESHOLD

Expand Down Expand Up @@ -259,26 +271,13 @@ def _log_callback(msg: bytes) -> None:
_log_native(str(msg.decode("utf-8")))


def _load_lib() -> ctypes.CDLL:
"""Load LightGBM library."""
lib_path = find_lib_path()
lib = ctypes.cdll.LoadLibrary(lib_path[0])
lib.LGBM_GetLastError.restype = ctypes.c_char_p
callback = ctypes.CFUNCTYPE(None, ctypes.c_char_p)
lib.callback = callback(_log_callback) # type: ignore[attr-defined]
if lib.LGBM_RegisterLogCallback(lib.callback) != 0:
raise LightGBMError(lib.LGBM_GetLastError().decode("utf-8"))
return lib


# we don't need lib_lightgbm while building docs
_LIB: ctypes.CDLL
# connect the Python logger to logging in lib_lightgbm
if environ.get("LIGHTGBM_BUILD_DOC", False):
from unittest.mock import Mock # isort: skip

_LIB = Mock(ctypes.CDLL) # type: ignore
else:
_LIB = _load_lib()
_LIB.LGBM_GetLastError.restype = ctypes.c_char_p
callback = ctypes.CFUNCTYPE(None, ctypes.c_char_p)
_LIB.callback = callback(_log_callback) # type: ignore[attr-defined]
if _LIB.LGBM_RegisterLogCallback(_LIB.callback) != 0:
raise LightGBMError(_LIB.LGBM_GetLastError().decode("utf-8"))


_NUMERIC_TYPES = (int, float, bool)
Expand Down Expand Up @@ -552,12 +551,6 @@ def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
self.path.unlink()


class LightGBMError(Exception):
"""Error thrown by LightGBM."""

pass


# DeprecationWarning is not shown by default, so let's create our own with higher level
# ref: https://peps.python.org/pep-0565/#additional-use-case-for-futurewarning
class LGBMDeprecationWarning(FutureWarning):
Expand Down
14 changes: 13 additions & 1 deletion python-package/lightgbm/libpath.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
# coding: utf-8
"""Find the path to LightGBM dynamic library files."""

import ctypes
from os import environ
from pathlib import Path
from platform import system
from typing import List

__all__: List[str] = []


def find_lib_path() -> List[str]:
def _find_lib_path() -> List[str]:
"""Find the path to LightGBM library files.

Returns
Expand All @@ -35,3 +37,13 @@ def find_lib_path() -> List[str]:
dll_path_joined = "\n".join(map(str, dll_path))
raise Exception(f"Cannot find lightgbm library file in following paths:\n{dll_path_joined}")
return lib_path


# we don't need lib_lightgbm while building docs
_LIB: ctypes.CDLL
if environ.get("LIGHTGBM_BUILD_DOC", False):
from unittest.mock import Mock # isort: skip

_LIB = Mock(ctypes.CDLL) # type: ignore
else:
_LIB = ctypes.cdll.LoadLibrary(_find_lib_path()[0])
Loading