Initial implementation of lazy JLLs for LinearAlgebra #57719

staticfloat · 2025-03-11T13:31:02Z

This alters CompilerSupportLibraries_jll, OpenBLAS_jll and
libblastrampoline_jll to use LazyLibrary objects and thereby be loaded
only upon first dlopen() or ccall() to the individual library
objects. Note that this is one of the more complicated cases, as
libblastrampoline must have OpenBLAS_jll added as a dynamic dependency
(as it does not actually have it listed in its shared object headers)
and also has some on-load callbacks that must be invoked.

Long-term, I would like to replace the bespoke JLLs here with actual JLL
source code, and vendor a version of LazyJLLWrappers in-tree to do the
actual code generation even for Base. That is left as future work.

[0] JuliaLang/LinearAlgebra.jl#1235

This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. This must be paired with the appropriate base Julia changes [0]. [0] JuliaLang/julia#57719

staticfloat · 2025-03-11T21:12:46Z

This is working locally now, I can confirm that Libdl.dllist() shows that libgfortran, libopenblas64_ and libblastrampoline are all not loaded until the first Linear Algebra call!

Note that this is attempting to conform to the LazyJLLWrappers API, which includes things like eager_mode(), which is called by JLLWrappers.jl. This is to provide backwards-compatibility with JLLs that expect all dependencies from all dependent JLLs to have already dlopen()'ed their libraries.

What this means is that for any realistic workload where there are JLLs that depend on LBT/OpenBLAS, everything is going to get eagerly loaded upon loading of that JLL anyway, until those JLLs are built to use LazyJLLWrappers instead of JLLWrappers, which functionally means waiting for BB2 to be ready, as it contains the new analysis passes that track things like library interdependencies, etc... However, for workloads that never use linear algebra workloads, this will be a nice savings, as the libraries will now never get loaded!

topolarity · 2025-03-11T22:48:07Z

The --trim failure should be fixed up by #57734

staticfloat · 2025-03-11T23:57:03Z

I've converted this to draft because I realized that while "real" JLLs that use JLLWrappers will correctly call eager_mode() on these lazy stdlib JLLs, other stdlib JLLs that are not lazy yet but that depend on any of these new lazy stdlib JLLs won't, I need to add that in here.

This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. This must be paired with the appropriate base Julia changes [0]. [0] JuliaLang/julia#57719

topolarity · 2025-03-12T02:19:13Z

It's also worth noting that once we land this PR, --trim will not support loading BLAS until we fix #57707

base/libdl.jl

stdlib/CompilerSupportLibraries_jll/src/CompilerSupportLibraries_jll.jl

topolarity · 2025-03-12T20:12:16Z

I think you might need a rebase for the trimming test to pass, and the following patch:

diff --git i/base/libdl.jl w/base/libdl.jl
index 77f608e77e..d94a2fd0e2 100644
--- i/base/libdl.jl
+++ w/base/libdl.jl
@@ -333,7 +333,7 @@ struct LazyLibraryPath
     pieces::Tuple{Vararg{Any}}
     LazyLibraryPath(pieces...) = new(pieces)
 end
-Base.string(llp::LazyLibraryPath) = joinpath(String[string(p) for p in llp.pieces])
+@inline Base.string(llp::LazyLibraryPath) = joinpath(String[string(p) for p in llp.pieces])
 Base.cconvert(::Type{Cstring}, llp::LazyLibraryPath) = Base.cconvert(Cstring, string(llp))
 # Define `print` so that we can wrap this in a `LazyString`
 Base.print(io::IO, llp::LazyLibraryPath) = print(io, string(llp))

The robust solution to this will require a callable of some kind, but for now massaging is required to make sure that the components of the path are visible to the optimizer

This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. This must be paired with the appropriate base Julia changes [0]. [0] JuliaLang/julia#57719

This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. Long-term, I would like to replace the bespoke JLLs here with actual JLL source code, and vendor a version of `LazyJLLWrappers` in-tree to do the actual code generation even for Base. That is left as future work. This must be paired with the appropriate `LinearAlgebra.jl` changes [0]. [0] JuliaLang/LinearAlgebra.jl#1235

This makes more sense, reduces work, and avoids that perennial thorn in my side, that in-tree builds store libraries in `${prefix}/lib`, whereas installed builds store libraries in `${prefix}/lib/julia`. Co-authored-by: Cody Tapscott <[email protected]>

We can no longer depend on the fact that these libraries will be eagerly loaded (huzzah!) so we must teach these helper functions to either inspect the lazy CSL JLL for the necessary information, or just directly load the library.

This includes the changes needed for lazy LinearAlgebra JLLs.

staticfloat · 2025-03-16T21:09:39Z

Tests passed, so I merged the LinearAlgebra PR, adjusted the final commit here to bump the LinearAlgebra stdlib, and once this goes green, we're good to merge!

nanosoldier · 2025-03-17T20:51:52Z

Your job submission was not accepted: invalid argument to vs keyword.

topolarity · 2025-03-17T20:53:08Z

@nanosoldier runtests(vs = "@04922e7e205237ab01cb009b2684bcbb97b053cf")

topolarity · 2025-03-17T23:44:19Z

It looks like this might have broken a test on FreeBSD: https://buildkite.com/julialang/julia-master/builds/45925#0195a580-58c7-45e6-a297-b1adcc81d8b6/770-1576

The commits just prior this on master are passing this test, in comparison

giordano · 2025-03-17T23:50:17Z

#57808

topolarity · 2025-03-18T00:09:30Z

Fix in #57811

nanosoldier · 2025-03-18T15:11:31Z

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

Report summary

❗ Packages that crashed

1 packages crashed only on the current version.

An internal error was encountered: 1 packages

4 packages crashed on the previous version too.

✖ Packages that failed

31 packages failed only on the current version.

Package fails to precompile: 5 packages
Illegal method overwrites during precompilation: 2 packages
Package has test failures: 5 packages
Package tests unexpectedly errored: 2 packages
Package requires a missing binary dependency: 4 packages
There were unidentified errors: 1 packages
Test duration exceeded the time limit: 12 packages

1518 packages failed on the previous version too.

✔ Packages that passed tests

21 packages passed tests only on the current version.

Other: 21 packages

5084 packages passed tests on the previous version too.

~ Packages that at least loaded

7 packages successfully loaded only on the current version.

Other: 7 packages

2817 packages successfully loaded on the previous version too.

➖ Packages that were skipped altogether

1 packages were skipped only on the current version.

Package could not be installed: 1 packages

909 packages were skipped on the previous version too.

topolarity · 2025-03-18T15:19:40Z

Some of the PkgEval failures (thankfully, not many!) look relevant, such as FastPolynomialRoots.jl and StateSpaceLearning.jl:

Failed to precompile GLMNet [8d5ece8b-de18-5317-b113-243142960cc6] to "/home/pkgeval/.julia/compiled/v1.13/GLMNet/jl_6YhP1U" (ProcessExited(1)).
ERROR: LoadError: InitError: could not load library "/home/pkgeval/.julia/artifacts/e80442d59af73a1877a108eb3fab2cd12d25dc56/lib/libglmnet.so"
libgfortran.so.5: cannot open shared object file: No such file or directory
Stacktrace:
  [1] #dlopen#3
    @ ./libdl.jl:120 [inlined]
  [2] dlopen
    @ ./libdl.jl:119 [inlined]
  [3] dlopen(s::String)
    @ Base.Libc.Libdl ./libdl.jl:119
  [4] __init__()
    @ glmnet_jll ~/.julia/packages/glmnet_jll/en9XD/src/wrappers/x86_64-linux-gnu-libgfortran5.jl:37

topolarity · 2025-03-18T17:30:33Z

Looks like the common theme is that the JLL's involved are so old (≥5 years) that they predate JLLWrappers

topolarity · 2025-03-18T17:38:11Z

stdlib/CompilerSupportLibraries_jll/src/CompilerSupportLibraries_jll.jl

 const LIBPATH = Ref("")
+const LIBPATH_list = String[]


Is it intentional that this dropped PATH_list?

Some very old JLL's (e.g. NL2sol_jll) fail now with:

ERROR: LoadError: InitError: UndefVarError: `PATH_list` not defined in `CompilerSupportLibraries_jll`

That's not good, that's documented: https://docs.binarybuilder.org/stable/jll/#The-wrappers

gitboy16 · 2025-03-19T16:43:04Z

Just a note that between this PR and the previous one 57752, the size of Julia windows installation folder jumped from 1.1G to 1.5G.

KristofferC · 2025-03-21T09:01:06Z

I removed the backport here because this is very much like a feature and feature freeze was many months ago.

topolarity · 2025-03-25T23:24:35Z

Just a note that between this PR and the previous one 57752, the size of Julia windows installation folder jumped from 1.1G to 1.5G.

Any idea which files / folders experienced the difference?

staticfloat mentioned this pull request Mar 11, 2025

Initial implementation of lazy JLLs for LinearAlgebra JuliaLang/LinearAlgebra.jl#1235

Merged

staticfloat force-pushed the sf/lazy_linear_algebra branch from 85eb73d to d537c45 Compare March 11, 2025 13:51

staticfloat force-pushed the sf/lazy_linear_algebra branch from d537c45 to b43712b Compare March 11, 2025 14:42

topolarity force-pushed the sf/lazy_linear_algebra branch from b43712b to 3db34a3 Compare March 11, 2025 14:59

staticfloat force-pushed the sf/lazy_linear_algebra branch 2 times, most recently from d21736d to 24bdde7 Compare March 11, 2025 21:02

staticfloat requested a review from topolarity March 11, 2025 21:07

topolarity added the backport 1.12 Change should be backported to release-1.12 label Mar 11, 2025

staticfloat marked this pull request as draft March 11, 2025 23:56

staticfloat force-pushed the sf/lazy_linear_algebra branch 3 times, most recently from b85e778 to 6109346 Compare March 12, 2025 00:25

staticfloat marked this pull request as ready for review March 12, 2025 00:30

staticfloat force-pushed the sf/lazy_linear_algebra branch 4 times, most recently from 94382bd to afc71c7 Compare March 12, 2025 15:36

topolarity reviewed Mar 12, 2025

View reviewed changes

base/libdl.jl Outdated Show resolved Hide resolved

stdlib/CompilerSupportLibraries_jll/src/CompilerSupportLibraries_jll.jl Show resolved Hide resolved

staticfloat force-pushed the sf/lazy_linear_algebra branch from 7dcb242 to b4cc2f8 Compare March 12, 2025 21:10

staticfloat force-pushed the sf/lazy_linear_algebra branch 2 times, most recently from 44bd276 to 41e8fb4 Compare March 13, 2025 14:28

staticfloat force-pushed the sf/lazy_linear_algebra branch 4 times, most recently from 3433d57 to f77f026 Compare March 16, 2025 04:33

staticfloat and others added 5 commits March 16, 2025 14:08

test/trimming: Use SHLIB_EXT in Makefile

3204987

Bump LinearAlgebra version to latest

f38a3b9

This includes the changes needed for lazy LinearAlgebra JLLs.

staticfloat force-pushed the sf/lazy_linear_algebra branch from f77f026 to f38a3b9 Compare March 16, 2025 21:08

staticfloat added the merge me PR is reviewed. Merge when all tests are passing label Mar 16, 2025

staticfloat merged commit 0d363a8 into master Mar 16, 2025
6 of 8 checks passed

staticfloat deleted the sf/lazy_linear_algebra branch March 16, 2025 23:04

topolarity reviewed Mar 18, 2025

View reviewed changes

KristofferC mentioned this pull request Mar 20, 2025

Backports release 1.12 #57536

Merged

KristofferC removed the backport 1.12 Change should be backported to release-1.12 label Mar 20, 2025

giordano removed the merge me PR is reviewed. Merge when all tests are passing label Mar 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial implementation of lazy JLLs for LinearAlgebra #57719

Initial implementation of lazy JLLs for LinearAlgebra #57719

staticfloat commented Mar 11, 2025 •

edited

Loading

staticfloat commented Mar 11, 2025

topolarity commented Mar 11, 2025

staticfloat commented Mar 11, 2025

topolarity commented Mar 12, 2025

topolarity commented Mar 12, 2025

staticfloat commented Mar 16, 2025

nanosoldier commented Mar 17, 2025

topolarity commented Mar 17, 2025

topolarity commented Mar 17, 2025

giordano commented Mar 17, 2025

topolarity commented Mar 18, 2025

nanosoldier commented Mar 18, 2025

❗ Packages that crashed

✖ Packages that failed

✔ Packages that passed tests

~ Packages that at least loaded

➖ Packages that were skipped altogether

topolarity commented Mar 18, 2025 •

edited

Loading

topolarity commented Mar 18, 2025

topolarity Mar 18, 2025

giordano Mar 18, 2025

gitboy16 commented Mar 19, 2025

KristofferC commented Mar 21, 2025

topolarity commented Mar 25, 2025

Initial implementation of lazy JLLs for LinearAlgebra #57719

Initial implementation of lazy JLLs for LinearAlgebra #57719

Conversation

staticfloat commented Mar 11, 2025 • edited Loading

staticfloat commented Mar 11, 2025

topolarity commented Mar 11, 2025

staticfloat commented Mar 11, 2025

topolarity commented Mar 12, 2025

topolarity commented Mar 12, 2025

staticfloat commented Mar 16, 2025

nanosoldier commented Mar 17, 2025

topolarity commented Mar 17, 2025

topolarity commented Mar 17, 2025

giordano commented Mar 17, 2025

topolarity commented Mar 18, 2025

nanosoldier commented Mar 18, 2025

❗ Packages that crashed

✖ Packages that failed

✔ Packages that passed tests

~ Packages that at least loaded

➖ Packages that were skipped altogether

topolarity commented Mar 18, 2025 • edited Loading

topolarity commented Mar 18, 2025

topolarity Mar 18, 2025

Choose a reason for hiding this comment

giordano Mar 18, 2025

Choose a reason for hiding this comment

gitboy16 commented Mar 19, 2025

KristofferC commented Mar 21, 2025

topolarity commented Mar 25, 2025

staticfloat commented Mar 11, 2025 •

edited

Loading

topolarity commented Mar 18, 2025 •

edited

Loading