Significant regression in TTFX, Julia 1.11

[jakobnissen]

With a nearly empty dummy package Foo, and running:

julia> using JET, Foo

julia> @time @eval report_package(Foo)

I get the following timings:
1.10.4: 11.770015 seconds (13.73 M allocations: 977.822 MiB, 10.09% gc time, 99.85% compilation time)
1.11-rc2: 33.809627 seconds (51.71 M allocations: 2.676 GiB, 4.95% gc time, 99.86% compilation time)

The high level of compilation and allocations suggests perhaps something could be done to reduce this?

[lgoettgens]

Now that julia 1.11 is out, the issue seems to have gotten even worse. I've done some timings on @report_opt and @report_call, as these are the things from JET that I used to use on a daily basis. But with these regressions, that seems no longer viable on 1.11.

All timings have been performed with JET v0.9.10
1.10.5:

julia> using JET

julia> @time @eval @report_opt 2+2
  0.246439 seconds (308.67 k allocations: 20.559 MiB, 99.73% compilation time)
No errors detected


julia> @time @eval @report_opt 2+2
  0.000364 seconds (488 allocations: 31.539 KiB, 0.14% compilation time)
No errors detected


julia> @time @eval @report_call 2+2
  0.106172 seconds (40.82 k allocations: 2.802 MiB, 99.48% compilation time)
No errors detected


julia> @time @eval @report_call 2+2
  0.000817 seconds (207 allocations: 12.602 KiB, 0.13% compilation time)
No errors detected

1.11.0:

julia> using JET

julia> @time @eval @report_opt 2+2
 13.131656 seconds (30.70 M allocations: 1.605 GiB, 6.32% gc time, 100.00% compilation time)
No errors detected


julia> @time @eval @report_opt 2+2
  0.000365 seconds (653 allocations: 29.164 KiB, 0.13% compilation time)
No errors detected


julia> @time @eval @report_call 2+2
 14.425664 seconds (24.85 M allocations: 1.291 GiB, 3.23% gc time, 100.00% compilation time)
No errors detected


julia> @time @eval @report_call 2+2
  0.000318 seconds (266 allocations: 12.039 KiB, 0.18% compilation time)
No errors detected

1.11.1:

julia> using JET

julia> @time @eval @report_opt 2+2
 13.215292 seconds (30.75 M allocations: 1.607 GiB, 9.01% gc time, 100.00% compilation time)
No errors detected


julia> @time @eval @report_opt 2+2
  0.000359 seconds (653 allocations: 29.164 KiB, 0.15% compilation time)
No errors detected


julia> @time @eval @report_call 2+2
 13.560778 seconds (24.86 M allocations: 1.292 GiB, 4.25% gc time, 99.99% compilation time)
No errors detected


julia> @time @eval @report_call 2+2
  0.000314 seconds (266 allocations: 12.039 KiB, 0.16% compilation time)
No errors detected

Edit: Re-ran everything on a different device, and added 1.11.1

[aviatesk]

This issue is more related to JET itself rather than the regression in Julia. That’s because JET’s precompilation is intentionally disabled in v1.11 and later (

JET.jl/src/JET.jl

Line 1211 in 4188f9b

@static v"1.11.0-DEV.1552" > VERSION && @setup_workload let

).
The reason for this is that Julia base has not yet implemented the ability to correctly cache the results of compilation by external AbstractInterpreter across sessions. It’s a bit nuanced to explain why it worked in v1.10, but the truth is that the correct caching mechanism wasn’t implemented there v1.10—JET’s precompilation just happened to work by chance in that version.