MAUI .NET 8.0 built with Full AOT Compilation reports a lot of Mono : AOT NOT FOUND

[crui3er]

Description

When I build maui application for Android with full aot compilation, I see a lot of aot not found log messags for mono aot logger.
Especially for generic class/methods. Even for the ones I expect that should be statically detected.
E.g.:
04-15 18:43:11.957 28518 28518 D Mono : AOT NOT FOUND: System.Collections.Concurrent.ConcurrentDictionary`2<Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceIdentifier, Microsoft.Extensions.DependencyInjection.ServiceProvider/ServiceAccessor>:.ctor ().

04-15 18:43:23.703 28518 28518 D Mono : AOT NOT FOUND: System.Runtime.CompilerServices.AsyncTaskMethodBuilder1<System.Threading.Tasks.VoidTaskResult>:GetStateMachineBox<MauiAotTest.Components.Pages.Weather/<OnInitializedAsync>d__2> (MauiAotTest.Components.Pages.Weather/<OnInitializedAsync>d__2&,System.Threading.Tasks.Task1<System.Threading.Tasks.VoidTaskResult>&).

See attached files: one with full log for the app and the one with aot not found only messages.

LogNote_47021f4c_20240415_18.43.07_tst2_aot_full_not_found.txt
LogNote_47021f4c_20240415_18.43.07_tst2_aot_full.txt

Steps to Reproduce

1. Create a template maui blazor app
2. Build it with AndroidEnableProfiledAot=false and install on Android device
3. Enable mono aot logging with command adb shell setprop debug.mono.log default,timing=bare,assembly,mono_log_level=debug,mono_log_mask=aot
4. Captute log with logcat
5. See a lot of aot not found (exclude the ones for wrappers) log messages.

Link to public reproduction project repository

https://github.com/crui3er/MauiAotTest

Version with bug

8.0.3 GA

Is this a regression from previous behavior?

Not sure, did not test other versions

Last version that worked well

Unknown/Other

Affected platforms

Android

Affected platform versions

No response

Did you find any workaround?

No

Relevant log output

No response

[crui3er]

It's also not clear whether should AOT compiler for Android process generic types and methods when $(AndroidEnableProfiledAot)==false
I found it's confusing when reading https://devblogs.microsoft.com/dotnet/dotnet-8-performance-improvements-in-dotnet-maui/#androidstripilafteraot
Looks like it works only for iOS.
Can @jonathanpeppers please shead a light on the subject.
What can I read or may be you can recommend how to reduce jit as much as possible during starting app.

[jonathanpeppers]

I think this is a known issue, some info at the bottom of:

• https://github.com/dotnet/maui/tree/main/src/ProfiledAot#testing-profiles

There are these tiny methods that are always JIT no matter what:

02-23 09:03:46.327 10401 10401 D Mono    : AOT NOT FOUND: (wrapper runtime-invoke) object:runtime_invoke_void (object,intptr,intptr,intptr).
02-23 09:03:46.334 10401 10401 D Mono    : AOT NOT FOUND: (wrapper managed-to-native) System.Diagnostics.Debugger:IsAttached_internal ().
02-23 09:03:46.367 10401 10401 D Mono    : AOT NOT FOUND: (wrapper native-to-managed) Android.Runtime.JNINativeWrapper:Wrap_JniMarshal_PPL_V (intptr,intptr,intptr).

But I don't think these are "actual methods", but tiny wrappers that enable generics or p/invoke. There is some JIT happening there currently on Android, which might be because we use MONO_AOT_MODE=Normal and allow JIT. I believe iOS/Catalyst would use MONO_AOT_MODE=Full, because those platforms require it.

[ivanpovazan]

@crui3er just for my understanding, is there a crash that happens with the application or the question is just about unexpected log messages?

Regarding full AOT compilation on Android, afaik you would have to set -p:AndroidAotMode=Full, @jonathanpeppers please confirm, that would force it (although I am not sure if we are officially supporting this mode). If it isn't set some wrappers will always fallback to JIT during runtime (and AOT NOT FOUND in logs is expected behaviour).

[crui3er]

@ivanpovazan Not, there is no crash, but there are unexpected log messages.
So I am aware that log messages for wrappers are ok (AOT NOT FOUND: (wrapper xxx)). @jonathanpeppers I do not take them in account.
In attached sample (in fact, it's template maui blazor app) jitting is used for stuff in MauiAotTest.Components.Pages.Weather.OnInitializedAsync method or during service provide usage (see my first comment).

In my maui app I working on there are a lot of log messages with AOT NOT FOUND. Even with using recorded aot profile there are jitted methods are which are in recorded profile.

So I am trying to figure out what is going wrong with this small sample app.

[ivanpovazan]

In my maui app I working on there are a lot of log messages with AOT NOT FOUND. Even with using recorded aot profile there are jitted methods are which are in recorded profile.

Could you share more information on that: like the aot profile that you are using and which methods you would not expect to see in the log?

[jonathanpeppers]

I don’t think Full will work on Android, does that mode prevent JIT?

[fanyang-mono]

Full AOT is what iOS uses, which does not JIT anything, since that's a requirement for iOS.

[crui3er]

I tested with -p:AndroidAotMode=Full and it fails on start wtih ExecutionEngineException.

F mono-rt : [ERROR] FATAL UNHANDLED EXCEPTION: System.ExecutionEngineException: Attempting to JIT compile method '(wrapper other) void Java.Interop.JavaVMInterface:PtrToStructure (intptr,object)' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.

[crui3er]

Could you share more information on that: like the aot profile that you are using and which methods you would not expect to see in the log?

I recorded custom aot profile https://github.com/crui3er/MauiAotTest/blob/master/custom.aprof and do test with it.
During recording and testing I do the following actions: start app, go to 'Counter' page, click 'Click me' button several times, then go to 'Weather' page and finally go again to 'Counter' page.

Then I compared AOT NOT FOUND log records and stat file generated from custom profile https://github.com/crui3er/MauiAotTest/blob/master/custom.aprof.stat.txt

Here is a few examples of aot not found records in the log for methods mentioned in profile.

stat:

void System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start (MauiAotTest.Components.Pages.Weather/<OnInitializedAsync>d__2&) 

log:

Mono    : AOT NOT FOUND: System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start<MauiAotTest.Components.Pages.Weather/<OnInitializedAsync>d__2> (MauiAotTest.Components.Pages.Weather/<OnInitializedAsync>d__2&).

stat:

void System.Collections.Concurrent.ConcurrentDictionary`2<Microsoft.AspNetCore.Components.Routing.RouteKey, Microsoft.AspNetCore.Components.Routing.RouteTable>:.ctor ()

log:

Mono    : AOT NOT FOUND: System.Collections.Concurrent.ConcurrentDictionary`2<Microsoft.AspNetCore.Components.Routing.RouteKey, Microsoft.AspNetCore.Components.Routing.RouteTable>:.ctor ().

stat:

System.Type System.Text.Json.Serialization.JsonConverter`1<System.Threading.Tasks.VoidTaskResult>:get_Type ()

log:

Mono    : AOT NOT FOUND: System.Text.Json.Serialization.JsonConverter`1<System.Threading.Tasks.VoidTaskResult>:get_Type ().

Here is AOT NOT FOUND log records excluding the one for wrappers.
LogNote_47021f4c_20240418_16.32.26_tst2_custom_profile_filtered.txt

Sample app is updated. Now it uses recorded aot profile.

[fanyang-mono]

@crui3er Thanks for providing more information about this. I will take a look at this.

[alexyakunin]

Hi there, just curious, are there any updates on that?

I want to add some context: the app we're working on shows ~20K methods in "AOT NOT FOUND" log entries - and that's just for the startup time. We were questioning whether AOT is even working at all - well, it does, coz there are also lots of "AOT FOUND" entries, but it's reasonable to assume that if you have even 50/50 split between these, you're probably getting just 50% of max. possible savings on JIT. And with AOT turned off, our startup time is almost 2x higher. In other words, if AOT issues would be fixed, our app's startup time would go from ~ 1.7s to probably just 0.3-0.5s or so, which is obviously a dramatic change.

Please let @crui3er know if you need anything else. We can provide logs from the actual app + give you instructions on how to reproduce the issue there, if a small sample won't be enough to identify the root cause(s).

And IMO it's super important to address this: compared to Native AOT, profile-based AOT (assuming it 100% works) is what most of mobile apps need. It allows to balance between the app size and the speed of the most crucial parts of the app, which is almost always the startup time for mobile apps. So nearly any MAUI and Blazor Hybrid app would benefit from this heavily.

[alexyakunin]

Also, should we change the title of this issue? It's not about just full AOT, it's about both full and profiled AOT. And it's understandable why full AOT may miss or intentionally omit some methods (the # of possible generic instances explodes exponentially with the codebase size), but profiled AOT is expected to produce AOT code at least for every method from the AOT profile.

[fanyang-mono]

I was able to build and run the app provided here. When setting AndroidEnableProfiledAot=true and enabling logging. I could see the 2/3 example @crui3er provided earlier, for example 1, they are not the same method.

I also noticed that this profile was gathered using the legacy profiler. Next, I will try the new profiler to see if this issue still exists. If so, I will investigate further.

[crui3er]

I also noticed that this profile was gathered using the legacy profiler. Next, I will try the new profiler to see if this issue still exists. If so, I will investigate further.

What is a new profiler and how to use it?

[fanyang-mono]

Actually, the new profiler is not fully supported on Android yet. The profiler I was talking about is this: https://github.com/dotnet/runtime/blob/main/docs/design/mono/profiled-aot.md

[crui3er]

Actually, the new profiler is not fully supported on Android yet. The profiler I was talking about is this: https://github.com/dotnet/runtime/blob/main/docs/design/mono/profiled-aot.md

I think @jonathanpeppers did experimental support https://github.com/jonathanpeppers/xamarin-android/blob/dotnet-pgo/src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.Aot.targets but it's not included in .NET8, is it?

[crui3er]

I was able to build and run the app provided here. When setting AndroidEnableProfiledAot=true and enabling logging. I could see the 2/3 example @crui3er provided earlier, for example 1, they are not the same method.

@fanyang-mono Why do you think that methods in example 1 are not the same?

[alexyakunin]

@fanyang-mono funny enough, we were trying to use it as well - not sure what's the state of .mibc format in the long run, but it seems currently it's not supported by release version of .NET 8, but you can see here we were trying both .aprof and .mibc formats here: https://github.com/Actual-Chat/actual-chat/blob/dev/src/dotnet/App.Maui/App.Maui.csproj#L265 Collecting .mibc isn't an issue - as well as merging these profiles. But it looks like Android build targets just don't use <AndroidMibcProfile> & fallback to full AOT (i.e. behave like there is no profile provided).

We created this issue because there are so many "AOT_NOT_FOUND" for regular .aprof output that it works much worse than full AOT for us (which still produces tons of "AOT_NOT_FOUND", but this is at least explainable). The fact profiled AOT doesn't really work is a huge issue affecting every MAUI Android app, and an existential issue for apps like ours, where the startup time is crucial. I wrote earlier that fully working profiled AOT is expected to drop our startup time to 0.5...1s on Android - vs current 2s. In other words, it's as different as night and day.

Just to illustrate how bad this is:

• Startup time on iPhohe 13 Pro in interpreter-only mode: 1.1s
• Startup time on Galaxy S23 Ultra in full AOT mode: 1.7-1.8s
• Startup time on Galaxy S23 Ultra with .aprof profile: 2.4s or more.

And unrelated, but: we spent a decent amount of time trying to enable AOT for iOS at least for some assemblies, but this inevitably leads to crashes. That's why for now the app works in interpreter mode there.

[alexyakunin]

Another illustration of how bad this is. Below is a screenshot from DotTrace showing where most of the time is spent in release build with full AOT - as I said, due to the issue listed here it's our best option for now. The timeline is constrained by [0..1.2s] interval. The original .nettrace was recorded via:

adb reverse tcp:9000 tcp:9000
start dotnet-dsrouter client-server -tcps 127.0.0.1:9000 -ipcc /tmp/maui-app --verbose debug
start dotnet-trace collect --diagnostic-port /tmp/maui-app --output "_Profiling/android.nettrace" --providers Microsoft-Windows-DotNETRuntime:0x1F000080018:5

As you see, JIT takes almost 74% of time there. So if profiled AOT would work, it could be just 0.3s or so.

[jonathanpeppers]

There is a PR adding .mibc support, that still has some open questions (blockers):

• [WIP] new Profiled AOT support android#7216

However, I don't think using .mibc would be able to avoid any additional AOT NOT FOUND messages. It seems like it would just be using a more modern format for profiled AOT than what we use for the built-in profile today.

Regarding:

Startup time on Galaxy S23 Ultra in full AOT mode: 1.7-1.8s
Startup time on Galaxy S23 Ultra with .aprof profile: 2.4s or more.

Are you able to share a .nettrace or .speedscope file of startup? There might be something we can fix in .NET MAUI or recommend.

You could also record your own AOT profile, if the file size of "full AOT" (AOT everything) is too large. This would just be a tradeoff where it wouldn't AOT everything, but just the methods called during your recording.

[fanyang-mono]

I was able to build and run the app provided here. When setting AndroidEnableProfiledAot=true and enabling logging. I could see the 2/3 example @crui3er provided earlier, for example 1, they are not the same method.

@fanyang-mono Why do you think that methods in example 1 are not the same?

One is a non-generic method -> System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start
The other is a generic method -> System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start<MauiAotTest.Components.Pages.Weather/<OnInitializedAsync>d__2>

The second one has a type argument and needs to be compiled differently than the first one.

However, I just check the method System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start. It is a generic method. https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs#L21

It seems that custome.aprof.stat.txt contains some inaccurate information. Hopefully, custome.aprof contains the correct information.

[fanyang-mono]

@alexyakunin @crui3er Could you compare the profile text files that you got from both methods to see if they are the same? I am curious about that. They might be the same.

[fanyang-mono]

I investigated the situation more and found out that

System.Type System.Text.Json.Serialization.JsonConverter`1<System.Threading.Tasks.VoidTaskResult>:get_Type ()

didn't pass the checks inside add_single_profile_method (https://github.com/dotnet/runtime/blob/main/src/mono/mono/mini/aot-compiler.c#L13822-L13870), which means it didn't meet the requirements to be AOT compiled. I believe the other two methods weren't compiled due to the same reason.

And the new profiler alos use add_single_profile_method to decide if a given method could be AOT compiled or not. Thus, @jonathanpeppers was correct, using the new profiler won't help more methods get AOT compiled either.

[kekekeks]

@alexyakunin would you be interested in a CoreCLR/NativeAOT-based "launcher" for Blazor?

We have a prototype of an Android backend for Avalonia that could be repurposed to host a WebView.

You can see the difference of running the same Avalonia-based app using CoreCLR+NativeAOT, xamarin maui-android mono with our custom interop and using full xamarin maui-android interop.

[alexyakunin]

@alexyakunin would you be interested in a CoreCLR/NativeAOT-based "launcher" for Blazor?

I'd say this is #1 thing Microsoft should be working on now; and if it's not them, probably, it's even better (this thread is one of proofs why). I don't care if it's MAUI or not, what we need is Blazor WebView running on iOS, Android, and non-mobile OSes. So if you need some testers of any preview tech, please reach me out. We'd be happy to contribute as well.

And yeah, I saw that demo + one other (w/ Avalonia controls). Yes, we need something similar in terms of startup time.

[jonathanpeppers]

I have an experiment here that shows how to use BlazorWebView without MAUI:

• https://github.com/jonathanpeppers/spice

It has comparisons of app size and startup time with MAUI+Blazor.

I did have to bring in some of the BlazorWebView source for it to work -- I think just minor changes, though.

If you are wanting the absolute fastest WebView experience, it seems like you could use a standard dotnet new android project with a WebView and not use MAUI. MAUI+Blazor is like including the entire .NET "kitchen sink" in your app.

To be honest, if you are looking for speed, maybe native controls are the way to go instead of web content?

[alexyakunin]

If you are wanting the absolute fastest WebView experience, it seems like you could use a standard dotnet new android project with a WebView and not use MAUI. MAUI+Blazor is like including the entire .NET "kitchen sink" in your app.

You probably know this is one of the most frequent feature requests related to Blazor: "Give us a cross-platform Blazor, but ditch MAUI!".

To be honest, if you are looking for speed, maybe native controls are the way to go instead of web content?

I mentioned it many times here:

• I am not looking for speed in general
• I am fine with even interpreter mode on iOS, if we're talking about the raw speed
• What I am not fine is ridiculously slow startup time on Android. It's nearly the same as browser-based WASM app startup time. And it's a consequence of a single issue this thread is about. There is no AOT on Android, period.

Now, speaking of switching to a lower-level APIs, such as native controls:

• It won't solve any problem unless it also implies "switch your whole client-side code to Kotlin or Java". And why you guys still work on C# than?
• In our case WebView with TypeScript-based client would also fit the bill. I certainly prefer a single cross-platform client vs N platform-specific ones.

[alexyakunin]

@jonathanpeppers

I have an experiment here that shows how to use BlazorWebView without MAUI: ...

Just checked it out - overall, getting rid of MAUI is step in the right direction. But this specific solution won't change anything in our case - we don't use most of MAUI, and we know it doesn't add much to the startup time (okey, there is some control registration code that eats up maybe 0.1s, but it's nothing compared to the rest).

As I wrote, the problem is that AOT mostly doesn't work on MAUI (ok, Xamarin for iOS and Android), and switching to "MAUI-less Blazor" (which still runs on top of Xamarin/Mono AOT) won't change this. Any async method and nearly any method parameterized with value type will still require JIT.

[alexyakunin]

If you are wanting the absolute fastest WebView experience, it seems like you could use a standard dotnet new android project with a WebView and not use MAUI.

This also won't change anything.

@jonathanpeppers could you please read what's the root cause here? Not sure how you guys make decisions on priority, but the fact you suggest such "solutions" tells me you misunderstand the root cause. A quick summary:

• We noticed no matter what you do, Mono AOT doesn't produce AOT code for generic methods parameterized with value types declared outside of System.Runtime (AFAIK there are more constraints - I don't know all of them, but that's ~ the key one)
• In particular, this means that any async method requires JIT - due to AsyncTaskMethodBuilder.Start<TStateMachine>() where TStateMachine is struct in Release builds
• Use of nearly any generic struct (ValueTask<T>, most of struct-based enumerators, or Result<T> in our case) leads to the same issue
• Fast serializers, such as MemoryPack and MessagePack, call ~ Register<T> for any type they support, including structs. As you might guess, all these calls require JIT. And I assume Avalonia's property registration code falls into the same category.

Think of structs like Nullable<Unit>, Nullable<Vector2>, or ValueTuple<int, MyClass>:

• All of them use types declared outside of System.Runtime, so generic methods parameterized w/ them will require JIT
• As you might guess, serializers like MemoryPack will call Register<T> for these types, if they're used in any other serializable type.

Overall, we see that in our case this triggers thousands of "AOT not found" in Mono debug log. I'll add a few screenshots showing how bad it is tomorrow.

[alexyakunin]

Okey, here are the screenshots - and you can easily reproduce all of that:

1. Download Actual Chat: https://play.google.com/store/apps/details?id=chat.actual.app
2. adb shell setprop debug.mono.log default,assembly,mono_log_level=debug,mono_log_mask=all
3. adb logcat -s "actual.chat" -s "Mono" > C:\Temp\_logcat.log
4. Start it on your device.

Here is what you'll see in the log right after the app renders (i.e. first 2-3 seconds):

15K "AOT NOT FOUND" entries:

It's easy to notice all these methods are generics; some are likely just full AOT "misses", but e.g. HostKind a struct, and if I scroll further, there will be quite broad categories like:

1. Serializer registration
   

2. Async methods and (Concurrent)Dictionary<Symbol / other custom struct, ...>:
   
   

3. The further we go, the more async methods are there:
   

4. There is also a fancy stuff like this:
   

I.e. think about this: Language is a custom struct, and JIT is used to generate GenericEqualityComparer<Nullable<Language>>.ctor here, + EqualityComparer<Nullable<Language>>.CreateComparer() - in other words, any custom struct will trigger JIT in 10+ methods no matter what just due to use of equality comparer for it and its "wrappers" (such as Nullable<T>).

5. Any async method requires JIT for a bunch of methods, actually:
   

Overall, ~30% of methods in this log contain AsyncTaskMethodBuilder in their full name:

And that's an impact of one of widely used generic structs in our case:

--

Finally, it worth showing that AOT was actually used:

The only problem is that it processed ~8K methods, and couldn't generate AOT code for another 15K.

That's why I am repeating here that AOT on .NET for Android is a fiction, and only brave guys from Microsoft somehow call it a feature.

[jonathanpeppers]

@alexyakunin to be clear, just because you see an AOT NOT FOUND message, that doesn't mean it's a noticeable slowdown.

You can get a JIT timing report from Mono:

• https://github.com/dotnet/android/blob/main/Documentation/guides/profiling.md#profiling-the-jit-compiler

Then we have a jit-times tool in dotnet/android that can parse this data into a table. If you can get one of those for your app, can you share it here?

@jonathanpeppers could you please read what's the root cause here? Not sure how you guys make decisions on priority, but the fact you suggest such "solutions" tells me you misunderstand the root cause. A quick summary:

I don't work on the Mono runtime, so I can't comment on their priorities. I am a C/C++ n00b at best, so I doubt I would be able to make a meaningful contribution to this issue. I was just trying my best to give you some options, thanks.

[alexyakunin]

@jonathanpeppers I'll try to get the timings as you suggested ASAP.

[alexyakunin]

Here is the output: jit-time-output.txt

The head:

Total (ms) |  Self (ms) | Method
     41.81 |      41.81 | System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start<Microsoft.Maui.Dispatching.DispatcherExtensions/<>c__DisplayClass2_0`1/<<DispatchAsync>b__0>d<ActualChat.Language>> (Microsoft.Maui.Dispatching.DispatcherExtensions/<>c__DisplayClass2_0`1/<<DispatchAsync>b__0>d<ActualChat.Language>&)
     24.22 |      24.22 | System.Collections.Frozen.FrozenDictionary:CreateFromDictionary<ActualChat.Users.ListeningMode, ActualChat.Users.ListeningModeInfo> (System.Collections.Generic.Dictionary`2<ActualChat.Users.ListeningMode, ActualChat.Users.ListeningModeInfo>)
     19.48 |      19.48 | System.Runtime.CompilerServices.DefaultInterpolatedStringHandler:AppendFormatted<double> (double,string)
     17.14 |      17.11 | ActualLab.Fusion.ComputedExt/<Changes>d__27`1<ActualLab.Api.ApiArray`1<ActualChat.UI.Blazor.App.Services.ActiveChat>>:MoveNext ()
     17.59 |      16.29 | System.Number:NumberToStringFormat<char> (System.Collections.Generic.ValueListBuilder`1<char>&,System.Number/NumberBuffer&,System.ReadOnlySpan`1<char>,System.Globalization.NumberFormatInfo)
     14.79 |      14.79 | System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.ValueTuple`2<ActualLab.Result`1<ActualChat.Users.AccountFull>, ActualLab.Fusion.Client.Internal.RpcOutboundComputeCall`1<ActualChat.Users.AccountFull>>>:AwaitUnsafeOnCompleted<System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1/ConfiguredTaskAwaiter<ActualChat.Users.AccountFull>> (System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1/ConfiguredTaskAwaiter<ActualChat.Users.AccountFull>&,System.Runtime.CompilerServices.IAsyncStateMachineBox)
     14.69 |      14.69 | ActualLab.Fusion.StateBoundComputed`1<bool>:.ctor (ActualLab.Fusion.ComputedOptions,ActualLab.Fusion.State`1<bool>)
     14.22 |      14.22 | System.Collections.Concurrent.ConcurrentDictionary`2<long, ActualLab.Rpc.Infrastructure.RpcOutboundCall>:GetValues ()
     13.17 |      13.17 | ActualLab.Fusion.Interception.ComputeMethodFunction`1/<Compute>d__8<System.ValueTuple`2<ActualChat.ChatId, bool>>:MoveNext ()
     11.91 |      11.91 | Microsoft.AspNetCore.Components.WebView.Maui.MauiDispatcher:InvokeAsync<ActualChat.LocalUrl> (System.Func`1<System.Threading.Tasks.Task`1<ActualChat.LocalUrl>>)
     11.68 |      11.68 | System.Runtime.CompilerServices.AsyncTaskMethodBuilder:AwaitUnsafeOnCompleted<System.Runtime.CompilerServices.ConfiguredTaskAwaitable/ConfiguredTaskAwaiter, ActualChat.Kvas.SyncedState`1/<ReadCycle>d__35<ActualChat.Users.UserOnboardingSettings>> (System.Runtime.CompilerServices.ConfiguredTaskAwaitable/ConfiguredTaskAwaiter&,ActualChat.Kvas.SyncedState`1/<ReadCycle>d__35<ActualChat.Users.UserOnboardingSettings>&)
     10.86 |      10.74 | System.Net.Sockets.SocketAsyncEventArgs:StartOperationCommon (System.Net.Sockets.Socket,System.Net.Sockets.SocketAsyncOperation)
     10.46 |      10.46 | System.Collections.Concurrent.ConcurrentDictionary`2<System.Reflection.MethodInfo, System.Func`2<ActualLab.Interception.Invocation, object>>:GetOrAdd<ActualLab.Interception.Invocation> (System.Reflection.MethodInfo,System.Func`3<System.Reflection.MethodInfo, ActualLab.Interception.Invocation, System.Func`2<ActualLab.Interception.Invocation, object>>,ActualLab.Interception.Invocation)
     10.30 |      10.30 | System.Net.Sockets.SafeSocketHandle:.ctor ()
     12.40 |      10.12 | System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1/AsyncStateMachineBox`1<ActualLab.Fusion.StateBoundComputed`1<ActualChat.UI.Blazor.App.Services.ReadPosition>, ActualLab.Fusion.State`1/<GetComputed>d__90<ActualChat.UI.Blazor.App.Services.ReadPosition>>:.cctor ()
      9.70 |       9.62 | System.Linq.Enumerable:ToList<ActualChat.UI.Blazor.App.Services.ActiveChat> (System.Collections.Generic.IEnumerable`1<ActualChat.UI.Blazor.App.Services.ActiveChat>)
      9.68 |       9.61 | ActualLab.Fusion.Interception.ComputeMethodFunction`1/<Compute>d__8<System.ValueTuple`2<System.Nullable`1<ActualChat.UI.Blazor.App.Services.RelatedChatEntry>, ActualChat.Chat.ChatEntry>>:MoveNext ()
      9.04 |       9.04 | System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1/AsyncStateMachineBox`1<System.Threading.Tasks.VoidTaskResult, ActualChat.UI.Blazor.App.Services.ChatAudioUI/<PushRealtimePlaybackState>d__63>:MoveNext (System.Threading.Thread)
      8.81 |       8.81 | System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Threading.Tasks.VoidTaskResult>:GetStateMachineBox<ActualLab.Fusion.ComputedState`1/<UpdateCycle>d__28<ActualChat.Users.UserLanguageSettings>> (ActualLab.Fusion.ComputedState`1/<UpdateCycle>d__28<ActualChat.Users.UserLanguageSettings>&,System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>&)
      8.58 |       8.58 | System.Number:FormatDouble<char> (System.Collections.Generic.ValueListBuilder`1<char>&,double,System.ReadOnlySpan`1<char>,System.Globalization.NumberFormatInfo)
      8.84 |       8.26 | System.Collections.Immutable.ImmutableHashSet:ToImmutableHashSet<ActualChat.ChatId> (System.Collections.Generic.IEnumerable`1<ActualChat.ChatId>,System.Collections.Generic.IEqualityComparer`1<ActualChat.ChatId>)
      8.01 |       8.01 | System.Buffers.SearchValues:TryGetSingleRange<char> (System.ReadOnlySpan`1<char>,char&,char&)
      7.41 |       7.41 | System.Net.Sockets.SocketAsyncEngine:EventLoop ()
      8.05 |       7.35 | ActualLab.Fusion.ComputeFunctionBase`1/<TryRecompute>d__17<int>:MoveNext ()
      7.09 |       7.09 | ActualLab.Fusion.Interception.ComputeMethodFunction`1/<Compute>d__8<ActualChat.UI.Blazor.App.Services.ChatAudioUI/RecordingBeepState>:MoveNext ()
     24.17 |       6.51 | ActualLab.Fusion.ComputeFunctionBase`1/<TryRecompute>d__17<ActualChat.UI.Blazor.App.Services.ChatAudioUI/RecordingBeepState>:MoveNext ()
      6.35 |       6.35 | System.Collections.Concurrent.ConcurrentDictionary`2<System.ValueTuple`2<System.Reflection.MethodInfo, System.Type>, System.Func`2<ActualLab.Interception.MethodDef, object>>:TryAddInternal (System.Collections.Concurrent.ConcurrentDictionary`2/Tables<System.ValueTuple`2<System.Reflection.MethodInfo, System.Type>, System.Func`2<ActualLab.Interception.MethodDef, object>>,System.ValueTuple`2<System.Reflection.MethodInfo, System.Type>,System.Nullable`1<int>,System.Func`2<ActualLab.Interception.MethodDef, object>,bool,bool,System.Func`2<ActualLab.Interception.MethodDef, object>&)
      5.99 |       5.99 | ActualLab.Fusion.Computed`1/<Update>d__35<ActualChat.ChatEntryId>:MoveNext ()
      5.87 |       5.87 | ActualLab.Api.ApiArray`1<ActualLab.Text.Symbol>:MemoryPack.IMemoryPackable<ActualLab.Api.ApiArray<T>>.Serialize<MemoryPack.Internal.ReusableLinkedArrayBufferWriter> (MemoryPack.MemoryPackWriter`1<MemoryPack.Internal.ReusableLinkedArrayBufferWriter>&,ActualLab.Api.ApiArray`1<ActualLab.Text.Symbol>&)
      5.82 |       5.82 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<Compute>d__15<ActualChat.Chat.ChatNews>:MoveNext ()
      7.00 |       5.81 | ActualLab.Fusion.Interception.ComputeMethodFunction`1/<Compute>d__8<int>:MoveNext ()
      7.82 |       5.79 | ActualLab.Fusion.Interception.ComputeMethodFunction`1/<Compute>d__8<ActualChat.ChatId>:MoveNext ()
      6.89 |       5.64 | System.Text.Ascii:EqualsIgnoreCase<byte, uint16, System.Text.Ascii/WideningLoader> (byte&,uint16&,uintptr)
      5.29 |       5.29 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<ComputeRpc>d__16<ActualChat.Users.Presence>:MoveNext ()
      5.25 |       5.25 | ActualLab.Fusion.ComputeFunctionBase`1/<TryRecompute>d__17<ActualChat.UI.Blazor.App.Services.ChatAudioUI/RecordingState>:MoveNext ()
      5.19 |       5.19 | ActualLab.Fusion.ComputedState`1/<UpdateCycle>d__28<ActualChat.Mathematics.Trimmed`1<int>>:MoveNext ()
      5.12 |       5.12 | ActualLab.Fusion.ComputedExt/<Changes>d__27`1<ActualChat.ChatEntryId>:MoveNext ()
      5.06 |       5.06 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<ApplyRpcUpdate>d__18<ActualLab.Mathematics.Range`1<long>>:MoveNext ()
      7.58 |       5.00 | System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<ActualChat.Users.UserAppSettings>:AwaitUnsafeOnCompleted<System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1/ConfiguredValueTaskAwaiter<ActualLab.Option`1<ActualChat.Users.UserAppSettings>>> (System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1/ConfiguredValueTaskAwaiter<ActualLab.Option`1<ActualChat.Users.UserAppSettings>>&,System.Runtime.CompilerServices.IAsyncStateMachineBox)
      4.89 |       4.89 | ActualLab.Fusion.ComputeFunctionBase`1/<TryRecompute>d__17<System.ValueTuple`2<System.Nullable`1<ActualChat.UI.Blazor.App.Services.RelatedChatEntry>, ActualChat.Chat.ChatEntry>>:MoveNext ()
      4.76 |       4.76 | ActualLab.Fusion.ComputedExt/<Changes>d__27`1<ActualChat.ChatId>:MoveNext ()
      4.68 |       4.68 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<ApplyRpcUpdate>d__18<ActualChat.Chat.ChatNews>:MoveNext ()
      4.65 |       4.65 | System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1<ActualLab.Mathematics.Range`1<string>>:OnTryRead (System.Text.Json.Utf8JsonReader&,System.Type,System.Text.Json.JsonSerializerOptions,System.Text.Json.ReadStack&,ActualLab.Mathematics.Range`1<string>&)
      4.63 |       4.63 | System.Collections.Concurrent.ConcurrentDictionary`2<ActualLab.Rpc.Infrastructure.RpcCallTypeKey, System.Func`2<ActualLab.Rpc.Infrastructure.RpcOutboundContext, ActualLab.Rpc.Infrastructure.RpcOutboundCall>>:TryAddInternal (System.Collections.Concurrent.ConcurrentDictionary`2/Tables<ActualLab.Rpc.Infrastructure.RpcCallTypeKey, System.Func`2<ActualLab.Rpc.Infrastructure.RpcOutboundContext, ActualLab.Rpc.Infrastructure.RpcOutboundCall>>,ActualLab.Rpc.Infrastructure.RpcCallTypeKey,System.Nullable`1<int>,System.Func`2<ActualLab.Rpc.Infrastructure.RpcOutboundContext, ActualLab.Rpc.Infrastructure.RpcOutboundCall>,bool,bool,System.Func`2<ActualLab.Rpc.Infrastructure.RpcOutboundContext, ActualLab.Rpc.Infrastructure.RpcOutboundCall>&)
      4.45 |       4.45 | System.Collections.Concurrent.ConcurrentDictionary`2<System.ValueTuple`2<System.Reflection.MethodInfo, System.Type>, System.Func`2<ActualLab.Interception.MethodDef, object>>:TryGetValueInternal (System.Collections.Concurrent.ConcurrentDictionary`2/Tables<System.ValueTuple`2<System.Reflection.MethodInfo, System.Type>, System.Func`2<ActualLab.Interception.MethodDef, object>>,System.ValueTuple`2<System.Reflection.MethodInfo, System.Type>,int,System.Func`2<ActualLab.Interception.MethodDef, object>&)
      4.31 |       4.31 | ActualLab.Fusion.Interception.ComputeMethodFunction`1<ActualChat.UI.Blazor.App.Services.ChatAudioUI/RecordingBeepState>:Compute (ActualLab.Fusion.ComputedInput,ActualLab.Fusion.Computed`1<ActualChat.UI.Blazor.App.Services.ChatAudioUI/RecordingBeepState>,System.Threading.CancellationToken)
      4.23 |       4.23 | System.SpanHelpers:NonPackedIndexOfValueType<int16, System.SpanHelpers/Negate`1<int16>> (int16&,int16,int)
      4.20 |       4.20 | MemoryPack.MemoryPackFormatterProvider:IsRegistered<ActualLab.Rpc.RpcMethodRef> ()
      4.04 |       4.01 | (wrapper native-to-managed) Android.Runtime.JNINativeWrapper:Wrap_JniMarshal_PPLIIIIIIII_V (intptr,intptr,intptr,int,int,int,int,int,int,int,int)
      3.97 |       3.97 | ActualChat.Kvas.StoredState`1<ActualChat.ChatId>:CreateComputed ()
      3.92 |       3.92 | ActualLab.Fusion.ComputedExt/<Changes>d__27`1<ActualChat.UI.Blazor.App.Services.ChatAudioUI/RecordingBeepState>:MoveNext ()
      3.87 |       3.87 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<ApplyRpcUpdate>d__18<ActualLab.Api.ApiArray`1<ActualChat.PlaceId>>:MoveNext ()
      3.83 |       3.83 | System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1<ActualLab.Mathematics.Range`1<int>>:OnTryRead (System.Text.Json.Utf8JsonReader&,System.Type,System.Text.Json.JsonSerializerOptions,System.Text.Json.ReadStack&,ActualLab.Mathematics.Range`1<int>&)
      4.05 |       3.82 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<TryRecompute>d__20<ActualLab.Mathematics.Range`1<long>>:MoveNext ()
      3.78 |       3.78 | System.DateTimeFormat:FormatCustomized<char> (System.DateTime,System.ReadOnlySpan`1<char>,System.Globalization.DateTimeFormatInfo,System.TimeSpan,System.Collections.Generic.ValueListBuilder`1<char>&)
      3.74 |       3.74 | ActualLab.Fusion.ComputeFunctionBase`1/<TryRecompute>d__17<ActualChat.Mathematics.Trimmed`1<int>>:MoveNext ()
      4.37 |       3.73 | System.Number:NumberToString<char> (System.Collections.Generic.ValueListBuilder`1<char>&,System.Number/NumberBuffer&,char,int,System.Globalization.NumberFormatInfo)
      7.96 |       3.68 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<Compute>d__15<ActualLab.Mathematics.Range`1<long>>:MoveNext ()
      5.87 |       3.67 | ActualLab.Fusion.Interception.ComputeMethodFunction`1/<Compute>d__8<ActualChat.Mathematics.Trimmed`1<int>>:MoveNext ()
      3.67 |       3.67 | ActualLab.Fusion.Computed`1/<Update>d__35<ActualLab.Api.ApiArray`1<ActualChat.UI.Blazor.App.Services.ActiveChat>>:MoveNext ()
      3.65 |       3.65 | ActualLab.Fusion.ComputeFunctionBase`1/<TryRecompute>d__17<bool>:MoveNext ()
      3.64 |       3.64 | ActualLab.Fusion.Computed/<Capture>d__83`1<ActualChat.ChatId>:MoveNext ()
      4.20 |       3.59 | System.Collections.Immutable.ImmutableHashSet`1<ActualChat.ChatId>:Union (System.Collections.Generic.IEnumerable`1<ActualChat.ChatId>,System.Collections.Immutable.ImmutableHashSet`1/MutationInput<ActualChat.ChatId>)
      3.48 |       3.48 | ActualLab.Fusion.ComputeFunctionBase`1/<TryRecompute>d__17<System.ValueTuple`2<ActualChat.ChatId, bool>>:MoveNext ()
      3.47 |       3.46 | ActualLab.Rpc.Infrastructure.RpcOutboundCall`1<ActualLab.Rpc.RpcNoWait>:.ctor (ActualLab.Rpc.Infrastructure.RpcOutboundContext)
      3.45 |       3.45 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<ApplyRpcUpdate>d__18<ActualLab.Api.ApiArray`1<ActualChat.ContactId>>:MoveNext ()
      3.42 |       3.42 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<ComputeCachedOrRpc>d__17<ActualLab.Api.ApiArray`1<ActualChat.ContactId>>:MoveNext ()
      3.31 |       3.31 | System.Number:FormatPercent<char> (System.Collections.Generic.ValueListBuilder`1<char>&,System.Number/NumberBuffer&,int,System.Globalization.NumberFormatInfo)
      3.56 |       3.30 | Pidgin.OneOfParser`2<char, char>:TryParse (Pidgin.ParseState`1<char>&,Pidgin.PooledList`1<Pidgin.Expected`1<char>>&,char&)
      3.30 |       3.30 | ActualLab.Fusion.Computed/<Capture>d__83`1<System.ValueTuple`2<ActualChat.ChatId, bool>>:MoveNext ()
      3.29 |       3.29 | ActualChat.Kvas.StoredState`1<ActualLab.Api.ApiArray`1<ActualChat.UI.Blazor.App.Services.ActiveChat>>:CreateComputed ()
      3.28 |       3.28 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<Compute>d__15<ActualChat.Users.Presence>:MoveNext ()
      3.46 |       3.26 | ActualLab.Result`1<System.ValueTuple`2<ActualChat.ChatId, bool>>:MemoryPack.IMemoryPackFormatterRegister.RegisterFormatter ()
      3.82 |       3.24 | ActualLab.Fusion.Computed/<Capture>d__83`1<ActualChat.UI.Blazor.App.Services.ChatAudioUI/RecordingBeepState>:MoveNext ()
      5.25 |       3.21 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<Compute>d__15<ActualLab.Api.ApiArray`1<ActualChat.ContactId>>:MoveNext ()
      3.19 |       3.19 | ActualLab.Fusion.Computed`1/<Use>d__37<ActualLab.Api.ApiArray`1<ActualChat.UI.Blazor.App.Services.ActiveChat>>:MoveNext ()
      3.16 |       3.16 | System.Collections.Generic.HashSet`1<ActualChat.UI.Blazor.App.Services.ActiveChat>:.ctor (System.Collections.Generic.IEqualityComparer`1<ActualChat.UI.Blazor.App.Services.ActiveChat>)
      4.61 |       3.13 | ActualLab.Fusion.Client.Interception.RemoteComputeMethodFunction`1/<ComputeCachedOrRpc>d__17<ActualLab.Mathematics.Range`1<long>>:MoveNext ()
      3.09 |       3.09 | System.Number:FormatDouble<byte> (System.Collections.Generic.ValueListBuilder`1<byte>&,double,System.ReadOnlySpan`1<char>,System.Globalization.NumberFormatInfo)
      3.09 |       3.09 | System.Number:NumberToString<byte> (System.Collections.Generic.ValueListBuilder`1<byte>&,System.Number/NumberBuffer&,char,int,System.Globalization.NumberFormatInfo)
      3.13 |       3.05 | System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1/AsyncStateMachineBox`1<System.Threading.Tasks.VoidTaskResult, ActualLab.Time.TimerSet`1/<OnRun>d__25<object>>:MoveNext (System.Threading.Thread)
      3.08 |       3.05 | System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1<long>:ConfigureIgnoreCondition (System.Nullable`1<System.Text.Json.Serialization.JsonIgnoreCondition>)
...

And the tail:

      0.88 |     -41.28 | Microsoft.Maui.Dispatching.DispatcherExtensions/<>c__DisplayClass2_0`1/<<DispatchAsync>b__0>d<ActualChat.Language>:MoveNext ()
Sum of self time (ms): 1410.85

[alexyakunin]

@jonathanpeppers this aligns quite well with our own measurements shown e.g. here - the total startup time (that's our internal metric which measures it till a certain point in rendering) was 2.1s during this specific recording - it's slightly higher usual, but 1.4/2.1 ~= 66%, which is quite close to our own estimate.

And here I even explicitly tagged you and wrote that 75% of our startup time is spent on JIT, no matter whether it's full or profiled AOT.

All of that data was shared in April-May. And no one moved a finger.

[alexyakunin]

As for System.Net.Sockets.SafeSocketHandle:.ctor, I guess it's here due to a workaround for #104599

[alexyakunin]

Two more runs - just wanted to check how consistent this is.

The total is ~ the same, but top methods are mostly different.

• out2.txt
• out3.txt

Oh, and I noticed there is negative "self-time" closer to the end in all these files.

Based on ChatGPT analysis of out3.txt, The sum of negative numbers in the "Self (ms)" column is −400.23 milliseconds.

[alexyakunin]

Okey... jit-times doesn't properly calculate JIT time, @jonathanpeppers

I created pull request addressing some of issues I spotted, but overall, I am not sure if we can even trust jit-times output - the tool seems to assume methods.txt is produced by a single-threaded app, so there are tons of warnings.

I'm attaching new outputs here:

• out1.txt
• out2.txt
• out3.txt
• example-warnings.txt

And that's what you'll find in the end now: Sum of self time (ms): 3735.00 - and now it's quite close to the time shown on this screenshot in terms of absolute value.

[alexyakunin]

Just in case, the original methods*.txt:

• methods1.txt
• methods2.txt
• methods3.txt

[alexyakunin]

Hmm, just found a whole category of methods which aren't generic ones:

@jonathanpeppers do you know why such methods aren't AOT-compiled in full AOT mode? Or does it mean any method appears in methods.txt, i.e. AOT compiled methods also appear there, but processing time is tiny for them? Based on the # of lines (~60K) it feels like it's the second case, i.e. every method is listed there: previously I wrote that Mono debug log shows 16K "AOT NOT FOUND" + 8K "AOT FOUND" entries, i.e. 24K in total, and here it's 30K, i.e. even more somehow.

If that's true, it makes sense to filter out all sub-0.01ms methods (looks like these are AOT compiled methods) + likely, all wrappers.

[alexyakunin]

Okey, so .NET 9 is released (I hope that explains why no one could help earlier), any chance someone looks at this now?

[vyacheslav-volkov]

It seems to me that there has been little to no noticeable progress between .NET 8 and .NET 9 for Android. While I haven’t conducted precise benchmarks yet, it subjectively feels like .NET 9 might even have a slower startup time compared to .NET 8

This issue has been open for more than six months, yet we still haven’t received a clear explanation of why it’s impossible to resolve or what specific challenges are blocking progress.

@jkotas Could you please involve someone from the mono-runtime team who could provide clarification and address the community's concerns?

[jkotas]

cc @steveisok @vitek-karas

Mono AOT compiler has number of limitations as you have found. I doubt that there is a simple quick way to address them in the existing releases. I am sorry. We have been discussing these topics among ourselves and how to address the issues like the one you have reported going forward. We will share our goals for .NET 10 once we have firm plans in place like we have done for past releases (likely after the New Year for this area).

[alexyakunin]

It subjectively feels like .NET 9 might even have a slower startup time compared to .NET 8

5-10% slower on Android in our case.

[alexyakunin]

@jkotas can you or someone who understands the limitations at least explain why it can't be fixed on Android, while the same scenario works on iOS (no JIT + no interpreter), and it's the same mono AOT compiler?

I am asking this because your response tells that MS already knows what's the root cause. So explaining it would be great in terms of at least restoring the trust.

[jkotas]

There are multiple problems. I do not think we have a complete list. One of them: Java interop uses Reflection.Emit. If you configure the runtime to have no JIT and no interpreter, Reflection.Emit is going to be break and Java interop too.

why it can't be fixed on Android

Nit: I have said that there is no simple quick way to fix it. It is all fixable in principle. For example, Java interop mentioned above can be refactored to avoid Reflection.Emit dependency and, in fact, we are looking into that.

[alexyakunin]

One of them: Java interop uses Reflection.Emit

But that's fine in this specific case:

1. I don't think there was an ask to have neither JIT nor interpreter - one of asks was to enable AOT + interpreter (as a way to possibly alleviate / workaround the root cause here, i.e. missing AOT code that triggers JIT), but e.g. I don't see why it even makes sense to disable both, and IMO Java interop is a relatively small thing in terms of cost - esp. in case with MAUI + Blazor apps, where interop is rare + you can push back whatever requires interop to some later stages of startup, etc.

2. The main issue here is that a code for specific kind of generic methods currently isn't generated - in particular, any async method currently requires JIT. Nevertheless, somehow it works on iOS. And that's why I asked for some details - if it's the same Mono AOT & more or less the same logic, why the fix for Android is complex?

[alexyakunin]

And just a feedback: it feels that you assume the "end game" for AOT is "just AOT, no JIT, no interpreter". And I think even though such a goal makes sense for some (maybe even many) apps, IMO it's a very bad fit for WASM & mobile apps. Here is what we care about:

• Startup time - and you know for sure that's crucial, coz that's ~ the only thing you measure for AOT, right?
• App size - super important for WASM, somewhat important for mobile.

And if you think of a perfect runtime that tackles these goals, it's simply a caching JIT:

• Startup time must be as good as it could be - except for the very first start, where the startup code is JITted.
• App size is as small as it can be.
• And there are no downsides - i.e. it's literally the same .NET as on "big OSes".

The only problem here is iOS, where JIT isn't an option. But even in this case I'd rather prefer profile-guided AOT + interpreter -- solely because of a smaller app size.

And if we can't get all of these options, profiled AOT + interpreter is the option I'd prefer 100% of time. Full AOT is nice in theory, but it is way more complex to deal with in practice:

• It tends to produce huge binaries (just a few generics can easily "explode" them). And it sucks to know that maybe 75+% of code there has zero chance to run - it's there "just because".
• Incompatibility w/ Reflection.Emit & other Reflection-based APIs sucks - and that's, by the way, is super annoying. E.g. we currently investigate why certain parts of our app don't work on .NET 9, even though it worked on .NET 8, and it's not even about AOT, but about trimming.

Overall, I am explicitly saying here that a goal of having "just AOT, no JIT, no interpreter" has zero value for us. We'd rather have a working profiled AOT + interpreter (or JIT).

[jkotas]

I don't think there was an ask to have neither JIT nor interpreter

You have asked why the config used on iOS (no JIT + no interpreter) does not just work on Android that is the question that I was answering.

if it's the same Mono AOT & more or less the same logic, why the fix for Android is complex?

Mono AOT compiler and runtime have a lot of hardcoded policies. If you enable the interpreter or if the runtime has JIT, it changes how the AOT compiler and runtime work. Look for all places that call methods like mono_aot_mode_is_interp, mono_aot_mode_is_full or check the various MONO_AOT_MODE enum values. It is several hundred places where these policies are hardcoded and they have complicated interplay with each other.

[alexyakunin]

@jkotas am I right you are saying that technically AOT should work the same way on Android as on iOS, but since these two targets were using different sets of AOT settings for a very long time & no one really tried certain combination of options, you don't know for sure whether certain options used on iOS (e.g. AOT for shared generics) will work with other Android-specific options (e.g. JIT, which were never used on iOS, Android-specific interop code, and so on)?

[alexyakunin]

@jkotas as for this:

You have asked why the config used on iOS (no JIT + no interpreter) does not just work on Android that is the question that I was answering.

I wasn't asking specifically this - I mentioned this mode as evidence that "proves" there is seemingly no fundamental gap in how AOT handles mentioned generic methods -- all because it works correctly on iOS.

[jkotas]

you don't know for sure whether certain options used on iOS (e.g. AOT for shared generics) will work with other Android-specific options (e.g. JIT, which were never used on iOS, Android-specific interop code, and so on)?

Right. Results of the experiments shared earlier in this discussion show that it does not work. For example: #101135 (comment)