Still running in CPU

[mmw1984]

Sorry that the screenshot is not in English

[mmw1984]

Device: Oneplus Ace 2 Pro
Chioset: Snapdragon 8 Gen 2
System: Android 14
RAM: 16GB

[danemadsen]

fixed

[ILOVEPIE]

No, it's not. The new changes include the x86_64 Linux build of the Vulkan SDK, not the correct ARM version, so maid_llm doesn't compile with Vulkan support.

[danemadsen]

Awe thats no fun

[danemadsen]

@ILOVEPIE any idea how to install that in github actions

[ILOVEPIE]

@ILOVEPIE any idea how to install that in github actions

The only thing I can think of off the top of my head is installing the dart binding for Vulkan, but I'm not sure if that'll work. I'll do a little more research.

[ILOVEPIE]

Apparently the Vulkan headers and libraries come as part of the Android SDK. I need to figure out where Flutter puts that so that you can adjust your CMake options to point to that.

[danemadsen]

No because the gitlab pipeline compiles it with vulkan fine.

download this build here:
https://gitlab.com/mobile-artificial-intelligence/maid/-/jobs/7663138914

its an issue with the github action thats causing it to not detect the vulkan headers

[danemadsen]

nvm its not working there either

[ILOVEPIE]

OK, I figgured it out... probably. the docs on this are terrible but i have an idea now. Ill test it on my fork then PR the changes.

[danemadsen]

Ok, much appreciated if you can get it working.

[ILOVEPIE]

Ok, much appreciated if you can get it working.

Assuming I don't need to make any other minor adustments to the build file, and the binary doesn't crash, I should be able to send that PR over shortly. I've already got it detecting the android vulkan library. There were some outdated paths in the CMakeLists though, which caused the build to fail when vulkan was available. So, fingers crossed that it doesn't crash when I install and test it.

[ILOVEPIE]

Ugg... The vulkan c++ headers aren't in the NDK, only the c ones.

[danemadsen]

You got this bro, i believe in you.

[ILOVEPIE]

You got this bro, i believe in you.

I have some good news and some bad news. I got the Vulkan support compiling. I haven't tested it yet, but I got it to find the headers and library in the compile phase. The problem is it's going to require a higher minimum version of android than we currently require, the llama.cpp vulkan implementation (for the version of llama.cpp we're currently using) requires (at least) vulkan 1.1 support (maybe higher) which means Android 14 minimum. The only solution I can come up with is having both a CPU-based version of the library and a GPU-based version of the library, and we detect if the user has Vulkan support and use the CPU library if they don't. And then we also give them an option to toggle GPU acceleration off, because phone SOC manufacturers aren't known for their GPUs and GPU Drivers. What I mean is phones tend to have fairly buggy GPUs, so we should at least have the option to turn the acceleration off.

[danemadsen]

The only solution I can come up with is having both a CPU-based version of the library and a GPU-based version of the library, and we detect if the user has Vulkan support and use the CPU library if they don't.

Its probably best to just ship 2 releases, 1 for vulkan / android 14 and 1 for versions below 14. Obviously it would be best to allow the user to switch between GPU / CPU (and eventually NPU) but for now shipping 2 apk's / bundles is probably a faster solution.

[ILOVEPIE]

The only solution I can come up with is having both a CPU-based version of the library and a GPU-based version of the library, and we detect if the user has Vulkan support and use the CPU library if they don't.

Its probably best to just ship 2 releases, 1 for vulkan / android 14 and 1 for versions below 14. Obviously it would be best to allow the user to switch between GPU / CPU (and eventually NPU) but for now shipping 2 apk's / bundles is probably a faster solution.

To be honest, I think it's about the same amount of work to do either option. So I'd rather do the more permanent solution.

[ILOVEPIE]

Just about to test the vulkan build.

[ILOVEPIE]

I've been trying to figure out why the Vulkan build is crashing. I'm not exactly sure. I'm getting some sort of weird illegal signal on ARM. I'm not too familiar with the ARM architecture. So I'm not sure if this is indicating an illegal instruction or some type of illegal register value or something. I don't know.

[ILOVEPIE]

I'm going to attempt to make a x86_64 build of the app to see if that will elucidate anything.