Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

llama.cpp sync for SVE support for Q4_K_Ms #109

Open
a-ghorbani opened this issue Jan 16, 2025 · 1 comment
Open

llama.cpp sync for SVE support for Q4_K_Ms #109

a-ghorbani opened this issue Jan 16, 2025 · 1 comment

Comments

@a-ghorbani
Copy link
Contributor

Apologies, I know you just synced a few days ago, but the numbers for this PR look amazing:
ggerganov/llama.cpp#11227 (comment)

@Vali-98
Copy link
Contributor

Vali-98 commented Jan 17, 2025

Hey there, wanted to ask if you actually tested this on device?

As far as I know, SVE isn't not actually implemented by most Android mobile SOCs, and the few which do have limited compatibility (Pixel devices are the biggest offender).

Most SVE implementations seem to be for server-grade ARM, like Graviton.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants