Use of all the CPU cores on Raspberry Pi 5 with llamafile

[ramkumarkoppu]

Hi,

To use all the CPU cores on Raspberry Pi 5 to speed up the inference, what configuration needs to be done with the llamafile?

[ChristianWeyer]

Maybe @jart can help here?

[Djip007]

Normally it will by default, no configuration is needed. (At least it do on x86)
if not try add --threads 4 and report it.

[jart]

You can speed up prompt processing by using F16 weights on RPI5 which is fast. for faster token generation speeds, your best bet at the moment would probably be Q4_0 or Q8_0. They'll have good performance all around. It's important to note Q4_0 is legacy and @ikawrakow might consider contributing new K quantizers for ARM64 soon.

[mofosyne]

Do we have a survey mechanism perhaps? E.g. what if we have a binary that runs though different benchmarks, then write a single sentence telling the user which mode they should use? And perhaps better if it has an optional "do you want to upload your spec" to a leaderboard?

In the readme you can tell users to download the benchmark first, run it though then pick a model from huggingface that matches the ability of your machine.