Llama-bench-0.8.6 with Mini Ryzens! #456
Replies: 4 comments 7 replies
-
Thanks for posting your numbers! The Threadripper 2970WX probably had underwhelming numbers due to NUMA. What OS are you running these benchmarks on? Was it Windows? Is there any chance you could boot into Linux and run again there? I'd like to know how much we stand to gain by adding Windows NUMA support. Thanks! |
Beta Was this translation helpful? Give feedback.
-
Ubuntu 24.04 on AMD Ryzen 9 7940HS with 64GB RAM (Massive improvement over Windows)
|
Beta Was this translation helpful? Give feedback.
-
On a Framwork 16 laptop with 64Go of RAM (@5600 cas40)
pretty the same ...
Nice isn't it.... But wait I wanted to know what I have if I offload only some Layer (and not all) So test with 1 layer on GPU other on CPU:
wait a minute... what's that??? I have add trace, look in model operator... made many hypotheses... And I can't figure what happen... That is the command line I use for the CPU: For the GPU: If someone have an idea why this happen? What change in the compute Layer on CPU if some Layer are build with a GPU (hip...) |
Beta Was this translation helpful? Give feedback.
-
Latest AMD Ryzen 9 8945HS with 96GB DDR5
Ran more tests on more models:
Another by itself (fastest):
|
Beta Was this translation helpful? Give feedback.
-
CPU Testing Experiments
Tests with Llamafile-bench on “Mini Ryzen” low power compute
(Table tests are output from llamafile-bench for the respective models)
Ryzen 9 7940HS TDP at 35 watts and 6900HX TDP at 45 watts and TR 2970WX TDP at 250 watts
Key observations:
Individual Tests
AMD Ryzen 9 7940HS CPU Tests (64GB RAM 5600Mhz):
(Exact Configuration: https://www.amazon.com/MINISFORUM-7940HS-Radeon-USB3-2-PCIe4-0/dp/B0CCX744RD/ $699 USD total)
Mixtral 8x7B
Llama 3 8B
Llama 3 70B
AMD Ryzen 9 6900HX (64GB RAM 4600Mhz):
(Exact configuration: https://www.amazon.com/MINISFORUM-PCIe4-0-Outputs-Graphics-Computer/dp/B0CNXSDLVN $504 USD total)
Mixtral 8x7B
Llama 3 8B
Llama 3 70B
AMD Ryzen Threadripper 2970WX 24-Core (96GB RAM):
Mixtral 8x7B
Mixtral 8x22B
Llama 3 8B
Llama 3 70B
Beta Was this translation helpful? Give feedback.
All reactions