fix ollama gpu acceleration #373924
fix ollama gpu acceleration #373924
Conversation
nix-owners
bot
requested review from
abysssol,
elohmeier,
RoyDubnium and
dit7ya
| @@ -199,6 +199,13 @@ goBuild { | |||
| make ${dist_cmd} -j $NIX_BUILD_CORES | |||
| ''; | |||
|
|
|||
| postInstall = lib.optionalString stdenv.hostPlatform.isLinux '' | |||
There was a problem hiding this comment.
It may be a good idea to replace the condition stdenv.hostPlatform.isLinux with (enableRocm || enableCuda).
The replacement:
- Means what it says: if GPU acceleration is enabled, copy GPU related libraries. Currently the copying is always attempted when building on Linux, even on a default, CPU only build (which shouldn't need any GPU libraries to be copied).
- Won't activate for CPU builds, which is (probably) good (I'm not entirely certain, though). This script was removed because build errors were encountered when the
cpwas executed without any arguments to copy (there were no libraries to copy). I think that this probably happened because the copy was executed on CPU builds, and there aren't any files to copy on CPU builds (though, strangely, that would imply that there used to be GPU libraries to copy on CPU builds, since this script used to work).
I'm not actually sure whether or not the CPU build does have any GPU libraries to copy, or whether my theory is at all accurate in the first place, so further testing is probably needed. See also my other comment on the topic.
There was a problem hiding this comment.
I think switching this would be good as it is more aligned to the intent of the build and isLinux feels like a hack. It may also cut down the build time.
We should test to confirm that there aren't any strange surprises.
There was a problem hiding this comment.
Feedback has been addressed (: I will test this on Nvidia and CPU (cannot test on AMD)
There was a problem hiding this comment.
Works on both machines! Nvidia and CPU are both good
There was a problem hiding this comment.
I'm not actually sure whether or not the CPU build does have any GPU libraries to copy, or whether my theory is at all accurate in the first place, so further testing is probably needed. See also my #365718 (comment) on the topic.
There are multiple types of CPU runners that use the same structure. A rocm + CPU build ends up with:
lib/ollama/runners/:
cpu_avx cpu_avx2 rocm_avx
Official ollama releases end up with cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx rocm_avx folders under runners.
There was a problem hiding this comment.
Okay, so maybe we want to copy it indiscriminately then?
There was a problem hiding this comment.
Yeah, accel flag checking approach means CPU ollama build on this PR is missing cpu_avx/cpu_avx2 😅
$ nix build github:nixos/nixpkgs/5e728b154d0e0f25cbc8f05b7006d001f20df52e#ollama
$ ls result/lib/ollama/runners
ls: cannot access 'result/lib/ollama/runners': No such file or directory
There was a problem hiding this comment.
CPU build looks correct now:
$ nix build github:nixos/nixpkgs/6ad8f0640cdf4077335177a93b319b94a7226f15#ollama
$ ls result/lib/ollama/runners
cpu_avx cpu_avx2
|
Is this compatible with the changes in #373234? |
|
|
@pbsds Looks like these changes are in that PR as well. Maybe we close this and just fix it in that, if that's getting merged soon-ish? |
scd31
force-pushed
the
ollama-gpu-bugfix
branch
from
7c3ded6 to
5e728b1
Compare
|
@pbsds Actually we should be able to merge this one and then rebase that PR |
|
scd31
force-pushed
the
ollama-gpu-bugfix
branch
from
5e728b1 to
6ad8f06
Compare
|
Since #373234 is a superset of this PR and it is older, let's close this PR in favor of the other. |
Fixed a bug where Ollama is no longer using GPU acceleration. Tested working with an Nvidia card. I am unsure if this also affected AMD cards
Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.