Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Greetings,
Image used:
redislabs/redisai:1.2.7-gpu-bionic
During my testing of RedisAI, I found that when running ONNX models, GPU could not be used, and every attempt failed. RedisAI would use the CPU instead.
When adding a model with
con.modelset('test', 'ONNX', 'GPU', m1)
RedisAI produced an error as bellow.redisai-redisai-1 | 2023-12-09 14:01:51.062001467 [E:onnxruntime:RedisAI, provider_bridge_ort.cc:964 Ensure] Failed to load library libonnxruntime_providers_shared.so with error: libonnxruntime_providers_shared.so: cannot open shared object file: No such file or directory
Looking at the container and the libraries located there I get this:
root@32f5fdc976e8:/# du -h /var/opt/redislabs/artifacts/redisai-gpu-onnxruntime.linux-bionic-x64.1.2.7.tgz 5.6M /var/opt/redislabs/artifacts/redisai-gpu-onnxruntime.linux-bionic-x64.1.2.7.tgz
After analyzing the code and performing some debugging, I discovered that
get_deps.sh
script downloads onnxruntime binaries fromhttps://s3.amazonaws.com/redismodules/onnxruntime
. Specifically, it fetches the filehttps://s3.amazonaws.com/redismodules/onnxruntime/onnxruntime-linux-x64-gpu-1.11.1.tgz
, which has a size of approximately 5MB. I noticed that this file size is significantly smaller than the official GPU release for ONNX Runtime available athttps://github.com/microsoft/onnxruntime/releases/download/v1.11.1/onnxruntime-linux-x64-gpu-1.11.1.tgz
, which exceeds 100MB in size.To make the RedisAI container run on GPU, I replaced the onnxruntime files by voluming the onnx directory. After this change, the container was able to run on GPU.
Based on my investigation, it appears that the root cause of the issue was the wrong onnxruntime file uploaded to the RedisAI AWS host. To fix this, I changed the
ORT_URL_BASE
in$OS == linux
to download from onnxruntime's Github page. This change enabled the container to run on GPU for ONNX models properly.here's a PR with the required changes.