Feature 'f16 arithemetic and compare instructions' requires .target sm_53 or higher

[shahizat]

Your current environment

Collecting environment information...
PyTorch version: 2.3.0
Is debug build: False
CUDA used to build PyTorch: 12.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (aarch64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.30.3
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.136-tegra-aarch64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Orin (nvgpu)
Nvidia driver version: N/A
cuDNN version: Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.8.9.4
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.9.4
/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.9.4
/usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.9.4
/usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.9.4
/usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.9.4
/usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.9.4
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Vendor ID: ARM
Model name: Cortex-A78AE
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 4
Socket(s): -
Cluster(s): 3
Stepping: r0p1
CPU max MHz: 2201.6001
CPU min MHz: 115.2000
BogoMIPS: 62.50
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp uscat ilrcpc flagm paca pacg
L1d cache: 768 KiB (12 instances)
L1i cache: 768 KiB (12 instances)
L2 cache: 3 MiB (12 instances)
L3 cache: 6 MiB (3 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-11
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, but not BHB
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] mypy==1.11.2
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] onnx==1.16.1
[pip3] onnx-graphsurgeon==0.3.12
[pip3] onnxruntime==1.19.2
[pip3] pyzmq==26.0.3
[pip3] sentence-transformers==3.0.1
[pip3] torch==2.3.0
[pip3] torchvision==0.18.0a0+6043bc2
[pip3] transformers==4.41.1
[pip3] tritonclient==2.48.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect

How would you like to use vllm

Greeting to all,

I want to run inference on the Nvidia AGX Orin dev kit. I don't know how to integrate it with vllm. I was building it using below command:

DOCKER_BUILDKIT=1 sudo docker build . --target vllm-openai --tag vllm/vllm-openai --build-arg max_jobs=10 --build-arg nvcc_threads=8 --build-arg torch_cuda_arch_list="8.7"

The error log:
`883.5 ptxas /tmp/tmpxft_000003ed_00000000-6_attention_kernels.ptx, line 4985278; error : Feature 'f16 arithemetic and compare instructions' requires .target sm_53 or higher
883.5 ptxas fatal : Ptx assembly aborted due to errors
883.5 ninja: build stopped: subcommand failed.

Dockerfile:136

135 | ENV CCACHE_DIR=/root/.cache/ccache
136 | >>> RUN --mount=type=cache,target=/root/.cache/ccache
137 | >>> --mount=type=cache,target=/root/.cache/pip
138 | >>> if [ "$USE_SCCACHE" != "1" ]; then
139 | >>> python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38;
140 | >>> fi
141 |

ERROR: failed to solve: process "/bin/sh -c if [ "$USE_SCCACHE" != "1" ]; then python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38; fi" did not complete successfully: exit code: 1`

Thanks in advance for your support!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[youkaichao]

you are building for aarch64 . please try #2021 (comment)

[gongchengli]

Your current environment

Collecting environment information... PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: 12.2 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (aarch64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.30.3 Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.136-tegra-aarch64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.2.140 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: Orin (nvgpu) Nvidia driver version: N/A cuDNN version: Probably one of the following: /usr/lib/aarch64-linux-gnu/libcudnn.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.9.4 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: ARM Model name: Cortex-A78AE Model: 1 Thread(s) per core: 1 Core(s) per cluster: 4 Socket(s): - Cluster(s): 3 Stepping: r0p1 CPU max MHz: 2201.6001 CPU min MHz: 115.2000 BogoMIPS: 62.50 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp uscat ilrcpc flagm paca pacg L1d cache: 768 KiB (12 instances) L1i cache: 768 KiB (12 instances) L2 cache: 3 MiB (12 instances) L3 cache: 6 MiB (3 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, but not BHB Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] mypy==1.11.2 [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.26.4 [pip3] onnx==1.16.1 [pip3] onnx-graphsurgeon==0.3.12 [pip3] onnxruntime==1.19.2 [pip3] pyzmq==26.0.3 [pip3] sentence-transformers==3.0.1 [pip3] torch==2.3.0 [pip3] torchvision==0.18.0a0+6043bc2 [pip3] transformers==4.41.1 [pip3] tritonclient==2.48.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: N/A vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: Could not collect

How would you like to use vllm

Greeting to all,

I want to run inference on the Nvidia AGX Orin dev kit. I don't know how to integrate it with vllm. I was building it using below command:

DOCKER_BUILDKIT=1 sudo docker build . --target vllm-openai --tag vllm/vllm-openai --build-arg max_jobs=10 --build-arg nvcc_threads=8 --build-arg torch_cuda_arch_list="8.7"

The error log:

`883.5 ptxas /tmp/tmpxft_000003ed_00000000-6_attention_kernels.ptx, line 4985278; error : Feature 'f16 arithemetic and compare instructions' requires .target sm_53 or higher
883.5 ptxas fatal : Ptx assembly aborted due to errors
883.5 ninja: build stopped: subcommand failed.

Dockerfile:136

135 | ENV CCACHE_DIR=/root/.cache/ccache

136 | >>> RUN --mount=type=cache,target=/root/.cache/ccache
137 | >>> --mount=type=cache,target=/root/.cache/pip
138 | >>> if [ "$USE_SCCACHE" != "1" ]; then
139 | >>> python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38;
140 | >>> fi
141 |
ERROR: failed to solve: process "/bin/sh -c if [ "$USE_SCCACHE" != "1" ]; then python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38; fi" did not complete successfully: exit code: 1`

Thanks in advance for your support!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

meet the same problem, maybe try --build-arg torch_cuda_arch_list="8.6" instead of "8.7" ?
I am still working on building it ...

[shahizat]

Hello @youkaichao, @gongchengli, I have some good news to share @dusty-nv and @johnnynunez have done a great job by porting vLLM onto Nvidia Jetson Orin devices.

For more information, you can check out this link: https://github.com/dusty-nv/jetson-containers/tree/dev/packages/llm/vllm

If someone is interested, I am share the command I'm using to run it. I'm still experimenting with parameters.

sudo docker run --runtime nvidia \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=YOUR_HF_TOKEN" \
    --env "HF_HOME=/root/.cache/huggingface" \
   --env "TRANSFORMERS_CACHE=/root/.cache/huggingface" \
    -p 8000:8000 \
    --ipc=host \
    dustynv/vllm:r36.4.0 \
    vllm serve /root/.cache/huggingface/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659 --gpu-memory-utilization 0.5  --enforce-eager --enable-prefix-caching --enable-chunked-prefill