support jetson AGX Orin

[johnnynunez]

No description provided.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mgoin]

If you want the wheel to support this, you need to also change the dockerfile

vllm/Dockerfile

Line 57 in 151ef4e

ARG torch_cuda_arch_list='7.0 7.5 8.0 8.6 8.9 9.0+PTX'

We can look at the impact on the wheel size after to help make a decision.

[johnnynunez]

If you want the wheel to support this, you need to also change the dockerfile

vllm/Dockerfile

Line 57 in 151ef4e

ARG torch_cuda_arch_list='7.0 7.5 8.0 8.6 8.9 9.0+PTX'

We can look at the impact on the wheel size after to help make a decision.

but it is arm, so the wheel of the dockerfile must be arm, it is different. So, I'm building vllm in jetson-containers.
Only put in on the scripts, it is sufficient
http://jetson.webredirect.org/jp6/cu126 https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/vllm

[mgoin]

Okay I think it is reasonable for us to allow building for this platform, although it would be nice if users wouldn't build for it by default

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @johnnynunez.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[johnnynunez]

@mgoin
jetson thor is coming, and jetson agx orin will be supported until 2032

@dusty-nv

[mgoin]

@johnnynunez please see this PR instead #9735

[johnnynunez]

@johnnynunez please see this PR instead #9735

Uoh perfect. We have right now running on jetson-containers but it is not optimized at all

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @johnnynunez.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mgoin]

Thanks - Closing as completed!