[QNN] MatMul Op Builder to Handle All Cases of ONNX's MatMul #22639
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
centwang
requested review from
skottmckay,
cloudhan,
adrianlizarraga and
jywu-msft
cloudhan
approved these changes
Oct 29, 2024
skottmckay
reviewed
Nov 7, 2024
onnxruntime/core/providers/qnn/builder/opbuilder/matmul_op_builder.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/qnn/builder/opbuilder/matmul_op_builder.cc
Outdated
Show resolved
Hide resolved
centwang
force-pushed
the
weicwang/matmul_op_builder
branch
from
cae0690 to
172d6c5
Compare
centwang
changed the title
skottmckay
reviewed
Nov 27, 2024
onnxruntime/core/providers/qnn/builder/opbuilder/matmul_op_builder.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/qnn/builder/opbuilder/matmul_op_builder.cc
Outdated
Show resolved
Hide resolved
adrianlizarraga
approved these changes
Jan 8, 2025
snnn
pushed a commit
that referenced
this pull request
Jan 8, 2025
ONNX's MatMul is same as numpy.matmul, which supports input tensors with rank >= 1. But QNN's MatMul can only support input tensors with rank >= 2. This PR is to add MatMulOpBuilder for QNN EP to build QNN graph to support all possible cases of ONNX's MatMul, by adding Reshape nodes if necessary, e.g., if Reshape 1D input to 2D if exists, and Reshape output to expected shape at the end. This PR also tries to use FullyConnected Op for MatMul if 2nd input is 2D initializer or 1D tensor because FullyConnected is faster than MatMul on QNN EP. If 2nd input is 2D tensor, we require it an initializer because FullyConnected requires 2nd input in [n, k] shape, we can transpose it when graph building if it's an initializer (we don't want to add extra Transpose node). Use swin_base model as example, which contains several MatMul nodes with 2nd input is 2D initializer (not followed by Add), running on Gen3 mobile device, before the change, it takes 34.8876 ms, after this change, it's 27.0639 ms.
tarekziade
pushed a commit
to tarekziade/onnxruntime
that referenced
this pull request
Jan 10, 2025
…ft#22639) ONNX's MatMul is same as numpy.matmul, which supports input tensors with rank >= 1. But QNN's MatMul can only support input tensors with rank >= 2. This PR is to add MatMulOpBuilder for QNN EP to build QNN graph to support all possible cases of ONNX's MatMul, by adding Reshape nodes if necessary, e.g., if Reshape 1D input to 2D if exists, and Reshape output to expected shape at the end. This PR also tries to use FullyConnected Op for MatMul if 2nd input is 2D initializer or 1D tensor because FullyConnected is faster than MatMul on QNN EP. If 2nd input is 2D tensor, we require it an initializer because FullyConnected requires 2nd input in [n, k] shape, we can transpose it when graph building if it's an initializer (we don't want to add extra Transpose node). Use swin_base model as example, which contains several MatMul nodes with 2nd input is 2D initializer (not followed by Add), running on Gen3 mobile device, before the change, it takes 34.8876 ms, after this change, it's 27.0639 ms.
guschmue
pushed a commit
that referenced
this pull request
Jan 12, 2025
ONNX's MatMul is same as numpy.matmul, which supports input tensors with rank >= 1. But QNN's MatMul can only support input tensors with rank >= 2. This PR is to add MatMulOpBuilder for QNN EP to build QNN graph to support all possible cases of ONNX's MatMul, by adding Reshape nodes if necessary, e.g., if Reshape 1D input to 2D if exists, and Reshape output to expected shape at the end. This PR also tries to use FullyConnected Op for MatMul if 2nd input is 2D initializer or 1D tensor because FullyConnected is faster than MatMul on QNN EP. If 2nd input is 2D tensor, we require it an initializer because FullyConnected requires 2nd input in [n, k] shape, we can transpose it when graph building if it's an initializer (we don't want to add extra Transpose node). Use swin_base model as example, which contains several MatMul nodes with 2nd input is 2D initializer (not followed by Add), running on Gen3 mobile device, before the change, it takes 34.8876 ms, after this change, it's 27.0639 ms.
adrianlizarraga
added a commit
that referenced
this pull request
Jan 17, 2025
…inputs (#23419) ### Description - Fixes regression for MatMul with two quantized/dynamic uint16 inputs. We need to convert input[1] to uint8 to pass QNN validation. - Separates translation of `ONNX MatMul -> QNN MatMul` and `ONNX MatMul -> QNN FullyConnected` to separate functions to make the code more readable. ### Motivation and Context The following PR updated the handling of MatMul. The logic to handle MatMul with two non-const uint16 inputs was not ported from [simple_op_builder.cc](https://github.com/microsoft/onnxruntime/blob/c64fa18834f0651b7d62507a34d802874b099c29/onnxruntime/core/providers/qnn/builder/opbuilder/simple_op_builder.cc#L107) to the new [matmul_op_builder.cc](https://github.com/microsoft/onnxruntime/blob/c64fa18834f0651b7d62507a34d802874b099c29/onnxruntime/core/providers/qnn/builder/opbuilder/matmul_op_builder.cc#L57). #22639
ashrit-ms
pushed a commit
that referenced
this pull request
Jan 23, 2025
…inputs (#23419) ### Description - Fixes regression for MatMul with two quantized/dynamic uint16 inputs. We need to convert input[1] to uint8 to pass QNN validation. - Separates translation of `ONNX MatMul -> QNN MatMul` and `ONNX MatMul -> QNN FullyConnected` to separate functions to make the code more readable. ### Motivation and Context The following PR updated the handling of MatMul. The logic to handle MatMul with two non-const uint16 inputs was not ported from [simple_op_builder.cc](https://github.com/microsoft/onnxruntime/blob/c64fa18834f0651b7d62507a34d802874b099c29/onnxruntime/core/providers/qnn/builder/opbuilder/simple_op_builder.cc#L107) to the new [matmul_op_builder.cc](https://github.com/microsoft/onnxruntime/blob/c64fa18834f0651b7d62507a34d802874b099c29/onnxruntime/core/providers/qnn/builder/opbuilder/matmul_op_builder.cc#L57). #22639
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ONNX's MatMul is same as numpy.matmul, which supports input tensors with rank >= 1. But QNN's MatMul can only support input tensors with rank >= 2. This PR is to add MatMulOpBuilder for QNN EP to build QNN graph to support all possible cases of ONNX's MatMul, by adding Reshape nodes if necessary, e.g., if Reshape 1D input to 2D if exists, and Reshape output to expected shape at the end.
This PR also tries to use FullyConnected Op for MatMul if 2nd input is 2D initializer or 1D tensor because FullyConnected is faster than MatMul on QNN EP. If 2nd input is 2D tensor, we require it an initializer because FullyConnected requires 2nd input in [n, k] shape, we can transpose it when graph building if it's an initializer (we don't want to add extra Transpose node).
Use swin_base model as example, which contains several MatMul nodes with 2nd input is 2D initializer (not followed by Add), running on Gen3 mobile device, before the change, it takes 34.8876 ms, after this change, it's 27.0639 ms.