[mlir] Do not set lastToken in AsmParser's resetToken function #290
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I plan to upstream this |
The member lastToken is the last token that was consumed by the parser. Resetting the lexer position to a different position does not cause any token to be consumed, so lastToken should not be updated. Setting it to curToken can cause the scopeLoc.end location of OperationDefinition to be off-by-one, pointing to the first token after the operation. An example for an operation for which the scopeLoc.end location was wrong before is: %0 = torch.vtensor.literal(dense_resource<__elided__> : tensor<768xbf16>) : !torch.vtensor<[768],bf16> . Here the scope end loc always pointed to the next token
jorickert
force-pushed
the
jrickert.parse_end_loc
branch
from
f7cc773 to
f357780
Compare
|
Can you please add a test case to show the improvement? |
|
LGTM, per here this |
|
@mgehre-amd Added a test |
mgehre-amd
reviewed
Aug 21, 2024
mgehre-amd
reviewed
Aug 21, 2024
jorickert
force-pushed
the
jrickert.parse_end_loc
branch
from
e363f41 to
5193e67
Compare
cferry-AMD
reviewed
Aug 21, 2024
This test checks the parsed ops' locations and scope locations. It uses the !llvm.array type in the test to check that locations are correct for ops that contain an extended-type. The parser for extended-types calls resetToken, which caused wrong locations in the past.
This getter is can easily be confused with the similar named allowUnregisteredDialects setter
jorickert
force-pushed
the
jrickert.parse_end_loc
branch
from
5193e67 to
5f083d4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The member lastToken is the last token that was consumed by the parser. Resetting the lexer position to a different position does not cause any token to be consumed, so lastToken should not be updated. Setting it to curToken can cause the end location of OperationDefinition to be off-by-one, pointing to the first token after the operation.
An example for an operation for which the scopeLoc.end location was wrong before is:
%0 = torch.vtensor.literal(dense_resource<elided> : tensor<768xbf16>)
: !torch.vtensor<[768],bf16> . Here the scope end loc always pointed to the next token