Tokenization testing #4
Conversation
| @@ -1,7 +1,7 @@ | |||
| { | |||
There was a problem hiding this comment.
Changes in this file are due to the "extra" change in this PR
There was a problem hiding this comment.
ah, it's going to be a blast of a merge conflict with my other small PR. which might be a reason to pull those changes out of this PR and apply them on main after merging that?
There was a problem hiding this comment.
Merged main into this branch, then resolved all conflicts in favor of main, then regenerated the file
| @@ -23,6 +23,10 @@ jobs: | |||
| java-version: 11 | |||
| distribution: 'temurin' | |||
| cache: 'maven' | |||
| - uses: actions/setup-node@v4 | |||
There was a problem hiding this comment.
add which version of node we need. and enable caching to improve CI times.
rascal-textmate-core/.gitignore
Outdated
| node_modules | ||
| package-lock.json | ||
| package.json |
There was a problem hiding this comment.
Normally package.json is not in the .gitignore
| @@ -0,0 +1,3 @@ | |||
| npx vscode-tmgrammar-test ^ | |||
There was a problem hiding this comment.
looks like the bat is the exact same.
how about we move it to package.json in the run block. and in rascal we just use npm run conversion-tests ?
There was a problem hiding this comment.
Fixed:
- Call
npxdirectly from Rascal instead of via the scripts - Remove the scripts
| @@ -1,7 +1,7 @@ | |||
| { | |||
There was a problem hiding this comment.
ah, it's going to be a blast of a merge conflict with my other small PR. which might be a reason to pull those changes out of this PR and apply them on main after merging that?
Previously, only structural properties of generated TextMate grammars could conveniently be tested from within Rascal (e.g., expected number of rules). This PR adds the infra to test also behavioral properties (i.e., expected tokenization of example input text).
Specifically, the
vscode-tmgrammar-testtool is used to: (1) tokenize example input text using generated TextMate grammars; (2) compare actual tokenization against expected tokenization. The tool uses the same tokenization engine as VS Code.CI is updated accordingly.