Change lexing

[kaleidawave]

This will be a long WIP PR to change how lexing works and experiment with some different ideas.

Currently the default mode from the lexer is a lexer in one thread that pushes tokens to a mspc::channel and another thread that reads

A few reasons

• There are limitations to expect_expression Cannot parse <T>(t: T) => T under JSX mode (problems with expression prefix) #165
• The parser is currently slower than the later type checker phase. Could changes improve performance?
• Simplify things with new lines, partial syntax and lookahead. Allow WASM to be under the same mode and to not buffer the whole source (currently buffers everything rather than

Things to change

• Move away in the lexer emitting tokens. Instead a lexer that the parser requests tokens from
• Allow the parser to request a different lexer mode based on the state of the parser
• Avoid intermediate tokens like ) ] } and other things via requested from expect_next.
• Avoid the peek() buffering system

Others (that might be other prs)

• Not using owned strings, instead references into a store (that may or may not be the FileSystem map from source-map)
• Using Vec/&[u8] instead of String/&str, that might reduce overhead in utf8 checker (or whatever)
• Changing token and AST definitions to be more performant (?)

Opening this as a PR at this early stage to track updates and not lose any progress.

Could use something like grit to automate large API changes?

Again there are better parsers out there now. Not all feature complete and as small. This isn't something that is needed but interesting to experiment with and helps with patterns in the checker

[kaleidawave]

Making some more progress on this:

• More compatibility, got a few more complex examples including the return type annotations in arrow functions inside a ternary conditional expression (https://x.com/kaleidawave/status/1837833122437632370)
• Parsing is roughly 2x faster just on this initial change (I think more could be done to improve this)

To add back

• Partial syntax
• Better errors (some are todos)
• The constant compilation stuff

To fix

• Some issues with type annotations being default
• Renaming some methods
• Some AST node positions are broken
• Fuzzing (I think because there are panic!s in place of actual errors)
• Tests (again maybe positions)