[RFC] Add PreLexerInterface

[VincentLanglet]

Inspired by @smnandre #3951 (comment)

Currently symfony/ux override the existing Lexer in order to pre-processing the code before the Lexer
https://github.com/symfony/ux/blob/2ded7546dc5f7ef2b85acb33ba6d4871b5376d7e/src/TwigComponent/src/Twig/ComponentLexer.php#L27-L42

One main issue could be that if another library has the same need, you have to chose between both library (since the override can only done once).

One idea could be to introduce a new extension point before the Lexer is called. Something like a PreLexer, then you can have as many PreLexer as you want.

Wording can be improved if accepted and tests might be needed.

Also, I dunno if there are similar use case

• For PreParser
• For PreCompiler

[smnandre]

TwigComponent PreLexer

The current implementation is one "per default" ... and it has a couple of drawbacks.

As these issues would also apply with your suggestion, let's discuss them here 😄

1- Source coherence

First: this pre-lexing step desynchronizes the Source object from the real content of the file on disk.
This can have very annoying DX effect on error messages, exposed line numbers, etc..

For debug too because source::getCode() can be very surprising for users not knowing how this pre-lex phase works.

2- No tools

Same cause but different consequence, it also forbid us to create any tool to optimize / validate the original code of the template... as it is "lost" after this step and invisible for the Parser, Optimizer, etc.

3- Lexer implementation

To make it work, the PreLexer contains a lot of duplicated/rewritten code. Because it needs to do things the Lexer does.. but does not expose nor allow to extend.

This make the TwigComponent PreLexer a bit fragile, and not 100% in phase with Twig configuration (example: custom open/close tags caracters), and above all sensible to Twig evolutions.

---

PreLexerInterface ?

PreLexer cooperation

I don't think it's possible to have multiple pre-lexer working together (as least, not if they work like the TwigComponent pre-lexer work..)
The current PreLexer would not cooperate well if another one did similar changes in the source code (but with another syntax).

This also raises the question of the order of different PreLexer runs

Needs ?

Do you have examples of needs/features that a PreLexer "chain" could help solve ?

I'm feeling they could/should be implemented in other parts of the templating process (the loader beeing a candidate)

[smnandre]

I'm very eager to have @fabpot and/or @stof opinion on this question (sorry for the hard-mention 😅 )

[fabpot]

I think we had a discussion about pre-lexing a while ago, and my gut feeling back then was that we should avoid doing this. And @smnandre has mentioned a lot of good reasons for not doing it.