5.2 support: Raw identifiers

[Julow]

This is the last missing piece for 5.2 support.

Thanks to @ccasin who made the work in janestreet#88

[hhugo]

Why is it ~constructor:true here ?

[hhugo]

We need raw idents to be preserved. if I use #effect in a 5.3 switch, I don't want it to be rewritten in a 5.2 switch.
The current implementation depends on Lexer.is_keyword that is version dependent.

[hhugo]

Thinking more about it, you could even imagine a mode that would help migrating to newer ocaml version by escaping idents that are becoming keyword.

[ccasin]

Sorry, I should have mentioned it, but I think my PR in the janestreet repo has a bug: escaped operators in types need special treatment.

The compiler itself also has this bug, see the report in ocaml/ocaml#13603 and a fix in ocaml/ocaml#13604. I'm in the process of adapting that fix in my ocamlformat pr to our branch.

[Julow]

I agree that we should preserve the raw identifier marker and not try to add it again later. I'm now investigating passing that information through a table, like we do for comments, as changing the AST for this might brings huge changes to the lexer and parser, making future backports harder.

Thinking more about it, you could even imagine a mode that would help migrating to newer ocaml version by escaping idents that are becoming keyword.

That should be possible on top of this work.

[hhugo]

changing the AST for this might brings huge changes to the lexer and parser, making future backports harder.

Can't you keep the escaping prefix during lexing ? Without touching the ast ?

[hhugo]

Sorry, I should have mentioned it, but I think my PR in the janestreet repo has a bug: escaped operators in types need special treatment.

If we go with preserving escaped idents, we no longer have special treatments

[hhugo]

Don't we need to update the lexer in parser-standard as well ?

[hhugo]

The upstream OCaml lexer is able to recognize different set of keywords. One can pass -keywords 5.2 on the command line

[Julow]

I'm currently investigating changing the AST to represent how identifiers looked in the source. If that's too much changes in the parser and AST, I'll switch to the table approach.

Don't we need to update the lexer in parser-standard as well ?

No, I'll let the parser-standard behave as it does upstream (\#foo is represented the same as foo).

The upstream OCaml lexer is able to recognize different set of keywords. One can pass -keywords 5.2 on the command line

This should be plugged to the ocaml-version option that we have but that won't solve the issue that \#effect must not be turned into effect, regardless of the value of ocaml-version. This option might be set wrong, or people writing \#effect might intent to set it to 5.3 soon.
Preserving the source code is the only approach I'm considering now.

[hhugo]

Don't we need to update the lexer in parser-standard as well ?

No, I'll let the parser-standard behave as it does upstream (\#foo is represented the same as foo).

Ok, turns out the standard lexer was already updated in #2512

[hhugo]

I'm currently investigating changing the AST to represent how identifiers looked in the source. If that's too much changes in the parser and AST, I'll switch to the table approach.

In #2619 (comment), I was suggesting to not change the Ast and just store the escaped ident in the LIDENT token. What do you think ?

[Julow]

I pushed the Parsetree approach here: #2620
The support in Fmt_ast is trivial in exchange for a +233 -198 change in the vendored parser. There's +88 -72 line changes in parser.mly, which will negatively impact future backports but which is less than I expected.

[Julow]

In #2619 (comment), I was suggesting to not change the Ast and just store the escaped ident in the LIDENT token. What do you think ?

That would have been simpler indeed. I'll investigate how that looks.

[Julow]

The lexer approach is in this PR: #2621
I'll close this, as the other approach is simpler and more complete.