feat(parse/md): markdown header support in lexer

[AugustinMauroy]

Summary

Supporting ATX heading for markdown lexer.

Blocking thing

• test don't pass and IDK how to fix it.

[codspeed-hq]

CodSpeed Performance Report

Merging #5208 will not alter performance

_{Comparing AugustinMauroy:markdown-header (769030c) with main (4df66a5)}

Summary

✅ 95 untouched benchmarks

[dyc3]

This change doesn't quite look right to me. We already have MdHeader defined above, but you've added 6 new nodes for headers. IMO, we should rename MdHeader into AnyMdHeader, and have it be a union of all the other header levels.

[AugustinMauroy]

okay, but how should we represent the level of the heading ?

[dyc3]

Having MdHeader1 through MdHeader6 is fine. I'm saying we should change MdHeader to:

AnyMdHeader = MdHeader1 | MdHeader2 | MdHeader3 | MdHeader4 | MdHeader5 | MdHeader6

[dyc3]

IIRC, I think the whitespace is required for it to become a heading. Do you have a source for this behavior?

[AugustinMauroy]

in this example space is skip https://spec.commonmark.org/0.31.2/#example-62
in this example spaces is also skiped https://spec.commonmark.org/0.31.2/#example-62

[dyc3]

At least one space or tab is required between the # characters and the heading’s contents, unless the heading is empty. Note that many implementations currently do not require the space.

The code here makes the whitespace optional when it is actually required. See example 64 in that doc

[dyc3]

Should this be a parsing error instead?

[ematipico]

Markdown, I think, shouldn't have parsing errors. What I mean is that at the end, the language is very lax and, worst case scenario, a paragraph is always emitted.

I've never seen an editor emitting a parsing error 🤔

[AugustinMauroy]

This part of checking if it's valid ATX heading or just paragphe is done in LEXER but rust ask us for fallback

[dyc3]

Now that I'm looking at this again, this isn't right. The purpose of the lexer is to convert source code into an "alphabet" of sorts for the parser to interpret into a syntax tree. So it should output the "leaf" token, which would be the token for #, which you can get the SyntaxKind for via the macro T![#]. To continue the "alphabet" analogy, the lexer needs to emit the "letters" (tokens) that make up the "words".

I recommend taking a look at how the HTML lexer works, particularly how consume_token works here:

biome/crates/biome_html_parser/src/lexer/mod.rs

Lines 54 to 64 in 766492f

	fn consume_token(&mut self, current: u8) -> HtmlSyntaxKind {
	match current {
	b'\n' | b'\r' | b'\t' | b' ' => self.consume_newline_or_whitespaces(),
	b'<' => self.consume_l_angle(),
	b'>' => self.consume_byte(T![>]),
	b'/' => self.consume_byte(T![/]),
	b'=' => self.consume_byte(T![=]),
	b'!' => self.consume_byte(T![!]),
	b'\'' | b'"' => self.consume_string_literal(current),
	_ if self.current_kind == T![<] && is_tag_name_byte(current) => {
	// tag names must immediately follow a `<`

[dyc3]

Alternatively, another approach here would be to make each header it's own token. In the ungram, it could look something like this:

MdHeader1 = '#' MdParagraph?
MdHeader2 = '##' MdParagraph?
MdHeader3 = '###' MdParagraph?
etc.

And then what you currently have here would make more sense. I'm not sure if I would recommend going that route though.

[AugustinMauroy]

with your proposal of grammar change:

thread 'main' panicked at /.cargo/registry/src/index.crates.io-6f17d22bba15001f/quote-1.0.38/src/runtime.rs:490:9:
"##_token" is not a valid Ident
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace