Serialize mdast to markdown

[bnchi]

fixes #64

TODO handlers :

• paragraph
• text
• strong
• emphasis
• heading
• html
• break
• thematic-break
• code
• blockquote
• • Complete the tests for blockquote
• image
• inline-code
• link
• list
• list-item
• • Add the tests for list item
• root
• definition
• image-reference
• link-reference

[bnchi]

While working on this, I noticed that the JS implementation relies heavily on regular expressions in some places. I believe we can rewrite some parts of the code in Rust instead of depending on the regex crate. Although the regex crate guarantees linear time complexity, we can eliminate the need for the regex engine in places where simpler Rust code would work. Maybe we can benchmark first but this won't happen until I move over most of the tests into Rust.

what do you think @wooorm ?

One more question what do you think make more sense here for the maintainability of this utility :

1. To create a new workspace for mdast to markdown ?
2. To create a separate module to group most of the related code for this util ?

[wooorm]

While the core of markdown-rs is no-std + alloc, that’s for the use case where you only really do markdown -> html, straight.
Here we are working with ASTs. I think it’s fine to depend on regexes there.
And, both JS and Rust will need to be maintained and kept in sync. So, fine to improve things if it’s fast, just, that has to be backported.
I’d recommend going with regexes first. To focus on porting the code. Instead of changing the code. That becomes confusing. Hard to manage.

As for workspace vs module, I have to particular preference currently. You?

[bnchi]

Since we're working with an AST and this is a utility a workspace sounds better to me this will help to isolate the tests when we quickly run cargo test and version this util by it's own too.

It's not that hard to change that decision later on, I updated the PR to use a workspace instead and kept the no_std constrain.

[bnchi]

I completed copying over the tests so now all of the tests are a match, I also did a quick review everything looks fine to me, defiantly a few performance improvements and refactors can be done in a separate PR but now since the tests are moved over it's easy to make changes without worrying about something breaking.

I've also added comments as proposed for some data structures.

[augustkline]

Just a heads-up, serializing a yaml node to markdown gives an error. I don't know if that was intentional, but if not you might be missing a yaml handler

[augustkline]

Thank you so much for this PR btw! Really needed this feature

[bnchi]

wooorm knows more about this than me, but I think Yaml and Toml are not part of the CommonMark specs they're part of the frontmatter extension this PR ports over the JS implementation here mdast-util-to-markdown

In order for Yaml and Toml to work we need to add support for frontmatter serialization I believe the JS version is here mdast-util-frontmatter

[augustkline]

Ah gotcha! Thanks for the info & links!

[ChristianMurphy]

Some extensions have built in parsers that can be turned on with an option.
Including frontmatter! https://github.com/wooorm/markdown-rs?tab=readme-ov-file#extensions

It could make sense to support serialization for those tools.
But this PR is pretty sizable already.
I'd lean towards having extensions in follow up PRs.

[ChristianMurphy]

Also additional testing, using a fuzzer to to validate that content can round trip.

In other words markdown can be parsed to a tree, be serialized again, and parsed again should have the same tree structure (though positions and exact characters may change).

That technique found a lot of quirks in the original micromark/remark parser and serialized.
And could be valuable to repeat here.
But is likely outside the scope of this PR.

References:
About fuzzing: https://rust-fuzz.github.io/book/
Current parser fuzz test: https://github.com/wooorm/markdown-rs/tree/main/fuzz

[wooorm]

But this PR is pretty sizable already.
I'd lean towards having extensions in follow up PRs.

Right yes, that indeed. Somewhere above @bnchi and I talked about this too and concluded the same.

I believe @bnchi has used up a lot of the time they have to work on this. So if @augustkline or others are up for follow up PRs, I’d love that!

[wooorm]

Only thing perhaps is some docs in this readme here?
And, how should this new code be used? Should it be published? As alpha?

[Enoumy]

Thank you so much for writing this PR!

[bnchi]

Only thing perhaps is some docs in this readme here?

Yeah I can add instructions on how to use this code in a readme

Should it be published? As alpha?

Alpha sounds alright, what do you think ?

[wooorm]

Sounds good to me! 👍