Extract code blocks by name

[panglesd]

This is an early prototype (on top of #1325) of an extract-code command, which extracts named code blocks into a file. Supersedes the venerable #303.

This demonstrates that the location is right:

test.mld:

{0 hello}

  {[

  let f x = x +. 2.

]}


  {[ let x = f 2

]}

$ dune exec -- odoc extract-code test.mld > ocaml.ml
$ ocaml ocaml.ml
File "test.mld", line 10, characters 16-17:
Error: This expression has type int but an expression was expected of type
         float
  Hint: Did you mean 2.?

[panglesd]

So, this PR currently implements the extract-code command.

This commands takes a mld file, and --line-directives flag, and a list of --name <name> argument, and an option -o/--output.

I then outputs (to the -o or stdout) the concatenation of "matching" code blocks, with line directives if the flag is set.

The rules for matching code blocks is:

• If no --name are provided, all ocaml code block matches. An ocaml code block is a code block without specified language or with "ocaml" specified as language.
• If some --name are provided, a code block match if it has a name that was passed in --name. A code block has a name if its "tags" (the string after the language) contain name=<name> (separated by spaces).

See the test for some examples.

Now, I wonder two things:

• Should we uniformize how tags are parsed? In this PR I use space as separators to look for name=<...>, but MDX seems to separate "labels" with ,: see this function called on odoc tags.
• What should we take as input in addition to mld files? Mli? cmti/cmi/cmt? (I don't think .odoc files should be taken as input as they move things around.)

[dbuenzli]

I had a quick look and the whole thing looks rather sound to me. Thank you very much @panglesd this is going to save me tons of time.

One thing I didn't find by having a quick look at the manual was the failure modes.

What happens if given --name N0 --name N1 … and a .mld file, one Ni did not match ? In my opinion one of these two things should happen:

1. Warning on stderr, output what matched (empty if none did), exit with zero.
2. Error on stderr, output what matched (empty if none did), exit with non-zero.

Or perhaps better 1. should happen by default and 2. on --warn-error.

• Should we uniformize how tags are parsed? In this PR I use space as separators to look for name=<...>, but MDX seems to separate "labels" with ,: see this function called on odoc tags.

I did not understand that question. Can you perhaps provide examples or point to documentation ?

• What should we take as input in addition to mld files? Mli? cmti/cmi/cmt? (I don't think .odoc files should be taken as input as they move things around.)

I think at least .cmti files would be nice. They are installed so you could instruct people to e.g. odoc --name="sample.ml $(opam var mylib:lib)/m.cmti > sample.ml to get a sample code out.

Regarding features, one thing I would add is a way to list the names found in a given file, one per line. This could be either a separate command or as a flag that triggers an other operating mode (e.g. --list). But this can be added later (it entails a bit of thinking on what you want to list, unique names, or unique combination of sorted names applied to blocks)

[panglesd]

Should we uniformize how tags are parsed?

I did not understand that question. Can you perhaps provide examples or point to documentation ?

Sure!

I realize it is not part of the documentation, but in addition to "language", code blocks have a part to store "tags":

{@language tag1 tag2[
content
]}

Internally, odoc-parser and odoc store code blocks in a record that look like that (omitting the location):

type codeblock = {
  content: string;
  language: string;
  tags: string
}

In the example above, the tags field would be "tag1 tag2". There is no specification from the parser on the format of this string.

Now, several tools might start to use this string:

• mdx uses it to get labels. Labels are parameters for the execution of the code block, and have a specific syntax. Labels are separated from each other using a ,:

  {@ocaml set-FOO=bar,set-BAR=foo[
  content
  ]}
  

  (example adapted from here.)
• extract-code uses tags to name code blocks. But in the implementation, unaware of mdx, I cut by space (and not ,) before looking for a name=<name> (certainly, influenced by the syntax inside pandoc's attributes)

  {@ocaml name=minimal name=complete[
  content
  ]}
  

  (This has two names to be included in both the "minimal" and the "complete" extraction)

So, it makes it hard to comply to both extract-code and mdx syntax for tags...

A possibility would be to say that the tags field of code blocks is string list, so that odoc handles the separation of tags, and each tool can ignore the tags that do not comply to their syntax.

[dbuenzli]

The mdx docs I'm pointed to looks like a hack of the style, oh we are just going to use String.split_on_char ',' and String.split_on_char '=' to get our tags and variable bindings.

Please just don't. These things end up being infuriating at some point.

Cater for the fact that the tags/bindings may want to contain the separators, including and especially spaces.

I think it's best to ignore whatever mdx has been doing and devise a proper grammar for tags and key values pairs.

In general the basics of sexps are always sound to use. If I assume we want both "tags" (atom) and "bindings" (atom=atom). Define a notion of atom as being either an unquoted atom or quoted atom. A tag is an atom and binding two atoms separated by = without intervening whitespace (for simplicity). Tags and bindings are separated by whitespace.

Very quickly

type codeblock = {
  content: string;
  language: string;
  info_string: [ `Tag of string | `Binding of string * string ] list
}

With info_string parsed by:

ws = *(%x0020 / %x0009)  # Not sure if newlines are allowed here 

unquoted-char = … # Char.Ascii.is_graphic without `=` or `"`
unquoted-atom = 1*unquoted-char

escape = %005C %x0022 # More can be added, e.g. if we want newlines
quoted-char = … # Anything except `"` (U+0022)
quoted-atom = %x0022 (quoted-char / escape) %x0022 

atom = unquoted-atom / quoted-atom

tag = atom 
binding = atom %003D atom 
info-string = *(ws (tag / binding)) 

[jonludlam]

In general the basics of sexps are always sound to use. If I assume we want both "tags" (atom) and "bindings" (atom=atom). Define a notion of atom as being either an unquoted atom or quoted atom. A tag is an atom and binding two atoms separated by = without intervening whitespace (for simplicity). Tags and bindings are separated by whitespace.

This sounds good to me. Mdx is for 'live' files - those that can be updated - and so we can update the syntax of the tags. They don't affect rendering so we've got no worries about existing mld files in released packages. We might want to add a warning into mdx if it notices the comma-separated syntax maybe? We can probably come up with some heuristic.

[dbuenzli]

Thanks @panglesd! Personally I'm fine with that being >= 4.10. I don't think it's worth lingering for too long on past OCaml versions.

[dbuenzli]

Ooops. Pushed wrong button :–)

[jonludlam]

Personally I don't think we need to support extract-code on older versions of OCaml. It's way less important than rendering the docs for older versions.

Also I'm not particularly convinced we need to go out of our way to try to support the syntax mdx currently uses. version<=4.14 and the like seem a little dodgy (version of what?) and we could have a more generic syntax, e.g. constraints="(ocaml <= 4.14)" or similar

[panglesd]

I think the location of quoted tags/bindings should include the quotes. Otherwise it's hard for programs that want to change the tags by modifying what is at this location...

[jonludlam]

Yes, that's fair.

[panglesd]

Thanks, the ast simplification is welcome, and the escaping is better.

The only important comment is that I think locations should include the quotes!

[jonludlam]

Thanks @panglesd