Support multiple YAML items without knowing how many

[jvsteiner]

Thanks for the awesome plugin. I'm using this to create custom and flexible export commands with pandoc. Pandoc has some flags that can be repeated, and just generally can require a lot of arguments to get it to do what you want. I'd love to be able to create a YAML tag like:

extra-pandoc-arguments:
 - --template foo
 - --lua-filter=script.lua
 - --toc

and then be able to glob them all into a shell command like:

pandoc --from markdown --to pdf source.md -o destination.pdf {{yaml_value:additional-pandoc-arguments}}

And then have those arguments get expanded with a space separator. as it is, I would need to access them like {{yaml_value:additional-pandoc-arguments.0}} and {{yaml_value:additional-pandoc-arguments.1}} etc, but that means I have to know ahead of time how many there are - which I dont.

Another possibility would be to create a new thing like {{yaml_value:additional-pandoc-arguments:all}} or something for lists

[Taitava]

Hi! Thank you for your suggestion! 🙂

as it is, I would need to access them like {{yaml_value:additional-pandoc-arguments.0}} and {{yaml_value:additional-pandoc-arguments.1}} etc, but that means I have to know ahead of time how many there are - which I dont.

True, this is the current situation.

Another possibility would be to create a new thing like {{yaml_value:additional-pandoc-arguments:all}} or something for lists

I'd probably create a new variable for this: {{yaml_values:property-name:separator}}. Here the name of the variable would directly indicate that the aim is to get multiple values.

Further questions to solve

Key-value pairs

What if the list contains also keys in addition to values? I'll use your YAML list in a bit different format:

extra-pandoc-arguments:
 template: foo
 lua-filter: script.lua
 toc: ~ # ~ denotes a null value, which could be interpreted as an empty string with no separator between the key and the empty value, i.e. no = after "toc".

We could have a few arguments for the {{yaml_values}} variable:

{{yaml_values:property-name:item-separator:key-value-pattern:key-only-pattern:value-only-pattern}}

• property-name: E.g. extra-pandoc-arguments.
• item-separator: A glue for chaining different list items together. E.g. a space .
• key-value-pattern: Denotes how an item's key and value should be presented in the result.
  • E.g. --$item_key=$item_value could become --template=foo.
  • Placeholders: $item_key and $item_value. Maybe also $item_index0 and $item_index1 for telling the order number of the item in the list (0-indexed and 1-indexed respectively)? And {{item_count0}} and {{item_count1}} for producing values like "item 1 of 5" or "item 0 of 4"?
  • To use a literal $, use \$. To use a literal \, make it double: \\.
• key-only-pattern: Similar to key-value-pattern, but used only when a value is null. (But not when the value is an empty string?)
  • Used e.g. when encountering a YAML value like: toc: ~
  • Has the same placeholders, but no $item_value.
  • Makes it possible to leave out a separator after $item_key. E.g. --$item_key could become --toc.
• value-only-pattern: Similar to key-value-pattern, but used when a list doesn't contain keys, i.e. when the list's items start with - .
  • Has the same placeholders, but no $item_key.
  • Makes it possible to use a static word in place of a key. E.g. --static-default-key=$item_value.

A complete example for your case

pandoc --from markdown --to pdf source.md -o destination.pdf {{yaml_values:extra-pandoc-arguments: :--$item_key=$item_value:--$item_key}}

(Note that the last argument value-only-pattern is left out from the example, as this example does not need it.)

YAML:

extra-pandoc-arguments:
 template: foo
 lua-filter: script.lua
 toc: ~ # ~ denotes a null value, which could be interpreted as an empty string with no separator between the key and the empty value, i.e. no = after "toc".

Result:

pandoc --from markdown --to pdf source.md -o destination.pdf --template=foor --lua-filter=script.lua --toc

However, see Escaping special characters below for a problem that needs to be solved.

Nested lists

How to handle e.g. this?

extra-pandoc-arguments:
 template: foo
 lua-filter: script.lua
 my-nested-list:
   nested1: example1
   nested2:
     deeply-nested: example2

I think the first development step could be this: If any of the list items is another list, show an error message telling that nested lists are not currently supported.

Later, there could be some way to virtually "flatten" the list:

extra-pandoc-arguments:
 template: foo
 lua-filter: script.lua
 nested1: example1 (extra placeholders `$parent_key` = `my-nested-list`, `$parent_keys` = `my-nested-list` `$count_parents` = `1`)
 deeply-nested: example2 (extra placeholders `$parent_key` = `nested2`, `$parent_keys` = `my-nested-list.nested2` `$count_parents` = `2`)

Escaping special characters

All values coming from {{variables}} are escaped by default. This means, that for {{yaml_values:extra-pandoc-arguments: :--$item_key=$item_value:--$item_key}}, also the dashes - and equal signs = will be escaped by preceding them with a backslash \. So, your command would actually look like:

pandoc --from markdown --to pdf source.md -o destination.pdf \-\-template\=foor\ \-\-lua\-filter\=script\.lua\ \-\-toc

This would not be a desired outcome for this case.

In this particular case, you could precede the variable name with an exclamation mark ! to prevent escaping: {{!yaml_values:extra-pandoc-arguments: :--$item_key=$item_value:--$item_key}}.

But I'd like to think about a solution for doing partial escaping, i.e. being able to choose which parts should be escaped and which not:

• item-separator
• $item_key
• $item_value
• ... other $placeholders
• Static, non-placeholder parts of patterns. E.g. -- and = in --$item_key=$item_value.

A different approach: Maybe use method syntax for this?

I have some ideas to implement a method syntax for variables. Shortly, methods would be "postfixes" that denote additional processing for variables. The syntax might be {{variable}}->method((argument1:argument2)).

In this particular case, there could be a method named ->join_properties(())) (maybe needs a better name):

• Usage: {{yaml_values:extra-pandoc-arguments}}->join_properties((item-separator:key-value-pattern:key-only-pattern))
• Otherwise same logic for the arguments.
• How to denote which parts of the result should be escaped and which not?
• If there will be other new variables that may retrieve key-value paired data, the method could be used with those other variables, too.

There could also be another method: ->join_values((item-separator:value-only-pattern)) for keyless lists. (Maybe needs a better name).

[jvsteiner]

A few thoughts:

1. Nested Lists: In my opinion, as a lone user, I think supporting it is an overkill - I wouldn't bother with it if I were you. In my view, the purpose of this part of the plugin's functionality is to map yaml content against command line input. It's hard for me to imagine a well structured commandline utility that would require arguments that would be well represented in nested yaml syntax. Conversely I have trouble imagining elegant yaml data, where a flattened version of them would be useful, non-surprising input to commandline utility.
2. The escaping part is indeed the main question I think. Indeed, in my notes to PR adding a special case, where an object, which is an array of strings … #416 I noted this issue, and had to use the ! syntax to ensure spaces were not escaped. This is because, some arguments, especially key-value pairs may naturally have a space in them, ie. --$item_key $item_value and not all utilities support a separator based syntax, ie. --$item_key=$item_value like pandoc does.
3. Method syntax seems powerful, but perhaps overly complex for most use cases.

In my PR, I took a very simple, possibly too simple, approach. Just look at the referenced item, ie. {yaml_values:path.to.arguments} and, if it is exactly a list of strings, instead of throwing an error, interpret it as multiple space separated arguments.

I think this approach is pretty ok, and I have been using it myself, on my fork, however I do think it can be improved in the following ways:

1. It would be nicer if the separator was able to be explicitly defined to be something else. There are some utilities which accept comma separated values, for example.
2. The fact that I have to use the literal ! syntax to turn off escaping is a little ugly. It would be better if this case automatically bypassed the escaper. I didn't work out how to do that, since the interaction between the parser and the escaper are at quite different levels of the code, and there doesn't seem to be an obvious simple way for them to communicate. maybe when the separator is chosen, it could be "pre-escaped" somehow.

[Taitava]

A few thoughts:

1. Nested Lists:  In my opinion, as a lone user, I think supporting it is an overkill - I wouldn't bother with it if I were you.

True, nested lists support can be left out, and reconsidered if someone comes up with a use-case that would really need it. 👍

2. The escaping part is indeed the main question I think. 

True. I need to continue thinking about this.

3. Method syntax seems powerful, but perhaps overly complex for most use cases.

Yeah, the method syntax could increase complexity. I'm considering it mainly for two points, that perhaps are not so prominent at first:

• a) The result of a variable returning multiple values like {{yaml_values}} could be consumed in different ways. There could be e.g. the following methods:
  • {{yaml_values:property}}->count(())
  • {{yaml_values:property}}->toJSON(())
  • {{yaml_values:property}}->contains((keyword:if yes:if no))
  • Just to mention some examples. I'm not certain if I'll implement these.
• b) It could separate arguments into logical groups:
  • E.g. instead of {{!yaml_values:extra-pandoc-arguments: :--$item_key=$item_value:--$item_key}}, this {{!yaml_values:extra-pandoc-arguments}}->join_properties(( :--$item_key=$item_value:--$item_key))) tells that the first argument is used for data retrieval, and the others relate to formatting.
  • When a user sees another variable that supports ->join_properties(()), they'll know immediately that the method uses exactly the same parameters they have used with {{!yaml_values}}.

(One complex thing would be that different variables would support different sets of methods. E.g. {{title}} returns a simple text value, so it would not support join_properties(()), but could support ->length(()). However, the autocomplete menu can help with this by hiding or dimming methods that are incompatible with a variable the user has just typed.)

In my PR, I took a very simple, possibly too simple, approach. Just look at the referenced item, ie. {yaml_values:path.to.arguments} and, if it is exactly a list of strings, instead of throwing an error, interpret it as multiple space separated arguments.

I think this approach is pretty ok, and I have been using it myself, on my fork

The reason why I'd like to have two separate variables, {{yaml_value}} (the current one) and {{yaml_values}} (the planned one) is that then users can communicate what they are expecting to receive:

• If they type {{yaml_value}}, they are expecting one value, and will see an error message, if their YAML accidentally contains an unexpected list.
• And vice versa: If they use {{yaml_values}} and it encounters a scalar value instead of a list, an error message will be shown, and the unexpected value will be caught early.
• If needed, an optional parameter could be developed to {{yaml_values}} for telling it to exceptionally accept a non-list value, too.

however I do think it can be improved in the following ways:

1. ...

2. The fact that I have to use the literal ! syntax to turn off escaping is a little ugly. It would be better if this case automatically bypassed the escaper. I didn't work out how to do that, since the interaction between the parser and the escaper are at quite different levels of the code, and there doesn't seem to be an obvious simple way for them to communicate. maybe when the separator is chosen, it could be "pre-escaped" somehow.

Yes, the escaping happens outside of the Variable class hierarchy. My main idea for this has been that everything a variable produces is escaped, and variables can't disable it (for now), not even partially. The aim is to seek for clarity when all variables act the same in this matter.

That said, I do want to find ways to partially disable escaping here. Maybe ! in front of !$item_key and !$item_value could be a syntactical way to do it in key-value-pattern? E.g. --!$item_key=$item_value would use the key part as-is, without escaping, but still enable escaping for the value part. Then just need to find some syntactical way to prevent escaping the static letters -- and =.

I'd need to do some refactoring to the escaping system so that variables could communicate, what parts the user wants to escape and what not. Perhaps a Variable class could return a set of objects like:

return [{
    resultText: "--template="
    escape: false,
}, {
    resultText: "foo",
    escape: true,
}]

[jvsteiner]

I forgot to respond to your suggestion about key value represented arguments, ie:

extra-pandoc-arguments:
 template: foo
 lua-filter: script.lua
 toc: ~ # ~ denotes a null value, which could be interpreted as an empty string with no separator between the key and the empty value, i.e. no = after "toc".

Indeed, this would be nicer, but not, I think, strictly needed for functionality. Obsidian already auto-suggests similar yaml keys, making entry speedy in case it's not the first time I use it anywhere, and the beauty of the yaml is not something I personally care much about.

If it is something that you want to support, then your suggested example:

pandoc --from markdown --to pdf source.md -o destination.pdf {{yaml_values:extra-pandoc-arguments: :--$item_key=$item_value:--$item_key}}

seems as good of a way as any.

[Taitava]

The key-value format could allow deciding what parts to escape and what not. If values ever get longer content (=harder to spot problematic characters with naked eyes) then it's better that at least they are escaped, even if the keys are not.

[Taitava]

First step: {{yaml_values:property-name:separator}}

@jvsteiner I've thought about this more now and decided today that we can proceed with the simple version of {{yaml_values:property-name:separator}} variable.

• It will support list values, but not key-value pairs in the beginning.
• The complex features I suggested earlier can be added later, if needed.

I'll use your pull request #416 as a base for this and do some modifications. If you want, I can put your name and link in the AUTHORS.md file?

[jvsteiner]

sure that would be fine, glad I could help

[jvsteiner]

I also wondered if the keys could pose some strangeness with yaml syntax, ie. if you had an item:

params:
   --one-key:  one-value
   -another_key: another

I guess it technically is valid yaml, but it sure looks weird. And you'd have to include the dashes, because different utilities have different requirements.

[Taitava]

Probably adding quotes would work:

params:
   "--one-key":  one-value
   "-another_key": another

---

I have now created the new variable. I still need to test it and write documentation etc. But the work can be seen in branch #424-{{yaml_values}}.

I also added you to AUTHORS.md. Please let me know if there's anything you'd like to change.

[Taitava]

Hi @jvsteiner ! This is now released in SC 0.23.0.

Please let me know if you have any feedback. 🙂

[jvsteiner]

Well, I haven't tried to break it yet, but I installed the update, and my existing scripts that relied on my customized version still worked as per usual, so thats a good sign!

[jvsteiner]

Ok, so I first had an opportunity to use this modification the way it was intended, and I can confirm that it was nice and easy, and overall a very nice experience. I really like how your version of my mod is much more general and extensible, and while I haven't had a reason to use its more advanced capabilities, I think I probably will. The only small thing I can think of to report is that, when I first started configuring the shell command, I didn't realize that it would not work unless I added a separator. I think that adding a default separator--for example, an implied : to make a space separator is probably a really bad idea, so I understand why you didn't do that. Perhaps throw an error if the separator is missing or include a more explicit line in the documentation? I think now, the absence is silently ignored, and the parameter is simply omitted.

[Taitava]

Ah, true, the separator can easily be forgotten. Also, the autocomplete menu does not show the needed arguments.

The variable parser is currently designed not to recognize variables with too few arguments, so it doesn't process them at all. Perhaps this could be changed so it would show an error message when there are too few arguments, like you suggested.

[Taitava]

I created issues #435 and #436 for these.

[jvsteiner]

It wasn't a huge burden to figure out. This plugin is obviously a labor of love for you - the attention to detail shows.