Bosh Templating Language RFC

[jpalermo]

Proposal for adding a new templating language for bosh releases

Preview

[rkoster]

Would it be possible to add some examples of what the new grammar would look like for at least the basic usecases (if_p, p('something').each do |prop|, multi line certs (with respecting indentation, some to_json/yaml, etc)?

[ameowlia]

ARP does A LOT of validation in the template. I always thought this was best practice because then the user gets an error far earlier in the deploy process.

Here are some examples of the validations we have:

• provided IPs are valid IPs
• integer ranges (must be larger than X or smaller than X)
• do domains end with a "."? If not, add one.
• if you set X, Y, Z property, then fail if a cert is not provided. Else it is fine.
• a provided string must be either "a" or "b"
• string length must be more than X

[ameowlia]

added more specific examples with links in this comment: #1004 (comment)

[ameowlia]

ARP releases do a lot of "check to see if this property is provided, if not then use the property from a bosh link". Is this type of logic someone that would be provided?

But really, I second what @rkoster said above asking for examples of what you want to include in your templating language, that way I can be more specific in my worries 😅

[ameowlia]

I come bearing examples of template logic!

• Most of our jobs use a function like this one to do the "check to see if this property is provided, if not then use the property from a bosh link" logic. link
• IP validation. We do this in many jobs.
• basic int logic greater than/less than. link
• string must be one of a set. link
• type checking, it must be an array. link
• if the property is provided, and once we strip whitespace, then it must be longer than 50 chars. link
• requiring certain properties, but only when other properties are set. link
• turn bosh machine info into json. ink
• if a certain thing is a certain number, then do all these other things. link
• cert validation link
• validating cert SANs link
• at least one CA has a certain SAN link
• and those examples are all from gorouter!
• here we use a property as a deprecation warning to alert users that nats (non-tls) is going away soon. link
• pulling out db params from a string link
• turn array into a comma separated list link

After going through all these files I see that nearly every one of ours has this helper method defined. It would be great if you owned these types of helper functions :)

  def parse_ip (ip, var_name)
    unless ip.empty?
      ip = ip.split(":")[0]
    begin
      parsed = IPAddr.new ip
    rescue  => e
      raise "Invalid #{var_name} '#{ip}': #{e}"
    end
    end
  end

[loewenstein-sap]

What is the percentage of structured vs. unstructured output being rendered? Background is, that https://cue-lang.org would be a great option to replace the release specs and include validation (like X : Int >3 but < 7) and for structured rendering output it could also directly be used for that. If we have a huge amount of unstructured formats, it might be a challenge. I could still see removing the need to validate from the templating language by using CUE.

Edit: I guess this would be relevant, when it comes to unstructured data and templating / interpolation: https://cuelang.org/docs/tour/expressions/interpolation/

[maxmoehl]

If we want to push logic out of the templates why not drop the central template rendering altogether and allow release authors to consume the YAML directly? The provided YAML would be placed in the config directory of the job and the release decides how to proceed with it.

This would also work around the core issue here which I think is dependency management: the release authors are implementing against a templating system which is controlled by the user of the release (e.g. the operator) and not the release author. By explicitly making the template rendering a part of the release itself we solve that issue. Bumping the template renderer becomes the responsibility of the release author which can properly test and resolve any issues with version bumps. And the operator no longer risks breaking a bunch of releases just because the ruby version or some required dependency is bumped in their environment.

If we still want BOSH to be able to distinguish between the templating and pre-start phase we could add an additional lifecycle step. If the release author provides a script for that step the ERB rendering is automatically disabled. And for release which don't want to switch a Ruby script which mimics the behaviour of the old template rendering could be provided / injected to speed up the transition.

Edit: as many jobs are running code written and maintained by the community this would allow to skip the intermediate config representation that often exists without any template rendering at all by incorporating the logic into the job itself. I guess this is technically already possible today, but the ERB templates might somewhat steer people to having an additional step?

Edit: this does not address the concern about failing before the deploy itself starts happening.