Skip to content

Latest commit

 

History

History
302 lines (267 loc) · 10.4 KB

01-generating-yaml-files-with-nix.md

File metadata and controls

302 lines (267 loc) · 10.4 KB
Raw

Generating YAML files with Nix

I hate YAML. Instead of writing an essay on why I hate YAML, I can just link to noyaml.com. In my personal projects I will never use it, preferring either JSON, TOML or even plain old INI files depending on the use case. However the ship has sailed already, there are tons of projects everywhere that uses YAML: from most CI systems (GitHub Actions, CircleCI, Travis, et tu builds.sr.ht), to Kubernetes, or in almost every Rails application.

One way to avoid at least some issues with the language is to write YAML in another language. I will show my solution in one of my personal repositories, writing Nix to generate GitHub Actions configuration files. Bonus points for validating the result against the schema of GitHub Actions, so the famous "this is supposed to be string instead of a list of strings" is gone.

Let's start with the basics: YAML is supposed to be a superset of JSON. What that means is that a JSON file can be parsed by a YAML parser. And Nix itself generates JSON natively, after all, Nix can be imagined as "JSON with functions".

To make things easier, I will assume that you have the nix-commands and flakes enabled as experimental-features in your Nix configuration. If not, go here.

Using the nix eval command, we can generate a JSON expression from Nix by:

$ nix eval --expr '{ foo = "bar"; }' --json
{"foo":"bar"}

However, typing long excerpts of Nix code inside the console would be impractical. We can write the following code inside a foo.nix file instead:

{
  foo = "bar";
}

And:

$ nix eval --file foo.nix --json
{"foo":"bar"}

While you can use a JSON output as an input for YAML parsers, it is probably not the best idea. Sadly (or maybe not), Nix has no native functionality to export data to YAML. However, since we are using Nix, it is trivial to use nixpkgs to use some program to convert from JSON to YAML.

To start, let's create a new directory, move our foo.nix file to it, create a new flake.nix file and put the following contents:

{
  description = "Generate YAML files with Nix";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = { nixpkgs, ... }:
    {
      packages.x86_64-linux =
        let
          inherit (nixpkgs) lib;
          pkgs = import nixpkgs { system = "x86_64-linux"; };
        in
        {
          toYAML = pkgs.runCommand "toYAML" {
            buildInputs = with pkgs; [ yj ];
            json = builtins.toJSON (import ./foo.nix);
            passAsFile = [ "json" ]; # will be available as `$jsonPath`
          } ''
            mkdir -p $out
            yj -jy < "$jsonPath" > $out/go.yaml
          '';
        };
    };
}

We are loading the ./foo.nix as a Nix file, converting it to JSON with builtins.toJSON function, and finally, using pkgs.runCommand and its passAsFile option to load the contents of the JSON file into yj, that converts between serialisation formats (-jy flag means "JSON to YAML"). The reason I choose yj is mostly because it is a single binary Go program, but you can use whatever you prefer.

By the way, there is a lib.generators.toYAML inside nixpkgs.lib, but as of the day of this post it only calls lib.strings.toJSON (that in turn, calls builtins.toJSON). So it doesn't really help here. Another option would be pkgs.formats.yaml.generate, that converts between formats, but it calls remarshal (in Python), so not my favorite choice.

If we run the following commands, we can see the result:

$ nix build .#packages.x86_64-linux.toYAML
$ cat result/foo.yaml
foo: bar

That is the basic idea. To have a more realistic example, let's convert the go.yml, that builds this blog, to Nix:

{
  name = "Go";
  on.push.branches = [ "main" ];

  jobs = {
    build = {
      runs-on = "ubuntu-latest";
      permissions.contents = "write";
      steps = [
        { uses = "actions/checkout@v4"; }
        {
          name = "Set up Go";
          uses = "actions/checkout@v4";
          "with".go-version = "1.21";
        }
        {
          name = "Update";
          run = "make";
        }
        {
          name = "Publish";
          run = "make publish";
          env.MATAROA_TOKEN = ''''${{ secrets.MATAROA_TOKEN }}'';
        }
        {
          name = "Commit";
          uses = "stefanzweifel/git-auto-commit-action@v5";
          "with".commit_message = "README/rss:update";
        }
      ];
    };
  };
}

Some interesting things to highlight: with is a reserved word in Nix, so we need to quote it. Not a problem, but something to be aware. And the template string in GitHub Actions uses the same ${} that Nix uses, so we need to escape.

And after running the following commands:

$ nix build .#packages.x86_64-linux.toYAML
$ cat result/go.yaml
jobs:
  build:
    permissions:
      contents: write
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Go
        uses: actions/checkout@v4
        with:
          go-version: "1.21"
      - name: Update
        run: make
      - env:
          MATAROA_TOKEN: ${{ secrets.MATAROA_TOKEN }}
        name: Publish
        run: make publish
      - name: Commit
        uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: README/rss:update
name: Go
"on":
  push:
    branches:
      - main

Yes, the keys are not in the same order as we defined, since Nix, like most programming languages (with the exception of Python), do not guarantee the insertion order in maps/dicts/attrsets/whatever. But I really hope whatever is consuming your YAML is not relying in the order the keys are defined (this would be more cursed than YAML already is).

So that is basically it. For the bonus points that I talked at the start of the post, we can modify pkgs.runCommand to run some kind of validator. I use action-validator, one that I particularly packaged in nixpkgs to use in those cases. But you could use e.g.: a validator of Kubernetes YAML. Or a generic YAML lint like this one. The possibilities are endless.

Let's modify our flake.nix to add the validation:

{
  # ...
  outputs = { nixpkgs, ... }:
    {
      packages.x86_64-linux =
        let
          inherit (nixpkgs) lib;
          pkgs = import nixpkgs { system = "x86_64-linux"; };
        in
        {
          toYAML = pkgs.runCommand "toYAML" {
            buildInputs = with pkgs; [ action-validator yj ];
            json = builtins.toJSON (import ./go.nix);
            passAsFile = [ "json" ];
          } ''
            mkdir -p $out
            yj -jy < "$jsonPath" > $out/go.yaml
            action-validator -v $out/go.yaml
          '';
        };
    };
}

And let's add an error in our go.nix file:

diff --git a/go.nix b/go.nix
index 25e0596..8c00033 100644
--- a/go.nix
+++ b/go.nix
@@ -5,7 +5,7 @@
   jobs = {
     build = {
       runs-on = "ubuntu-latest";
-      permissions.contents = "write";
+      permissions.contents = [ "write" ];
       steps = [
         { uses = "actions/checkout@v4"; }
         {

Finally, let's try to build our YAML file again:

$ nix build .#packages.x86_64-linux.toYAML
error: builder for '/nix/store/j8wr6j1pvyf986sf74hqw8k31lvlzac5-toYAML.drv' failed with exit code 1;
       last 25 log lines:
       >                                 "Additional property 'runs-on' is not allowed",
       >                             ),
       >                             path: "/jobs/build",
       >                             title: "Property conditions are not met",
       >                         },
       >                         Properties {
       >                             code: "properties",
       >                             detail: Some(
       >                                 "Additional property 'steps' is not allowed",
       >                             ),
       >                             path: "/jobs/build",
       >                             title: "Property conditions are not met",
       >                         },
       >                         Required {
       >                             code: "required",
       >                             detail: None,
       >                             path: "/jobs/build/uses",
       >                             title: "This property is required",
       >                         },
       >                     ],
       >                 },
       >             ],
       >         },
       >     ],
       > }
       For full logs, run 'nix log /nix/store/j8wr6j1pvyf986sf74hqw8k31lvlzac5-toYAML.drv'.

Yes, the output of action-validator is awfully verbose, but it is still better than making "8 commits/push in one hour".

If you are interested in how a more advantage usage of this technique is, including usage of functions and constants to share common steps between different actions, please take a look at the actions (permalink) in my nix-config repository.