Identify the best solutions for CI/CD and builds

[girodav]

(This discussion serves as a placeholder to consolidate any discussion around what to use as CI/CD and build tool for this repo)

CI

Requirements

• Ease of use
• Needs to be compatible with the build tool of choice
• Whatever runs on the CI needs to be reproducible locally / in a dev env. No need to do "trial and error" on PRs to figure out if the CI would pass

Possible alternatives

• BuildKite
• Github Actions
• ?

Build tool

Requirements

• Ease of use
• Should be multi-platform (Windows, macos, linux...)
• Needs to work with the CI of choice

Alternatives

• Magefiles
• Taskfile
• Makefiles
• ?

[girodav]

I am doing some tests with Taskfile on https://github.com/girodav/inputrunner/tree/taskfile , if anyone wants to have a look.
From my initial tests, I find it much easier to use than Magefiles. With 0 knowledge of it, I managed to replicate the current magefile in few mins. I also personally find it clearer to read than Magefiles.

Example:

version: '3'

tasks:
  go:mod:download:
    cmds:
      - go mod download
  go:build:
    cmds:
      - go build -v .
  go:test:
    cmds:
      - go test ./... -coverprofile=coverage.out
  go:tool:coverage:
    cmds:
      - go tool cover -html=coverage.out -o coverage.html
  staticcheck:
    cmds:
      - go install honnef.co/go/tools/cmd/staticcheck@latest
      - staticcheck -f=stylish ./...
  gosec:
    cmds:
      - go install github.com/securego/gosec/v2/cmd/gosec@latest
      - gosec ./...

  build:
    desc: Build the project
    deps:
      - task: go:build
      - task: go:mod:download
  test:
    desc: Run tests and produce a coverage report
    deps:
      - task: go:tool:coverage
      - task: go:test
  check:
    desc: Run a static analysis check and look for any security vulnerabilities
    deps: [gosec, staticcheck]

It can be integrated in Github actions very easily with https://github.com/arduino/setup-task

[gsantoro]

just for reference, I have been using Taskfile for the past 6 months. I have started a repo to share some of those automation scripts with a bigger audience https://github.com/elastic/obs-infraobs-team.

Also I have been getting some feedback from the rest of my team and from @endorama .

[tommyers-elastic]

my hot take -

• mage is used in most elastic go repos today, it's well understood within the company so it's a good idea to use it
• HALP NO MOAR YAML
• to keep builds reproducible we enforce some simple guidelines, e.g. CI jobs must run a single build target. if we have configuration (e.g. for integration tests) that differs between local/CI builds, then we build in local workflows that handle it.
• avoid jenkins - it seems like the company is favouring buildkite for new CD workflows
• do we really need to support windows??

more personal notes on mage - whilst it's more verbose, i think the flexibility is great. plus it's one less syntax to keep in your head.

[girodav]

mage is used in most elastic go repos today, it's well understood within the company so it's a good idea to use it

This is a valid point, but we can argue if it's that easy to use compared to Taskfile or other solutions, particularly for newcomers.

HALP NO MOAR YAML
more personal notes on mage - whilst it's more verbose, i think the flexibility is great.
plus it's one less syntax to keep in your head.

Another valid point. Not a big fan of yaml either, but are we sure that writing go code to build go is the best? I do not particularly find it simpler or better to read that the yaml example above.

Regarding the flexibility, could you elaborate more? What do you think Mage can do that other solutions cannot? I am not an expert of Mage, so any extra datapoint can be useful :)

avoid jenkins - it seems like the company is favouring buildkite for new CD workflows

Agreed, Let's remove it from the alternatives

do we really need to support windows??

I agree that it is not very likely to happen, but since this repo is here to stay, what happens if a new dev chooses a windows laptop? As far as I know there is no strict requirement about what OS we can use.

[gsantoro]

mage is used in most elastic go repos today, it's well understood within the company so it's a good idea to use it

that's true but everyone I have ever asked an opinion about mage they told me they hate it as much or more than jenkins.

[tommyers-elastic]

@gsantoro what are the complaints about mage that you've heard?

[gsantoro]

Off top of my head:

• since most of the time Mage is just used to build and run a bash command, it would be great to log which command was run in case you want to replicate an issue without mage. This is currently not the case as far as I know
• Currently the Mage code in the beats repo is quite complex and hard to follow. Since we don't have tests, making a quick change is quite complicated. Taskfiles have no tests either but there is a lot less code to debug and the list of commands run in bash is always logged
• Debugging code in Mage is not as straightforward as debugging golang code. I have managed to do that in the past but it wasn't without struggle. I have instructions in this open public issue at Document debugging mage files in VSCode magefile/mage#280 (comment)
• Finally in multiple occasions I thought about migrating from Taskfile to Mage but I haven't found yet lots of things I can't do in a simpler way in Taskfile. If it was me and Taskfile wasn't an option, I will avoid Mage and implement everything in plain old Golang code instead mostly for the extra complexity of tests and debugging.

[dmathieu]

My own hot take: something that can run out of the box on any machine without the need to setup something else is better.
For example, I've seen mentions of devenv in observability-dev, which seems great. But relies on Nix, which is an awesome, but very intrusive software.

[ruflin]

This is one of the discussions very we can invest a lot of time and not come to a conclusion. I expect the initial tasks we need to build are pretty simple. Key for me is, that what we can do in CI must be reproducible locally. Lets pick one and if we hit limits, lets move to another one. If no decision can be reached quickly, happy to make one.

[tommyers-elastic]

agree let's not burn time on this. i would rather stick with mage+GH actions for now for our simple build/test/lint workflows. if we hit issues with like @gsantoro mentioned we will have a better case for investing time in an alternative.

[girodav]

Yep, last thing we need is to debate on this topic in a similar way as "what's the best programming language?". I also agree that for now we can use more or less whatever solution, since our needs are basic (i.e build PRs, run tests...). Nevertheless, I think it's important to not end up with "one-way doors" and have these discussions sooner rather than later. It's true that for now, and short term, we can switch from a solution to another more or less painlessly. However, this won't hold forever.

[endorama]

TL;DR: I think that choosing mage is a sort of one way door. I would approach it with the simplest solution first and grow in complexity if the need arise. As of today GitHub Actions + Task seems to me the best trade-offs.

I agree with Davide that we need to have this conversation, and sooner is better than later so below you find a bit of reasoning behind my opinions.

The too long

Consider all that follow as a reasoning coming from a lot of research and observations that should in no way be a blocker. There is no "best" here, but I think there is "better in the long run".

I consider this discussion as composed by these separate problems:

• developer toolchain; answers to "how do I install all the needed requirements?"
• reproducibility across environments; answers to "how to I run the same tools that others are running?"
• command discovery; answers to "which is the command to perform this task?" and make sure everyone is using the same command
• toubleshooting; answers to "this is not working for me, who may I get help from?"

All of them are intertwined, but not so much that can't be considered each one by itself.
Nicolas also made a very compelling point: these discussion can go on for ages, and we should not let this happen. A key point is that all solutions have trade-offs and there is no real one-size-fits-all: we should find what works enough for us as a trade-off between complexity/maintenance.

To summarise what I think is the best way and tools to approach this:

1. Developer toolchain
   Being able to install tools across all 3 OSes reliably is a challenge that is hard to solve, the trade off cost/benefit is complex, as is not a frequent occurrence.
   I think this should be not part of this current discussion. Tools are expected to be available, and their requirement is documented so each dev can do this on their own. There is room for future improvements.
2. Reproducibility across environments
   There are different tools to tackle this, depending on the depth we want to achieve. The most basic way is to provide version information within the git repo, that tools like goenv or asdf can then leverage. Being files and part of the repo is easy to leverage them in other contexts, like CI (es a GitHub Actions job I wrote but you can also simply use cat | grep)
   The next step here would be Docker probably, but I would first check out if BuildKite has some opinions here (i.e. Docker would not make that much sense with GitHub Actions for example).
3. Command discovery
   This is where I think tools like mage and make do not really shine. They can be used for complex things, and with increased complexity the cognitive load to use them increases as well. They also don't or partially support out of the box features like "dry run" mode or "list all the tasks". Task does, and in general is more oriented to being a task runner only.
   The more complex the tool, the more cognitive demanding its maintenance, and as our focus is on the product having heavy maintenance toolchain is a big risk in the long run in my opinion.
4. Troubleshooting
   Linter failed on CI for someone else, but I can't reproduce. This kind of issues can be addressed only by making all the other step explicit enough that helping another dev does not require deep or specific knowledge of the toolchain (like "you need to ask Jane for this" or deep dive into CI worker setup or "how do I debug this").
   This is the area where I think the Unix philosophy of small single purpose tools shines and make our life easier. Instead of trying to have a single tool to do everything, make it small and precise, but with a high degree of reliability.
   For any setup we choose there will always be something that do not work, but are devs able to unblock themselves or will they get stuck and need help from someone else? Will they be able to get help by anyone or just very specific (busy) people?

In my experience the key element to aid in this choice is dependencies: when the tool you use allow to express complex dependencies across tasks the maintenance greatly suffers.

The limitations imposed by tools like Task make them very effective in the long run: the cost for understanding what is going on is low even for people not familiar with them or with the task, there is no room for complex dependencies, making errors is more difficult, lock in is small, is possible to delegate to other tools when needed and replacing them has low overhead. These tools tends also to be more newcomer friendly.