From 33d6cf873ff6072abf098d83ca99255c12c00408 Mon Sep 17 00:00:00 2001 From: Priti Desai Date: Thu, 5 Oct 2023 21:51:37 -0700 Subject: [PATCH] TEP-0147 Dynamic Matrix Proposing a new section for matrix which allows to specify the explicit list of combinations dyanmically Signed-off-by: Priti Desai --- teps/0147-dynamic-matrix.md | 379 ++++++++++++++++++++++++++++++++++++ teps/README.md | 1 + 2 files changed, 380 insertions(+) create mode 100644 teps/0147-dynamic-matrix.md diff --git a/teps/0147-dynamic-matrix.md b/teps/0147-dynamic-matrix.md new file mode 100644 index 000000000..2249a86b9 --- /dev/null +++ b/teps/0147-dynamic-matrix.md @@ -0,0 +1,379 @@ +--- +status: implementable +title: Dynamic Matrix +creation-date: '2023-10-05' +last-updated: '2023-10-18' +authors: [ + '@pritidesai' +] +see-also: + - TEP-0118 + - TEP-0090 +--- + +# TEP-0147: Dynamic Matrix + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) + - [Use Cases](#use-cases) +- [Proposal](#proposal) + - [`matrix.strategy` initialized by the `pipelineRun`](#matrixstrategy-initialized-by-the-pipelinerun) + - [`matrix.strategy` specified by a task result](#matrixstrategy-specified-by-a-task-result) + - [GitHub Actions](#github-actions) +- [Alternative](#alternatives) +- [Future Work](#future-work) +- [References](#references) + + +## Summary + +Matrix feature of Tekton Pipelines allows pipeline authors to specify multiple concurrent execution of the same task. +The same task can be executed in parallel based on the number of input parameters or a combination of parameters. + +Matrix supports specifying [implicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#generating-combinations) +and [explicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#explicit-combinations) combinations of +parameters. + +- With implicit combinations, a `pipelineTask` is defined with a list of parameters and one or more values for each +parameter. Tekton controller generates an exhaustive list of combinations based on the specified parameters. Based on +the generated list of combinations, a `pipelineTask` is fanned out and executed with each combination. + +```yaml + matrix: + params: + - name: platform + value: $(params.platforms[*]) + - name: browser + value: $(params.browsers[*]) + ... +``` + +- With explicit combinations, a `pipelineTask` is defined with a list of combinations and Tekton controller fans out the +task based on the number of combinations. Each running instance receives unique combination of input parameters. + +```yaml + matrix: + include: + - params: + - name: platform + value: "linux" + - name: browser + value: "chrome" + - params: + - name: platform + value: "mac" + - name: browser + value: "safari" + ... +``` + +Both implicit and explicit combinations fit well with many use cases. At the same time, creating a sharable pipeline with +existing `matrix` syntax is not possible when a list of combinations are specified at the runtime through `pipelineRun`. + +```yaml +apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: cd-pipeline +spec: + workspaces: + - shared-workspace + params: + - name: image-1 + - name: image-2 + - name: image-3 + - name: dockerfile-1 + - name: dockerfile-2 + - name: dockerfile-3 + tasks: + - name: build + taskRef: + name: kaniko + workspaces: + - name: source + workspace: shared-workspace + matrix: + include: + - name: build-1 + params: + - name: IMAGE + value: $(params.image-1) + - name: DOCKERFILE + value: $(params.dockerfile-1) + - name: build-2 + params: + - name: IMAGE + value: $(params.image-2) + - name: DOCKERFILE + value: $(params.dockerfile-2) + - name: build-3 + params: + - name: IMAGE + value: $(params.image-3) + - name: DOCKERFILE + value: $(params.dockerfile-3) +--- + +apiVersion: tekton.dev/v1beta1 +kind: PipelineRun +metadata: + generateName: cd-pipelinerun- +spec: + pipelineRef: + name: cd-pipeline + params: + - name: image-1 + value: "image-1" + - name: dockerfile-1 + value: "path/to/1/Dockerfile" + - name: image-2 + value: "image-2" + - name: dockerfile-2 + value: "path/to/2/Dockerfile" + - name: image-3 + value: "image-3" + - name: dockerfile-3 + value: "path/to/3/Dockerfile" +``` + +## Motivation + +It is a common practice to execute a `pipeline` either by creating a `PipelineRun` object or through `TriggerTemplate`. +Generally, a list of `params` are specified as part of the `PipelineRun` which might be initialized statically or +through a [trigger template params](https://github.com/tektoncd/triggers/blob/main/docs/triggertemplates.md#structure-of-a-triggertemplate). +Now, running a shared `pipeline` with `matrix` where the values for each instance of fanned out task is dynamically specified +at the runtime is not possible with the existing `matrix` syntax. In this proposal, we would like to extend `matrix` +syntax to provide an option to dynamically specify a list of explicit combinations + +### Goals + +- Be able to author a sharable pipeline with `matrix` such that a task can fan out based on the explicit combinations +which are specified dynamically. + +### Non-Goals + +- Matrix explicit combination mechanism only supports a `param` of type `string` and not adding support for any +other type. + +### Use Cases + +Let's revisit the example of build pipeline from [TEP-0118](0118-matrix-with-explicit-combinations-of-parameters.md#define-explicit-combinations-in-the-matrix). +A pipeline `pipeline-to-build-images-from-a-single-repo` has three explicit combinations defined in the pipeline such +that `kaniko` can fan out and create three separate `taskRun`s. The values for each combination can be furthered +dynamically specified using the `params` as seen in the example in the [summary](#summary). + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: pipeline-to-build-images-from-a-single-repo +spec: + workspaces: + - name: shared-workspace + tasks: + - ... + - name: kaniko-build + taskRef: + name: kaniko + workspaces: + - name: source + workspace: shared-workspace + matrix: + include: + - name: build-1 + params: + - name: IMAGE + value: "image-1" + - name: DOCKERFILE + value: "path/to/Dockerfile1" + - name: build-2 + params: + - name: IMAGE + value: "image-2" + - name: DOCKERFILE + value: "path/to/Dockerfile2" + - name: build-3 + params: + - name: IMAGE + value: "image-3" + - name: DOCKERFILE + value: "path/to/Dockerfile3" + - ... +``` + +This `pipeline` is part of a catalog and shared across multiple teams. Now, the teams are building their own applications +using this `pipeline` and different teams have different number of images to built as part of their application. This way +of specifying `matrix` combinations in a `pipeline` is constant and can not be utilized for an application with +different number of images. + +## Proposal + +We propose adding a field - `strategy` of type `string` - within the `matrix` section. This allows pipeline authors to +maintain a single pipeline in a catalog and allow the users to specify the list of explicit combinations for each +instance dynamically through `pipelineRun` or a trigger template or a task result. + +The `matrix.strategy` is of type `string` where its value is a list of objects in key:value pairs. The controller reads +a stringified JSON payload in a form of `{"include": a list of combinations}` and creates an equivalent number of +instances based on the length of the specified list. + +When `matrix.strategy` is specified, no other `matrix` fields will be allowed i.e. `matrix.params` and `matrix.include` +are not permitted with `matrix.strategy`. This restriction will help avoid any undesired conflicts after the `matrix` +is resolved. + +**NOTE:** `matrix.include` support an optional field `name`. `name` is for information purpose only and has no real +significance in how combinations are being generated. `name` is useful from the readability perspective with +the explicit combinations when specified in the pipeline definition itself. With this proposal, `name` is not included. +The reason behind not includeing `name` is, JSON specifications inherently does not support an optional fields. +The users have to specify empty string at the minimum. + +### `matrix.strategy` initialized by the `pipelineRun` + +The `Pipeline` addressing this use case will be defined as shown below: + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: pipeline-to-build-images-from-a-single-repo +spec: + params: + - name: matrix-strategy + workspaces: + - name: shared-workspace + tasks: + - ... + - name: kaniko-build + taskRef: + name: kaniko + workspaces: + - name: source + workspace: shared-workspace + matrix: + strategy: $(params.matrix-strategy) +``` + +A `PipelineRun` can be created to execute the above `pipeline` to build 1 image: + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: PipelineRun +metadata: + generateName: pipelineRun- +spec: + pipelineRef: + name: pipeline-to-build-images-from-a-single-repo + params: + - name: matrix-strategy + value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"}]}" +``` + +`PipelineRun` can be created to execute the above `pipeline` to build 3 images: + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: PipelineRun +metadata: + generateName: pipelineRun- +spec: + pipelineRef: + name: pipeline-to-build-images-from-a-single-repo + params: + - name: matrix-strategy + value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"},{\"IMAGE\":\"image-2\",\"DOCKERFILE\":\"path/to/Dockerfile2\"},{\"IMAGE\":\"image-3\",\"DOCKERFILE\":\"path/to/Dockerfile3\"}]}" +``` + +### `matrix.strategy` specified by a task result + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: pipeline-to-build-images-from-a-single-repo +spec: + workspaces: + - name: shared-workspace + tasks: + - ... + - name: pre-requisite + taskSpec: + results: + - name: matrix-strategy + steps: + - name: read-strategy + image: alpine + script: | + # this strategy can be read from a JSON file in the repo + echo -n "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"}]}" | tee $(results.matrix-strategy.path) + - name: kaniko-build + params: + - name: matrix-strategy + value: $(tasks.read-matrix-strategy.results.matrix-strategy) + taskRef: + name: kaniko + workspaces: + - name: source + workspace: shared-workspace + matrix: + strategy: $(params.matrix-strategy) +--- + +apiVersion: tekton.dev/v1beta1 +kind: PipelineRun +metadata: + generateName: pipelineRun- +spec: + pipelineRef: + name: pipeline-to-build-images-from-a-single-repo +``` + +### GitHub Actions + +This syntax is inspired by the GitHub actions syntax similar to the rest of the sections of `matrix`. + +[GitHub actions](https://docs.github.com/en/actions/learn-github-actions/expressions#example-returning-a-json-object) +does support specifying a list of jobs in which a JSON payload is set in one job and passed to the next job using +`output` and `fromJSON`. + +```yaml +name: build +on: push +jobs: + job1: + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.set-matrix.outputs.matrix }} + steps: + - id: set-matrix + run: echo "matrix={\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}" >> $GITHUB_OUTPUT + job2: + needs: job1 + runs-on: ubuntu-latest + strategy: + matrix: ${{ fromJSON(needs.job1.outputs.matrix) }} + steps: + - run: build +``` + +## Alternatives + +- A feature is requested in general to support [nested parameters](https://github.com/tektoncd/pipeline/issues/7069). + The idea here is to be able to specify array and object params within array and object params. This + feature can help support [dynamic specifications](https://github.com/tektoncd/pipeline/issues/7069#issuecomment-1721670122) + by using an array of object i.e. a list of key value pairs. + +## Future Work + +- The proposal here is to support a string version of JSON object right now but this can be extended in future to support + an expression language such as CEL. The current proposal does not support an expression language but provides an + opportunity to extend the existing proposal. + +## References + +- [#7170]: https://github.com/tektoncd/pipeline/issues/7170 +- [Tekton Pipelines PoC]: https://github.com/tektoncd/pipeline/pull/7180 +- [Dynamic Matrix in GitHub Actions]: https://stackoverflow.com/questions/65056670/is-it-possible-to-have-a-dynamic-strategy-matrix-in-a-workflow-in-github-actions +- [GitHub Actions Matrix]: https://adamtheautomator.com/github-actions-matrix/ +- [Nested Params]: https://github.com/tektoncd/pipeline/issues/7069 \ No newline at end of file diff --git a/teps/README.md b/teps/README.md index 0976ede7a..e4d80c4b3 100644 --- a/teps/README.md +++ b/teps/README.md @@ -131,3 +131,4 @@ This is the complete list of Tekton TEPs: |[TEP-0140](0140-producing-results-in-matrix.md) | Producing Results in Matrix | implementable | 2023-08-21 | |[TEP-0141](0141-platform-context-variables.md) | Platform Context Variables | proposed | 2023-08-21 | |[TEP-0145](0145-cel-in-whenexpression.md) | CEL in WhenExpression | implementable | 2023-10-01 | +|[TEP-0147](0147-dynamic-matrix.md) | Dynamic Matrix | implementable | 2023-10-18 |