TEP-0147 Dynamic Matrix

Proposing a new section for matrix which allows to specify the explicit list of combinations dyanmically Signed-off-by: Priti Desai <[email protected]>
tektoncd · Oct 6, 2023 · d68baad · d68baad
1 parent f069a45
commit d68baad
Show file tree

Hide file tree

Showing 2 changed files with 211 additions and 0 deletions.
diff --git a/teps/0147-dynamic-matrix.md b/teps/0147-dynamic-matrix.md
@@ -0,0 +1,210 @@
+---
+status: implementable
+title: Dynamic Matrix
+creation-date: '2023-10-05'
+last-updated: '2023-10-05'
+authors: [
+  '@pritidesai'
+]
+see-also:
+  - TEP-0118
+  - TEP-0090
+---
+
+# TEP-0142: Dynamic Matrix
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+  - [Use Cases](#use-cases)
+- [Proposal](#proposal)
+- [References](#references)
+<!-- /toc -->
+
+## Summary
+
+Matrix feature of Tekton Pipelines allows pipeline authors to specify multiple concurrent execution of the same task.
+The same task can be executed in parallel based on the number of input parameters or a combination of parameters.
+
+Matrix supports specifying [implicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#generating-combinations)
+and [explicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#explicit-combinations) combinations of
+parameters.
+
+- With implicit combinations, a `pipelineTask` is defined with a list of parameters and one or more values for each
+parameter. Tekton controller generates an exhaustive list of combinations based on the specified parameters. Based on
+the generated list of combinations, a `pipelineTask` is fanned out and executed with each combination.
+
+- With explicit combinations, a `pipelineTask` is defined with a list of combinations and Tekton controller fans out the
+task based on the number of combinations. Each running instance receives unique combination of input parameters.
+
+Both implicit and explicit combinations fit well with many use cases. At the same time, creating a sharable pipeline with
+existing Matrix syntax is not possible when a list of combinations are specified at the runtime through `pipelineRun`.
+
+## Motivation
+
+It is a common practice to execute a `pipeline` either by creating a `PipelineRun` object or through `TriggerTemplate`.
+Generally, a list of `params` are specified as part of the `PipelineRun` which might be initialized statically or
+through a [trigger template params](https://github.com/tektoncd/triggers/blob/main/docs/triggertemplates.md#structure-of-a-triggertemplate).
+Now, running a `pipeline` with `matrix` where the values for each instance of fanned out task is dynamically specified
+at the runtime is not possible with the existing `matrix` syntax. In this proposal, we would like to extend `matrix`
+syntax to provide an option to dynamically specify a list of explicit combinations
+
+### Goals
+
+- Be able to author a sharable pipeline with `matrix` such that a task can fan based on the explicit combinations which
+are specified dynamically.
+
+### Non-Goals
+
+- Matrix explicit combination mechanism only supports a `param` of type `string` and not adding support for any
+other type.
+
+### Use Cases
+
+Let's revisit the example of build pipeline from [TEP-0118](0118-matrix-with-explicit-combinations-of-parameters.md#define-explicit-combinations-in-the-matrix).
+A pipeline `pipeline-to-build-images-from-a-single-repo` has three explicit combinations defined in the pipeline such
+that `kaniko` can fan out and create three separate `taskRun`s. The values for each combination can be further
+dynamically specified using the `params`.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  name: pipeline-to-build-images-from-a-single-repo
+spec:
+  workspaces:
+    - name: shared-workspace
+  tasks:
+    - ...
+    - name: kaniko-build
+      taskRef:
+        name: kaniko
+      workspaces:
+        - name: source
+          workspace: shared-workspace
+      matrix:
+        include:
+          - name: build-1
+            params:
+              - name: IMAGE
+                value: "image-1"
+              - name: DOCKERFILE
+                value: "path/to/Dockerfile1"
+          - name: build-2
+            params:
+              - name: IMAGE
+                value: "image-2"
+              - name: DOCKERFILE
+                value: "path/to/Dockerfile2"
+          - name: build-3
+            params:
+              - name: IMAGE
+                value: "image-3"
+              - name: DOCKERFILE
+                value: "path/to/Dockerfile3"
+    - ...
+```
+
+This `pipeline` is part of a catalog and shared across multiple teams. Now, the teams are building their own applications
+using this `pipeline` and different teams have different number of images to built as part of their application. This way
+of specifying `matrix` combinations in a `pipeline` is constant and can not be utilized for an application with
+variety of images.
+
+## Proposal
+
+We propose adding a field - `strategy` of type `string` - within the `matrix` section. This allows pipeline authors to
+maintain a single pipeline in a catalog and allow the users to specify the list of explicit combinations of `params`
+dynamically through `param` (`pipelineRun` or a trigger template) or a task result of a parent task.
+
+The `matrix.strategy` is of type `string`  with a JSON payload `{"include": a list of combinations}`. This aligns with
+`matrix.include` syntax and allows to specify dynamic list of combinations.
+
+The `Pipeline` addressing this use case will be defined as shown below:
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  name: pipeline-to-build-images-from-a-single-repo
+spec:
+  params:
+    - name: matrix-strategy
+  workspaces:
+    - name: shared-workspace
+  tasks:
+    - ...
+    - name: kaniko-build
+      taskRef:
+        name: kaniko
+      workspaces:
+        - name: source
+          workspace: shared-workspace
+      matrix:
+        strategy: $(params.matrix-strategy)
+```
+
+A `PipelineRun` can be created to execute the above `pipeline` to build 1 image:
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  generateName: pipelineRun-
+spec:
+  pipelineRef:
+    name: pipeline-to-build-images-from-a-single-repo
+  params:
+    - name: matrix-strategy
+      value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"}]}"
+```
+
+`PipelineRun` can be created to execute the above `pipeline` to build 3 images:
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  generateName: pipelineRun-
+spec:
+  pipelineRef:
+    name: pipeline-to-build-images-from-a-single-repo 
+  params:
+    - name: matrix-strategy
+      value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"},{\"IMAGE\":\"image-2\",\"DOCKERFILE\":\"path/to/Dockerfile2\"},{\"IMAGE\":\"image-3\",\"DOCKERFILE\":\"path/to/Dockerfile3\"}]}"
+```
+
+This syntax is inspired by the GitHub actions syntax similar to the other sections of `matrix`.
+
+[GitHub actions](https://docs.github.com/en/actions/learn-github-actions/expressions#example-returning-a-json-object)
+does support specifying a list of jobs in which a JSON payload is set in one job and passed to the next job using
+`output` and `fromJSON`.
+
+```yaml
+name: build
+on: push
+jobs:
+  job1:
+    runs-on: ubuntu-latest
+    outputs:
+      matrix: ${{ steps.set-matrix.outputs.matrix }}
+    steps:
+      - id: set-matrix
+        run: echo "matrix={\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}" >> $GITHUB_OUTPUT
+  job2:
+    needs: job1
+    runs-on: ubuntu-latest
+    strategy:
+      matrix: ${{ fromJSON(needs.job1.outputs.matrix) }}
+    steps:
+      - run: build
+```
+
+## References
+
+- https://github.com/tektoncd/pipeline/issues/7170
+- https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategy
+- https://github.com/tektoncd/pipeline/compare/main...pritidesai:matrix-strategy?expand=1
+- https://tomasvotruba.com/blog/2020/11/16/how-to-make-dynamic-matrix-in-github-actions/
+  https://stackoverflow.com/questions/65056670/is-it-possible-to-have-a-dynamic-strategy-matrix-in-a-workflow-in-github-actions
diff --git a/teps/README.md b/teps/README.md
@@ -130,3 +130,4 @@ This is the complete list of Tekton TEPs:
 |[TEP-0138](0138-decouple-api-and-feature-versioning.md) | Decouple api and feature versioning | proposed | 2023-07-27 |
 |[TEP-0140](0140-producing-results-in-matrix.md) | Producing Results in Matrix | implementable | 2023-08-21 |
 |[TEP-0141](0141-platform-context-variables.md) | Platform Context Variables | proposed | 2023-08-21 |
+|[TEP-0142](0147-dynamic-matrix.md) | Dynamic Matrix | implementable | 2023-10-05 |