forked from cncf-tags/green-reviews-tooling
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Co-authored-by: locomundo <[email protected]> Signed-off-by: nikimanoledaki <[email protected]>
- Loading branch information
1 parent
d4f4522
commit 991b8f0
Showing
2 changed files
with
223 additions
and
0 deletions.
There are no files selected for viewing
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,223 @@ | ||
| # Run the bechmark tests as part of the automated pipeline | ||
|
|
||
| ## Authors | ||
|
|
||
| - @locomundo | ||
| - @nikimanoledaki | ||
|
|
||
| ## Status | ||
|
|
||
| Draft | ||
|
|
||
| ## Table of Contents | ||
| <!-- toc --> | ||
| - [Run the bechmark tests as part of the automated pipeline](#run-the-bechmark-tests-as-part-of-the-automated-pipeline) | ||
| - [Authors](#authors) | ||
| - [Status](#status) | ||
| - [Table of Contents](#table-of-contents) | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| - [Goals](#goals) | ||
| - [Non-Goals](#non-goals) | ||
| - [Linked Docs](#linked-docs) | ||
| - [Proposal](#proposal) | ||
| - [User Stories (Optional)](#user-stories-optional) | ||
| - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) | ||
| - [Risks and Mitigations](#risks-and-mitigations) | ||
| - [Design Details](#design-details) | ||
| - [Graduation Criteria (Optional)](#graduation-criteria-optional) | ||
| - [Drawbacks (Optional)](#drawbacks-optional) | ||
| - [Alternatives](#alternatives) | ||
| - [Infrastructure Needed (Optional)](#infrastructure-needed-optional) | ||
| <!-- /toc --> | ||
|
|
||
| ## Summary | ||
|
|
||
| <!-- | ||
| A good summary is at least a paragraph in length and should be written with a wide audience | ||
| in mind. | ||
| It should encompass the entire document, and serve as both future documentation | ||
| and as a quick reference for people coming by to learn the proposal's purpose | ||
| without reading the entire thing. | ||
| Both in this section and below, follow the guidelines of the [documentation | ||
| style guide]. In particular, wrap lines to a reasonable length, to make it | ||
| easier for reviewers to cite specific portions, and to minimize diff churn on | ||
| updates. | ||
| [documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md | ||
| --> | ||
|
|
||
| ## Motivation | ||
|
|
||
| <!-- | ||
| This section is for explicitly listing the motivation, goals and non-goals of | ||
| this proposal. Describe why the change is important, how it fits into the project's | ||
| goals and the benefits to users. | ||
| It is helpful to frame this to answer the question: "What is the problem this proposal | ||
| is trying to solve?" | ||
| --> | ||
|
|
||
| This proposal is part of the pipeline automation of the Green Review for Falco. Currently, we are using Flux to watch the upstream Falco repository and run the benchmark tests constantly. For example, [this benchmark test](https://github.com/falcosecurity/cncf-green-review-testing/blob/main/kustomize/falco-driver/ebpf/stress-ng.yaml#L27-L32) is setup as a Kubernetes Deployment that runs an endless loop of [`stress-ng`](https://wiki.ubuntu.com/Kernel/Reference/stress-ng), which applies stress to the kernel. Instead, this proposal aims to provide a solution for how to deploy the benchmark tests only when they are needed. | ||
|
|
||
| Secondly, automating the way we run benchmark tests in this pipeline will help to make it easier and faster to add new benchmark tests. It will enable both the WG Green Reviews and CNCF Project Maintainers to come up with new benchmark tests and run them to get feedback faster. | ||
|
|
||
| ### Goals | ||
|
|
||
| <!-- | ||
| List the specific goals of the proposal. What is it trying to achieve? How will we | ||
| know that this has succeeded? | ||
| --> | ||
|
|
||
| - Describe the actions to take immediately after the trigger from Proposal 1 https://github.com/cncf-tags/green-reviews-tooling/issues/84 | ||
| - Describe how the pipeline should _fetch_ the benchmark tests either from this repository (`cncf-tags/green-reviews-tooling`) or from an upstream repository (Falco's [`falcosecurity/cncf-green-review-testing`](https://github.com/falcosecurity/cncf-green-review-testing)). | ||
| - Describe how the pipeline should _run_ the benchmark tests through GitHub Actions for a specific project e.g. Falco | ||
| - Communicate the changes needed to be made by the Falco team to change the benchmark test to a GitHub Action file. | ||
| - Provide _modularity_ for the benchmark tests. | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| <!-- | ||
| What is out of scope for this proposal? Listing non-goals helps to focus discussion | ||
| and make progress. | ||
| It is important to remember that non-goals are still equally important things | ||
| which will be dealt with one day but are not things which need to be dealt with immediately | ||
| within the scope of this work. This helps make sure everyone is crystal clear on the outcomes. | ||
| --> | ||
|
|
||
| * Defining or designing the content of benchmark tests themselves, or assigning responsibility for who should write them. | ||
|
|
||
| ### Linked Docs | ||
|
|
||
| <!-- | ||
| Provide links to previous discussions, Slack threads, motivation issues or any other document | ||
| with context. It is really helpful to provide a "source of truth" for the work | ||
| so that people aren't searching all over the place for lost context. | ||
| --> | ||
|
|
||
| * [Slack discussion on benchmark test framework](https://cloud-native.slack.com/archives/C060EDHN431/p1708416918423089?thread_ts=1708348336.259699&cid=C060EDHN431) | ||
|
|
||
| ## Proposal | ||
|
|
||
| <!-- | ||
| This is where we get down to the specifics of what the proposal actually is: | ||
| outlining your solution to the problem described in the Motivation section. | ||
| This should have enough detail that reviewers can understand exactly what | ||
| you're proposing, but should not include things like API designs or | ||
| implementation. The "Design Details" section below is for the real | ||
| nitty-gritty. | ||
| --> | ||
|
|
||
| ### User Stories (Optional) | ||
|
|
||
| <!-- | ||
| Detail the things that people will be able to do if this proposal is implemented. | ||
| Include as much detail as possible so that people can understand the "how" of | ||
| the system. The goal here is to make this feel real for users without getting | ||
| bogged down. | ||
| --> | ||
|
|
||
|
|
||
| * As a CNCF Project Maintainer, | ||
|
|
||
| * As a Green Review WG reviewer, | ||
|
|
||
| * As a Green Review WG cluster maintainer, | ||
|
|
||
| ### Notes/Constraints/Caveats (Optional) | ||
|
|
||
| <!-- | ||
| What are the caveats to the proposal? | ||
| What are some important details that didn't come across above? | ||
| Go in to as much detail as necessary here. | ||
| This might be a good place to talk about core concepts and how they relate. | ||
| --> | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| <!-- | ||
| What are the risks of this proposal, and how do we mitigate? | ||
| Think broadly. For example, consider how this will impact or be impacted | ||
| by scaling to support more CNCF Projects. | ||
| How will this affect the benchmark tests, CNCF Project Maintainers, pipeline maintainers, etc? | ||
| --> | ||
|
|
||
| ## Design Details | ||
|
|
||
| <!-- | ||
| This section should contain enough information that the specifics of your | ||
| change are understandable. This may include manifests or workflow examples | ||
| (though not always required) or even code snippets. If there's any ambiguity | ||
| about HOW your proposal will be implemented, this is the place to discuss them. | ||
| --> | ||
|
|
||
| Workflows can be fetched from other GitHuborganizations and repositories using the `jobs.<job_id>.uses` syntax defined here: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_iduses | ||
|
|
||
| ``` | ||
| jobs: | ||
| call-workflow-in-another-repo: # job_id | ||
| uses: octo-org/another-repo/.github/workflows/workflow.yml@v1 # refer to self or other repo | ||
| ``` | ||
|
|
||
| This syntax can be configured to use `@main` or `@another-branch` which would be nice for versioning and testing specific releases. | ||
|
|
||
| This means there is no need to authenticate to any repository as long as the external repositories are public. | ||
|
|
||
| How will the job run in the cluster? We assume that the workflow already contains a kubeconfig to authenticate with the test cluster and Falco has already been deployed to it. | ||
|
|
||
| The next steps describe how to deploy a benchmark test to the cluster, taking `stress-ng` as an example. | ||
|
|
||
| The workflow can use an existing job that contains a reference for how to deploy the benchmark test to the cluster: `octo-org/another-repo/.github/workflows/workflow.yml@v1` | ||
|
|
||
| For example, Falco has define their `stress-ng` test in a Deployment manifest which is ready to be applied to a cluster. This Deployment manifest can be applied in a similar way to how Falco was deployed to the cluster, using infrastructure-as-code in an _ad hoc_ rather than _continuous_ way. For example, using the FLux CLI: | ||
|
|
||
| ``` | ||
| flux create kustomization stress-ng-benchmark-test \ | ||
| --target-namespace=falco \ | ||
| --source=<falco-test-repo> | ||
| ``` | ||
|
|
||
| Essentially, the command above would deploy the benchmark test ad hoc. The CLI arguments would be similar to how Falco is currently deployed in [clusters/projects/falco/falco.yaml](https://github.com/cncf-tags/green-reviews-tooling/blob/main/clusters/projects/falco/falco.yaml). The main difference is that, using the CLI, it is deployed _ad hoc_ rather than _continuously_. | ||
|
|
||
| For future benchmark tests, it is not necessary to deploy the test using Flux. This framework works for now but other ways could be considered if other CNCF Project Maintainers would like to suggest a different way to do this. | ||
|
|
||
| In addition, The test should run the test on a cluster which is configurable. The workflow receives parameters to indicate and authenticate on the cluster where it should e.g. define which namespace, pass the kubeconfig to the workflow. | ||
|
|
||
| ### Graduation Criteria (Optional) | ||
|
|
||
| <!-- | ||
| List criteria which would allow progression from one maturity level to another. | ||
| eg. What needs to have been accomplished/demonstrated to move from Alpha to Beta. | ||
| If applicable, what is the milestone marker which will allow deprecation of the | ||
| replaced capability? | ||
| --> | ||
|
|
||
| ## Drawbacks (Optional) | ||
|
|
||
| <!-- | ||
| What other approaches did you consider, and why did you rule them out? These do | ||
| not need to be as detailed as the proposal, but should include enough | ||
| information to express the idea and why it was not acceptable. | ||
| --> | ||
|
|
||
| ## Alternatives | ||
|
|
||
| <!-- | ||
| What other approaches did you consider, and why did you rule them out? These do | ||
| not need to be as detailed as the proposal (pros and cons are fine), | ||
| but should include enough information to express the idea and why it was not acceptable | ||
| as well as illustrate why the final solution was selected. | ||
| --> | ||
|
|
||
| ## Infrastructure Needed (Optional) | ||
|
|
||
| <!-- | ||
| Use this section if you need things from the project/SIG. Examples include a | ||
| new subproject, repos requested, or GitHub details. Listing these here allows a | ||
| SIG to get the process for these resources started right away. | ||
| --> |