Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADR 45: Guidelines for developing tasks #216

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions ADR/0045-guidelines-for-developing-tasks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# 45. Guidelines for developing tasks

* Date: 2024-11-06

## Status

Proposed

## Context

One of the architectural goals for Konflux is extensibility. The core system provides opinionated [build pipelines]
and [release pipelines], but lets users extend those and/or create their own. Beyond this goal, however, there has
not been any clear guidance about where contributions should be added or the types of functionality that are typical
for the stages of software development.

## Decision
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would mentioning Task policies be of value here? As in: "Tasks contributed to Konflux need to adhere to the Build and Task policies"


* If any process is required to modify artifacts themselves, it should be done in [build pipelines]. This will ensure
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not aware any use cases that modify the artifacts from the build pipeline. Perhaps a footnote/appendix with examples of such modifications could be useful. Some OLM use cases I'm not aware of?

This could be even broader/stricter, e.g. "any production of the artifact must be performed on the build pipelines with the intent that the produced artifact is shipped as-is".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

konflux-ci/release-service-catalog#656 modifies the sbom, which is an artifact from the build pipeline

that Konflux users can appropriately test and verify these artifacts before triggering releases with them.
* Any tasks which need to be included in the artifacts' provenance (enabling verification with EC policies) must be
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

included in the [build pipelines].
* Whenever possible, a task that just changes the format of an artifact (for example creating a tar from a container image)
should be performed in the same pipeline that produced the artifact. These modified artifacts should be attached using
the referrer's API and the blob digest reported via results to enable verification that the artifacts were produced
from a trusted task.
Comment on lines +22 to +25
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? Converting to a different format can be done during release or even post-release without loss of trust

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It enables users to verify that the format change is done properly before those artifacts are released.

I am not sure what you mean by post-release here. These are modifications that are needed before releases can progress.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It enables users to verify that the format change is done properly before those artifacts are released.

Maybe I'd need a better example than the one in this doc. When I skopeo copy docker://registry.access.redhat.com/ubi9/ubi-micro:9.4 oci:/tmp/ubi, I don't need to verify anything.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have an integration test scenario that does not use the ubi-migro image in registry but instead uses a tarball, you would need that tarball to verify the artifact.

The need to verify isn't always required by a human but by some process. It should be possible to have all artifacts-to-be-released verified as is. If testing/verifying, a user or process shouldn't have to replicate the modifications done during a release pipeline within integration pipelines.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The need to verify isn't always required by a human but by some process. It should be possible to have all artifacts-to-be-released verified as is. If testing/verifying, a user or process shouldn't have to replicate the modifications done during a release pipeline within integration pipelines.

Agreed, but still think that the example is really bad. The use case is purely hypothetical, nobody is ever going to create an integration test that requires an image to be in the weird format before testing the image. And in this example the release pipeline doesn't do any modifications. If we're calling skopeo copy a modification, then the release pipeline always modifies images on release

* Artifact-modifying steps should generally be avoided in [release pipelines]. When interacting with produced artifacts,
the release pipeline should be limited to extracting artifacts from storage (i.e. an OCI registry), copying/pushing
artifacts to target locations, and updating metadata associated with the artifacts.
* Reusable tasks or pipelines that are intended to be used in IntegrationTestScenarios should be contributed to
[pipeline samples].

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to add a bullet point to where/how Tasks are contributed to the build (or even release) pipelines here?

## Consequences

* All artifacts produced in Konflux should have provenance attestations. These attestations will include information
about which tests have run.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The word "test" could be a bit ambiguous, perhaps

Suggested change
about which tests have run.
about which artifact or source checks have run.

* Whenever possible, the provenance attestations should be made available with the released artifacts. It may be
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how this relates to the topic of developing tasks.

necessary to update this metadata during the release process (for example, signing provenance with a different key
or updating the provenance to indicate that a build was hermetic). If it is not possible to publish the provenance
as-is, a summary attestation can be used instead.
* It should be possible to test any artifact that is going to be released.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps not related to the topic of developing tasks?


## Footnotes

* It is not currently possible to produce trusted build-time tasks without pushing them to the [build definitions]
repository. Once the process for trusting tasks from various locations has been established, a new ADR should be
created to update the recommendations presented here.
* Once it is possible to generate attestations for integration tests, the requirement to add all gating tasks to the
[build definitions] can be relaxed. This will, of course, require that the tasks and their results can be
appropriately trusted.

[build pipelines]: https://github.com/konflux-ci/build-definitions/
[build definitions]: https://github.com/konflux-ci/build-definitions/
[release pipelines]: https://github.com/konflux-ci/release-service-catalog
[pipeline samples]: https://github.com/konflux-ci/pipeline-samples