H4HIP: Helm Sequencing Proposal

[Jeb135]

proposal to add resource sequencing support to Helm for v4

[scottrigby]

Nice proposal 👏 Definitely has come up in conversations a lot. Added some suggestions, and questions.

[scottrigby]

Does this proposal also include sequencing for uninstall, or only install/upgrade? For example, waiting to uninstall a resource until all that depends on it have completely uninstalled?

[Jeb135]

Good point I didn't think about that. At least with uninstall it can just be done backwards to the installation, but upgrade would add some complexity that I'm not sure how to tackle.

[Jeb135]

I added a section to clarify that this would apply to all 3, installation/uninstall/upgrade

[joejulian]

and rollback, I assume

[scottrigby]

To be clear, helm.sh/depends-on/chart refers to some chart that is not listed as a dependency of the current chart? Is that right?

• assuming this because the wait flag should already check that dependent charts are fully installed before installing the current chart
• is the idea that this is limited to other charts that have already been deployed to the same namespace, or is there an idea to check charts in other namespaces (assuming the user has correct RBAC permissions)?

My other question here is, do we want to allow depending on other named k8s resources that are not defined by a chart? Eg, helm.sh/depends-on/resource? Or is the idea that users could annotate other resources not defined by a chart with helm.sh/layer, declaring that they are considered part of the same layer of a chart?

[Jeb135]

helm.sh/depends-on/chart is referring to any direct chart dependencies. I was unsure if Helm waits to install the current chart until all dependent charts are installed and assumed not. If Helm is already waiting for dependent charts to be installed, then this section can probably be removed?

[banjoh]

assuming this because the wait flag should already check that dependent charts are fully installed before installing the current chart

helm.sh/depends-on/chart meant for both dependencies and subcharts, will wait regardless of --wait/--wait-for-jobs CLI flags being present. Unless we want to have a way to opt-out from sequencing, which we can then assume if --wait/--wait-for-jobs are absent, not sequencing. Today, --wait will wait for certain resources only. In the proposal we have helm.sh/resource-ready which a chart developer can use to define what they mean by "readiness".

I think we need to add a section explaining how --wait/--wait-for-jobs will work with sequencing annotations. I added it to Open Issues

My other question here is, do we want to allow depending on other named k8s resources that are not defined by a chart? Eg, helm.sh/depends-on/resource? Or is the idea that users could annotate other resources not defined by a chart with helm.sh/layer, declaring that they are considered part of the same layer of a chart?

I think using helm.sh/depends-on/resource to specify a resource makes sense. Helm would eventually interpret layers as a group of resources to wait for

[banjoh]

A suggestion came my way regarding reasoning around sequencing chart dependencies/subcharts. Chart authors would want to place such annotations on a chart-level definition rather than deep inside chart templates. Defining once also makes sense because only one reference of the chart dependency is needed. Any other references are repetition since helm will wait for the chart before deploying the parent chart. We came up with something like below. We'd move helm.sh/depends-on/charts from chart templates

In the parent Chart.yaml

name: MyChart
annotations:
  # List of charts that need to be installed before MyChart
  helm.sh/depends-on/charts: "subchart, minio, rabbitmq"
dependencies:
- name: rabbitmq
  version: "1.2.3"
- name: minio
  version: "5.0.0"

[scottrigby]

Is the idea that this would be a check that Helm could use to override—kstatus ready condition (as an example implementation)? https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition

[Jeb135]

Yeah

[scottrigby]

Personally, I quite like the general direction of proposal 1—as a replacement of numeric weights entirely—over proposal 2, which extends that. My point of view is I would like to move the numeric weighting into the "alternatives considered" section. But would like to hear other maintainers and community feedback on this.

I have wanted to introduce a depends_on functionality for helm resources—as a dynamic alternative to weights—to allow users to specify a specific order they are applied (a similar concept is used by used by projects like terraform, flux, and docker-compose).

My main reasoning for this is numbered weights are static—you must know the entire list of how all resources should be ordered, and this often takes a lot of finagling and fussing to get it right.

While named dependencies are dynamic—creating a DAG on the fly, based only on when and if certain resources have an actual dependency on one or more other resources.

It might be good to note some of this in the proposal. I could sketch up some suggested text for this if you prefer, but would also be happy for you to take a stab at it in your HIP. What do you think?

[Jeb135]

Go ahead! I'll add you as a contributor, I appreciate the assistance.

[Jeb135]

I moved the weights proposal to rejected ideas and tried to capture some of the points you made here in why.

[mattfarina]

This is making me wonder if we need a new version of charts to handle the compatibility issues.

[mattfarina]

When it comes to sequencing, I'm not sure what you mean is clear in the document. I could come to two different conclusions:

1. The ordering of resources when they are uploaded to the k8s API server in one push
2. An ordered set of communications to the k8s API server where set it sent and completed prior to the next one being sent

Can you please provide some more clarity on what you mean.

[Jeb135]

2 is what we're going for here. I'll make a note to try and make that more clear.

[Jeb135]

Clarified what "resource sequencing" meant in the HIP

[mattfarina]

The # at the front denotes a comment. I'm not sure the k8s yaml parser, that we currently use and is supported by the k8s project, provides a means to parse and use data in comments. This needs to be investigated before this possibility could be considered. See https://github.com/kubernetes-sigs/yaml

[banjoh]

For reference, the ## Layer: <name> would be above/below # Source:. Maybe there is no need for 2 #. We can have it be # Layer: <name>.

helm template from an example chart

---
# Source: foo/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:

[banjoh]

I made some changes to allow a chart author to build a dependency of subcharts. Its based on #373 (comment) post I had made earlier

[banjoh]

This HIP will depend on #374 HIP

[joejulian]

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

[joejulian]

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

Which also makes me think about storage. Will each stage need to be a release in order to manage failures and rollbacks? What will that do to storage requirements?

[scottrigby]

What happens when it breaks? How will partial successes be rolled back? What if a rollback fails a stage?

I think this HIP should not change the way rollbacks work today, only the order in which resources within a chart and subcharts are applied, and how waiting for readiness works if --wait is passed when sequencing is specified.

Which also makes me think about storage. Will each stage need to be a release in order to manage failures and rollbacks? What will that do to storage requirements?

This HIP should not change storage either. A chart's resources and and its subcharts should continue to be stored the same way for a release as if it didn't have any sequencing of those specified.

To clarify for some of the questions above, @banjoh / @Jeb135 do you think this is a fair summary of sequencing, scope, and waiting for readiness in this HIP?

1. Sequencing scope:

   • order of RESOURCES can only be specified for resources within the same chart
   • order of SUBCHARTS can only be specified for subcharts of the same parent chart (this can of course be recursive as subcharts work in Helm 3)

2. Sequencing order during upgrade/install/uninstall actions:

   • Helm will build a "Subcharts DAG" for a parent chart being acted upon, using the order specified by the parent and it's subchart siblings. This will start with the parent chart and recursively include subcharts, and any of their subcharts.
   • Helm will also build a "Resources DAG" for each individual subchart (and the top level parent chart) within the Subcharts DAG
   • Install and upgrade actions will start with the lowest level of the Charts DOG, and handle one subchart at a time moving to the top level parent chart. Uninstall action will follow the reverse order.
   • When each individual subchart (and top-level chart) is being handled following the correct Charts DAG order, the resource groups ("layers") within that individual subchart/chart will be handled following the Resources DAG for that chart/subchart until complete. The Resources DAG order within an individual subchart/chart will follow the same ASC/DESC logic above depending on the install/upgrade/uninstall action.
   • After the resources within each individual subchart/chart are handled for it's Resources DAG, Helm will move onto the next subchart/chart in the Charts DAG until complete.

3. Waiting for readiness during Install/Upgrade:

   • the --wait flag for each action allows the user to opt-in to waiting for readiness of each layer in the Resources DAG, and each subchart/chart in the Charts DAG, before moving onto the next. If the --wait flag is not set, Helm will not wait for readiness of any chart-specified sequencing.
   • As Helm moves through the Charts DAG, the --wait flag will wait for all the resources in each subchart/chart to be ready before moving onto then next. Similar to how this works for all resources in a chart in Helm 3, but now with stages in a specified order.
   • At each subchart/chart in the Charts DAG above—as Helm moves through the Resources DAG for that subchart/chart—the --wait flag will wait for all resources within each group/layer of the Resources DAG to be Ready before moving onto the next. When all layers are complete, this signals that all resources within that subchart/chart are Ready, so the next chart/subchart in the Charts DAG can take its turn.

[scottrigby]

@banjoh / @Jeb135 do you also think the summary of functionality in this comment is accurate? #375 (comment)

Specifically that this HIP defines these two use cases:

• Allow chart authors to specify ordering RESOURCES WITHIN A SINGLE CHART
• Allow chart authors to specify ordering SUBCHARTS WITHIN A PARENT CHART