Support per-component tolerations (+ manager)

[loganmzz]

Should partially fix aws/containers-roadmap#2515

Issue #, if available:

aws/containers-roadmap#2515

Description of changes:

1. Let's support per-component tolerations
2. Add tolerations for manager

Behavior

• Root tolerations is applied by default
• Manager tolerations is disabled by default ([] value, see below)
• Each component may override tolerations
• Tolerations can be disabled by using empty list [] (root or specific)
• If specific tolerations is null (nil in Go/Template), root ones are used.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[sky333999]

It might not be very intuitive to set another boolean just to express that the tolerations at the specific component level should be respected.

Could we instead have the tolerations commented out at the specific component level, such as:

# tolerations: []  # Uncomment this line to override root-level tolerations.

Once you merge from latest main, you will notice the root tolerations have been defined & only apply to daemonsets and do NOT apply to the controller manager deployment. So just for controller manager alone, we truly do want to explicitly set tolerations: [] and let it override the root level ones to maintain compatibility i.e. not comment that line.

We can then probably have logic that does something like:

{{- if hasKey .Values.manager "tolerations" -}}
  {{- toYaml .Values.manager.tolerations -}}
{{- else -}}
  {{- toYaml .Values.tolerations -}}
{{- end -}}
{{- end -}}

Could probably put this in a shared method in _helpers.tpl and call it for the all components.

The components to cover would be:

• manager:
• containerLogs.fluentBit
• agent
• dcgmExporter
• neuronMonitor

[loganmzz]

Hi @sky333999,

Thank you for the feedback!

The PR has been designed to be conservative to avoid unexpected impact for existing deployment. I will check new code changes and update my PR accordingly.

[loganmzz]

@sky333999

I have updated the PR based on:

• Root tolerations is applied by default
• Manager tolerations is disabled by default ([] value, see below)
• Each component may override tolerations
• Tolerations can be disabled by using empty list [] (root or specific)
• If specific tolerations is null (nil in Go/Template), root ones are used.

[sky333999]

Thats great! Thank you so much for the changes. Looks good to me.

[sky333999]

Now that I think about it a bit more, you could probably get away with something like:

  {{- with (.Values.agent.tolerations | default .Values.tolerations ) }}
  tolerations: {{- toYaml . | nindent 2}}
  {{- end }}

Im debating if this is just simpler for each of the components instead of the helper function. Im fine either way.

[loganmzz]

Sorry I miss your two last review feedbacks.

Because nil and empty ([]) are handled differently, default function won't do what is expected.

[sky333999]

Thanks for the updates @loganmzz . Would you mind adding some testing information to the description so we have that captured before I merge this in?
Some manual test suggestions if you could please try out locally:

• Render the chart prior to and after your change to make sure the rendered manifests are identical since your change should be a no-op
• Override the deployment tolerations to something other than [] and ensure its reflected correctly in the rendered manifests
• Override one of the daemonset tolerations to [] or something eles to ensure its reflected correctly in the rendered manifests

[loganmzz]

Thanks for the updates @loganmzz . Would you mind adding some testing information to the description so we have that captured before I merge this in? Some manual test suggestions if you could please try out locally:

• Render the chart prior to and after your change to make sure the rendered manifests are identical since your change should be a no-op
• Override the deployment tolerations to something other than [] and ensure its reflected correctly in the rendered manifests
• Override one of the daemonset tolerations to [] or something eles to ensure its reflected correctly in the rendered manifests

For sure here the values I have tested on:

# 00 - Empty 

# 01 - Disable all tolerations with empty
tolerations: []

# 02 - Disable all tolerations with null
tolerations:

# 03 - Set manager tolerations
manager:
  tolerations:
  - key: "specific"
    operator: "Equal"
    value: "manager"

# 04 - Set manager tolerations to default
manager:
  tolerations:

# 05 - Set agent tolerations
agent:
  tolerations:
  - key: "specific"
    operator: "Equal"
    value: "agent"

# 06 - Disable agent tolerations
agent:
  tolerations: []