Overridden variables not handled properly in child profiles

[JM1]

TL;DR: When computing cmdline_cpu_part both variables not_isolated_cpumask and not_isolated_cores_expanded are not read from the child profile but from the parent profile. The child profile is:

[main]
include=[...],cpu-partitioning,[...]

[variables]
isolated_cores=3
not_isolated_cores=0-1
not_isolated_cores_expanded=${f:cpulist_unpack:${not_isolated_cores}}
not_isolated_cores_online_expanded=${f:cpulist_online:${not_isolated_cores_expanded}}
not_isolated_cpumask=${f:cpulist2hex:${not_isolated_cores_expanded}}

[bootloader]
cmdline_cpu_part=+nohz=on rcu_nocbs=${isolated_cores} tuned.non_isolcpus=${not_isolated_cpumask} systemd.cpu_affinity=${not_isolated_cores_expanded}

Long story:

OpenShift's Cluster Node Tuning Operator (NTO) defines a TuneD config openshift-node-performance which includes TuneD's cpu-partitioning profile and defines a variable isolated_cores={{.IsolatedCpus}}. It uses variables not_isolated_cpumask (and not_isolated_cores_expanded) which is defined in parent profile's tuned.conf as following:

not_isolated_cores_expanded=${f:cpulist_invert:${isolated_cores_expanded}}
not_isolated_cores_online_expanded=${f:cpulist_online:${not_isolated_cores_expanded}}
not_isolated_cpumask=${f:cpulist2hex:${not_isolated_cores_expanded}}

(Variable not_isolated_cores_expanded is also defined in openshift-node-performance in the same way not_isolated_cores_expanded=${f:cpulist_invert:${isolated_cores_expanded}}.)

Now, I want to change the child profile openshift-node-performance to set the not_isolated_cores* variables independently from isolated_cores*. For example:

[variables]
isolated_cores=3
not_isolated_cores=0-1
not_isolated_cores_expanded=${f:cpulist_unpack:${not_isolated_cores}}
not_isolated_cores_online_expanded=${f:cpulist_online:${not_isolated_cores_expanded}}
not_isolated_cpumask=${f:cpulist2hex:${not_isolated_cores_expanded}}

The child profile then would use those variables to set the kernel's cmdline:

[bootloader]
cmdline_cpu_part=+nohz=on rcu_nocbs=${isolated_cores} tuned.non_isolcpus=${not_isolated_cpumask} systemd.cpu_affinity=${not_isolated_cores_expanded}

Here, not_isolated_cpumask should resolve to 00000003 and not_isolated_cores_expanded to 0,1. But in practice not_isolated_cpumask is 00000000 and not_isolated_cores_expanded to `` (empty string)!?!

(Complete NTO patch can be found here.)

As a workaround, change line not_isolated_cores_expanded=${f:cpulist_unpack:${not_isolated_cores}} to not_isolated_cores_expanded=${f:cpulist_unpack:0-1}:

[main]
include=openshift-node,cpu-partitioning,[...]

[variables]
isolated_cores=3
not_isolated_cores=0-1

# TODO: Why does the following expand to ''?
# TODO: Why is the following not inherited from parent profile cpu-partitioning?
#not_isolated_cores_expanded=${f:cpulist_unpack:${not_isolated_cores}}
#
# Workaround because the above is broken
not_isolated_cores_expanded=${f:cpulist_unpack:0-1}

# The following variables actually inherited from parent profile cpu-partitioning 
#not_isolated_cores_online_expanded=${f:cpulist_online:${not_isolated_cores_expanded}}
#not_isolated_cpumask=${f:cpulist2hex:${not_isolated_cores_expanded}}


[bootloader]
cmdline_cpu_part=+nohz=on rcu_nocbs=${isolated_cores} tuned.non_isolcpus=${not_isolated_cpumask} systemd.cpu_affinity=${not_isolated_cores_expanded}

(Complete NTO patch can be found here.)

EDIT: Using OpenShift's latest development branch 4.19 with RHCOS based on RHEL9.6:

[root@w1 core]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION="419.96.202502141056-0"
VERSION_ID="4.19"
VARIANT="CoreOS"
VARIANT_ID=coreos
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 419.96.202502141056-0"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos::coreos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://docs.okd.io/latest/welcome/index.html"
BUG_REPORT_URL="https://access.redhat.com/labs/rhir/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.19"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.19"
OPENSHIFT_VERSION="4.19"
RHEL_VERSION=9.6
OSTREE_VERSION="419.96.202502141056-0"


[root@w1 core]# uname -a
Linux w1 5.14.0-565.el9.x86_64+rt #1 SMP PREEMPT_RT Fri Feb 7 16:08:17 EST 2025 x86_64 x86_64 x86_64 GNU/Linux

Cluster Node Tuning Operator's tuned pod is based on RHEL9.4:

bash-5.1# /usr/bin/python3 -Es /usr/sbin/tuned --version 
tuned 2.24.0


bash-5.1# cat /etc/os-release  
NAME="Red Hat Enterprise Linux"
VERSION="9.4 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"

[zacikpa]

Thanks for filing the bug, @JM1. This is a similar problem to the one hacked around in #239.

I'll explain what's happening with two simpler profiles:

parent profile:

[variables]
isolated_cores=0-2
not_isolated_cores_expanded=${f:cpulist_invert:${f:cpulist_unpack:${isolated_cores}}}

child profile:

[main]
include=parent

[variables]
not_isolated_cores=3-5
not_isolated_cores_expanded=${f:cpulist_unpack:${not_isolated_cores}}

When the child profile includes the parent, the two profiles are merged into one. The options in each profile section are ordered as they appear, first in the parent, then in the child. However, when there is a collision, the child option overwrites the parent option in its place. In the example, we get:

[variables]
isolated_cores=0-2
not_isolated_cores_expanded=${f:cpulist_unpack:${not_isolated_cores}}
not_isolated_cores=3-5

As is now clear, the not_isolated_cores variable is not yet defined when it's used in the definition of not_isolated_cores_expanded.

Personally, I think this behavior definitely requires changes; your use case is completely valid and it does not make sense to work around the problem like in #239.

Morever (and funnily enough) I found another related bug when debugging this: if you would attempt to apply the profile for the second time (without restarting TuneD), it would apply properly because TuneD does not currently discard old variable definitions. You can guess the confusing behavior that occurs when you change the definition of not_isolated_cores between the two attempts...