What is the intended behavior of sos collect when provided a --config /etc/sos/custom.conf

[TrevorBenson]

I'm noticing that when I run sos collect with a customized sos configuration, that the values under the [collect] section are honored. Values under the [report] section appear to be ignored.

Environment

/etc/sos/sos.conf

[global] section

• batch = yes
• all-logs = yes

[report] section

• only-plugins (set to 8 specific report plugins)

[plugin_options] section

• enables specific options for the enabled plugins

/etc/sos/custom.conf

[global] section

• batch = yes
• all-logs = no
• log-size = 1 (to limit any collection from 25MB to 1MB if a plugin does not honor all-logs)

[collect] section

• upload-protocol = s3
  Additional options for upload-s3-XYZ are defined for endpoint, bucket, access-key and secret-key

[report] section

• skip-plugins = networking

[plugin_options] section

• enables specific options for the enabled plugins

Observations

After testing with the above custom configuration files is when running sos collect --config /etc/sos/custom.conf [...] the [global] and [collect] sections of the custom.conf file are being honored by sos collect. However, the sos report execution on the local and remote node appears to not use any options from [report] and [plugin_options] sections of /etc/sos/custom.conf. It also does not appear to use the /etc/sos/custom.conf on the remote node, which the file exists with identical values to the local/collector node.

Instead what appears to happen is that the local and remote node both load the /etc/sos/sos.conf file values enable-logs, only-plugins running the 8 specific plugins, and the plugin options etc..

I suspect this is not the intended behavior. However, there is limited information on --config-file in man page for sos-collect or details for collect in man sos.conf. I believe collect should should execute report passing any of the included report and plugin options to the local and remote nodes, or maybe even pass --config-file /etc/sos/custom.conf to the remote nodes if it exists? Having the options for all reports come from /etc/sos/sos.conf to me seems unexpected.

@TurboTurtle @pmoravec

Is what I observe the intended behavior in this use case? Is my above assumption correct and I should open a ticket on the issue?

Thanks in advance.

[TurboTurtle]

I suspect this is not the intended behavior.

However, the sos report execution on the local and remote node appears to not use any options from [report] and [plugin_options] sections of /etc/sos/custom.conf.

Local not using custom.conf would be a bug. Remote not using it is not so much "intended behavior with collect" so much as we've not had the request to pass local config file options to remote.

It also does not appear to use the /etc/sos/custom.conf on the remote node, which the file exists with identical values to the local/collector node.

We don't pass the config-file value today, so this too is less "intended" behavior as it is "haven't thought to do it this way".

This would require some re-work on options handling, but I think this is what I'd like to see to support this kind of functionality:

• By default, local should use the given --config-file for both collect and the locally collected report. Remote nodes should act according to the (existing) config file on that host.
• A new --node-config-file (or whatever) option enables users to specify the config file on the remote nodes to use
• A new --inherit-config-file (or whatever) option causes collect to transfer the supplied config file to remote nodes and implicitly sets the per-node sos command to use that transferred config file

Totally open to hearing alternatives as to defaults or the general approach here.

For --inherit-config-file, I have a slight preference to writing the config file on the remote host to parsing the config and setting those options, but I'm not married to it and can be convinced otherwise.

[TrevorBenson]

I thought about how this ended up this way. For background, the initial changes to sos.conf were from a project/product. They modified the sos.conf as they wanted, all good (mostly). Later a new feature providing a "call home" like functionality was being added. The feature requires information beyond the limited 8 plugins. Thus the custom.conf is essentially a feature.conf.

By making the first change a new product.conf instead of modifying sos.conf the behavior is much closer to what is needed/expected. The options in the features custom.conf file are not honored on each node, but at least a "default behavior" from sos is observed which is much closer to the intended behavior.

---

Overall, I like your combination of suggestions. It opens up the most functionality of relying on an existing custom config on each host as well as inheriting one defined on the collector. This seems simpler to include a --config-file through to execute_sos_command() (or wherever needed) than attempting to pass the correct subset of self.opts and parsing the individual self.only_plugins, etc., differently than they already are.

At the moment it brings me to a partial/temporary standstill for the feature. The hosts being used are RHEL8 and this does not sound like a bugfix, but instead a feature enhancement. The former, I think, might get backported into RHEL 8 versions of sos during the Maintenance Support portion of the lifecycle. I suspect the latter would be added to only RHEL 9 and greater since the Full Support Phase on RHEL 8 ended in May this year.

I'm unfamiliar with Ubuntu's lifecycle and enhancement vs. maintenance updates enough to comment on them compared to RHEL.

[arif-ali]

I'm unfamiliar with Ubuntu's lifecycle and enhancement vs. maintenance updates enough to comment on them compared to RHEL.

To answer this specifically

All upstream releases:

• snap releases same time as upstream -> but we only migrate to latest/stable after some time
• We upload to Debian into unstable (sid) which after a few days migrates to testing (trixie)
• Ubuntu dev releases (currently 25.04 plucky) after auto-sync from Debian testing so takes a few more days

When it comes to stable releases, this is a different story, we do this typically every 6 months, as the verification process can some time. This is typically valid for all our supported distributions. For example our plan for 4.8.2 when it is released next month will update the following distributions

• 20.04 (focal)
• 22.04 (jammy)
• 24.04 (noble)
• 25.10 (oracular)

Hopefully that explains the life-cycle on what we plan

[TrevorBenson]

@TurboTurtle @arif-ali I have a working proof of concept implementing the suggested logic. I haven't opened a PR yet though, I was hoping to use a temporary directory for the inherit config logic for non local nodes. At the moment I have not tracked down where the code is for the remote temporary directory so that the remote config gets cleaned up at the end with everything else.

Otherwise its available for a quick peak in this branch/commit.

If anyone has a pointer what to look at for the remote node temporary directory I'd gladly accept a nudge in the right direction. Otherwise I'll try to take a look at it later this evening or tomorrow so I can get the PR opened.

[TrevorBenson]

#3852 opened for review.