dsc_host keeps crashing

[johanburati]

dsc_host keeps crashing.

• dsc_host Version 1.0.0000 from omsconfig-1.1.1-930.x86_64 package
• A couple of ppl have already reported this issue: dsc_host crashing on CentOS 8 OMS-Agent-for-Linux#1282
• I was able to reproduce it on Centos 8.1, Centos 8.2, RHEL 8.1 and RHEL 8.2:

kernel: dsc_host[9960]: segfault at 28 ip 0000000000433fb4 sp 00007ffc964cb850 error 4 in dsc_host[400000+94000]
kernel: Code: 89 4c 24 30 64 48 8b 04 25 28 00 00 00 48 89 84 24 a8 00 00 00 31 c0 4d 85 e4 b0 04 0f 84 94 01 00 00 49 c7 04 24 00 00 00 00 <48> 8b 6f 28 49 89 fe 49 8b 50 18 48 8d 05 3e 19 04 00 49 89 cf 4c

[Klaas-]

We also opened a support case with Microsoft

[johanburati]

It's kind of the same thing as for RHEL 8 and other extensions: Azure/WALinuxAgent#1719

WORKAROUND

This should prevent dsc_host from crashing and filling up the fs with core files until the devs push a new release that fix the issue for good.

python is missing on those images, you can create and make a link and make it point to python3 with the following command:

sudo alternatives --set python /usr/bin/python3

Or if you are seeing issue with python3, you can install python2 and make python point to python2 :

sudo dnf install python2 -y
sudo alternatives --set python /usr/bin/python2

Tried it on a couple of VMs and did not see any crash after using either option.

@Klaas- could you try on your VMs and confirm ?

[Klaas-]

Hi,
I would not consider this a solution but more a workaround :) Waagent/oms do not need this kind of workaround anymore. I am guessing you should either default to python3 directly or build something to recognize the python version:
https://github.com/microsoft/OMS-Agent-for-Linux/blob/73d35e11a5ec4a2541731d737816d3ad2ff6018f/tools/LogCollector/source/omslinux_agentlog.sh#L49-L68
https://github.com/microsoft/OMS-Agent-for-Linux/blob/b6f2658136f5f84974be7401d9f18945ef0bf541/installer/bundle/bundle_skel.sh#L652-L661

Red Hat does not recommend to set an unversioned python link ( see man unversioned-python ) on a recent rhel.

[johanburati]

@Klass- right, I changed "solution" for "workaround", at least it will prevent the program from crashing and generating core files until the developers release a new version that fix the issue. I hope this will help other ppl having this issue.

[robfordww]

Do you know if I somehow can disable this feature on Ubuntu 20.04 running on Azure. It fills up the entire dmesg log system. I could try the python thing mentioned above, but why not remove the entire thing?

[johanburati]

@robfordww You need to disconnect the VMs within your OMS workplace and all the OMS components will get uninstalled. Please cut a support ticket if you need help with that.

[sbonds]

This directly relates to MS case 121012721003494. I'll use the workaround for now, but will also be eagerly awaiting a RHEL8-compatible solution. (Honestly, though, why did RedHat need to break "#!/usr/bin/env python"? That's the method that's supposed to always work!)

[sbonds]

Note that the advice to use python3 as the alternative seems to work best on RHEL8 and is the closest to the RHEL8 general guidelines to extremely preferentially use python3. However, this advice contradicts the advice given in this Microsoft document:

https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux#python-requirement

That advice is given in the context of "if you must use an older agent" and I'll suggest an improvement to make that clearer.

[Klaas-]

@sbonds I think this is a more general recommendation by python developers -- see "man unversioned-python" on an EL8 system or https://www.python.org/dev/peps/pep-0394/

[Klaas-]

This seems to be fixed by omsconfig-1.1.1-930.x86_64, you should no longer recommend the python workarounds in docs :)