Skip to content

Latest commit

 

History

History
228 lines (186 loc) · 12.8 KB

Troubleshooting.md

File metadata and controls

228 lines (186 loc) · 12.8 KB
Raw

Troubleshooting Guide for OMS Agent for Linux

The following document provides a quick series of steps and procedures to diagnose and troubleshoot common issues. If none of these steps work for you the following channels for help are also available

Table of Contents

Important Log Locations

File Path
OMS Agent for Linux Log File /var/opt/microsoft/omsagent/log/omsagent.log
OMS Agent Configuration Log File /var/opt/microsoft/omsconfig/omsconfig.log

Important Configuration Files

Catergory File Location
Syslog /etc/syslog-ng/syslog-ng.conf or /etc/rsyslog.conf or /etc/rsyslog.d/95-omsagent.conf
Performance, Nagios, Zabbix, OMS output and general agent /etc/opt/microsoft/omsagent/conf/omsagent.conf
Additional configurations /etc/opt/microsoft/omsagent/conf.d/*.conf

Note: Editing configuration files for performance counters & syslog is overwritten if Portal Configuration is enabled. Disable configuration in the OMS Portal (all nodes) or for single nodes run the following:

sudo su omsagent -c /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable

Enable Debug Logging

OMS output plugin debug

FluentD allows for plugin specific logging levels allowing you to specify different log levels for inputs and outputs. To specify a different log level for OMS output edit the general agent configuration at /etc/opt/microsoft/omsagent/conf/omsagent.conf:

In the OMS output plugin, near the bottom of the configuration file, change the log_level property from info to debug

<match oms.** docker.**>
 type out_oms
 log_level debug
 num_threads 5
 buffer_chunk_limit 5m
 buffer_type file
 buffer_path /var/opt/microsoft/omsagent/state/out_oms*.buffer
 buffer_queue_limit 10
 flush_interval 20s
 retry_limit 10
 retry_wait 30s
</match>

Debug logging allows you to see batched uploads to the OMS Service seperated by type, number of data items, and time taken to send:

Example debug enabled log:

Success sending oms.nagios x 1 in 0.14s
Success sending oms.omi x 4 in 0.52s
Success sending oms.syslog.authpriv.info x 1 in 0.91s

Verbose output

Instead of using the OMS output plugin you can also output Data Items directly to stdout which is visible in the OMS Agent for Linux log file.

In the OMS general agent configuration file at /etc/opt/microsoft/omsagent/conf/omsagent.conf:

Comment out the OMS output plugin by adding a # in front of each line

#<match oms.** docker.**>
#  type out_oms
#  log_level info
#  num_threads 5
#  buffer_chunk_limit 5m
#  buffer_type file
#  buffer_path /var/opt/microsoft/omsagent/state/out_oms*.buffer
#  buffer_queue_limit 10
#  flush_interval 20s
#  retry_limit 10
#  retry_wait 30s
#</match>

Below the output plugin, uncomment the following section by removing the # in front of each line

<match **>
  type stdout
</match>

My forwarded Syslog messages are not showing up!

Probable Causes

  • The configuration applied to the Linux server does not allow collection of the sent facilities and/or log levels
  • Syslog is not being forwarded correctly to the Linux server
  • The number of messages being forwarded per second are too great for the base configuration of the OMS Agent for Linux to handle

Resolutions

  • Check that the configuration in the OMS Portal for Syslog has all the facilities and the correct log levels
    • OMS Portal > Settings > Data > Syslog
  • Check that native syslog messaging daemons (rsyslog, syslog-ng) are able to recieve the forwarded messages
  • Check firewall settings on the Syslog server to ensure that messages are not being blocked
  • Simulate a Syslog message to OMS using logger command
    • `logger -p local0.err "This is my test message"

I'm unable to connect through my proxy to OMS

Probable Causes

  • The proxy specified during onboarding was incorrect
  • The OMS Service Endpoints are not whitelistested in your datacenter

Resolutions

  • Re-onboard to the OMS Service with the OMS Agent for Linux using the following command with the option -v enabled. This allows verbose output of the agent connecting through the proxy to the OMS Service.
    • /opt/microsoft/omsagent/bin/omsadmin.sh -w <OMS Workspace ID> -s <OMS Workspace Key> -p <Proxy Conf> -v
    • Review documentation for OMS Proxy located here
  • Double check that the following OMS Service endpoints are whitelisted
Agent Resource Ports
*.ods.opinsights.azure.com Port 443
*.oms.opinsights.azure.com Port 443
ods.systemcenteradvisor.com Port 443
*.blob.core.windows.net/ Port 443

I'm getting a 403 when I'm trying to onboard!

Probable Causes

  • Date and Time is incorrect on Linux Server
  • Workspace ID and Workspace Key used are not correct

Resolution

  • Check the time on your Linux server with the command date. if the data is +/- 15 minutes from current time then onboarding fails. To correct this update, the date and/or timezone of your Linux server.
  • New! The latest version of the OMS Agent for Linux now notifies you if the time skew is causing the onboarding failure
  • Re-onboard using correct Workspace ID and Workspace Key instructions

I'm seeing a 500 Error and 404 Error in the log file right after onboarding

This is a known issue an occurs on first upload of Linux data into an OMS workspace. This does not affect data being sent or service experience.

My Nagios data is not showing up in the OMS Portal!

Probable Causes

  • omsagent user does not have permissions to read from Nagios log file
  • Nagios source and filter have not been uncommented from omsagent.conf file

Resolutions

  • Add omsagent user to read from Nagios file instructions
  • In the OMS Agent for Linux general configuration file at /etc/opt/microsoft/omsagent/conf/omsagent.conf ensure that both the Nagios source and filter are uncommented
<source>
  type tail
  path /var/log/nagios/nagios.log
  format none
  tag oms.nagios
</source>

<filter oms.nagios>
  type filter_nagios_log
</filter>

I'm not seeing any Linux data in the OMS Portal

Probable Causes

  • Onboarding to the OMS Service failed
  • Connection to the OMS Service is blocked
  • OMS Agent for Linux data is backed up

Resolutions

  • Check if onboarding the OMS Service was successful by checking if the following file exists: /etc/opt/microsoft/omsagent/conf/omsadmin.conf
  • Re-onboard using the omsadmin.sh command line instructions
  • If using a proxy, check proxy troubleshooting steps aboce
  • In some cases, when the OMS Agent for Linux cannot talk to the OMS Service, data on the Agent is backed up to the full buffer size: 50 MB. The OMS Agent for Linux should be restarted by running the following command service omsagent restart or systemctl restart omsagent.
  • Note: This issue is fixed in Agent version >= 1.1.0-28

My portal side configuration for (Syslog/Linux Performance Counter) is not being applied

Probable Causes

  • The OMS Agent for Linux Configuration Agent has not picked up the latest portal side configuration
  • The changed settings in the portal were not applied

Resolutions

Background: omsconfig is the OMS Agent for Linux configuration agent that looks for new portal side configuration every 5 minutes. This configuration is then applied to the OMS Agent for Linux configuration files located at /etc/opt/microsoft/omsagent/conf/omsagent.conf.

  • In some cases the OMS Agent for Linux configuration agent might not be able to communicate with the portal configuration service resulting in latest configuration not being applied.

  • Check that the omsconfig agent is installed

  • dpkg --list omsconfig or rpm -qi omsconfig

  • If not installed, reinstall the latest version of the OMS Agent for Linux

  • Check that the omsconfig agent can communicate with the OMS Portal Service

    • Run the following command sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/GetDscConfiguration.py'
    • This command returns the Configuration that agent sees from the portal including Syslog settings, Linux Performance Counters, and Custom Logs
    • If this command fails run the following command sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/PerformRequiredConfigurationChecks.py. This command forces the omsconfig agent to talk to the OMS Portal Service and retrieve latest configuration.

I'm not seeing my Linux Custom Log Data in the OMS Potal

Probable Causes

  • Onboarding to OMS Service failed
  • The setting "Apply the following configuration to my Linux Servers" has not been check marked
  • omsconfig has not picked up the latest Custom Log from the portal
  • OMS Agent for Linux user omsagent is unable to access the Custom Log due to permissions or not being found
  • [DATETIME] [warn]: file not found. Continuing without tailing it.
  • [DATETIME] [error]: file not accessible by omsagent.
  • Known Issue with Race Condition fixed in OMS Agent for Linux version 1.1.0-217

####Resolutions

  • Check if onboarding the OMS Service was successful by checking if the following file exists: /etc/opt/microsoft/omsagent/conf/omsadmin.conf

  • Re-onboard using the omsadmin.sh command line instructions

  • In the OMS Portal under Settings ensure that the following checkbox is checked

  • Check that the omsconfig agent can communicate with the OMS Portal Service

    • Run the following command sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/GetDscConfiguration.py'
    • This command returns the Configuration that agent sees from the portal including Syslog settings, Linux Performance Counters, and Custom Logs
    • If this command fails run the following command sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/PerformRequiredConfigurationChecks.py. This command forces the omsconfig agent to talk to the OMS Portal Service and retrieve latest configuration.

Background: Instead of the OMS Agent for Linux user running as a privileged user, root - The OMS Agent for Linux runs as the omsagent user. In most cases explicit permission must be granted to this user in order for certain files to be read.

  • To grant permission to omsagent user run the following commands

  • Add the omsagent user to specific group sudo usermod -a -G <GROUPNAME> <USERNAME>

  • Grant universal read access to the required file sudo chmod -R ugo+rw <FILE DIRECTORY>

  • There is a known issue with a Race Condition in OMS Agent for Linux version <1.1.0-217. After updating to the latest agent run the following command to get the latest version of the output plugin

  • sudo cp /etc/opt/microsoft/omsagent/sysconf/omsagent.conf /etc/opt/microsoft/omsagent/conf/omsagent.conf