Service Monitoring

The power of Icinga 2 lies in its modularity. There are thousands of community plugins available next to the standard plugins provided by the Monitoring Plugins project.

Requirements

Plugins

All existing Nagios or Icinga 1.x plugins work with Icinga 2. Community plugins can be found for example on Icinga Exchange.

The recommended way of setting up these plugins is to copy them to a common directory and create a new global constant, e.g. CustomPluginDir in your constants.conf configuration file:

# cp check_snmp_int.pl /opt/monitoring/plugins
# chmod +x /opt/plugins/check_snmp_int.pl

# cat /etc/icinga2/constants.conf
/**
 * This file defines global constants which can be used in
 * the other configuration files. At a minimum the
 * PluginDir constant should be defined.
 */

const PluginDir = "/usr/lib/nagios/plugins"
const CustomPluginDir = "/opt/monitoring/plugins"

Prior to using the check plugin with Icinga 2 you should ensure that it is working properly by trying to run it on the console using whichever user Icinga 2 is running as:

# su - icinga -s /bin/bash
$ /opt/monitoring/plugins/check_snmp_int.pl --help

Additional libraries may be required for some plugins. Please consult the plugin documentation and/or the included README file for installation instructions. Sometimes plugins contain hard-coded paths to other components. Instead of changing the plugin it might be easier to create a symbolic link to make sure it doesn't get overwritten during the next update.

Sometimes there are plugins which do not exactly fit your requirements. In that case you can modify an existing plugin or just write your own.

CheckCommand Definition

Each plugin requires a CheckCommand object in your configuration which can be used in the Service or Host object definition.

Please check if the Icinga 2 package already provides an existing CheckCommand definition. If that's the case, throroughly check the required parameters and integrate the check command into your host and service objects.

Please make sure to follow these conventions when adding a new command object definition:

Use command arguments whenever possible. The command attribute must be an array in [ ... ] for shell escaping.
Define a unique prefix for the command's specific arguments. That way you can safely set them on host/service level and you'll always know which command they control.
Use command argument default values, e.g. for thresholds.
Use advanced conditions like set_if definitions.

This is an example for a custom my-snmp-int check command:

object CheckCommand "my-snmp-int" {
  command = [ CustomPluginDir + "/check_snmp_int.pl" ]

  arguments = {
    "-H" = "$snmp_address$"
    "-C" = "$snmp_community$"
    "-p" = "$snmp_port$"
    "-2" = {
      set_if = "$snmp_v2$"
    }
    "-n" = "$snmp_interface$"
    "-f" = {
      set_if = "$snmp_perf$"
    }
    "-w" = "$snmp_warn$"
    "-c" = "$snmp_crit$"
  }

  vars.snmp_v2 = true
  vars.snmp_perf = true
  vars.snmp_warn = "300,400"
  vars.snmp_crit = "0,600"
}

For further information on your monitoring configuration read the Monitoring Basics chapter.

If you have created your own CheckCommand definition, please kindly send it upstream.

Plugin API

Currently Icinga 2 supports the native plugin API specification from the Monitoring Plugins project. It is defined in the Monitoring Plugins Development Guidelines.

Create a new Plugin

Sometimes an existing plugin does not satisfy your requirements. You can either kindly contact the original author about plans to add changes and/or create a patch.

If you just want to format the output and state of an existing plugin it might also be helpful to write a wrapper script. This script could pass all configured parameters, call the plugin script, parse its output/exit code and return your specified output/exit code.

On the other hand plugins for specific services and hardware might not yet exist.

Common best practices when creating a new plugin are for example:

Choose the pragramming language wisely
Scripting languages (Bash, Python, Perl, Ruby, PHP, etc.) are easier to write and setup but their check execution might take longer (invoking the script interpreter as overhead, etc.).
Plugins written in C/C++, Go, etc. improve check execution time but may generate an overhead with installation and packaging.
Use a modern VCS such as Git for developing the plugin (e.g. share your plugin on GitHub).
Add parameters with key-value pairs to your plugin. They should allow long names (e.g. --host localhost) and also short parameters (e.g. -H localhost)
-h|--help should print the version and all details about parameters and runtime invocation.
Add a verbose/debug output functionality for detailed on-demand logging.
Respect the exit codes required by the Plugin API.
Always add performance data to your plugin output

Example skeleton:

# 1. include optional libraries
# 2. global variables
# 3. helper functions and/or classes
# 4. define timeout condition

if (<timeout_reached>) then
  print "UNKNOWN - Timeout (...) reached | 'time'=30.0
endif

# 5. main method

<execute and fetch data>

if (<threshold_critical_condition>) then
  print "CRITICAL - ... | 'time'=0.1 'myperfdatavalue'=5.0
  exit(2)
else if (<threshold_warning_condition>) then
  print "WARNING - ... | 'time'=0.1 'myperfdatavalue'=3.0
  exit(1)
else
  print "OK - ... | 'time'=0.2 'myperfdatavalue'=1.0
endif

There are various plugin libraries available which will help with plugin execution and output formatting too, for example nagiosplugin from Python.

Note

Ensure to test your plugin properly with special cases before putting it into production!

Once you've finished your plugin please upload/sync it to Icinga Exchange. Thanks in advance!