Nagios also offers a tool for sending passive check results for hosts and services over a network. It is called NSCA (Nagios Service Check Acceptor). It can be used to send results from one Nagios instance to another. It can also be used by third-party applications running on different machines to send passive check results to a central Nagios server.

This mechanism includes password protection, along with encryption, to prevent injection of false results in to Nagios. In this way, NSCA communication sent over the Internet is more secure.

What are Passive Checks?

Previous parts of this book often mentioned Nagios performing checks on various software and machines. In such cases, Nagios decides when a check is to be performed, runs the check and stores the result. These types of checks are called Active Checks.

Nagios also offers another way to work with the statuses of hosts and services. It is possible to configure Nagios so that it will receive status information sent over a command pipe. In such a case, checks are done by other programs, and their results are sent to Nagios. Nagios will still handle all notifications, event handlers, and dependencies between hosts and services.

Active checks are most common in the Nagios world. They have a lot of advantages and some disadvantages. One of the problems is that such checks can take only a couple of seconds to complete—a typical timeout for an active check to complete is 10 or 30 seconds. In many cases, the time taken is not enough, as some checks need to be performed over a longer period of time to have satisfactory results. A good example might be running a check that takes several hours to complete—in this case, it does not make sense to raise the global service_check_timeout option, but rather to schedule these checks outside of Nagios and only report the results back to it.

There are also different types of checks including external applications or devices that want to report information directly to Nagios. This can be done to gather all critical errors to a single, central place. These types of checks are called Passive Checks.

For example, when a web application cannot connect to the database, it will let Nagios know about it immediately. It can also send reports after a database recovery, or periodically, even if connectivity to the database has been consistently available, so that Nagios has an up-to-date status. This can be done in addition to active checks, to identify critical problems earlier.

Another example is where an application already processes information such as network bandwidth utilization. In such a case, adding a module that reports current utilization along with the OK/WARNING/CRITICAL state to Nagios seems much easier than using active checks for the same job.

Often, there are situations where active checks obviously fit better. In other cases, passive checks are the way to go. In general, if a check can be done quickly and does not require long running processes, it should definitely be done as an active service. If the situation involves reporting problems from other applications or machines, it is definitely a use case for a passive check. In cases where the checks require the deployment of long-running processes or monitoring information constantly, this should be done as a passive service.

Another difference is that active checks require much less effort to be set up when compared to passive checks. In the first case, Nagios takes care of the scheduling, and the command only needs to perform the actual checks and mark the results as OK/WARNING/CRITICAL based on how a check command is configured. Passive checks require all the logic related to what should be reported and when it should be checked to be put in an external application. This usually calls for some effort.

The following diagram shows how both active and passive checks are performed by Nagios. It shows what is performed by Nagios in both cases and what needs to be done by the check command or an external application for passive checks.

Nagios also offers a way of combining the benefits of both active and passive checks. Often, you have situations where other applications can report if a certain service is working properly or not. But if the monitoring application is not running or some other issue prevents it from reporting, Nagios can use active checks to keep the service status up–to-date.

A good example would be a server that is a part of an application, processing job queues using a database. It can report each problem when accessing the database. We want Nagios to monitor this database, and as the application is already using it, we can add a module that reports this to Nagios.

The application can also periodically let Nagios know if it succeeded in using the database without problems. However, if there are no jobs to process and the application is not using it, Nagios will not have up-to-date information about the database.

Configuring Passive Checks

The first thing that needs to be done in order to use passive checks for your Nagios setup is to make sure that you have the following options in your main Nagios configuration file:

accept_passive_service_checks=1
accept_passive_host_checks=1

It would also be good to enable the logging of incoming passive checks—this makes determining the problem of not processing a passive check much easier. The following directive allows it:

log_passive_checks=1

Setting up hosts or services for passive checking requires an object to be defined and set up so as not to perform active checks. The object needs to have the passive_checks_enabled option set to 1 for Nagios to accept passive check results over the command pipe.

The following is an example of the required configuration for a host that accepts passive checks and has active checks disabled:

define host
{
use                           generic-host
host_name                     linuxbox01
address                       10.0.2.1
active_checks_enabled         0
passive_checks_enabled        1 
}

Configuring services is exactly the same as with hosts. For example, to set up a very similar service, all we need to do is to use the same parameters as those for the hosts:

define service
{
use                              ping-template
host_name                        linuxbox01
service_description              PING
active_checks_enabled            0
passive_checks_enabled           1
}

In this case, Nagios will never perform any active checks on its own and will only rely on the results that are passed to it.

We can also configure Nagios so that if no new information has been provided within a certain period of time, it will use active checks to get the current status of the host or service. If up-to-date information has been provided by a passive check during this period, then it will not perform active checks.

In order to do this, we need to enable active checks by setting the active_checks_enabled option to 1 without specifying the normal_check_interval directive. For Nagios to perform active checks when there is no up-to-date result from passive checks, you need to set the check_freshness directive to 1 and set freshness_threshold to the time period after which a check should be performed. The time performed is specified in seconds.

The first parameter tells Nagios that it should check whether the results from the checks are up-to-date. The next parameter specifies the number of seconds after which Nagios should consider the results to be out of date. Attributes can be used for both hosts and services.

A sample definition for a host that runs an active check if there has been no result provided within the last two hours:

define host
{
use                             generic-host
host_name                       linuxbox02
address                         10.0.2.2
check_command                   check-host-alive
check_freshness                 1
freshness_threshold             7200
active_checks_enabled           1
passive_checks_enabled          1
}

The following is an illustration showing when Nagios would invoke active checks:

Each time there is at least one passive check result that is still valid (i.e., was received within the past two hours), Nagios will not perform any active checks. However, two hours after the last passive or active check result was received, Nagios would perform an active check to keep the results up-to-date.

Passive Checks—Hosts

Nagios allows applications and event handlers to send out passive check results for host objects. In order to use them, the host needs to be configured to accept passive checks results.

In order to be able to submit passive check results, we need to configure Nagios to allow the sending of passive check results, and set the host objects to accept them.

Submitting passive host check results to Nagios requires sending a command to the Nagios external command pipe. This way, the other applications on your Nagios server can report the status of the hosts.

The command to submit passive checks is PROCESS_HOST_CHECK_RESULT (visit http://www.nagios.org/developerinfo/externalcommands/commandinfo.php?command_id=115). This command accepts the host name, status code, and the textual output from a check. The host status code should be 0 for an UP state, 1 for DOWN and 2 for an UNREACHABLE state.

The following is a sample script that will accept the host name, status code, and output from a check and will submit these to Nagios:

#!/bin/sh

NOW='date +%s'
HOST=$1
STATUS=$2
OUTPUT=$3

echo "[$NOW] PROCESS_HOST_CHECK_RESULT;$HOST;$STATUS;$OUTPUT" 
     >/var/nagios/rw/nagios.cmd

exit 0

As an example of the use of this script, the command that is sent to Nagios for host01, status code 2 (UNREACHABLE) and output router 192.168.1.2 down would be as follows:

[1206096000] PROCESS_HOST_CHECK_RESULT;host01;2;router
192.168.1.2 down

When submitting results, it is worth noting that Nagios might take some time to process them, depending on the intervals between Nagios’ checks of the external command pipe.

Unlike active checks, Nagios will not take network topology into consideration by default. This is very important in situations where a host behind a router is reported to be down because the router is actually down.

By default, Nagios handles results from active and passive checks differently. When Nagios plans and receives results from active checks, it takes the actual network topology into consideration and performs a translation of the states based on this. This means that if Nagios receives a result indicating that a host is DOWN, it assumes that all child hosts are in an UNREACHABLE state.

When a passive result check comes in to Nagios, Nagios expects that the result already has a network topology included. When a host is reported to be DOWN as a passive check result, Nagios does not perform a translation from DOWN to UNREACHABLE. Even if its parent host is currently DOWN, the child host state is also stored as DOWN.

The following illustration shows how results from active and passive checks are treated differently by Nagios:

 

 

In both the cases, a check result stating that the host is down is received by Nagios. When it comes in as a passive check, no state translation is done and Nagios stores the host and all child nodes being down. When it is an active check result, Nagios takes the fact that switch1 is down into account and maps the child node’s result into an UNREACHABLE state.

How Nagios process handles passive check results can be defined in the main Nagios configuration file. In order to make Nagios treat passive host check results in the same way as active check results, we need to enable the following option:

translate_passive_host_checks=1

By default, Nagios treats host results from passive checks as hard results. This is because, very often, passive checks are used to report host and service statuses from other Nagios instances. In such cases, only reports regarding hard state changes are propagated across Nagios servers.

If you want Nagios to treat all passive check results for hosts as if they were soft results, you need to enable the following option in the main Nagios configuration file:

passive_host_checks_are_soft=1

Passive Checks—Services

Passive service checks are very similar to passive host checks. In both the cases, the idea is that Nagios receives information about host statuses over the external commands pipe.

As with passive checks of hosts, all that is needed is to enable the global Nagios option to accept passive check results, and also enable this option for each service that should allow the passing of passive check results.

The results are passed to Nagios in the same way as they are passed for hosts. A command to submit passive checks is PROCESS_SERVICE_CHECK_RESULT (visit http://www.nagios.org/developerinfo/externalcommands/commandinfo.php?command_id=114). This command accepts the host name, service description, status code, and the textual output from a check. Service status codes are the same as those for active checks—0 for OK, 1 for WARNING, 2 for CRITICAL, and 3 for an UNKNOWN state.

The following is a sample script that will accept the host name, status code, and output from a check and will submit these to Nagios:

#!/bin/sh

CLOCK='date +%s'
HOST=$1
SVC=$2
STATUS=$3
OUTPUT=$4

echo "[$CLOCK] PROCESS_SERVICE_CHECK_RESULT;$HOST;$SVC;$STATUS;
       $OUTPUT"
     >/var/nagios/rw/nagios.cmd

exit 0

As an example of the use of this script, the command that is sent to Nagios for host01, service PING, status code 0 (OK) and output RTT=57 ms is as follows:

[1206096000] PROCESS_SERVICE_CHECK_RESULT;host01;PING;0;RTT=57 ms

A very common scenario for using passive checks is a check that takes a very long time to complete.

As with submitting host check results, it is worth mentioning that Nagios will take some time to process passive check results as they are polled periodically from the external commands pipe.

A major difference between hosts and services is that service checks differentiate between soft and hard states. When new information regarding a service gets passed to Nagios via the external commands pipe, Nagios treats it the same way as if it had been received by an active check.

If a service is set up with a max_check_attempts directive of 5, then the same number of passive check results would need to be passed in order for Nagios to treat the new status as a hard state change.

Passive service checks are often used to report the results of long lasting tests asynchronously. A good example of such a test is checking whether there are bad blocks on a disk. This requires trying to read the entire disk directly from the block device (such as /dev/sda1) and checking if the attempt has failed. This can’t be done as an active check as reading the device takes a lot of time to complete—larger disks might require several hours to complete.

For this reason, the only way to perform such a check is to schedule them from the system—for example, using the cron daemon (visit http://man.linuxquestions.org/index.php?query=cron). The script should then post results to the Nagios daemon.

The following is a script that runs the dd system command (visit http://man.linuxquestions.org/index.php?query=dd) to read an entire block device. Based on whether the read was successful or not, the appropriate status code, along with plugin output, is sent out.

#!/bin/sh

 SVC=$1
 DEVICE=$2
 TMPFILE=/tmp/ddlog.$$
 NOW='date +%s'
 PREFIX="['date +%s'] [$NOW] PROCESS_SERVICE_CHECK_
RESULT;localhost;$SVC"
 
 # try to read the device
 dd if=$DEVICE of=/dev/null >$TMPFILE 2>&1
 CODE=$?
 RESULT='grep copied <$TMPFILE'
 rm $TMPFILE

 if [ $CODE == 0 ] ; then
    echo "$PREFIX;0;$RESULT"
 else
    echo "$PREFIX;2;Error while checking device $DEVICE"
 fi

 exit 0