Check process output with Monit workaround

We are monitoring our client's daemons with Monit. This works just fine, but sometimes we would like to run a process and have monit do something depending on its result which is not supported...

So you could just go and run the command from a cron script and do all actions from the cron script. But if you restart a service maybe you run into problems/conflicts with monit, and if you want alerts you have to configure them some other way, so it ends up bing a bit messy.

Another option would be write status to syslog and check for matching lines in monit but this would cause alerts/actions to be performed every time the matching line is logged.

So we're using the following workaround:

• Run command periodically with cron and save output to a status-file


• Monitor the status-file checking for a regular expressing

Monit's file content checking is thought for logs - so it tries to read from where it left off last time unless the file shrinks or the inode changes. To make monit reread the file everytime it changes we're using the following wrapper script "monitwrapper.sh":

#!/bin/sh

FILE=$1
shift

#make sure file exists
touch $FILE

#run command and gather output (exit status and stdout)
NEWSTDOUT=`$@`
STATUS=$?
NEWV=$STATUS:$NEWSTDOUT

#get output of last run
OLDV=`cat $FILE`

#if status has changed:
#change inode of the file to make monit reread
if [ "$NEWV" != "$OLDV" ]; then
    echo $NEWV > $FILE.tmp
    mv $FILE.tmp $FILE 
fi

This leaves us with the return status and the stdout written to the status file only if the content is different.

We can run this from cron:

*/5 * * * * root monitwrapper.sh /var/run/teststatus testcmd

And monitor with monit:

check file status with path /var/run/teststatus
    if not match "^0" then alert