Selective alerting

The forum for help and support with FreeNATS as well as any useful hints and tips
Post Reply
paullanders
Posts: 92
Joined: Thu Sep 04, 2008 9:48 pm

Selective alerting

Post by paullanders » Thu Oct 02, 2008 9:58 pm

Dave, I have found that most of my server "failures" are simple glitches, and most of them occur during the night when I'm asleep and don't really care for my cell to ring for a hiccup :) Is it possible to configure FreeNATS to continue testing and recording 24/7 but suppress alerts during a particular time-frame? For example, even if a real failure occurs at 3:00 a.m., FreeNATS would continue silently testing and recording but my first alert would occur at the 7:00 am test?

Thanks!

dave
Site Admin
Posts: 260
Joined: Fri May 30, 2008 9:09 pm
Location: UK
Contact:

Re: Selective alerting

Post by dave » Thu Oct 02, 2008 10:21 pm

Hi,

Yes, sort of. You can (from a recent update) give each alert action a schedule when it will run (the same as test schedules). With this you can have the nodes tested 24x7 but the alert action only active 0800-1900 or whatever.

What this won't do is send an alert at that point if the thing has already failed so if a server fails at 3am then the 3am alert is thrown away and at 8am the alert is already open.

If you want it to alert you still i.e. if a server fails 3am-3.01am then ignore but if it failed at 3am and is still failed then alert at 8am then I think some sort of event handler/queuer or cron script (to close open alerts at 7.59) would be the way to go - I should be able to knock something together pretty easily.

Cheers,

Dave.

paullanders
Posts: 92
Joined: Thu Sep 04, 2008 9:48 pm

Re: Selective alerting

Post by paullanders » Thu Oct 02, 2008 10:52 pm

Yeah, that would be great! But no hurry.

Thanks, Dave!

dave
Site Admin
Posts: 260
Joined: Fri May 30, 2008 9:09 pm
Location: UK
Contact:

Re: Selective alerting

Post by dave » Fri Oct 03, 2008 11:13 pm

Okay,

The best way to implement this is by turning the nodealert flag on and off (the "node alerts" option in the interface). This means that though the node is tested, and will display/record failures no alert is created.

To allow this to work easily I have made a tiny change to the alert code. Because it's core code this is released as an alpha (1.01.8a) now available for download.

What you then want to do is use the following script - I recommend you put it in server/base/site/ (not in the tests or events sub-dirs) or somewhere totally outside of the FreeNATS structure (to avoid it ever being overwritten) - called something like set_node_alert.php

Code: Select all

<?php
// set_node_alert - sets node alerting on or off for all or a set of nodes
$BaseDir="../"; // must point to server/base/ with trailing /
require($BaseDir."nats.php");
$NATS->Start();
$exit=false;
if ($argc<2) $exit=true; // not enough options
else if ($argv[1]=="on") $nodealert=1;
else if ($argv[1]=="off") $nodealert=0;
else $exit=true; // illegal option

if ($exit)
	{
	echo "Usage: php set_node_alert.php on|off [nodes...]\n";
	echo "Sets nodealert on all or specified nodes\n\n";
	echo "php set_node_alert.php on\n";
	echo " turns on nodealert on all nodes\n\n";
	echo "php set_node_alert.php off node1 node2\n";
	echo " turns off nodealerts on node1 and node2\n\n";
	exit();
	}

$data=array("nodealert" => $nodealert); // data array
$nodes=array();

if ($argc>2) // command-line specified nodes
	{
	for ($a=2; $a<$argc; $a++)
		$nodes[]=ss($argv[$a]); // safe-string it
	}
else // all nodes
	{
	$query="SELECT nodeid FROM fnnode";
	$result=$NATS->DB->Query($query);
	while ($noderow=$NATS->DB->Fetch_Array($result))
		$nodes[]=$noderow['nodeid'];
	$NATS->DB->Free($result);
	}

echo "Setting nodealert=".$nodealert."\n";
foreach($nodes as $node)
	{
	echo $node;
	$NATS->SetNode($node,$data);
	echo "\n";
	}
$NATS->Stop();
	
?>
Hopefully it's pretty self-explanitory. You'll want to setup a CRON job to run it whenever you want to turn alerts on or off. With no nodes individually specified it does them all. It uses the API and so it not very efficient but should continue to work even if the underlying method changes.

Hope that's ok - let me know how you get on.

Cheers,

Dave.

paullanders
Posts: 92
Joined: Thu Sep 04, 2008 9:48 pm

Re: Selective alerting

Post by paullanders » Fri Oct 17, 2008 10:51 pm

Hi Dave.

I think I will give this a try, but I have 2 questions:

1. You said "by turning the nodealert flag on and off (the "node alerts" option in the interface)". I'm not seeing a "node alerts" option in the interface?

2. How would I be selective so that some nodes continue to send alerts but others don't?

Thanks! I don't know how I ever lived without FreeNATS! :-)

dave
Site Admin
Posts: 260
Joined: Fri May 30, 2008 9:09 pm
Location: UK
Contact:

Re: Selective alerting

Post by dave » Sat Oct 18, 2008 11:32 am

Hi Paul,

Sorry - in the interface it's the "alerts active" checkbox.

To use the script:

First put the script into server/base/site called something like set_node_alert.php

To use the script from the command line you do:

php set_node_alert.php on|off [nodes...]

So...

php set_node_alert.php off node1 node2 node3
Turns off "alerts active" for node1, node2 and node3

php set_node_alert.php on
Turns on "alerts active" for ALL nodes as none are specified

What you'll want to do is create a CRON script that at say 17:00 turns off "alerts active" on the nodes in question and then turns it back on at 08:00 or whenever you want.

e.g.

17:00 script

Code: Select all

#!/bin/sh
cd /opt/freenats/server/base/site
php set_node_alerts.php off node1 node2 node3 > /dev/null
08:00 script

Code: Select all

#!/bin/sh
cd /opt/freenats/server/base/site
php set_node_alerts.php on node1 node2 node3 > /dev/null
Hope that helps,

Dave.

paullanders
Posts: 92
Joined: Thu Sep 04, 2008 9:48 pm

Re: Selective alerting

Post by paullanders » Tue Oct 21, 2008 10:15 pm

Hi Dave.

I've tested this using a 10 minute cron interval between OFF and ON under 1.01.9/a and it works just fine. A failed node was detected in Live Monitor but no alert was sent. After the script toggled alerting back ON I then received an an alert. Beautiful!

I will test it "for real" tonight.

Thanks!

Paul

UPDATE - I tested overnight and it performed perfectly. Thank you!

dave
Site Admin
Posts: 260
Joined: Fri May 30, 2008 9:09 pm
Location: UK
Contact:

Re: Selective alerting

Post by dave » Mon Oct 27, 2008 5:18 pm

Excellent news! Glad it worked.

Cheers,

Dave.

Post Reply