May 252015
 

For my current task at work – moving away from CA Spectrum to Zabbix – I’ve had to integrate NMIS with Zabbix for SMS alerting as we wanted all alerts to filter through Zabbix for easy tracking of SMSes, and also for the ServiceNow integration functions that I have built into my Zabbix instance.

To build the integration between NMIS and Zabbix, I had to create a custom script that NMIS would call which would then send an SNMP trap to Zabbix for processing.

On the Zabbix side, I needed to set up snmptt to process the traps so that Zabbix will recognise them as traps and action them as alerts.

The following sections will detail the bits I’ve setup to get this integration working

NMIS Configuration

NMIS Alert Script

This script is what NMIS calls when an alert is generated.

I’ve named this script “snmptrap.pm”, it’s based off the example script in /usr/local/nmis8/lib/Notify on the appliance.
If you name it something different, make sure you update the relevant lines in the script file.

package Notify::snmptrap; ## Update this if you change the name

require 5;

use strict;

use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);

use Exporter;
use JSON::XS;
use File::Path;
use Net::SNMP;

$VERSION = 1.00;

@ISA = qw(Exporter);

@EXPORT = qw(
snmptrap ## Update this if you change the name
);

@EXPORT_OK = qw( );

my $dir = "/tmp/customsnmptrap"; ## This is the log directory, and can also be changed

sub sendNotification {
my %arg = @_;
my $contact = $arg{contact};
my $event = $arg{event};
my $message = $arg{message};

my $trapdestination = "10.0.0.5"; ## This should be your Zabbix server IP Address
my $trapcommunity = "public"; # Use your community string here
my $oid = "1.3.6.1.4.1.4818.1"; # I've used the OPMANTEK MIB OID here, but you can use your own if you want

if ( not -d $dir ) {
my $permission = "0770";

my $umask = umask(0);
mkpath($dir,{verbose => 0, mode => oct($permission)});
umask($umask);
}

# add the time now to the event data.
$event->{time} = time;

$event->{email} = $contact->{Email};
$event->{mobile} = $contact->{Mobile};

my ($sess, $err) = Net::SNMP->session(
-hostname => $trapdestination,
-version => 1, #trap() requires v1
-port => 162
);

if (!defined $sess) {
print "Error connecting to target ". $trapdestination . ": ". $err;
next;
}

my @vars = qw();
my $varcounter = 1;

push (@vars, $oid . '.' . $varcounter);
push (@vars, OCTET_STRING);

# This is where you set up the variables for the SNMP Trap message
push (@vars,$event->{level}.' : '.$event->{node}.' : '.$event->{element}.' : '.$event->{event});

my $result = $sess->trap(
-varbindlist => \@vars,
-enterprise => $oid,
-specifictrap => 1,
);

if (! $result)
{
print "An error occurred sending the trap: " . $sess->error();
}

my $fcount = 1;
my $file ="$dir/$event->{startdate}-$fcount.json";
while ( -f $file ) {
++$fcount;
$file ="$dir/$event->{startdate}-$fcount.json";
}

my $mylog;
$mylog->{contact} = $contact;
$mylog->{event} = $event;
$mylog->{message} = $message;

open(LOG,">$file") or logMsg("ERROR, can not write to $file");
print LOG JSON::XS->new->pretty(1)->encode($mylog);
close LOG;
# good to set permissions on file.....
}
1;

NMIS Alert Configuration

Now NMIS needs to be setup to call the new snmptrap.pm script when an alert is generated.
To setup NMIS, you’ll need to locate the escalation that requires the SNMP trap.
Open the escalations table by going to Setup --> Emails, Notifications and Escalation.
In the escalation table, find the alert that you need SNMP traps for, and add this to the escalation level that requires traps –
snmptrap:Contact – Replace Contact with a contact in NMIS.

Zabbix Configuration

Once NMIS has been setup to send traps to the Zabbix server, some configuration needs to be done on the Zabbix side.

SNMPTT Configuration

You’ll need to set up SNMPTT to receive SNMP traps from the OID you’re using in the snmptrap script.
In this example, I’m just going to set up a catchall to catch the SNMP traps, however you can use SNMPTT to parse different traps and generate different alerts.

Create a file in /etc/snmp called snmptt.conf.catch
In that file, put the following lines in
EVENT general .* "SNMP Catchall" Normal
FORMAT ZBXTRAP $aA $ar

You’ll also need to modify /etc/snmp/snmptt.ini to add the newly created file to the snmptt_conf_files configuration variable.
This can be done by appending the path to the list like this –

snmptt_conf_files = <<END
/etc/snmp/snmptt.conf.catch
END

If there are already lines there, then add the line to the block of text right before the last END

Zabbix Item Configuration

I’ve used the basic snmptrap.fallback method to catch all traps, but you can set up specific alerts in SNMPTT to generate different messages.
On the NMIS server, add an item of type SNMP Trap, and with a key of snmptrap.fallback.
This item will now get any SNMP traps from the Server, you can create a trigger if required to alarm on the SNMP traps, or just keep them for history.

Share
May 112015
 

I never noticed this before since I didn’t use Zabbix for production monitoring, but Zabbix out of the box does not have any alerts set up to tell you that an SNMP agent is unresponsive.
This isn’t an issue if you’re doing monitoring using the Zabbix Agent, or just monitoring server ups and downs, but when you’re using Zabbix to gather metrics such as CPU and Memory usage, this can become an issue.

The solution is to create a trigger for SNMP hosts to alert when Zabbix does not get any data for more than a certain amount of time.

Creating the trigger

I’ve chosen to create the trigger on the Template SNMP Generic template so that all SNMP devices will get this trigger.
To create the trigger, click on Configuration ==> Templates, and then find Template SNMP Generic. To the right, click on Triggers
Once the Triggers page has loaded, click on Create Trigger in the top right.
Give the trigger a name, and use the following Expression {Template SNMP Generic:sysUpTime.nodata(5m)}=1
Trigger Configuration
Optionally, give it a Description, and then set the Severity of the alert that you want to generate, and then click on Add.

The trigger should then apply to any devices that are linked to the Template SNMP Generic template.

Share
Apr 222015
 

I’ve wanted to get some temperature stats for some of my boxes for a while now to replace my aging Cacti install.
Since I already had Zabbix, that was the first place I looked for the functionality, however it does not have any templates set up out of the box, so I decided to set up my own templates for Temperature monitoring via SNMP.

I’m using Zabbix 2.2 at the moment, but the instructions should be applicable to 2.4 as well.
I’m using the Linux SNMP agent to get the temperature stats – the relevant packages on Debian are snmpd and lm-sensors.

First Things first

We need to install the snmp daemon if not already installed – apt-get install snmpd lm-sensors
After installing those the snmp daemon and lm-sensors, you may need to run sensors-detect to make sure the sensors are configured correctly.

Once the snmp daemon and lm-sensors is configured, running a snmpwalk for temperatures should result in something like this –

user@debian:~$ snmpwalk -v 2c -c public 127.0.0.1 1.3.6.1.4.1.2021.13.16.2
iso.3.6.1.4.1.2021.13.16.2.1.1.1 = INTEGER: 1
iso.3.6.1.4.1.2021.13.16.2.1.1.2 = INTEGER: 2
iso.3.6.1.4.1.2021.13.16.2.1.1.16 = INTEGER: 16
iso.3.6.1.4.1.2021.13.16.2.1.1.17 = INTEGER: 17
iso.3.6.1.4.1.2021.13.16.2.1.1.18 = INTEGER: 18
iso.3.6.1.4.1.2021.13.16.2.1.2.1 = STRING: "Core 0"
iso.3.6.1.4.1.2021.13.16.2.1.2.2 = STRING: "Core 1"
iso.3.6.1.4.1.2021.13.16.2.1.2.16 = STRING: "temp1"
iso.3.6.1.4.1.2021.13.16.2.1.2.17 = STRING: "temp2"
iso.3.6.1.4.1.2021.13.16.2.1.2.18 = STRING: "temp3"
iso.3.6.1.4.1.2021.13.16.2.1.3.1 = Gauge32: 39000
iso.3.6.1.4.1.2021.13.16.2.1.3.2 = Gauge32: 36000
iso.3.6.1.4.1.2021.13.16.2.1.3.16 = Gauge32: 39000
iso.3.6.1.4.1.2021.13.16.2.1.3.17 = Gauge32: 42000
iso.3.6.1.4.1.2021.13.16.2.1.3.18 = Gauge32: 4294965296

It looks like gibberish at a glance, but it’s actually telling us that it can detect 5 sensors.
The top 5 lines – the ones that have INTEGER are the identifiers for the sensors,
The next 5 lines – the ones that have STRING are the names of the sensors,
and the last 5 lines are the values of the sensors to 3 decimal places, just without the actual decimal point.

So that’s the Linux part all set up. On to Zabbix…

Zabbix Configuration

Regex

First up, we need to setup a RegEx to catch the sensors we want to monitor. In my case, I wanted to monitor all of them so I used the following regex which I named Sensors for Discovery –
^(temp[0-9]*|Core [0-9]*)$
The RegEx configuration is located in the Admin Tab, then drop down the menu on the right to get to “Regular expressions”

Template

Once that is done, we’ll need to create a new template. I’ve called mine “Template SNMP Sensors” and added it into the group “Templates”.
Create a new Discovery rule on the Template with the following settings
discovery rule

I’ve used {#SNMPVALUE} for the Macro, and @Sensors for Discovery for the Regexp.
You can use any value for the Key, that is a value internal to Zabbix.
And to save you some typing, the SNMP OID that is in the image is .1.3.6.1.4.1.2021.13.16.2.1.2

Item Prototype

Once the Discovery Rule is setup, you will need to create an Item prototype.
Here’s one I prepared earlier
item prototype

Again, the Key is internal to Zabbix, however the [{#SNMPVALUE}] is essential.
And again, here’s the SNMP OID to save some typing – .1.3.6.1.4.1.2021.13.16.2.1.3.{#SNMPINDEX}

Apply the Template

Once the Discovery and Item Prototype is setup, you’ll need to apply the template to a server in order for Zabbix to discover the sensors.
Once the sensors are discovered, they should show up in latest data with some values. The discovery itself may take a while unless you adjust the Interval on the Discovery Rule in the Template.
latest data

Share
Apr 162015
 

I’ve been setting up SNMP Traps on Zabbix 2.4 to replace our current in place monitoring solution.
One of the hurdles that I’ve come across is trying to get all the traps setup.

An easy way of doing this is getting the MIB files for the traps that you’re getting, and converting them into configuration files for SNMPTT to use to parse the traps.
The snmpttconvertmib command will take a MIB file as an input, and spit out a configuration file suitable for SNMPTT.
Using an Oracle MIB file as an example –

snmpttconvertmib --in=ORACLE-ENTERPRISE-MANAGER-4-MIB.mib --out=/etc/snmp/snmptt.conf.ora-em4

This will produce a file for SNMPTT but Zabbix will not parse the traps yet as the FORMAT line isn’t quite what we need yet.
Next, we’ll use sed to do a global search and replace to make sure the FORMAT lines conform to the format that Zabbix requires.

sed -i 's/FORMAT/FORMAT ZBXTRAP $aA/g' /etc/snmp/snmptt.conf.ora-em4

The configuration file then needs to be added to the list of files that SNMPTT uses to parse the traps.
Open /etc/snmp/snmptt.ini file – assuming it’s in the default location – and scroll right down to the bottom of the file.
You will see the following lines –

snmptt_conf_files = <<END
/etc/snmp/snmptt.conf

Add the file you’ve just created to the end like so –

snmptt_conf_files = <<END
/etc/snmp/snmptt.conf
/etc/snmp/snmptt.conf.ora-em4

And you should start getting SNMP traps appearing in Zabbix – assuming you’ve already set up the item.

Share
Mar 102014
 

I’ve been trying to use Cacti to graph my ADSL’s Sync rate and SNR/Attenuation Ratios for the past few weeks as I’ve been having issues with my ADSL.

Originally, I was using a BigPond Thomson ST536v6, but unfortunately, the SNMP agent on the Thomson will only expose the Sync Rate, and not the SNR and Attenuation.

So I have decided to use an old SpeedStream 4200 instead. The default SNMP community string is ‘public’ but I wanted to change it to my private one.

To change the SNMP community string, you need to telnet onto the modem to change it.
telnet 192.168.254.254

Once you’ve telnetted in, you can show the current snmp settings with this command –
xsh> cfg snmp
The output will show you the current configuration.
snmp
comm#[0..6]
nam = ""
rd = n
wr = n
dsbl = n

These settings just mean that the default settings are applied.
To update the snmp community string, you need to use the following command –

cfg snmp{comm#0{nam=skynet
cfg snmp{comm#0{rd=y

Those 2 lines will set the community string to “skynet” and set the permissions to readonly.

After setting these, run the command cfg save to save the configuration, and then reboot the modem. This will allow the new settings to take affect.

Share