May 042015
 

I’ve had some issues with some of my VMWare guests crashing for some odd reason.
The guests were my pfSense routers, so when they crashed, the house lost internet which was causing some issues as you could imagine!

Since I couldn’t work out why the guests kept crashing at first, I configured Zabbix to just reboot the virtual machines each time they went down so that the internet connectivity would be restored automatically. This meant that the emails that told me that my routers had crashed would actually be sent so that I would know that my internet had a hiccup, and for me to check to make sure any downloads I had running either finished properly, or that I had to redownload.

Zabbix has the ability to run remote commands on hosts via SSH, and my VMWare hosts had SSH enabled so I could run commands that I needed to reboot the hosts when the hosts went down.

Finding the VM

First thing I needed to do was to find the ID of the VM that I needed to reboot.
To get those, I needed to SSH onto my VM host and run the command vim-cmd vmsvc/getallvms | grep pfsense-2 which outputs this –
70 pfsense-2 [vmhost-500gb] pfsense-2/pfsense-2.vmx freebsd64Guest vmx-08 pfSense backup node
This command gave me the ID of 70 that I needed to use to reboot the VM from the command line.

Configuring the Zabbix Action

In order to run a command when a host went down, I created a new action to be run when a certain host goes down.
I used the following 3 conditions –
(A) Trigger value = PROBLEM
(B) Trigger = Template ICMP Ping: Template ICMP Ping is unavailable by ICMP
(C) Host = pfsense-2

This would make sure that the actions that run are only for the host that I have the ID for

In the Operations section of the action, I created a new step with the following settings –

Operation Type: Remote Command
Target : vmhost
Type : SSH
Username : root
Password : password
Port : 22
Commands : /bin/vim-cmd vmsvc/power.off 70 && /bin/vim-cmd vmsvc/power.on 70

The commands that I have used instructs VMWare to forcefully power off the VM with an ID of 70 – which in this case is my pfsense-2 guest, and then power it back on.

This was done on an ESX 5.1 host, but should work on anything newer as long as SSH is enabled.

Share
Apr 222015
 

I’ve wanted to get some temperature stats for some of my boxes for a while now to replace my aging Cacti install.
Since I already had Zabbix, that was the first place I looked for the functionality, however it does not have any templates set up out of the box, so I decided to set up my own templates for Temperature monitoring via SNMP.

I’m using Zabbix 2.2 at the moment, but the instructions should be applicable to 2.4 as well.
I’m using the Linux SNMP agent to get the temperature stats – the relevant packages on Debian are snmpd and lm-sensors.

First Things first

We need to install the snmp daemon if not already installed – apt-get install snmpd lm-sensors
After installing those the snmp daemon and lm-sensors, you may need to run sensors-detect to make sure the sensors are configured correctly.

Once the snmp daemon and lm-sensors is configured, running a snmpwalk for temperatures should result in something like this –

user@debian:~$ snmpwalk -v 2c -c public 127.0.0.1 1.3.6.1.4.1.2021.13.16.2
iso.3.6.1.4.1.2021.13.16.2.1.1.1 = INTEGER: 1
iso.3.6.1.4.1.2021.13.16.2.1.1.2 = INTEGER: 2
iso.3.6.1.4.1.2021.13.16.2.1.1.16 = INTEGER: 16
iso.3.6.1.4.1.2021.13.16.2.1.1.17 = INTEGER: 17
iso.3.6.1.4.1.2021.13.16.2.1.1.18 = INTEGER: 18
iso.3.6.1.4.1.2021.13.16.2.1.2.1 = STRING: "Core 0"
iso.3.6.1.4.1.2021.13.16.2.1.2.2 = STRING: "Core 1"
iso.3.6.1.4.1.2021.13.16.2.1.2.16 = STRING: "temp1"
iso.3.6.1.4.1.2021.13.16.2.1.2.17 = STRING: "temp2"
iso.3.6.1.4.1.2021.13.16.2.1.2.18 = STRING: "temp3"
iso.3.6.1.4.1.2021.13.16.2.1.3.1 = Gauge32: 39000
iso.3.6.1.4.1.2021.13.16.2.1.3.2 = Gauge32: 36000
iso.3.6.1.4.1.2021.13.16.2.1.3.16 = Gauge32: 39000
iso.3.6.1.4.1.2021.13.16.2.1.3.17 = Gauge32: 42000
iso.3.6.1.4.1.2021.13.16.2.1.3.18 = Gauge32: 4294965296

It looks like gibberish at a glance, but it’s actually telling us that it can detect 5 sensors.
The top 5 lines – the ones that have INTEGER are the identifiers for the sensors,
The next 5 lines – the ones that have STRING are the names of the sensors,
and the last 5 lines are the values of the sensors to 3 decimal places, just without the actual decimal point.

So that’s the Linux part all set up. On to Zabbix…

Zabbix Configuration

Regex

First up, we need to setup a RegEx to catch the sensors we want to monitor. In my case, I wanted to monitor all of them so I used the following regex which I named Sensors for Discovery –
^(temp[0-9]*|Core [0-9]*)$
The RegEx configuration is located in the Admin Tab, then drop down the menu on the right to get to “Regular expressions”

Template

Once that is done, we’ll need to create a new template. I’ve called mine “Template SNMP Sensors” and added it into the group “Templates”.
Create a new Discovery rule on the Template with the following settings
discovery rule

I’ve used {#SNMPVALUE} for the Macro, and @Sensors for Discovery for the Regexp.
You can use any value for the Key, that is a value internal to Zabbix.
And to save you some typing, the SNMP OID that is in the image is .1.3.6.1.4.1.2021.13.16.2.1.2

Item Prototype

Once the Discovery Rule is setup, you will need to create an Item prototype.
Here’s one I prepared earlier
item prototype

Again, the Key is internal to Zabbix, however the [{#SNMPVALUE}] is essential.
And again, here’s the SNMP OID to save some typing – .1.3.6.1.4.1.2021.13.16.2.1.3.{#SNMPINDEX}

Apply the Template

Once the Discovery and Item Prototype is setup, you’ll need to apply the template to a server in order for Zabbix to discover the sensors.
Once the sensors are discovered, they should show up in latest data with some values. The discovery itself may take a while unless you adjust the Interval on the Discovery Rule in the Template.
latest data

Share
Apr 162015
 

I’ve been setting up SNMP Traps on Zabbix 2.4 to replace our current in place monitoring solution.
One of the hurdles that I’ve come across is trying to get all the traps setup.

An easy way of doing this is getting the MIB files for the traps that you’re getting, and converting them into configuration files for SNMPTT to use to parse the traps.
The snmpttconvertmib command will take a MIB file as an input, and spit out a configuration file suitable for SNMPTT.
Using an Oracle MIB file as an example –

snmpttconvertmib --in=ORACLE-ENTERPRISE-MANAGER-4-MIB.mib --out=/etc/snmp/snmptt.conf.ora-em4

This will produce a file for SNMPTT but Zabbix will not parse the traps yet as the FORMAT line isn’t quite what we need yet.
Next, we’ll use sed to do a global search and replace to make sure the FORMAT lines conform to the format that Zabbix requires.

sed -i 's/FORMAT/FORMAT ZBXTRAP $aA/g' /etc/snmp/snmptt.conf.ora-em4

The configuration file then needs to be added to the list of files that SNMPTT uses to parse the traps.
Open /etc/snmp/snmptt.ini file – assuming it’s in the default location – and scroll right down to the bottom of the file.
You will see the following lines –

snmptt_conf_files = <<END
/etc/snmp/snmptt.conf

Add the file you’ve just created to the end like so –

snmptt_conf_files = <<END
/etc/snmp/snmptt.conf
/etc/snmp/snmptt.conf.ora-em4

And you should start getting SNMP traps appearing in Zabbix – assuming you’ve already set up the item.

Share
Apr 302014
 

I have Spectrum integrated with Service Now in order to raise Incidents automatically when certain critical alerts come through on CA Spectrum

As a pre-requisite, Spectrum needs a user configured in Service Now with the ability to create records in the Incident table.

  1. Spectrum Configuration
  2. I have alarm notifiers setup to catch events that require an automated incident. The alarm notifier SetScript then calls a custom perl script in order to send the web services request to Service Now. The alarm notifiers are configured

    The SetScript has also been modified to parse out some extra parameters from the policy’s Notification Data. The notification data is accessed from the script through the variable “$NOTIFDATA”.
    I use a comma as a delimiter in the notification data to create fields, so the notification data within the policy looks like this

    Network Team,Network,Infrastructure

    The NOTIFDATA variable is expanded, and then assigned a variable inside the SetScript with the following bash script snippet

    declare -a ENOTIFDATA
    OldIFS=$IFS
    IFS=','
    for x in $NOTIFDATA; do
    ENOTIFDATA=( "${ENOTIFDATA[@]}" "$x" )
    done

    AssignmentGroup=${ENOTIFDATA[0]}
    Category=${ENOTIFDATA[1]}
    SubCat=${ENOTIFDATA[2]}

    The SetScript then calls the Perl script with some arguments. The perl script will make the calls to Service Now.

    $SPECROOT/custom/scripts/SNow/RaiseInc.pl $AID "$AssignmentGroup" "Autogenerated - A $SEV alarm has occurred on $MNAME" "$EVENTMSG" "$Category" "$SubCat" "$SEV"

  3. Perl Script
  4. The Perl script is built from examples available from ServiceNow.

    The following example will need to be modified to suit your environment – specifically the parameters as different organisations will have different configurations for ServiceNow.

    #!/usr/bin/perl

    #use lib '/usr/lib/perl5/custom';

    # declare usage of SOAP::Lite

    use SOAP::Lite;
    use feature 'switch';

    # specifying this subroutine, causes basic auth to use
    # its credentials when challenged
    sub SOAP::Transport::HTTP::Client::get_basic_credentials {
    # login as the itil user

    return 'spectrum_user' => 'spectrum_password';
    }

    # declare the SOAP endpoint here
    my $soap = SOAP::Lite
    -> proxy('https://instance.service-now.com/incident.do?SOAP');

    # calling the insert function
    my $method = SOAP::Data->name('insert')
    ->attr({xmlns => 'http://www.service-now.com/'});

    # create a new incident with the following short_description and category
    my @params = ( SOAP::Data->name(short_description => $ARGV[2]) );
    push(@params, SOAP::Data->name(u_requestor => 'Spectrum 9.3') );
    push(@params, SOAP::Data->name(contact_type => 'Auto Monitoring') );
    push(@params, SOAP::Data->name(description => $ARGV[3]) );
    push(@params, SOAP::Data->name(u_business_service => $ARGV[4]) );
    push(@params, SOAP::Data->name(assignment_group => $ARGV[1] ) );

    given ($ARGV[6]) {
    when ("MINOR") {
    push(@params, SOAP::Data->name(urgency => '3') );
    }
    when ("MAJOR") {
    push(@params, SOAP::Data->name(urgency => '3') );
    }
    when ("CRITICAL") {
    push(@params, SOAP::Data->name(urgency => '3') );
    }
    default {
    push(@params, SOAP::Data->name(urgency => '3') );
    }
    }

    # invoke the SOAP call
    my $result = $soap->call($method => @params);

    # print any SOAP faults that get returned
    print_fault($result);
    if ($result->fault) {
    exec 'echo Incident Raising Error. Please check spectrum logs | mail -h smtp.dmz.localnet -s "Issue Raising Incident" admin@example.com';
    }
    # print the SOAP response that get return
    print_result($result);

    # convenient subroutine for printing all results
    sub print_result {
    my ($result) = @_;
    }

    # convenient subroutine for printing all SOAP faults
    sub print_fault {
    my ($result) = @_;

    if ($result->fault) {
    print "faultcode=" . $result->fault->{'faultcode'} . "\n";
    print "faultstring=" . $result->fault->{'faultstring'} . "\n";
    print "detail=" . $result->fault->{'detail'} . "\n";
    }
    }


References
http://wiki.servicenow.com/index.php?title=Perl_Web_Services_Client_Examples

Share
Nov 142013
 

I wanted to rename a whole bunch of models to transform their names into all lowercase with just the hostname rather than the FQDN.
I used this script to do it with bash and vnmsh. The script will loop through all models found by the query with a model type handle, and then renames then with a vnmsh update command.


#!/bin/bash
export CLIMNAMEWIDTH=70
OldIFS=$IFS
IFS=$'\n'
WORKPATH="/opt/CA/Spectrum/vnmsh"

$WORKPATH/connect

## Pingable
MDLLIST=`$WORKPATH/show models mth=0x10290`
MDLLIST=`echo "$MDLLIST" | grep -vi mname`

for x in $MDLLIST; do
MDLHANDLE=`echo $x | awk -F '[ |.]+' ' { print $1 } '`
MDLNAME=`echo $x | awk -F '[ |.]+' ' { print tolower($2) }'`
echo $MDLHANDLE
echo $MDLNAME

$WORKPATH/update mh=$MDLHANDLE attr=0x1006e,val=$MDLNAME
done
$WORKPATH/disconnect
IFS=$OldIFS

Share