SWick

Die Ferne ruft mich wieder und nun bin ich am überlegen, wohin die Reise dieses Jahr gehen soll.
Japan oder China? Nochmal nach Süd-Korea oder alle auf einmal?
Noch hab ich nicht entschieden und daher erstmal Reiseführer zu China und Japan bestellt.

Heute morgen kamen meine bestellten Reiseführer zu Japan und China. Als ich im Lonely Planet im Kultur-Kapitel schmökerte, las ich die bisher beste Beschreibung zu Anime-Filmen:

Anime films encompass all genres, from science-fiction and action adventure
to romance and historical drama. Anime targets all age and social groups.
Anime films include deep explorations of philosophical questions and social issues,
humorous entertainment and bizarre fantasies.
The films offer breathtakingly realistic visuals, exquisite attention to detail,
complex and expressive characters and elaborate plots.

I tried using the existing Net-SNMP Module but had no immediate success. To use it under 64-Bit systems, you have to change the Makefile. But even after that I only got zeros when walking the snmp tree. So then, I decided going for Python :-)

This is how it will look like:

First, our Python script getting the values. Deploy it on your load-balancer running IPVS and make it executable with chmod +x /opt/ipvs_stats.py.

/opt/ipvs_stats.py

#!/usr/bin/env python

import sys

filename = "/proc/net/ip_vs_stats"

try:
    f = open(filename,'r')

except IOError:
    print "Sorry, could not read file " + "'" + filename + "'"
    sys.exit()

data = f.read()


def hex2dec(s):
    """return the integer value of a hexadecimal string s"""
    return int(s, 16)

# first create a list of lists
data_list = [line.split()  for line in data.split('\n')]

stats = {}

stats['total_conns_sec']    = hex2dec(data_list[5][0])
stats['incoming_pkts_sec']  = hex2dec(data_list[5][1])
stats['outgoing_pkts_sec']  = hex2dec(data_list[5][2])
stats['incoming_bytes_sec'] = hex2dec(data_list[5][3])
stats['outgoing_bytes_sec'] = hex2dec(data_list[5][4])

if __name__ == '__main__':

    if len(sys.argv) < 2:
        print "\nError:\tNo arguments given.\n"
        print "Try:"

        for argument in stats:
            print "\t", sys.argv[0], argument
        sys.exit()


    if sys.argv[1] == 'total_conns_sec':
        print stats['total_conns_sec']

    if sys.argv[1] == 'incoming_pkts_sec':
        print stats['incoming_pkts_sec']

    if sys.argv[1] == 'outgoing_pkts_sec':
        print stats['outgoing_pkts_sec']

    if sys.argv[1] == 'incoming_bytes_sec':
        print stats['incoming_bytes_sec']

    if sys.argv[1] == 'outgoing_bytes_sec':
        print stats['outgoing_bytes_sec']

The script reads the values from proc and as they are hexadecimal, we have a function for converting them to integers. The script can get you 5 values but in this HowTo I left out the Outgoing values because I don't use LVS-NAT in my setup. But you can simply add them from the examples here...

Next step is to prepare SNMPd with our script so we can get the stats via SNMP.

/etc/snmp/snmpd.conf

# Monitoring IPVS

extend total_conns_sec    /opt/ipvs_stats.py total_conns_sec
extend incoming_bytes_sec /opt/ipvs_stats.py incoming_bytes_sec
extend incoming_pkts_sec  /opt/ipvs_stats.py incoming_pkts_sec

You can test it with

snmpwalk -v 2c -c <community> <IP> nsExtendOutline

NET-SNMP-EXTEND-MIB::nsExtendOutLine."total_conns_sec".1 = STRING: 3
NET-SNMP-EXTEND-MIB::nsExtendOutLine."incoming_pkts_sec".1 = STRING: 62
NET-SNMP-EXTEND-MIB::nsExtendOutLine."incoming_bytes_sec".1 = STRING: 31928

As you can see, I use nsExtendOutline instead of nsExtendResult. The reason is, that our values are surely greater than 127 and most systems require the exit value to be in the range 0-127, and produce undefined results otherwise. Therefore in our script we don't use sys.exit() but print the values to STDOUT instead.

The OID is an ASCII representation of your chosen string after the "extend" command. To see it, use -On in your snmpwalk.

snmpwalk -On -v 2c -c <community> <IP> nsExtendOutline

.1.3.6.1.4.1.8072.1.3.2.4.1.2.15.116.111.116.97.108.95.99.111.110.110.115.95.115.101.99.1 = STRING: 4
.1.3.6.1.4.1.8072.1.3.2.4.1.2.17.105.110.99.111.109.105.110.103.95.112.107.116.115.95.115.101.99.1 = STRING: 74
.1.3.6.1.4.1.8072.1.3.2.4.1.2.18.105.110.99.111.109.105.110.103.95.98.121.116.101.115.95.115.101.99.1 = STRING: 33322

Now we can configure OpenNMS for collecting the data

/etc/opennms/datacollection-config.xml

...
...
...

      <group name="ipvs" ifType="ignore">
        <mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.15.116.111.116.97.108.95.99.111.110.110.115.95.115.101.99" 
                instance="1" alias="ipvsTotalConnsSec" type="octetstring" />
        <mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.17.105.110.99.111.109.105.110.103.95.112.107.116.115.95.115.101.99" 
                instance="1" alias="ipvsPktsSecIn" type="octetstring" />
        <mibObj oid=".1.3.6.1.4.1.8072.1.3.2.4.1.2.18.105.110.99.111.109.105.110.103.95.98.121.116.101.115.95.115.101.99" 
                instance="1" alias="ipvsBytesSecIn" type="octetstring" />
      </group>
...
...
...

      <systemDef name="Net-SNMP">
        <sysoidMask>.1.3.6.1.4.1.8072.3.</sysoidMask>
        <collect>
          <includeGroup>mib2-host-resources-system</includeGroup>
          <includeGroup>mib2-host-resources-memory</includeGroup>
          <includeGroup>mib2-X-interfaces</includeGroup>
          <includeGroup>net-snmp-disk</includeGroup>
          <includeGroup>openmanage-coolingdevices</includeGroup>
          <includeGroup>openmanage-temperatureprobe</includeGroup>
          <includeGroup>openmanage-powerusage</includeGroup>
          <includeGroup>ucd-loadavg</includeGroup>
          <includeGroup>ucd-memory</includeGroup>
          <includeGroup>ucd-sysstat</includeGroup>
          <includeGroup>ucd-sysstat-raw</includeGroup>
          <includeGroup>ucd-sysstat-raw-more</includeGroup>
          <!-- <ipvs> -->
          <includeGroup>ipvs</includeGroup>
          <!-- </ipvs> -->
        </collect>
      </systemDef>
...
...
...

Note the type of "octetstring". If you look at the type of this OID from the walk above, you’ll see it is "string". RRDtool and JRobin can’t store string data, thus it needs to be converted to a number. Setting the type to "octetstring" causes this to happen (it is converted to a gauge).

And finally we can build pretty graphs out from it:

/etc/opennms/snmp-graph.properties

...
...
...

reports=...\
..., \
ipvs, ipvs.incoming.bytes, ipvs.incoming.packets \
...
...
...
report.ipvs.name=IPVS Stats
report.ipvs.columns=ipvsTotalConnsSec
report.ipvs.type=nodeSnmp
report.ipvs.command=--title="IPVS Stats" \
 DEF:totalconns={rrd1}:ipvsTotalConnsSec:AVERAGE \
 LINE2:totalconns#DE0056:"Total Connections/sec" \
 GPRINT:totalconns:AVERAGE:"Avg \\: %10.2lf %s" \

report.ipvs.incoming.bytes.name=IPVS Stats Incoming Bytes
report.ipvs.incoming.bytes.columns=ipvsBytesSecIn
report.ipvs.incoming.bytes.type=nodeSnmp
report.ipvs.incoming.bytes.command=--title="IPVS Stats - Incoming Bytes" \
 DEF:bytes={rrd1}:ipvsBytesSecIn:AVERAGE \
 LINE2:bytes#DE0056:"Bytes/sec" \
 GPRINT:bytes:AVERAGE:"Avg \\: %10.2lf %s" \

report.ipvs.incoming.packets.name=IPVS Stats Incoming Packets
report.ipvs.incoming.packets.columns=ipvsPktsSecIn
report.ipvs.incoming.packets.type=nodeSnmp
report.ipvs.incoming.packets.command=--title="IPVS Stats - Incoming Packets" \
 DEF:packets={rrd1}:ipvsPktsSecIn:AVERAGE \
 LINE2:packets#DE0056:"Packets/sec" \
 GPRINT:packets:AVERAGE:"Avg \\: %10.2lf %s" \

...
...
...