Wednesday, May 6, 2009

Configuring MRTG, RRDtool, and Routers2 on Fedora 10

Since I run MRTG on my Syslog server I had to update that procedure as well. It’s taken me some time because I had to deal with some other unrelated issues over the past two weeks. In the past I just used MRTG by itself but I felt it time to step it up a bit. Although I chose to run this on my Syslog sever the procedure is designed for a standalone setup. Also depending on the size of your environment and the resources available you may not want the double duty. As always if you have any suggestions, recommendations, or comments please let me know.

Here are some links to additional documentation and software:

The last thing I want to point out before getting into the procedure is that MRTG has one limitation and that is the 5 minute report time. This hides short term spikes in traffic by flattening them out over a five minute period. I have read some articles on changing MRTG to run every minute but have yet to actually implement this change. If anyone is interested in modifying my procedure to include one minute resolution please let me know I’d be happy to work with you on it. If your interested I found some information here but haven’t had time to really look into it, some day.

Now without further ado:

1. Install Fedora with MySQL and Apache both with PHP support also add in the dev tools so you can compile anything else required. For mine I do not install a GUI only CLI so no Xwindows required unless you want it. Some other tidbits:
    a. Set the time zone to UTC - I do this on all networking equipment and synchronize with NTP
    b. Enable SELinux
    c. When the installation is complete perform a yum update
    d. Static IP address
2. Now lets secure Linux a bit and setup some basic services
    1. Create a user
        a. Add a user for to login with
        useradd -c "John Doe" jdoe
        passwd jdoe
    2. SSH
        a. SSH into the box and logon as this user.
        b. su - and use the root password
        c. nano -w /etc/ssh/sshd_config
        d. Find '#PermitRootLogin yes' and change it to 'PermitRootLogin no'
        e. Find '#PermitEmptyPasswords no' and change it to 'PermitEmptyPasswords no'
        f. service sshd restart
    3. Turn off unneeded services: netfs, nfslock, acpid, bluetooth, cpuspeed, cups, gpm, haldaemon
        chkconfig --levels 0123456 netfs off
        chkconfig --levels 0123456 nfslock off
        chkconfig --levels 0123456 acpid off
        chkconfig --levels 0123456 bluetooth off
        chkconfig --levels 0123456 cpuspeed off
        chkconfig --levels 0123456 cups off
        chkconfig --levels 0123456 gpm off
        chkconfig --levels 0123456 haldaemon off
    4. Install and configure NTP - if you happened to be running this in a Virtual machine this may not work.
        a. yum install ntp.i386
        b. Optionally edit the ntp.conf file and update with your own NTP server (nano -w /etc/ntp.conf)
        c. ntpdate -u time.nist.gov
        d. service ntpd start
        e. chkconfig --levels 234 ntpd on
        d. service ntpd start
        f. NOTE: if you run into issues because this is a VM. Put this ntpdate -u time.nist.gov into crontab.
            EDITOR=nano
            export EDITOR
            crontab -e
            * 00 * * * /usr/sbin/ntpdate -u time.nist.gov
        g. You can use ntpstat, ntpdc -p, and ntpq -p to check on the status of ntpd
    5. Update iptables -
        a. nano -w /etc/sysconfig/iptables
            # Firewall configuration written by system-config-securitylevel
            # Manual customization of this file is not recommended.
            *filter
            :INPUT ACCEPT [0:0]
            :FORWARD ACCEPT [0:0]
            :OUTPUT ACCEPT [0:0]
            :RH-Firewall-1-INPUT - [0:0]
            -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
            -A INPUT -p icmp -j ACCEPT
            -A INPUT -i lo -j ACCEPT
            -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
            -A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
            -A INPUT -j REJECT --reject-with icmp-host-prohibited
            -A FORWARD -j REJECT --reject-with icmp-host-prohibited
            COMMIT
        b. service iptables restart
    6. Configure crontab editor and add updatedb to schedule
        EDITOR=nano
        export EDITOR
        crontab -e
        Add Line: 00 00 * * * /usr/bin/updatedb
        crontab -l
3. Install some additional required components 
    yum install mrtg.i386 rrdtool.i386 rrdtool-perl.i386 perl-GD.i386
4. Create required directories 
    1. mkdir /etc/mrtg/thresholds
5. Create some MRTG configuration scripts and configure MRTG for each device
These scripts may work on some switches however others may not. You can use snmpwalk to check, see notes bellow. Some may have multiple values for different modules such as the Cisco CAT 6500. Your best bet is to find a value you would want to alert on and use that some experimentation may be needed at first.
    1. Create tempscript
    nano /etc/mrtg/tempscript
        #-------------------- Temperature --------------------
        $head_lines .= <<ECHO;
        #---------------------------------------------------------------------
        ECHO
        my $target_name = $router_name . "_temp";
        $target_lines .= <<ECHO;

        Target[$target_name]:1.3.6.1.4.1.9.9.13.1.3.1.3.1&1.3.6.1.4.1.9.9.13.1.3.1.3.2:$router_connect
        MaxBytes[$target_name]: 100
        Title[$target_name]: $router_name Chassis temperature
        PageTop[$target_name]: <h1>$router_name Chassis Temperature</H1>
        Options[$target_name]: growright,absolute,gauge,integer,nopercent
        YLegend[$target_name]: Degrees C.
        Unscaled[$target_name]: dwmy
        ShortLegend[$target_name]: Degrees
        Legend1[$target_name]: Ambient temperature
        Legend2[$target_name]: Chassis temperature
        ThreshMaxI[$target_name]: 50
        ThreshMaxO[$target_name]: 50
        LegendI[$target_name]: &nbsp;Ambient:
        LegendO[$target_name]: &nbsp;Chassis:
        ECHO
    2. Create cpuscript
    nano /etc/mrtg/cpuscript
        #-------------------- CPU --------------------
        $head_lines .= <<ECHO;
        #---------------------------------------------------------------------
        ECHO
        my $target_name = $router_name . "_cpu";
        $target_lines .= <<ECHO;

        Target[$target_name]:1.3.6.1.4.1.9.2.1.57.0&1.3.6.1.4.1.9.2.1.58.0:$router_connect
        MaxBytes[$target_name]: 100
        Title[$target_name]: $router_name CPU Utilization
        PageTop[$target_name]: <h1>$router_name CPU Utilization</H1>
        Options[$target_name]: growright,absolute,gauge,integer
        YLegend[$target_name]: Precent Utilization
        Unscaled[$target_name]: dwmy
        ShortLegend[$target_name]: %
        Legend1[$target_name]: 1 Minute Average
        Legend2[$target_name]: 5 Minute Average
        ThreshMaxI[$target_name]: 75
        ThreshMaxO[$target_name]: 75
        LegendI[$target_name]: &nbsp;1 Minute Average:
        LegendO[$target_name]: &nbsp;5 Minute Average:
        ECHO
    3. For a device that can support both CPU and Temp readings combine the two scripts into one file
    nano /etc/mrtg/cputempscript
        #-------------------- CPU & Temperature --------------------
        $head_lines .= <<ECHO;
        #---------------------------------------------------------------------
        ECHO
        my $target_name_cpu = $router_name . "_cpu";
        my $target_name_temp = $router_name . "_temp";
        $target_lines .= <<ECHO;

        Target[$target_name_cpu]:1.3.6.1.4.1.9.2.1.57.0&1.3.6.1.4.1.9.2.1.58.0:$router_connect
        MaxBytes[$target_name_cpu]: 100
        Title[$target_name_cpu]: $router_name CPU Utilization
        PageTop[$target_name_cpu]: <h1>$router_name CPU Utilization</H1>
        Options[$target_name_cpu]: growright,absolute,gauge,integer
        YLegend[$target_name_cpu]: Percent Utilization
        Unscaled[$target_name_cpu]: dwmy
        ShortLegend[$target_name_cpu]: %
        Legend1[$target_name_cpu]: 1 Minute Average
        Legend2[$target_name_cpu]: 5 Minute Average
        ThreshMaxI[$target_name_cpu]: 75
        ThreshMaxO[$target_name_cpu]: 75
        LegendI[$target_name_cpu]: &nbsp;1 Minute Average:
        LegendO[$target_name_cpu]: &nbsp;5 Minute Average:

        Target[$target_name_temp]:1.3.6.1.4.1.9.9.13.1.3.1.3.1&1.3.6.1.4.1.9.9.13.1.3.1.3.2:$router_connect
        MaxBytes[$target_name_temp]: 100
        Title[$target_name_temp]: $router_name Chassis temperature
        PageTop[$target_name_temp]: <h1>$router_name Chassis Temperature</H1>
        Options[$target_name_temp]: growright,absolute,gauge,integer,nopercent
        YLegend[$target_name_temp]: Degrees C.
        Unscaled[$target_name_temp]: dwmy
        ShortLegend[$target_name_temp]: Degrees
        Legend1[$target_name_temp]: Ambient temperature
        Legend2[$target_name_temp]: Chassis temperature
        ThreshMaxI[$target_name_temp]: 50
        ThreshMaxO[$target_name_temp]: 50
        LegendI[$target_name_temp]: &nbsp;Ambient:
        LegendO[$target_name_temp]: &nbsp;Chassis:
        ECHO
    3.Run cfgmaker for each device. Modified the community, host IP, host template, and output file for each one. You may want to put this into a script so you can rerun it periodically as you add interfaces. You will need to specify correct email information for the Threshold settings as well as set your desired threshold max input and output values.
        /usr/bin/cfgmaker \
        --noreversedns \
        --ifref descr \
        --ifdesc descr \
        --show-op-down \
        --global "WorkDir: /var/www/html/mrtg" \
        --global "PathAdd: /usr/bin/" \
        --global "LogFormat: rrdtool" \
        --global "Options[_]: bits,growright" \
        --global "ThreshDir: /etc/mrtg/thresholds" \
        --global "ThreshMaxI[_]: 75%" \
        --global "ThreshMaxO[_]: 75%" \
        --global "ThreshMailServer: mail.company.com" \
        --global "ThreshMailSender: mrtg@company.com" \
        --global "ThreshMailAddress[_]: admin@company.com" \
        --host-template=/etc/mrtg/cputempscript \
        --output /etc/DEVICE.mrtg.cfg \
        community@hostip1
6. Now let’s download and setup Routers2
    1. cd ~
    2. wget http://www.steveshipway.org/software/rrd/routers2-v2.19.tar.gz
    3. tar -xvzf routers2-v2.19.tar.gz
    4. cd ~/routers2-v2.19
    5. perl install.pl
        FINDING OUT ABOUT YOUR SYSTEM
        0. /etc/httpd/conf
        1. [/var/www/html]?
        2. [/var/www/cgi-bin/]
        3. /etc
        4. *.mrtg.cfg
        5. [/var/www/html/mrtg]
        6. [/usr/bin/perl]
        7. [/var/www/html/mrtg/routers2.conf]
        ASKING OPTIONS
        1. Net::SNMP does not appear to be installed.
           GD Perl Library is detected.
        2. The Compact Summary pages will be enabled.
        3. no
        4. [none]
        5. [no] (I could not say yes)
        6. [no]? yes or no optional
        Continue to install [no]? yes
        ** ALL COMPLETE **
7. Start the web server
    1. setsebool httpd_enable_cgi 1
    2. setsebool allow_httpd_sys_script_anon_write 1
    3. chcon -R -t httpd_sys_content_rw_t /var/www/html/graphs
    4. service httpd start
    5. chkconfig --levels 234 httpd on
8. Schedule each device in crontab (if you have a number of devices you may want to stagger groups of them depending on system resources)
    EDITOR=nano
    export EDITOR
    crontab -e
    00,05,10,15,20,25,30,35,40,45,50,55  * * * *  env LANG=C /usr/bin/mrtg /etc/DEVICE.mrtg.cfg
9. Now access the site with a browser and watch the data come in.
    https://mrtg.company.com/cgi-bin/routers2.cgi

SNMPWALK
If you need to test the OID for CPU, Temperature, or some other utilization value you can use SNMPWALK.
NOTE: Use caution when you select the OID as you can pull a ton of data from the device.
1. yum install net-snmp-utils.i386
2. snmpwalk -v1 -c community host1p OID
   for Example: snmpwalk -v1 -c public 192.168.1.1 1.3.6.1.4.1.9.9.13.1.3.1.3.1

No comments:

Post a Comment