Tuesday, April 6, 2010

Troubleshooting Lessons Learned

I’ve told this story many times as it has been one of my many anecdotes for Job Interviews and while chatting with fellow geeks for some time now. The story comes early on in my career about two years out of college. A time when I was just a little wet behind the ears and probably a little more cocky then I should have been. I was a Network Administrator for a small company and had recently enabled Internet email on a Microsoft Exchange Server. Prior to this Exchange was only utilized for sending internal messages. It was a very simple network, a Cisco PIX520 firewalls and a Cisco Router with an Internet T1.

The problem I faced was that random users would complain that emails they sent were not being received. I noticed that the messages were being held in the outbound message queue. I asked to be copied on some of these messages and did receive them internally however when sent to my external email address they were held up in the queue. I also noticed that these messages had attachments. At this point I opened a trouble ticket with Microsoft to work on the issue.

We discovered that when Exchange was configured for MIME encoding the message would get hung up and when set to Uuencoding  it would work fine. We made the change to UUencoding and left it figuring we had solved the issue with a workaround. A month or so later I discover that messages would still randomly get hung up. Less frequently then with MIME encoding but the problem did persist. So I went back to Microsoft for additional troubleshooting. At this point Microsoft had me perform some sniffing to see where the problem was. They found that something was missing from the network side of things and suggested a firewall issue. This didn’t sound right to me since port 25 was open for outbound traffic and the PIX doesn’t allow filtering at the application layer (attachments) however I opened a case with Cisco. Maybe it was a bug or something else strange.

Cisco had me go a little further with the sniffing and I monitored traffic both inside and outside the firewall for comparison. They found the same information internally and externally and thus told me that I was missing traffic from the Internet. At this point I was a little perplexed so I decided to build an Exchange server outside my firewall for testing. Within DNS I created a bogus DNS zone to send messages to and configured this domain on the test Exchange server. I then sent mail from the production Exchange server to the test server and low and behold it worked fine.  By now I also discovered one more unusual piece of information, with MIME encoding enabled the only attachments that had an issue were Microsoft Office documents.  I could send PDF or JPG files without issue.

So the challenge now was to convince the ISP that they were blocking MIME encoded Microsoft Office documents and no other Internet traffic was impacted. I opened a ticket with them and described my troubleshooting process to the Engineer. They did not believe they could be the cause and although the evidence pointed in their direction I was a little concerned that I missed something along the way. So their only idea was that it was some form of virus. So they had me build a laptop with a CD installation of Windows with Outlook Express. This system was never attached to my internal network to avoid infection. I used a crossover cable to connect it to the Internet router directly and used Outlook Express to send email using both MIME and UU encoding. Sure enough UU encoding worked and MIME encoding did not. Still unconvinced and determined to prove me wrong the Engineer and his boss drove from Virginia (I think but I remember it was far) to NJ with a router and laptop to test with. When they arrived in the office they connected to our T1 and experienced the same issue. I must say that this took all of 15 minutes, much less time then it took them to drive up. Surprised but convinced they contacted the LEC and initiated a call with them to check the circuit.

When the LEC engineer arrived he attached his T-Bird (T-BERD) to run test patterns on the circuit. Every single test passed until he ran a 1’s and 8’s test from the CO. This last test failed so he tried to re-punch the DMARC. At this point the circuit completely failed and he could not get the circuit back up reusing the original pair of wires between the multiplexer and the DMARC. He simply replaced the pair of wires with a free pair and reran his tests successfully. Once the line came up I also tested sending MIME encoded Office documents through the Exchange server successfully.

A number of months later I was discussing a different issue with an engineer at the same ISP. During our conversation he mentioned that the issue we were working on was strange. Now of course I mentioned the email issue as being significantly stranger. When I started the story he exclaimed “You’re the email guy?” They had found this problem so intriguing that they incorporated it into their customer service training. The key point being sometimes the customer is right.

Think about the probability of an issue with a pair of wires filtering MIME encoded Microsoft Office documents sent via SMTP with no other noticeable issues and I hope you will come to the same conclusion as I did. At some point when you’ve exhauster all logical explanations the illogical becomes possible and probable.

I hope you’ve enjoyed my little trip down troubleshooting lane. As always comments and feedback are welcome.

Tuesday, March 23, 2010

RackTables Datacenter Asset Management

Since my last post a lot has transpired, after the new year I started a new - old position, returning to a former employer as a Network Security Engineer. I was pleasantly surprised at how much knowledge I retained of the environment even after being away for more than two years. Since I left the network has grown significantly and documenting such a complex environment has moved beyond Word and Excel documents. So I started researching methods for documenting IP Address allocations and came across a wonderful open source project called RackTables.  The application installs very easily on a LAMP (Linux Apache MySQL PHP) server and maintains equipment (Objects), IP Address allocation, rack usage, and Load Balancer information. So far my experience has been very positive. The advantage over a spreadsheet is that each IP address allocation can be tied to an object. If for example that object is a Windows server you can document each IP address assigned to it. If that server is ever reallocated or renamed you simple rename the object in RackTables and all the associated IP addresses, and Racks are updated automatically. This way you will never have the one network that didn’t get updated to the new server name. Retire an object and the IP address and rack space are automatically made available for new assignments. It is also very customizable with different Dictionaries, Attributes, and Tags.

Tags for lack of a better term are meta data you can assign to different objects within RackTables. For example you can create different tags for various locations or organizational groups. I’ve also created tags for various VLAN IDs and Network Types (Corporate, Production, Internet, DMZ, etc.) This enables you to filter the various screens within RackTables to specific tags. For example under the IP allocation screen I can limit the IP block to a specific location or to a specific VLAN. This makes finding the network or object you’re looking for much easier. Also filtering this way in the Rack Space screen limits the racks to a given location. Which comes in handy if you have multiple data centers or even document multiple small offices with one or two racks each.

Attributes and Dictionaries go hand in hand. Attributes are informational items that can be added to objects. For example one of the predefined attributes is “contact person” which is then defined for specific types of objects such as Server, Routers, etc. Attributes can have different types of values, a string, an integer, or a dictionary record. When set to a dictionary record you specify which dictionary to use for a given object type. For example for object type “Network security” the  “Hardware Type” attribute uses values from the dictionary “network security models.” Now in the “network security models” dictionary is a list of various vendor's security equipment. Each one of the dictionaries can be customized to add equipment that does not exist by default.

Another nice feature is the ability to link server interfaces to patch panels, or switches together. You can really document your entire infrastructure with this tool.

As wonderful an application as RakeTables is I do have one complaint. The reporting functionality is very limited. I would love to have the ability to export some of the data to PDF or CSV based on selected tags. For example it would be nice to generate a rack usage report which has a page for each rack and the specific objects in it. Another would be port utilization, for example generate a report that shows which ports on a switch are utilized and which are free. With all the information this system stores there are a number of reports that should be easy to generate.

Lastly some recommendations for future versions:

  1. Allow for the automatic parsing of “show interface” from Cisco equipment to add Interfaces. On the same note ifconfig and ipconfig in the Linux and Windows world respectfully. I’ve made a spread sheet to speed this up for now but it still takes time when you need to add a switch with 300 ports.
  2. Allow for multiple blank “L2 Addresses” for ports that do not have L2 addresses such as Consoles (RS-232), KVM, etc. Here’s a simple solution (inc/database.php):

      function alreadyUsedL2Address ($address, $my_object_id)
              if($address == '')
                      return false;

  3. Custom IP Address allocation types. This is currently a fixed enum in table IPv4Allocation.
  4. Add Attributes to IP address blocks.
  5. Provide a mechanism  for NAT IP linking. Have a physical address on a server which is NATed to an outside IP address. I’d like to see a NAT from and NAT to link in the IP address allocation field. It would also need to allow for multiple translations.
  6. Have a virtualization area that allows virtual servers to be assigned to a physical hardware. I would basically like to document which Virtual Servers are on which physical servers or server farms if they can be migrated automatically with tools such as vMotion. Also to link the virtual NIC with the physical ones would be nice. This isn’t currently possible since you can only link one port with another.

So to wrap this all up RackTables is a great tool and those that have worked on the development have done an excellent job. I highly recommend it to anyone looking for an Open Source solution for documenting the various aspects of their Data Center.

I hope this article was useful to you and as always if you have any comments or suggestions I’d be happy to hear from you.

Thursday, October 1, 2009

Presentation - Overview of a SQL Injection Attack

This is a slightly modified version of a presentation I gave on an overview of a SQL Injection attack. I performed a forensics investigation on this attack as well as the desired effects on systems that browsed the site afterward. Additional information on this attack can be found on the Internet Storm Center in the following diary entries:

The 10.000 web sites infection mystery solved

SQL Injection: More of the same

If you have any questions or would like to discuss this please let me know. As always comments are welcome.

Sunday, August 23, 2009

My Experience with Untangle an Open Source Network Gateway

It has been some time since I’ve written so I figured I better get on this one while I had a little time. I recently demoed a TippingPoint IPS as a potential security product for my network. I’ll refrain from going into too much detail but due to budgetary issues we where unable to gain approval for the actual purchase. I was also given a challenge, we had to do something without any money. I’m sure many of you will point out the risks and liability associated with this. I recently read a post on LinkedIn Answers on this very topic. To summarize, when an organization purchases security equipment they transfer some of their risk and liability to the company selling the product. To put it another way, you have someone else to blame when things go wrong. The problem with ‘free’ software is that their is no one else to blame, no technical support to call, you’re on your own. I’ll leave the bantering at that, just know that these are not ideal circumstances and a more ideal solution can be found with a proper budget.

So on to Google I went, with the requirements that it needed to be an inline IPS that was reasonably simple to setup and maintain. Now in the past I’ve used Snort as an IDS and it worked well enough for the particulars of that setup but I was looking for an inline IPS. So I started searching for inline Snort in IPS mode and many articles came up but my requirement of a simple setup where far from met. I had to consider that others unfamiliar with Snort or Linux would need to assist in the maintenance of this system. So after some creative searching the Google Gods shined on me and I stumbled upon Untangle. Now there have been many free applications I’ve been impressed with but Untangle takes the prize. It incorporates both free and paid components into a very simple to install framework that even a novice can use.

There are two options for running Untangle, Untangle Server and Untangle for Windows. Untangle for Windows is advertised to run on your desktop PC and will work in environments of 10 or fewer PC’s. For my setup I downloaded the Untangle Server which is said to be ideal for networks with more then 10 people connecting to the Internet. Simply burn the ISO to CD and boot a computer with 2 or more network interfaces and follow the bouncing ball. Once installed the system works just like any other security appliance, no real need for KVM unless you have an issue. All management is performed through the web interface (which you can also access with KVM if needed).

The Untangle interface resembles a computer room rack and has places for the different components that are available for installation. Installing the different components is very straight forward. You select one on the left side menu and a popup prompts you to download if its free or provides pricing information if not. You can also install several components at once by selecting the Open Source or Professional packages. The paid packages are more cost effective over the individual paid components.

So what does Untangle do? Well it all depends on what you want it to do. The components that met my needs where Spyware Blocker, Web Filter, Virus Blocker, Intrusion Prevention, Attack Blocker, and Reports. All of which utilize open source freely available software. For example the Virus Blocker is based on ClamAV and Intrusion Prevention is based on Snort. The other free components are Spam Blocker, Ad Blocker, Phish Blocker, Firewall, Routing & QoS, Protocol Control, and OpenVPN. All the components I installed where very simple to configure and provided a very intuitive and homogenous interface. The most difficult component to configure was Intrusion Prevention but only because of the number of signatures available.

The only complaints I have thus far are limitations in the reporting and event logging. The daily and weekly reports are fine for review but there is no option to generate adhoc reports. There are also a few limitations in the event log areas. For one there is an individual event log for each component. A centralized event viewer that displays an event source would be helpful. There are also no filtering options for events, I think it would be helpful to filter based on source or destination address, ports, or some other relevant condition. This would aid in troubleshooting issues. Lastly in the Web Filter it doesn’t provide which category blocked a specific site. This would come in handy if you’re unfamiliar with a specific site and are unsure what category it falls under. Your organization may react one way if for example someone browses to a game site but another if its pornography.

Overall the developers at Untangle have integrated a number of tools into a nice simple to build appliance. As always there is room for improvement but I think they’ve done an excellent job so far. If you have any questions about my experience with Untangle or if you have any experience yourself please let me know.

Thursday, May 14, 2009

Security Awareness Thoughts & McAfee Threats Report Summary

As always my colleagues and  I monitor the global security landscape and how it affects our organization. McAfee one of the most well known virus protection software companies has recently released their first quarterly report for 2009. “McAfee Threats Report: First Quarter 2009

To summarize the report goes into the types of threats that are prevalent on the Internet and has some predications as to what may come and when. Although I don’t expect everyone to read the article I wanted to share some of its insights.

  1. They have detected nearly twelve million new IP addresses operating as “zombies,” computers under the control of spammers and others.
  2. It is now apparent that cybercriminals will attack any target of opportunity they can find including targets in their own Country.
  3. Spam as a percentage of all email sent has fallen bellow 90% for the first time since 2006 due to the shutdown of the Web hosting service provider McColo. However they fully expect spam volumes to regain their 2008 levels.
  4. Although Conficker was more media hype than threat it still infected a large number of systems worldwide.

My concern with the last point is that the unfulfilled hype undermines the credibility of threats in general and there are a number of threats that did not gain media publicity however infected more systems and pose a larger security threat in general.

I’d also like to add that today’s malware is more difficult to detect, and that undetected threats are possible. Looking at the “Failures in Detection (Last 24 Hours)” graph on VirusTotal.com’s Statics page you will gain a scary piece of knowledge. For a single 24 hour period (5/14/2009) 50936 infected files where scanned. Out of them only 360 where detected by all of the 39 antivirus engines they utilize.  There is a very significant statement in these numbers: “Not one Virus protection software package detects all known Viruses.”

So what does this all mean to an Organization? Simple, you need to be very aware of the possibility of security threats including viruses and malicious applications as well as attacks. You must be observant to any abnormal behavior in system activity and react quickly as to minimize its affect. I’m sure if you monitor incoming traffic from your network by IDS or simple Firewall logs you will notice traffic that is not quite right. For example traffic coming from Asia when you know you don’t have any clients outside of the USA. You may even notice some brute force attacks or some other attempt at breeching your security in server logs.

Do yourself and your organization a favor if you experience any weird behavior on your desktop or laptop, please take the precaution of disconnecting it from the network immediately and then contact your Information Security team. If you’re part of your organization’s Info Sec Team then you’re more then aware of the ramifications of having some malicious software reeking havoc on your network. You need to educate your company on the likely possibility of threats and the immediate actions they should take. Develop a strong incident response plan that is endorsed at the corporate level. This way you have the authority to react quickly.

I hope this article was useful to you and as always if you have any comments or suggestions I’d be happy to hear from you.