Skip to main content

Analysis of network issues with tcpdump and wireshark

I've recently had to deal with the application connectivity issue (details on stackoverflow) that appeared after the migration of the application to a new server. It resulted in "Connection timed out" Java exceptions in certain cases. The answer was on the surface but I didn't know where to look at exactly. So I had to investigate and apply network sniffing tools such as tcpdump and Wireshark. Here I'd like to share my experience with the network analysis.
  1. The issue and the cause.
  2. Wireshark.
  3. Tcpdump.
The issue and the cause
The following exception was thrown by Saxon XSLT processor when the document function was invoked:
Caused by: org.apache.commons.lang.exception.NestableRuntimeException: net.sf.saxon.trans.DynamicError: net.sf.saxon.trans.DynamicError: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.Socket.connect(Socket.java:519)
        at java.net.Socket.connect(Socket.java:469)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:157)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
It clearly shows that some resource was not accessible although I could easily access the URL that was an argument of the document function. Finally it appeared that there was a DOCTYPE declaration of a DTD resource in the target xml file. So Saxon apparently failed when it could not access that DTD to perform the validation. To find out this root cause I had to use network tools.

Wireshark
First of all, I tried to reproduce the issue on my developer's Windows-based laptop but could not do this that confirmed it's a server configuration issue. However, before debugging on QA environment I analyzed the application network activity locally with Wireshark that appears to be the most popular network packet analyzer with GUI. This tool provides numerous filtering options so you can locate whatever you want on the network. It helped me to capture the HTTP request sent by Saxon and to see all request headers. Afterwards I simulated the identical request with the same headers with wget and curl on the QA environment. But generally it didn't help so I had to move to the QA environment.

Tcpdump
On the QA environment we have RHEL system installed so tcpdump appeared to be the best fit. It is another very popular network packet analyzer but only command-line. To investigate the issue further, I've used tcpdump to record the network activity for the current host for two scenarios: sending suspicious HTTP request with curl and doing it via the application itself. Curl worked fine, so the issue proved to be application-related. Indeed curl did not send another request to fetch the DTD file while Saxon did. So this is how curl command looked like:
curl -v -H "Pragma: no-cache" -H "User-Agent: Java/1.6.0_21" -H "Cache-Control: no-cache" -H "Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2" -H "Connection: keep-alive" http://suspicious-request-URL-here
This is how tcpdump command looked like:
sudo /usr/sbin/tcpdump -i eth2 -s 512 -l -A host 134.27.100.153
It prints all the requests coming to and from the specified host. I'm not pasting those results here as they are too big. But having compared the tcpdump outputs for two scenarios, I've found out there were additional requests sent from the application. It did not reveal the exact URL though but it helped me to guess the root cause and locate that DTD declaration. To conclude, the network packet analysis can be very useful for debugging. I'm pretty confident with client-side tools built into browsers (e.g. Firebug). As for the server-side, I'm still not an expert here and it may be possible to discover much more details with the tools I mentioned above.

Comments

Popular posts from this blog

DynamicReports and Spring MVC integration

This is a tutorial on how to exploit DynamicReports reporting library in an existing Spring MVC based web application. It's a continuation to the previous post where DynamicReports has been chosen as the most appropriate solution to implement an export feature in a web application (for my specific use case). The complete code won't be provided here but only the essential code snippets together with usage remarks. Also I've widely used this tutorial that describes a similar problem for an alternative reporting library.
So let's turn to the implementation description and start with a short plan of this how-to:
Adding project dependencies.Implementing the Controller part of the MVC pattern.Modifying the View part of the MVC pattern.Modifying web.xml.Adding project dependencies
I used to apply Maven Project Builder throughout my Java applications, thus the dependencies will be provided in the Maven format.

Maven project pom.xml file:
net.sourceforge.dynamicreportsdynamicrepo…

Do It Yourself Java Profiling

This article is a free translation of the Russian one that is a transcript of the Russian video lecture done by Roman Elizarov at the Application Developer Days 2011 conference.
The lecturer talked about profiling of Java applications without any standalone tools. Instead, it's suggested to use internal JVM features (i.e. threaddumps, java agents, bytecode manipulation) to implement profiling quickly and efficiently. Moreover, it can be applied on Production environments with minimal overhead. This concept is called DIY or "Do It Yourself". Below the lecture's text and slides begin.
Today I'm giving a lecture "Do It Yourself Java Profiling". It's based on the real life experience that was gained during more than 10 years of developing high-loaded finance applications that work with huge amounts of data, millions currency rate changes per second and thousands of online users. As a result, we have to deal with profiling. Application profiling is an i…

Using Oracle impdp utility to reload database

Here I'll show an example of using Oracle Data Pump Import (impdp) utility. It allows importing Oracle data dumps. Specifically, below is the list of steps I used on an existing Oracle schema to reload the data from a dump. Steps to reload the data from an Oracle dump
We start with logging into SQL Plus as sysdba to be able to manage users. sqlplus sys/password@test as sysdba Dropping the existing user. CASCADE clause will ensure that all schema objects are removed before the user. SQL> DROP USER test CASCADE; Creating a fresh user will automatically create an empty schema with the same name. SQL> CREATE USER test IDENTIFIED BY "testpassword"; Granting DBA role to the user to load the dump later. Actually, it's an overkill and loading the dump can be permitted using a more granular role IMP_FULL_DATABASE. SQL> GRANT DBA TO test; Registering the directory where the dump is located. SQL> CREATE DIRECTORY dump_dir AS '/home/test/dumpdir'; Runnin…