Skip to main content

Analysis of network issues with tcpdump and wireshark

I've recently had to deal with the application connectivity issue (details on stackoverflow) that appeared after the migration of the application to a new server. It resulted in "Connection timed out" Java exceptions in certain cases. The answer was on the surface but I didn't know where to look at exactly. So I had to investigate and apply network sniffing tools such as tcpdump and Wireshark. Here I'd like to share my experience with the network analysis.
  1. The issue and the cause.
  2. Wireshark.
  3. Tcpdump.
The issue and the cause
The following exception was thrown by Saxon XSLT processor when the document function was invoked:
Caused by: org.apache.commons.lang.exception.NestableRuntimeException: net.sf.saxon.trans.DynamicError: net.sf.saxon.trans.DynamicError: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.Socket.connect(Socket.java:519)
        at java.net.Socket.connect(Socket.java:469)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:157)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
It clearly shows that some resource was not accessible although I could easily access the URL that was an argument of the document function. Finally it appeared that there was a DOCTYPE declaration of a DTD resource in the target xml file. So Saxon apparently failed when it could not access that DTD to perform the validation. To find out this root cause I had to use network tools.

Wireshark
First of all, I tried to reproduce the issue on my developer's Windows-based laptop but could not do this that confirmed it's a server configuration issue. However, before debugging on QA environment I analyzed the application network activity locally with Wireshark that appears to be the most popular network packet analyzer with GUI. This tool provides numerous filtering options so you can locate whatever you want on the network. It helped me to capture the HTTP request sent by Saxon and to see all request headers. Afterwards I simulated the identical request with the same headers with wget and curl on the QA environment. But generally it didn't help so I had to move to the QA environment.

Tcpdump
On the QA environment we have RHEL system installed so tcpdump appeared to be the best fit. It is another very popular network packet analyzer but only command-line. To investigate the issue further, I've used tcpdump to record the network activity for the current host for two scenarios: sending suspicious HTTP request with curl and doing it via the application itself. Curl worked fine, so the issue proved to be application-related. Indeed curl did not send another request to fetch the DTD file while Saxon did. So this is how curl command looked like:
curl -v -H "Pragma: no-cache" -H "User-Agent: Java/1.6.0_21" -H "Cache-Control: no-cache" -H "Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2" -H "Connection: keep-alive" http://suspicious-request-URL-here
This is how tcpdump command looked like:
sudo /usr/sbin/tcpdump -i eth2 -s 512 -l -A host 134.27.100.153
It prints all the requests coming to and from the specified host. I'm not pasting those results here as they are too big. But having compared the tcpdump outputs for two scenarios, I've found out there were additional requests sent from the application. It did not reveal the exact URL though but it helped me to guess the root cause and locate that DTD declaration. To conclude, the network packet analysis can be very useful for debugging. I'm pretty confident with client-side tools built into browsers (e.g. Firebug). As for the server-side, I'm still not an expert here and it may be possible to discover much more details with the tools I mentioned above.

Comments

Popular posts from this blog

DynamicReports and Spring MVC integration

This is a tutorial on how to exploit DynamicReports reporting library in an existing Spring MVC based web application. It's a continuation to the previous post where DynamicReports has been chosen as the most appropriate solution to implement an export feature in a web application (for my specific use case). The complete code won't be provided here but only the essential code snippets together with usage remarks. Also I've widely used this tutorial that describes a similar problem for an alternative reporting library.
So let's turn to the implementation description and start with a short plan of this how-to:
Adding project dependencies.Implementing the Controller part of the MVC pattern.Modifying the View part of the MVC pattern.Modifying web.xml.Adding project dependencies
I used to apply Maven Project Builder throughout my Java applications, thus the dependencies will be provided in the Maven format.

Maven project pom.xml file:
net.sourceforge.dynamicreportsdynamicrepo…

Choosing Java reporting tool - part 2

I've provided a general overview of possible solutions to get a reporting/exporting functionality in the previous post. This is the second overview of alternatives based on JasperReports reporting engine.

Since the previous part I've done the following:
Implemented a simple report using both DynamicJasper and DynamicReports to compare them from technical side.Investigated JasperServer features and tried to implement a simple report for JasperServer instance (it appeared we already have a ready licensed installation of JasperServer that makes it unreasonable to install a fresh one).
First, the comparison results of Java libraries (DynamicJasper and DynamicReports):
Both libraries suffer from poor-quality or missing Java docs but they look a bit better in DynamicJasper.Taking into account the point 1, a developer has to use online documentation and to review the code. Here the code looks definitely nicer and more readable for DynamicReports. With respect t…

Do It Yourself Java Profiling

This article is a free translation of the Russian one that is a transcript of the Russian video lecture done by Roman Elizarov at the Application Developer Days 2011 conference.
The lecturer talked about profiling of Java applications without any standalone tools. Instead, it's suggested to use internal JVM features (i.e. threaddumps, java agents, bytecode manipulation) to implement profiling quickly and efficiently. Moreover, it can be applied on Production environments with minimal overhead. This concept is called DIY or "Do It Yourself". Below the lecture's text and slides begin.
Today I'm giving a lecture "Do It Yourself Java Profiling". It's based on the real life experience that was gained during more than 10 years of developing high-loaded finance applications that work with huge amounts of data, millions currency rate changes per second and thousands of online users. As a result, we have to deal with profiling. Application profiling is an i…