As a web developer I often have to debug HTTP problems. At local.ch we also do the frontend/backend communication using the HTTP protocol, so sometimes I also have to debug that. So over the whole time I’ve been doing that, some tips have accumulated.

LiveHTTPHeaders

LiveHTTPHeaders is a Firefox extension that allows to see the headers exchanged between the browser and server. It adds a “Headers” tab in the page information. This window can be accessed with Apple+I on the Mac or Ctrl+I on Linux and Windows. The headers tab shows a list of the headers that Firefox sent to the server and a separate list of the response headers sent by the server.

I use that mostly to see the Cookie and Accept-Language headers that Firefox sent.

Firebug

The Firefox extension Firebug is a debugging tool. It has JavaScript debugging, a DOM inspector, debug messages and a lot more. But the most useful feature for this article is XMLHttpRequest logging. After opening the console you can enable the option “Show XMLHttpRequests” in Firebug’s Options menu. This logs the URL including complete request and response of every XMLHttpRequest done.

GET requests with netcat

Netcat is in my experience one of the most overlooked tools for network debugging. It allows you to easily send network requests to any TCP or UDP server. While you can also open TCP connections with good old telnet, netcat offers one big advantage: it reads the data to be sent through the standard input channel. This allows you to replay requests a lot easier than with telnet. For example I usually do the following in a bash shell:

<code>echo "GET / HTTP/1.1
Host: weblog.patrice.ch
Connection: Close

" | nc weblog.patrice.ch 80 | less</code>

This opens / and shows the full HTTP responses in less. Let me explain the three requests lines for those who never have seen raw HTTP:

  • GET / HTTP/1.1: Get the page at the absolute path / using HTTP 1.1.
  • Host: weblog.patrice.ch: This tells the web server that we’d like to retrieve the absolute path given in the previous line from the weblog.patrice.ch host. This must be specified with HTTP 1.1 and allows the server to serve different sites from the same IP address.
  • Connection: Close: This turns off Keep-Alive. The server can free resources more quickly and the TCP connection is closed immediately.
  • Empty line: This three lines are followed by an empty line. In HTTP the first empty line separates the headers from the body.

The HTTP request is piped to the nc command, netcat. The two parameters signify that we want to open a connection to weblog.patrice.ch on port 80. Instead of weblog.patrice.ch you could also provide it’s IP address.

POST requests with netcat

Even more useful is doing POST requests with netcat. An example would be:

<code>echo "POST /test.php HTTP/1.1
Host: myserver
Connection: Close
Content-Length: 10

key=value" | nc myserver 80 | less</code>

This sends the data key=value to the server. You have to specify the Content-Length header so the server knows how much to read. When the request data is all in one line you can easily count the number of characters in your editor. Just place the whole line in your editor, position your cursor at the end of the line and look at the column that your editor gives you. That number minus one has to be used as the content length.

With longer requests I usually edit the request in a file saved on the disk. I then invoke netcat with the following command:

<code>nc myserver 80 <request.txt | less</code>

That’s a bit more comfortable for large data than editing everything in the shell.

Netcat is included with Mac OS X. Thank you Apple.

tcpdump

When investigating the backend/frontend communication I sometimes have a need to see exactly what’s going over the wire. Tcpdump shows you the communication in that case.

<code>sudo tcpdump -efli en1 -XX -s0 | tee /tmp/tcpdump.log</code>

The params I use are the following:

  • -e: Print the link-level header on each dump line.
  • -f: Print `foreign’ IPv4 addresses numerically rather than symbolically (I like IP addresses more then hostnames in this context)
  • -i en1: Listen on interface en1 (my wireless interface). Have to use en0 for my network card
  • -XX: Print packets in hex and ASCII
  • -s0: Print whole packets, not only first 68 chars.

The tee command shows me the data on the screen while everything is also logged into /tmp/tcpdump.log for later consultation.

If you’re only interested in one specify communication, you can add some filters. For example:

<code>sudo tcpdump -efli en1 -XX -s0 'port 8080' | tee /tmp/tcpdump.log</code>

This command will only include output going to or coming form port 8080. The tcpdump man page gives extended information about the possible filters.

Tcpdump is included with Mac OS X. Thank you Apple.

dig

Often I need to do some DNS debugging. This is not directly related to HTTP but very often the problem is actually in the DNS system when something doesn’t work. A lot of people use nslookup for doing name server lookups. I prefer dig. Let me show you a few commands:

  • Look up a host: dig www.local.ch
  • Very short output: dig +short www.local.ch
  • Reverse-lookup of an IP address: dig -x 195.141.88.144 (no need to manually put together the in-addr.arpa address.
  • Again with very short output: dig +short -x 195.141.88.144
  • Check at a specific name server: dig @ns1.local.ch mail.local.ch
  • Get MX record: dig local.ch MX (If you leave out the type dig searches for A records)

I think dig is part of the BIND distribution. It’s included with Mac OS X by default. Thank you Apple. (Yes I know I repeat myself)

Conclusion

I presented the following five tools:

Those tools are basically everything I use for debugging of HTTP requests.

Please leave any questions or contributions in the comments.