Joe was pondering to make the new version a commercial release. Instead he has now decided to leave it Open Source but asks for donations.
LiveHTTPHeaders is a Firefox extension that allows to see the headers exchanged between the browser and server. It adds a “Headers” tab in the page information. This window can be accessed with Apple+I on the Mac or Ctrl+I on Linux and Windows. The headers tab shows a list of the headers that Firefox sent to the server and a separate list of the response headers sent by the server.
I use that mostly to see the Cookie and Accept-Language headers that Firefox sent.
GET requests with netcat
Netcat is in my experience one of the most overlooked tools for network debugging. It allows you to easily send network requests to any TCP or UDP server. While you can also open TCP connections with good old telnet, netcat offers one big advantage: it reads the data to be sent through the standard input channel. This allows you to replay requests a lot easier than with telnet. For example I usually do the following in a bash shell:
echo “GET / HTTP/1.1
“ | nc weblog.patrice.ch 80 | less
This opens / and shows the full HTTP responses in
less. Let me explain the three requests lines for those who never have seen raw HTTP:
GET / HTTP/1.1: Get the page at the absolute path / using HTTP 1.1.
Host: weblog.patrice.ch: This tells the web server that we’d like to retrieve the absolute path given in the previous line from the weblog.patrice.ch host. This must be specified with HTTP 1.1 and allows the server to serve different sites from the same IP address.
Connection: Close: This turns off Keep-Alive. The server can free resources more quickly and the TCP connection is closed immediately.
- Empty line: This three lines are followed by an empty line. In HTTP the first empty line separates the headers from the body.
The HTTP request is piped to the
nc command, netcat. The two parameters signify that we want to open a connection to
weblog.patrice.ch on port
80. Instead of
weblog.patrice.ch you could also provide it’s IP address.
POST requests with netcat
Even more useful is doing POST requests with netcat. An example would be:
echo “POST /test.php HTTP/1.1
key=value” | nc myserver 80 | less
This sends the data
key=value to the server. You have to specify the
Content-Length header so the server knows how much to read. When the request data is all in one line you can easily count the number of characters in your editor. Just place the whole line in your editor, position your cursor at the end of the line and look at the column that your editor gives you. That number minus one has to be used as the content length.
With longer requests I usually edit the request in a file saved on the disk. I then invoke netcat with the following command:
nc myserver 80 <request.txt | less
That’s a bit more comfortable for large data than editing everything in the shell.
Netcat is included with Mac OS X. Thank you Apple.
When investigating the backend/frontend communication I sometimes have a need to see exactly what’s going over the wire. Tcpdump shows you the communication in that case.
sudo tcpdump -efli en1 -XX -s0 | tee /tmp/tcpdump.log
The params I use are the following:
- -e: Print the link-level header on each dump line.
- -f: Print `foreign’ IPv4 addresses numerically rather than symbolically (I like IP addresses more then hostnames in this context)
- -i en1: Listen on interface en1 (my wireless interface). Have to use en0 for my network card
- -XX: Print packets in hex and ASCII
- -s0: Print whole packets, not only first 68 chars.
tee command shows me the data on the screen while everything is also logged into /tmp/tcpdump.log for later consultation.
If you’re only interested in one specify communication, you can add some filters. For example:
sudo tcpdump -efli en1 -XX -s0 'port 8080' | tee /tmp/tcpdump.log
This command will only include output going to or coming form port 8080. The tcpdump man page gives extended information about the possible filters.
Tcpdump is included with Mac OS X. Thank you Apple.
Often I need to do some DNS debugging. This is not directly related to HTTP but very often the problem is actually in the DNS system when something doesn’t work. A lot of people use nslookup for doing name server lookups. I prefer dig. Let me show you a few commands:
- Look up a host:
- Very short output:
dig +short www.local.ch
- Reverse-lookup of an IP address:
dig -x 184.108.40.206(no need to manually put together the in-addr.arpa address.
- Again with very short output:
dig +short -x 220.127.116.11
- Check at a specific name server:
dig @ns1.local.ch mail.local.ch
- Get MX record:
dig local.ch MX(If you leave out the type dig searches for A records)
I think dig is part of the BIND distribution. It’s included with Mac OS X by default. Thank you Apple. (Yes I know I repeat myself)
I presented the following five tools:
Those tools are basically everything I use for debugging of HTTP requests.
Please leave any questions or contributions in the comments.