cURL and HTTP Transaction Timing
curl is one of the most used commands for HTTP transactions. It’s a Swiss army knife for network and application engineers when they want to quickly test whether a server application is responsive, and it also has a myriad of options to configure proxy servers, username-password pairs, encryption protocols, redirections, and much more. Our purpose here is to give a quick overview of how you can use curl to break down the timing of an HTTP transaction from name lookup to the completion of the HTTP request. For an exhaustive list of options, you can take a look here: http://curl.haxx.se/docs/manpage.html
How to Use cURL
In the following examples, I am using an Ubuntu-based console. However, curl has been ported to all Unix flavors and you can also find Microsoft Windows implementations online.
Step 1: Fetch an HTML page with curl
The basic command syntax for curl to fetch a page such as google.com is as follows:
$curl 'google.com' 301 Moved The document has moved here.
Step 2: Follow page redirections with curl
As you can see, curl printed on the console whatever was returned by Google’s server. Apparently “google.com” is a redirection to “http://www.google.com/.” curl doesn’t follow redirection, unless we give the input option -L. So, now try the following:
$curl –L ‘google.com’ <!doctype html>
I only included the beginning of what curl returns on the console. The whole return text includes 7 lines, 462 words, and 19098 characters. It’s quite ugly for humans.
Step 3: Print only necessary information with curl
It’s interesting to see what hides behind Google’s home page, but it’s not what we normally want to look at when we use curl. So, we can make all that source code disappear by redirecting it to /dev/null as follows:
$curl -L –output /dev/null ‘google.com’ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 219 100 219 0 0 1610 0 –:–:– –:–:– –:–:– 3590 0 0 0 19106 0 0 64463 0 –:–:– –:–:– –:–:– 186k
Now, curl printed some information about the progress of the page fetch and nothing more than that. If you don’t want to print the progress info you can use the option --silent as follows:
$curl -L --output /dev/null --silent 'google.com'
If you run this command you won’t see any output on the console. Not even any error messages. For example, if you used “googleasdf.com” curl wouldn’t even print that it can’t find the host. In order to tell curl to print error messages we add the --show-error option as follows:
$curl -L --output /dev/null --silent --show-error 'googleasdf.com' curl: (6) Couldn't resolve host 'googleasdf.com'
As you can see now curl tells us that it failed to resolve “googleasdf.com.”
Step 4: Print timing breakdown with curl
But we still don’t see the timing breakdown of the HTTP request as promised. If we go through all the options supported by curl (http://curl.haxx.se/docs/manpage.html) we will see that it has the capability of returning timing information on the hostname lookup (time_namelookup), the time it takes to connect to the remote host (time_connect), the time it takes to start transferring data (time_pretrasnfer), the total time (total_time) and many others. In order to display this information on our console in a pretty and human readable format copy and paste the following command:
$ curl -L --output /dev/null --silent --show-error --write-out 'lookup: %{time_namelookup}\nconnect: %{time_connect}\nappconnect: %{time_appconnect}\npretransfer: %{time_pretransfer}\nredirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal: %{time_total}\n' 'google.com'
lookup: 0.036
connect: 0.053
appconnect: 0.000
pretransfer: 0.053
redirect: 0.105
starttransfer: 0.118
total: 0.253
appconnect is the time, in seconds, it took from the start until the SSL/SSH/etc. connect/handshake to the remote host was completed. In the example above it is zero because we don’t use the HTTPS protocol. If you try the same command with “https://google.com” you get the following nonzero value for appconnect:
$ curl -L –output /dev/null –silent –show-error –write-out ‘lookup: %{time_namelookup}\nconnect: %{time_connect}\nappconnect: %{time_appconnect}\npretransfer: %{time_pretransfer}\nredirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal: %{time_total}\n’ ‘https://google.com’
lookup: 0.038
connect: 0.063
appconnect: 0.177
pretransfer: 0.178
redirect: 0.225
starttransfer: 0.679
total: 0.929
Curl Timing Values
Here is what each value represents, according to curl’s manual page:
- lookup: The time, in seconds, it took from the start until the name resolving was completed.
- connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
- appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
- pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
- redirect: The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
- starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
- total: The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.
In the above commands, if you replace “google.com” with “www.google.com” then there is no redirection to be taken, and obviously the “redirect” value will be zero.
cURL Benefits
Let’s say that a user complains about an application being slow. With the above curl command, we can check how long it takes to resolve the hostname of the server, how long it takes to connect to the server, and how long to fetch the HTML application page from the server. If some values are unreasonable (e.g. name lookup taking 3 seconds) we can get a step closer to the root cause of the problem.
Obviously, to simulate as closely as possible what the user experiences, you would have to run the curl command on the user’s workstation. When this is not possible, you would have to either walk to the user’s location and run the command on your workstation or connect remotely to a machine at the user’s location. Of course, curl doesn’t take into account any delay that has to do with page rendering. It breaks down the name lookup, protocol negotiation timing, and the byte transfer timing. Nonetheless, this can help us know if the server is accessible, responsive, and how fast it is. At the very least, it can help us prove that the problem is not with the network or the application and focus on the end user’s workstation.
If you want to learn more how NetBeez implements web performance monitoring I encourage you to schedule a demo.
