HTTP transaction timing breakdown with curl

curl-refined
curl is one of the most used commands for HTTP transactions. It’s a Swiss army knife for network and application engineers when they want to quickly test whether a server application is responsive, and it also has a myriad of options to configure proxy servers, username-password pairs, encryption protocols, redirections, and much more. Our purpose here is to give a quick overview of how you can use curl to break down the timing of an HTTP transaction from namelookup to the completion of the HTTP request.

For an exhaustive list of options, you can take a look here: http://curl.haxx.se/docs/manpage.html

In the following examples,  I am using an Ubuntu-based console. However, curl has been ported to all Unix flavors and you can also find Microsoft Windows implementations online.

Step 1: Fetch an HTML page

The basic command syntax for curl to fetch a page such as google.com is as follows:

301 Moved

The document has moved
here.

Step 2: Follow page redirections

As you can see, curl printed on the console whatever was returned by Google’s server. Apparently “google.com” is a redirection to “http://www.google.com/.” curl doesn’t follow redirection, unless we give the input option -L. So, now try the following:

I only included the beginning of what curl returns on the console. The whole return text includes 7 lines, 462 words, and 19098 characters. It’s quite ugly for humans.

Step 3: Print only necessary information

It’s interesting to see what hides behind Google’s home page, but it’s not what we normally want to look at when we use curl. So, we can make all that source code disappear by redirecting it to /dev/null as follows:

Now, curl printed some information about the progress of the page fetch and nothing more than that. If you don’t want to print the progress info you can use the option --silent as follows:

If you run this command you won’t see any output on the console. Not even any error messages. For example, if you used “googleasdf.com” curl wouldn’t even print that it can’t find the host. In order to tell curl to print error messages we add the --show-error option as follows:

As you can see now curl tells us that it failed to resolve “googleasdf.com.”

(Watch the video review on NetBeez that Joel Snyder released on Network World here.)

Step 4: Print timing breakdown

But we still don’t see the timing breakdown of the HTTP request as promised. If we go through all the options supported by curl (http://curl.haxx.se/docs/manpage.html) we will see that it has the capability of returning timing information on the hostname lookup ( time_namelookup), the time it takes to connect to the remote host ( time_connect), the time it takes to start transferring data ( time_pretrasnfer), the total time ( total_time) and many others. In order to display this information on our console in a pretty and human readable format copy and paste the following command:

appconnect is the time, in seconds, it took from the start until the SSL/SSH/etc. connect/handshake to the remote host was completed. In the example above it is zero because we don’t use the HTTPS protocol. If you try the same command with “https://google.com” you get the following nonzero value for appconnect:

Here is what each value represents, according to curl’s manual page:

  • lookup: The time, in seconds, it took from the start until the name resolving was completed.
  • connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
  • appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
  • pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
  • redirect: The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
  • starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
  • total: The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.

In the above commands, if you replace “google.com” with “www.google.com” then there is no redirection to be taken, and obviously the “redirect” value will be zero.

But how is this useful?

Let’s say that a user complains about an application being slow. With the above curl command, we can check how long it takes to resolve the hostname of the server, how long it takes to connect to the server, and how long to fetch the HTML application page from the server. If some values are unreasonable (e.g. name lookup taking 3 seconds) we can get a step closer to the root cause of the problem.

Obviously, to simulate as closely as possible what the user experiences, you would have to run the curl command on the user’s workstation. When this is not possible, you would have to either walk to the user’s location and run the command on your workstation or connect remotely to a machine at the user’s location. Of course, curl doesn’t take into account any delay that has to do with page rendering. It breaks down the name lookup, protocol negotiation timing, and the byte transfer timing. Nonetheless, this can help us know if the server is accessible, responsive, and how fast it is. At the very least, it can help us prove that the problem is not with the network or the application and focus on the end user’s workstation.

If you want to learn more how NetBeez implements end-user network and application monitoring I encourage you to schedule a demo with a representative from our team or attend on July the 30th at 2:00PM EDT our webinar that will show how to use NetBeez to monitor DNS and cloud services.

  • Kevin

    Your last 2 examples need to use “–” for the options and use single quotes instead of a back tick. The backtick may just be how the page renders characters, but for options that are words, — is needed.

    > curl -L –output /dev/null –silent –show-error –write-out ‘lookup: %{time_namelookup}nconnect: %{time_connect}nappconnect: %{time_appconnect}npretransfer: %{time_pretransfer}nredirect: %{time_redirect}nstarttransfer: %{time_starttransfer}ntotal: %{time_total}n’ ‘google.com’
    lookup: 0.005
    connect: 0.025
    appconnect: 0.000
    pretransfer: 0.025
    redirect: 0.057
    starttransfer: 0.078
    total: 0.158

    • Panagiotis Vouzis

      Kevin, thanks for catching this. WordPress converted the double dash to a single dash. As you can see, it did the same with your comment. I will try to fix this. Thanks for catching it.

      Panos

  • Pingback: Top 10 Post on NetBeez Blog for 2015 | NetBeez – Distributed network monitoring()