Speedtest Comparison: Ookla, NDT, NetFlix, HTML, iPerf

(TL;DR: this is a write up of the talk I gave at the Wireless LAN Professionals conference in February 2018. You can find the slide deck and video here.)

If you have a few years’ experience in network troubleshooting, you know that when you receive a ticket from a user complaining about “slowness”, in most cases, it’s the user’s device or the application. But you are still tasked with proving that, “It’s Not the Network.”

There are many ways to do this, but the ‘quick and dirty’ way to test the network connection is to run a speedtest – such as the Ookla speedtest. There are a dozen similar tests (with every major TelCo provider having their own speedtest) and a number of free and paid mobile device apps available. In this post, we’ll cover five of them, their different features and how they can be used in different scenarios.

It’s pretty straightforward how all of these speedtests work: a device uploads and downloads a large file to a server, and based on how long it takes, it calculates the connection bandwidth. In addition, it can measure some other metrics as well such as latency and jitter.

1. Ookla

This is the most well known and most commonly used speedtest. You can ask the user that complained to go to speedtest.net, click the “Go” button and tell you what they see on their browser.

If these numbers are high enough (high enough being dependent on the user’s requirements) then you can at least know that the bandwidth between the user’s device and the Internet (where the Ookla server is) is not an issue.

Since this test uses an Internet-based server the test path is the following:

The path includes the WiFi (or wired network), the backhaul, the Internet connection between the edge of the infrastructure and the server, and finally the server. Consequently, if the measured bandwidth is low, we don’t know what part of the connection is responsible. In addition, if the public server is running more than one speedtests then it might not be able to give accurate and consistent measurements. There are around 7,000 public Ookla servers listed here.

 

2. Google NDT

If you google “speedtest” you will find the following prompt:

Google has partnered with the Measurement Lab (M-Lab) and promotes their speedtest, NDT. This is similar to Ookla, in the sense that you can use public servers maintained by third parties for testing. There are around 150 public NDT servers, however, NDT is an open-source project; this opens the option to install a test server in your own network and enables you to isolate parts of it as follows:

By having full control of the server the measurements are more accurate and consistent, and you are able to verify if bandwidth issues are caused by internal networking issues or not. In addition, NDT has a command line test utility that you can use to script or automate this measurement.

 

3. Netflix Fast

Netflix maintains their own speedtest at fast.com, in order to give their users ISP-independent testing that also use Netflix servers. The difference with Ookla is that it uses Netflix servers exclusively and only reports download bandwidth. Obviously, streaming customers are more interested in their download rather than their upload bandwidth, and Netflix provides a simple and ad-free test for that. I like Fast because of its simplicity: as soon as fast.com loads, the test starts running and immediately provides value. There is no button to click. It can’t be dumber than that.

4. Open Source HTML

This is another open source HTML based test that you can find here. There are no public servers for this test, so if you want to use it, you’d have to set up your own server and install the software.

It gives more or less the same information as Ookla and NDT, with the addition of jitter which can be useful when debugging VoIP issues.

 

5. iPerf

iPerf is a utility that can be used on the command line or on a GUI to test bandwidth. It’s not browser-based, and it requires to download or install software on the host and the server used for the testing. In that regard, it’s a bit more involved to use, but it gives far more flexibility and options. For example, you can test TCP and UDP traffic, and specify parameters such as the TCP window size, the type-of-service (TOS) for outgoing packets, the TCP maximum segment size, and others. There are a few public iPerf servers but are not well maintained and reliable.  For more details on how to use iPerf you can read this post.

The command line output of iPerf looks like this:

Comparison Table

In the following table, I have sorted all these tests with respect to how easy they are to use, with fast.com being the easiest, and iPerf the least easy. The benefit of using iPerf is that you have the ability to customize many parameters that are not possible to customize with the HTML-based tests.

 

Easy Custom Private Server Scripting # Public

servers

Notes
NetFlix Fast 4 1 No No ? Only download
Ookla 3 1 No Yes ~7000 3rd party scripting, App
NDT 3 2 Yes Yes ~150 Native scripting
HTML Speedtest 2 2 Yes Yes 1 Jitter
iPerf 1 4 Yes Yes 5-10 TCP/UDP, jitter, loss, App

Consistency

If you run all of the previous five tests on the same hosts and on the same network you will notice significant discrepancies in the bandwidth measurements they provide. In addition, if you try the same test on two different browsers you might also see discrepancies in the measurements. This is because each speedtest implementation might handle data and timing differently. For example, some tests discard the first few seconds worth of measurements because those values tend to be lower than the average while the server and the device ramp up the upload and download process. According to my experience, the most reliable and accurate test is iPerf.

If you were to ask, “Then…which test should I use?” I would answer: “Pick one and be consistent”. The important observation with these tests is to look for changes in the measurements, and if you have some historical measurements and a baseline you can detect if there are any issues that might affect your users.

 

NetBeez

At NetBeez, we use network monitoring to detect issues that affect the end user. NetBeez works by installing agents on both wireless and wired networks that run periodic tests to measure latency, HTTP response time, bandwidth and more. On the NetBeez dashboard, you can automate the execution of bandwidth testing with iPerf and Ookla to run on a specific schedule and collect the historical information to build statistics, baselines, and get alerted when something goes wrong.

In addition with NetBeez, you can collect measurements on dozens or hundreds of agents which allows you to capture user experience data from all your remote sites and be proactive with any issues that may come up. You can detect and troubleshoot WiFi and wired issues proactively even before they affect your end user.