What is the Purpose of Measurement Swiss Army-Knife (MSAK)

At NetBeez, we’re pretty familiar with the different tools that help measure upload and download speeds, latency, jitter, and bandwidth. There are a lot of options out there. Previously, we talked about how speed tests from Ookla, NDT, Fast.com, Cloudflare, and more work and how they differ from each other. Our monitoring tool can analyze data from hundreds or thousands of end users by orchestrating and collecting measurements in an automated way. Why do we need another speed test when we already have so many options?

The answer is that FCC has laid out requirements of how end-user ISP bandwidth, latency, jitter, and packet loss should be measured to have a standard way of reporting and comparing values from different sources, and be able to submit them to the Broadband Data Collection Program. Actually, the FCC released their own speedtest mobile app that helps dispute, or “challenge,” coverage data submitted by providers. The Measurement Lab (M-Lab) has also introduced an open source speedtest that fits right in with the FCC’s rules. They call it the “Measurement Swiss-Army Knife,” or MSAK for short.

MSAK vs. NDT

M-Lab currently offers an expansive network of servers and an open-source speedtest client. NDT doesn’t run speedtest according to FCC’s requirements. Namely, it doesn’t support multi-stream testing and UDP-based latency test that runs together with the multi-stream test. MSAK is going to close this gap, and they are currently calibrating the new test by allowing users to run two back-to-back tests between the production NDT and the under-development MSAK. You can test that at https://speed.measurementlab.net/#/ 

The end result looks like this:

If you want to run the MSAK client locally, you can follow the instructions on their github repository: https://github.com/m-lab/msak

MSAK Server/Client

The MSAK client and server are written in Go, so you first need to install Go on your system. After that you can install the following:

go install github.com/m-lab/msak/cmd/msak-server@latest
go install github.com/m-lab/msak/cmd/msak-client@latest
go install github.com/m-lab/msak/cmd/minimal-download@latest
go install github.com/m-lab/msak/cmd/msak-latency@latest

You can also host your own MSAK server if interested. The client performs an upload, download, and latency test. Use the minimal download client if you want to run a download and latency test. Lastly, use the latency client to just run a latency test. 

Here is an example of running my own MSAK server and testing from the same host:

Server:

msak-server  2>&1 | grep -v WARN
2024/07/02 15:31:11 INFO <msak-server/server.go:117> About to listen for ws tests endpoint=:8080
2024/07/02 15:31:11 INFO <latency1/latency1.go:286> Accepting UDP packets...

Client:

msak-client  -server localhost:8080 -scheme ws
Starting download stream (server: localhost:8080)
Starting download stream (server: localhost:8080)
Starting download stream (server: localhost:8080)
…
Elapsed: 4.93s, Goodput: 46383.428887 Mb/s, MinRTT: 0
…
Starting upload stream (server: localhost:8080)
Starting upload stream (server: localhost:8080)
Starting upload stream (server: localhost:8080)
…
Elapsed: 4.97s, Goodput: 49323.453477 Mb/s, MinRTT: 0

I summarized the output for clarity, but as you can see by default the client starts a download test with 3 parallel streams and then follows with an upload test with 3 streams as well. The upload and download are in the order for 50Gbps, which is expected since the client and server run on the same host. What I noticed is that the Round Trip Time (RTT) always appears as 0. I assume this is a bug that needs to be fixed.

If I run the client without any arguments I get the following:

msak-client 
Starting download stream (server: msak-mlab3-nuq06.mlab-oti.measurement-lab.org)
Starting download stream (server: msak-mlab3-nuq06.mlab-oti.measurement-lab.org)
Starting download stream (server: msak-mlab3-nuq06.mlab-oti.measurement-lab.org)
…
Elapsed: 4.35s, Goodput: 252.563276 Mb/s, MinRTT: 0
…
Starting download stream (server: msak-mlab2-nuq04.mlab-oti.measurement-lab.org)
Starting download stream (server: msak-mlab2-nuq04.mlab-oti.measurement-lab.org)
Starting download stream (server: msak-mlab2-nuq04.mlab-oti.measurement-lab.org)
…
Elapsed: 4.91s, Goodput: 244.633808 Mb/s, MinRTT: 0

As you can see now the client found the closets available server, and the measurements it took are around 250 Mbs for download and upload. That’s on par with the web-based speedtest from the screenshot above.

Similarly, the minimal download client output looks like this:

minimal-download                         
Connecting: wss://msak-mlab3-nuq03.mlab-oti.measurement-lab.org/throughput/v1/download?...
…
Download server #1 rate: 280.83 Mbps, rtt 11.86ms, elapsed 5.0104s, application r/w: 0/179332695, network r/w: 0/179581633 kernel* r/w: 1295/175887693
------
Download total average:  281.29 Mbps, MinRTT  3.80ms, elapsed 5.1003s, application r/w: 0/179333746

This output shows a similar download measurement, but also shows the latency and a summary of the results. This is something that’s missing from the msak-client above.

Finally, the latency client output looks like this:

msak-latency                  
2024/07/02 15:41:25 Argument locate.url=https://locate.measurementlab.net/v2/nearest/
2024/07/02 15:41:25 Argument mid=
2024/07/02 15:41:25 Argument prometheusx.listen-address=:9990
2024/07/02 15:41:25 Argument scheme=http
2024/07/02 15:41:25 Argument server=
Attempting to connect to: http://msak-mlab1-nuq08.mlab-oti.measurement-lab.org/latency…
…

rtt min/avg/max: 5.634/6.440/13.516 ms, loss: 0.0

Conclusion

MSAK is designed to align with FCC requirements for measuring internet speed, addressing gaps in existing tools like NDT. It supports multi-stream testing and UDP-based latency testing. Currently, M-Lab has made this tool available, and it was great to test it. Can’t wait for a better version with all the bugs and problems fixed.