How to Monitor Packet Drops with Dropwatch

We’ve talked about Linux tools that enable you to manipulate kernel networking parameters such as TCP window size and packet loss or latency with TC. These kinds of tools help optimize or troubleshoot network traffic either at the application or kernel level.

As the name suggests, dropwatch is a tool that allows you to monitor and track packet loss within the Linux networking stack. This utility is available for download at https://github.com/nhorman/dropwatch.

Dropwatch Installation

It doesn’t look like dropwatch is readily available within Linux repositories, so you will most likely need to install it from source by downloading it from here. Once downloaded, you have to run the following commands to install it:

./autogen.sh
./configure
make
make install

If you get any errors, you may be missing one or more libraries. Running the following command should fix this problem:

sudo apt-get install -y libnl-3-dev libnl-genl-3-dev binutils-dev libreadline6-dev

Once these libraries are downloaded, re-run the installation commands.

Dropwatch Usage

Dropwatch doesn’t have many options, as you can see from its documentation

If you try to run dropwatch and get the following error, it means that the kernel of your system doesn’t support dropwatch’s functionality.

netbeez.net$ dropwatch -lkas
Unable to find NET_DM family, dropwatch can't work
Cleaning up on socket creation error

The most common way is to start it with the -lkas and when prompted type “Start.”

Then you will see the following output.

netbeez$ dropwatch -lkas
Initializing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
4 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]

And this will keep going until you hit Ctrl-C to stop it.

As you can see there are some dropped packages related to software, and dropwatch gives us the kernel symbol or function related to each packet drop. If you don’t use the -lkas option you will just get the address of the symbol (e.g. 0xffffffff9d5123d0) which is much more cryptic.

Testing Packet Loss

To test a bit more dropwatch we can try to force some packet drops on the host, and one way to do this is by running an iPert test while dropwatch is running and monitoring for dropped packets. iPerf is a utility that tests network bandwidth by pushing as much traffic as possible between two hosts. As you know, when the OS hits the limit of the interface bandwidth limit some packets will be dropped.

Here is how this looks like:

netbeez@ :src:$ ./dropwatch -lkas
Initializing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
5 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
3 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at tcp_v4_rcv+158 (0xffffffff9d57b6c8) [software]
4 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
1738 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at unix_stream_connect+2e8 (0xffffffff9d5bd3f8) [software]
2988 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2585 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2585 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2573 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2833 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2387 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2545 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2704 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
3055 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
408 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
1 drops at tcp_v4_rcv+158 (0xffffffff9d57b6c8) [software]
1 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
1 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
1 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]

In the first few iterations we see a handful of packet drops reported. Once iPerf kicks in, the number of packets dropped increases. Once iperf ends, this number will reduce.

decoration image

Get your free trial now

Monitor your network from the user perspective

You can share

Twitter Linkedin Facebook

Let's keep in touch

decoration image