Linux for Network Engineers: How to Monitor Packet Drops with Dropwatch

By July 29, 2020Linux

We’ve talked about Linux tools that enable you to manipulate kernel networking parameters such as TCP window size and packet loss or latency. These kinds of tools are used when trying to optimize or troubleshoot network traffic either at the application or kernel level.

When it comes to measuring the impact of your changes, one of the tools you can use is dropwatch (https://github.com/nhorman/dropwatch). As the name suggests, dropwatch enables you to monitor where packets are dropped on the Linux networking stack.

Dropwatch Installation

It doesn’t look like dropwatch is readily available within Linux repositories, so you will most likely need to install it from source by downloading it from here. Once downloaded, you have to run the following commands to install it:

./autogen.sh
./configure
make
make install

If you get any errors, you may be missing one or more libraries. Running the following command should fix this problem:

sudo apt-get install -y libnl-3-dev libnl-genl-3-dev binutils-dev libreadline6-dev

Once these libraries are downloaded, re-run the installation commands.

Dropwatch Usage

Dropwatch doesn’t have many options, as you can see from its documentation

If you try to run dropwatch and get the following error, it means that the kernel of your system doesn’t support dropwatch’s functionality.

netbeez.net$ dropwatch -lkas
Unable to find NET_DM family, dropwatch can't work
Cleaning up on socket creation error

The most common way is to start it with the -lkas and when prompted type “Start.”

Then you will see the following output.

netbeez$ dropwatch -lkas
Initializing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
4 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]

And this will keep going until you hit Ctrl-C to stop it.

As you can see there are some dropped packages related to software, and dropwatch gives us the kernel symbol or function related to each packet drop. If you don’t use the -lkas option you will just get the address of the symbol (e.g. 0xffffffff9d5123d0) which is much more cryptic.

Testing Packet Loss

To test a bit more dropwatch we can try to force some packet drops on the host, and one way to do this is by running an iPert test while dropwatch is running and monitoring for dropped packets. iPerf is a utility that tests network bandwidth by pushing as much traffic as possible between two hosts. As you know, when the OS hits the limit of the interface bandwidth limit some packets will be dropped.

Here is how this looks like:

netbeez@ :src:$ ./dropwatch -lkas
Initializing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
5 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
3 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at tcp_v4_rcv+158 (0xffffffff9d57b6c8) [software]
4 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
1738 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2 drops at unix_stream_connect+2e8 (0xffffffff9d5bd3f8) [software]
2988 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2585 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2585 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2573 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2833 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2387 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2545 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
2704 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
3055 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
408 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
1 drops at tcp_v4_rcv+158 (0xffffffff9d57b6c8) [software]
1 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
1 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]
1 drops at net_tx_action+0 (0xffffffff9d5123d0) [software]

In the first few iterations we see a handful of packet drops reported. Once iPerf kicks in, the packet drop increases and then it dies off again once it’s done.