Simulating network failure with iptables

There’s many situations where it’s desirable to have a network failure during testing, however this can sometimes seem like an impossible thing to replicate in a predictable manner for testing your failure mode. Let’s talk about iptables.

This post was originally appeared on the Adzuna Engineering Blog.

A blank iptables filter looks like this:

# Generated by iptables-save v1.6.1 on Wed Jul 29 09:56:26 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Wed Jul 29 09:56:26 2020

You might have other rules if your machine is on a shared network or such, but let’s go from the default and work from there.

Take a backup of your existing config before we begin and then copy it:

sudo iptables-save > iptables-save-`date +%Y%m%d`
cp iptables-save-`date +%Y%m%d iptables-save-new

Note that if you are working on a machine remotely, messing with iptables can lock you out of the machine until it’s rebooted so take care.

We have a MySQL database that we want to replicate failure to, let’s use 10.10.10.10 as the IP address of that DB. We want to add a new chain to the filter table.

Let’s add networktest:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:networktest - [0:0]
COMMIT

Now we want to send traffic destined for MySQL to that chain:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:networktest - [0:0]

# Send MySQL traffic to network test chain
-A OUTPUT -d 10.10.10.10 -p tcp -m tcp --destination-port 3306 -j networktest

COMMIT

Let’s test:

root@iann-desktop:~# telnet 10.10.10.10 3306
Trying 10.10.10.10...
^C
root@iann-desktop:~# iptables --list OUTPUT -v
Chain OUTPUT (policy ACCEPT 2766 packets, 459K bytes)
 pkts bytes target     prot opt in     out     source               destination
    3   180 networktest  tcp  --  any    any     anywhere             10.10.10.10          tcp dpt:mysql

Now we can control access to the destination with a simple command, add a DROP rule to the networktest chain:

sudo iptables -A networktest -j DROP

This will drop outgoing packets as if they were never sent. If you want a faster response and your service to be aware, use REJECT instead.

Restore access by flushing the networktest chain (remove all rules):

sudo iptables -F networktest

Restore your backup once you’re done if you don’t want to persist this:

sudo iptables-restore < iptables-save-`date +%Y%m%d`

It’s left as an exercise for the reader to look at further iptables extensions man page for things like rate limiting and hashlimit that allow a more nuanced approach to packet loss rather than total loss.