Using docker with multiple internet connections
An early iteration of the probe setup for blocked.org.uk used a single docker host which had four mobile internet USB adaptors connected. The probe server ran a probe docker container for each of the network interfaces. This article describes the linux routing that allowed each of the docker containers to use a different mobile internet connection for its outbound traffic.
This setup uses the Routing table management features in Linux. This was used back in 2015, so there might be a more "dockerish" way to do them now.
Mobile internet setup
The mobile internet connections aren't the main focus of the article, so I'll skip over those a bit. Each of the mobile internet connections was set up in the usual way, making a note of the local subnet[1] and default gateway[2] that was set for each connection (these details are needed later). The pppd configs for the mobile dongles were then set so that they didn't set the default gateway when the connections were started up.
The routing table (given with the ip route show command) would display the normal routing table for the docker host's lan:
default via 192.168.28.254 dev en0 proto dhcp metric 600
192.168.28.0/24 dev en0 proto kernel scope link src 192.168.28.83 metric 600
Setting up docker networks
A docker network was created for each of the four containers:
docker network create --subnet 192.168.100.0/24 vodafone
docker network create --subnet 192.168.101.0/24 o2
docker network create --subnet 192.168.102.0/24 three
docker network create --subnet 192.168.103.0/24 ee
Setting up routing tables
A new linux routing table was created for each of the four mobile networks, using the subnet and default gateway recorded at [1] and [2] above:
# vodafone
ip route table add 1100 to <mobile subnet> dev ppp0
ip route table add 1100 default via <mobile default gateway> dev ppp0
# O2
ip route table add 1101 to <mobile subnet> dev ppp0
ip route table add 1101 default via <mobile default gateway> dev ppp1
# Three
ip route table add 1102 to <mobile subnet> dev ppp0
ip route table add 1102 default via <mobile default gateway> dev ppp2
# EE
ip route table add 1103 to <mobile subnet> dev ppp0
ip route table add 1103 default via <mobile default gateway> dev ppp3
Setting outbound rules
The ip rule command was used to assign the outbound traffic from each container network to a different mobile network. Similar to iptables rules being used to filter traffic, these rules are used to direct traffic out of the docker host.
For each mobile interface and docker network, a rule was added to link the two
ip rule add from 192.168.100.0/24 lookup 1100 priority 1100
ip rule add from 192.168.101.0/24 lookup 1101 priority 1101
ip rule add from 192.168.102.0/24 lookup 1102 priority 1102
ip rule add from 192.168.103.0/24 lookup 1103 priority 1103
Each of these rules tells the kernel which new routing table goes with traffic coming from each of the subnets.
Finally, an instance of the probe container was started, one on each of the four docker networks. The network traffic coming out of the probe would be sent to whichever mobile internet device was required.
docker run --name probe_vodafone --network vodafone probe:latest
docker run --name probe_o2 --network o2 probe:latest
docker run --name probe_three --network three probe:latest
docker run --name probe_ee --network ee probe:latest
This allowed the probes to be isolated from each other and to only send traffic over their assigned mobile device. The normal iptables masquerade rules ensure that traffic from the docker networks is NATted to give the correct source IP over the mobile internet.
Other applications
To make a docker container use a VPN connection instead of the defaut route of the docker host, you could set up the VPN connection on tun0 (for openvpn), making a note of its local subnet (10.69.1.0/24 in this example) and default gateway (10.69.1.1). The vpn would be configured not to set the default gateway in the main routing table.
You could then create a new routing table:
route add table 20000 to 10.69.1.0/24 dev tun0
route add table 20000 default via 10.69.1.1 dev tun0
and add a routing rule based on the container's IP:
ip rule add from 172.17.0.2 lookup 20000 priority 20000
Running ip route show on the docker host would give you:
default via 192.168.28.254 dev en0 proto dhcp metric 600 # <-- LAN interface
10.69.1.0/24 dev tun0 proto kernel scope link src 10.69.1.89 metric 300 # <-- VPN interface
192.168.28.0/24 dev en0 proto kernel scope link src 192.168.28.83 metric 600
Inside the container, using ping, wget, curl or making any other kind of network request would see the traffic being sent only through the VPN connection.