Setup Multiple Gateways and Multiple Internet Connections One Host

These days, most organizations have multiple circuits and connectivity for egress traffic outside the core network environment. There may be certain use cases where a host needs to have connectivity to multiple gateways or ISPs for different network connections. In the Windows world, this is virtually impossible to do natively within Windows as even when you have multiple network cards, it is highly recommended never to run multiple gateways. In other words, you don’t want to have more than one network connection assigned with a gateway address. This is noted in the Microsoft KB article found here. However, in the Linux world, there are mechanisms in place where you can have multiple gateways defined and actually use them effectively. Let’s take a look at this topic – how to setup multiple gateways and multiple internet connections one host.

Why are two gateways on a host a problem?

Many may ask – why is it a problem to have more than one “gateway” or Internet connection.  When we refer to “gateway” we are generally speaking of the “gateway of last resort”.  For 99% of hosts configured, they general have one gateway even if they have more than one network card.  Usually more than one network card is used to connect disjointed networks or specific VLANs for various use cases.  However, one connection only has a gateway defined that handles network traffic that doesn’t live on the local networks the host is aware of.  Generally, we think of this as “Internet” traffic or traffic that exists in the outside world.

Multiple connections with one gateway presents a problem of routing “paths”.  Most routers or firewalls configured today will not accept return traffic that returns to it on a different path than it expects it to.  So if a host is able to receive traffic on a certain interface and send that traffic out another interface that has the gateway defined, this is a problem as mentioned, most routers and firewalls will not accept this kind of traffic where source and return paths are different.

In the Windows world, you can have multiple gateways defined, albeit with a warning from Windows itself.  The problem with multiple gateways in Windows is that you aren’t really able to utilize these as you would think.  Windows assigns metrics to both gateways and the lower metric is always used.  If a failure in that gateway is detected, the other gateway is used, most likely resulting in routing issues nonetheless.  The problem comes down to the fact that Windows can only have one routing table defined.  Even if you have multiple connections and gateways, each connection will reference the same routing table with only a primary gateway.  So. despite having two network cards potentially connected to two different ISPs, only one will be used.

Note  There may be a third party utility or software out there for Windows that will do this, however, I am not aware of one.

In the Linux world, we CAN take advantage of more than one gateway defined and successfully route traffic as well as more than one routing table.  This is accomplished by implementing policy based routing on the host.  If traffic meets the criteria of policy defined, we can steer it out and back in the correct interface connected to a network/ISP.

Setup Multiple Gateways and Multiple Internet Connections One Host

In general, it is fairly rare to have a need to do this.  For most environments that sit behind a firewall or other router, it makes the decisions on which “pipe” or Internet connection that traffic egresses out.  However, there can be certain corner cases for wanting to do this.  A host or virtual machine with multiple connections may be sending out certain traffic that needs to return on the same link.  This could be some type of web traffic, email traffic, etc.

To do this effectively we need a Linux host.  Linux natively is a much more powerful networking platform than Windows.  With a Linux distro, we can accomplish this in only a few short and simple tasks.  I did run into a challenge of putting all the information together as there are a number of blog posts out there detailing many parts of the process, but there were various pieces for my use case in particular that were found elsewhere.  The following are the steps that allowed setting up multiple gateways connected to multiple Internet connections.  Details in particular:

Ubuntu 16.04 LTS, (2) Internet connections, (2) gateways

• Install iproute2
• Add two new routing tables
• Add ip routes and ip rules to the /etc/network/interfaces config file
• Test routing paths

Install iproute2 and add routing tables

Installing the iproute2 utility is simple enough in most Linux distros.  In Ubuntu it was simply running the command:

• sudo apt-get install iproute2

After installing iproute2, adding the routing tables involves editing the /etc/iproute2/rt_tables file and adding your new routing tables.  The following is this file after editing.  The bottom contains the two new routing tables.  I added routing tables named the same as my interfaces for clarity and simplicity.  The ens160 and ens192 tables will route my specific interface traffic.

#
# reserved values
#
255 local
254 main
253 default
0   unspec
#
# local
#
#1 inr.ruhep
1   ens160
2   ens192

Adding IP Routes and IP Rules for routing to specific multiple gateways

Now that we have the new routing tables setup, we can edit the /etc/network/interfaces config file in Ubuntu to add addressing and also the ip routes and ip rules for directing traffic.  The advantage of doing your configuration in the file is that it allows the configuration to be persistent.  At the bottom of this section you will find the completed file.  However, let’s walk through a couple of configurations that I found necessary to get this to work correctly.

The following command I found for me was the key to the puzzle.  Without adding the command, even though I had traffic correctly traveling over each interface, when pinging out to the Internet, pings that would go out my primary connection to the Internet, but not the secondary connection.  The command allows setting up a dual default gateways for each connection and assigning weights.  However, it still works by selecting the route that applies to each connection.

ip route add default scope global nexthop via 192.168.1.1 dev ens160 weight 1 nexthop via 192.168.30.1 dev ens192 weight 2

For each interface, you can see we are adding routes to each specific routing table we have created.  Below for each interface, we specify the local subnet associated and the default gateway used for traffic on that interface.

ip route add 192.168.1.0/24 dev ens160 table ens160
ip route add default via 192.168.1.1 dev ens160 table ens160

Then using the ip rule command, we are adding these in the processing order for processing before the main default table:

ip rule add from 192.168.1.182/32 table ens160
ip rule add to 192.168.1.182/32 table ens160

The completed /etc/network/interfaces file is below.  Note, we use the post-up directive to apply these rules just after each interface is brought up.

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface

auto ens160
iface ens160 inet static
    address 192.168.1.182
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    dns-nameservers 192.168.1.10 8.8.8.8
    post-up ip route add default scope global nexthop via 192.168.1.1 dev ens160 weight 1 nexthop via 192.168.30.1 dev ens192 weight 2
    post-up ip route add 192.168.1.0/24 dev ens160 table ens160
    post-up ip route add default via 192.168.1.1 dev ens160 table ens160
    post-up ip rule add from 192.168.1.182/32 table ens160
    post-up ip rule add to 192.168.1.182/32 table ens160

auto ens192
iface ens192 inet static
    address 192.168.30.10
    netmask 255.255.255.0
    network 192.168.30.0
    broadcast 192.168.30.255
    dns-nameservers 192.168.1.10 8.8.8.8
    post-up ip route add 192.168.30.0/24 dev ens192 table ens192
    post-up ip route add default via 192.168.30.1 dev ens192 table ens192
    post-up ip rule add from 192.168.30.10/32 table ens192
    post-up ip rule add to 192.168.30.10/32 table ens192

Commands helpful in troubleshooting iproute2 and multiple gateways

The following commands are very helpful in troubleshooting iproute2 and multiple gateways defined:

route, ip route show – both of these command can quickly show configured default gateways and other routes
ip route show table <table name> – This allows you to view new configured routing tables created by iproute2
ip rule – This shows the processing order of configured ip rules
ping -I <interface> 8.8.8.8 – This forces ping traffic over a specified source interface
ip route get 8.8.8.8 dev <interface> – This shows you the route taken to get to a specific IP address for traffic from a specific device.

Thoughts

The process to Setup Multiple Gateways and Multiple Internet Connections One Host is really not too difficult on a Linux host.  It allows one to funnel traffic out a specific Internet connection by utilizing multiple routing tables.  Using Linux policy based routing, one can specific specific traffic based on subnets and interface that allows correctly directing traffic as expected.  While this may be relegated to certain corner cases, it is great to know utilizing a Linux host, we can make this happen quite effectively.