Multiple EC2 Network Interfaces on Red Hat / CentOS 7

networkmessIf you’re not running Amazon Linux with the built in network interface management tools, adding multiple ENIs on the same subnet can be a confusing experience.  We use this sometimes to run multiple elastic IPs on separate network interfaces so we can bind to them separately.

We worked through this with Amazon support recently and thought we should share a quick overview of how to do this on Red Hat / CentOS 7.

1. Force your default gateway to be eth0

Edit /etc/sysconfig/network and add:

GATEWAYDEV=eth0

Not doing this left the default gateway of the main routing table set to the last interface to be configured, which caused some strange behavior.

2. Configure each additional interface you’ve added

In /etc/sysconfig/network-scripts, create an ifcfg-ethX for each new interface.

Modify:

1. The DEVICE name to match the ENI.

DEVICE="eth1"
BOOTPROTO="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
PEERDNS="yes"
IPV6INIT="no"
PERSISTENT_DHCLIENT="1"

3. Add a custom route for each additional interface

Again in /etc/sysconfig/network-scripts, create a route-ethX file for each interface.

Modify:

1. The device name
2. Increment the table number
3. The gateway to your VPC subnet’s gateway.
4. Change the source IP to the assigned internal network address of the ENI.

default via 10.0.0.1 dev eth1 table 2
10.0.0.0/24 dev eth1 src 10.0.0.10 table 2

4. Add a custom rule for each additional interface

Also in /etc/sysconfig/network-scripts, create a rule-ethX for each interface.

Modify:

1. Increment the table number to match route-ethX
2. Change the IP to the assigned internal network address of the ENI.

from 10.0.0.10/32 table 2

Restart the network service and you should be up and running. You can confirm with “ip rule”:

# ip rule
0:	from all lookup local 
32764:	from 10.0.0.10 lookup 3 
32765:	from 10.0.0.11 lookup 2 
32766:	from all lookup main 
32767:	from all lookup default

Note that Amazon suggested a custom route and rule for eth0, but we found allowing eth0 to use the default main routing table not only worked but was more flexible.