Individual IP routing in CentOS/KVM/Libvirt with Port Security

6/27/2012 by gadicc in Tech, Coding, Sysadmin

Note: if all your IPs are in the same subnet, you’d be better off using the instructions in this post.

Introduction

Generally when hosting VMs on publically accessible IPs, we create a network bridge and add the host’s network interface and all guest interfaces to that bridge. This is efficient and works great, except in situations where out hosting provider has “port security” enabled on their switches. This limits each port on the switch to one MAC address (the port on the switch being the port which our network card is physically connected to by cable). In the default bridge setup, every virtual network card will create traffic with it’s own MAC address, which isn’t allowed by port security… usually traffic from these addresses will be silently ignored, or even worse, our entire switch port might be disconnected, preventing access to the host machine too.

There are two solutions, the one is a ‘proxy arp’ setup, where our host will masquerade all VM traffic using it’s own MAC address, but transparently modify traffic accordingly so all works intended. The second is a routed setup, where our host acts as another router between our VMs and our ISP. While these are essentially almost the same thing, the advantage of the latter approach is that we can still use DHCP to configure our clients (which is impossible with proxy_arp since we don’t know the VM’s actual MAC address on the host’s side of the bridge).

Previously, the only way I found for adding single IPs involved setting up a tunnel interface (e.g. tap0, etc) for each IP address. Besides being quite cumbersome, it required a vast reduction in security measures, which made it less than ideal. This new method, involves setting up a new bridge, but one that isn’t directly bridged to the ethernet card in the host, and having the VM appear as a router in between. So, the switch still only deals with one MAC address, but we can still use DHCP, etc.

Note: everything below works, but I’m still working on scripts to better automate the process. (June, 2012).

On the Host

Network Definition

1. Generate a UUID for the network

$ uuidgen 
0fdc9175-4800-4499-a320-9fadf87a5bd3

2. Create a MAC address for our the router interface. KVM MAC addresses should look like this 52:54:00:XX:XX:XX, where the XX are random hex digits. We can generate one like this:

$ echo 52:54:00:`openssl rand -hex 3 | sed 's/\(..\)/\1:/g; s/.$//'`
52:54:00:e0:50:66

3. Create routed.xml, replacing relevant fields with data you generated above:

<network>
  <name>routed</name>
  <uuid>0fdc9175-4800-4499-a320-9fadf87a5bd3</uuid>
  <forward mode='route'/>
  <bridge name='virbr1' dev="eth1" />
  <mac address='52:54:00:e0:50:66'/>
  <ip address='10.0.0.1' netmask='255.255.255.255'>
  </ip>
</network>

4. Activate:

$ virsh net-define routed.xml
$ virsh net-start routed
$ # start automatically in future
$ virsh net-autostart routed
5) Allow forwarding over the bridge in your firewall with either:
# Forward all packets over libvirt routed network
iptables -A FORWARD -i virbr1 -j ACCEPT
iptables -A FORWARD -o virbr1 -j ACCEPT

or, even safer (but more resource intensive), a rule for each IP:

# Forward traffic for hostname
iptables -A FORWARD -s X.X.X.X -j ACCEPT
iptables -A FORWARD -d X.X.X.X -j ACCEPT

Save the current ruleset to be loaded on boot:

service iptables save

Note: If you “virsh net-destroy routed” after the above rules are in effect, you’ll need to apply them again after recreating the network with “virsh net-start routed”.

You can verify you ruleset like this:

[[Ignore this. TODO, verify; libvirt incorrect iptables rules, correct them with script. Write about libvirt hook inadequacies for net-start, create script to compensate]]

6. Create start-up scripts:

# If they don't already exist
mkdir /etc/libvirt/hooks
touch /etc/libvirt/hooks/qemu
chmod a+x /etc/libvirt/hooks/qemu 

# Critical if /etc/libvirt/hooks/qemu didn't exist before
service libvirt restart

Edit /etc/libvirt/hooks/qemu to include the following:

#!/bin/sh

# Add individual IPs for our routed network to the routing table
#
# Since no hook exists for net-start, the best we can do is check if
# all the IPs are added everytime a VM is launched, without re-adding.
# When a net-destroy occurs, the routes will be autoamtically removed.
. `dirname $0`/routed-ips
if [ "$2" == "start" ]; then
    for IP in $ROUTED_IPS ; do
        if [ "`ip route list | grep $IP`" == "" ] ; then
             ip route add $IP via $ROUTED_GW dev $ROUTED_DEV
        fi
    done
fi
exit 0

And create /etc/libvirt/hooks/routed-ips with something like:

ROUTED_GW="10.0.0.1"
ROUTED_DEV="virbr1"
ROUTED_IPS="X.X.X.X Y.Y.Y.Y Z.Z.Z.Z"

Guest definition files

In your VMs definition xml file, have the network section look similar to this:

<interface type='network'>
      <mac address='00:16:36:7b:4e:64'/>
      <source network='routed'/><!-- ◀ -->
      <forward mode='route'/><!-- ◀ -->
      <model type='virtio'/>
</interface>

On the Guest

Assumes eth1 is main network device.

Example /etc/sysconfig/network-scripts/ifcfg-eth1:

DEVICE="eth1"
NM_CONTROLLED="no"
ONBOOT=yes
HWADDR=52:54:00:b7:3c:c2
TYPE=Ethernet
BOOTPROTO="static"
IPADDR="X.X.X.X" # <--
NETMASK="255.255.255.255"
DNS1="Y.Y.Y.Y"
DNS2="Z.Z.Z.Z"
# GATEWAY=  not needed, defined below

Most importantly, create /etc/sysconfig/network-scripts/route-eth1:

10.0.0.1 dev eth1
default via 10.0.0.1 dev eth1

That’s it!

Let me know if you have any problems, comments on the method, etc.