Individual IP routing in CentOS/KVM/Libvirt with Port Security

Note: if all your IPs are in the same subnet, you’d be better off using the instructions in this post.

Introduction

Generally when hosting VMs on publically accessible IPs, we create a network bridge and add the host’s network interface and all guest interfaces to that bridge. This is efficient and works great, except in situations where out hosting provider has “port security” enabled on their switches. This limits each port on the switch to one MAC address (the port on the switch being the port which our network card is physically connected to by cable). In the default bridge setup, every virtual network card will create traffic with it’s own MAC address, which isn’t allowed by port security… usually traffic from these addresses will be silently ignored, or even worse, our entire switch port might be disconnected, preventing access to the host machine too.

There are two solutions, the one is a ‘proxy arp’ setup, where our host will masquerade all VM traffic using it’s own MAC address, but transparently modify traffic accordingly so all works intended. The second is a routed setup, where our host acts as another router between our VMs and our ISP. While these are essentially almost the same thing, the advantage of the latter approach is that we can still use DHCP to configure our clients (which is impossible with proxy_arp since we don’t know the VM’s actual MAC address on the host’s side of the bridge).

Previously, the only way I found for adding single IPs involved setting up a tunnel interface (e.g. tap0, etc) for each IP address.  Besides being quite cumbersome, it required a vast reduction in security measures, which made it less than ideal.  This new method, involves setting up a new bridge, but one that isn’t directly bridged to the ethernet card in the host, and having the VM appear as a router in between.  So, the switch still only deals with one MAC address, but we can still use DHCP, etc.

Note: everything below works, but I’m still working on scripts to better automate the process.  (June, 2012).

On the Host

Network Definition

1. Generate a UUID for the network

$ uuidgen 
0fdc9175-4800-4499-a320-9fadf87a5bd3

2) Create a MAC address for our the router interface. KVM MAC addresses should look like this 52:54:00:XX:XX:XX, where the XX are random hex digits. We can generate one like this:

$ echo 52:54:00:`openssl rand -hex 3 | sed 's/\(..\)/\1:/g; s/.$//'`
52:54:00:e0:50:66

3) Create routed.xml, replacing relevant fields with data you generated above:

<network>
  <name>routed</name>
  <uuid>0fdc9175-4800-4499-a320-9fadf87a5bd3</uuid>
  <forward mode='route'/>
  <bridge name='virbr1' dev="eth1" />
  <mac address='52:54:00:e0:50:66'/>
  <ip address='10.0.0.1' netmask='255.255.255.255'>
  </ip>
</network>

4) Activate:

$ virsh net-define routed.xml
$ virsh net-start routed
$ # start automatically in future
$ virsh net-autostart routed

5) Allow forwarding over the bridge in your firewall with either:

# Forward all packets over libvirt routed network
iptables -A FORWARD -i virbr1 -j ACCEPT
iptables -A FORWARD -o virbr1 -j ACCEPT

or, even safer (but more resource intensive), a rule for each IP:

# Forward traffic for hostname
iptables -A FORWARD -s X.X.X.X -j ACCEPT
iptables -A FORWARD -d X.X.X.X -j ACCEPT

Save the current ruleset to be loaded on boot:

service iptables save

Note:  If you “virsh net-destroy routed” after the above rules are in effect, you’ll need to apply them again after recreating the network with “virsh net-start routed”.

You can verify you ruleset like this:

[[Ignore this.  TODO, verify; libvirt incorrect iptables rules, correct them with script.  Write about libvirt hook inadequacies for net-start, create script to compensate]]

6)  Create start-up scripts:

# If they don't already exist
mkdir /etc/libvirt/hooks
touch /etc/libvirt/hooks/qemu
chmod a+x /etc/libvirt/hooks/qemu 

# Critical if /etc/libvirt/hooks/qemu didn't exist before
service libvirt restart

Edit /etc/libvirt/hooks/qemu to include the following:

#!/bin/sh

# Add individual IPs for our routed network to the routing table
#
# Since no hook exists for net-start, the best we can do is check if
# all the IPs are added everytime a VM is launched, without re-adding.
# When a net-destroy occurs, the routes will be autoamtically removed.
. `dirname $0`/routed-ips
if [ "$2" == "start" ]; then
    for IP in $ROUTED_IPS ; do
        if [ "`ip route list | grep $IP`" == "" ] ; then
             ip route add $IP via $ROUTED_GW dev $ROUTED_DEV
        fi
    done
fi
exit 0

And create /etc/libvirt/hooks/routed-ips with something like:

ROUTED_GW="10.0.0.1"
ROUTED_DEV="virbr1"
ROUTED_IPS="X.X.X.X Y.Y.Y.Y Z.Z.Z.Z"

Guest definition files

In your VMs definition xml file, have the network section look similar to this:

<interface type='network'>
      <mac address='00:16:36:7b:4e:64'/>
      <source network='routed'/>
      <forward mode='route'/>
      <model type='virtio'/>
</interface>

On the Guest

Assumes eth1 is main network device.

Example /etc/sysconfig/network-scripts/ifcfg-eth1:

DEVICE="eth1"
NM_CONTROLLED="no"
ONBOOT=yes
HWADDR=52:54:00:b7:3c:c2
TYPE=Ethernet
BOOTPROTO="static"
IPADDR="X.X.X.X"
NETMASK="255.255.255.255"
DNS1="Y.Y.Y.Y"
DNS2="Z.Z.Z.Z"
# GATEWAY=  not needed, defined below

Most importantly, create /etc/sysconfig/network-scripts/route-eth1:

10.0.0.1 dev eth1
default via 10.0.0.1 dev eth1

 That’s it!

Let me know if you have any problems, comments on the method, etc.

This entry was posted in kvm, linux, networking and tagged , , , , . Bookmark the permalink.

7 Responses to Individual IP routing in CentOS/KVM/Libvirt with Port Security

  1. Bart says:

    Is 10.0.0.1 IP supposed to come up on the KVM host or the VM?

    If KVM host I don’t really understand how this can work (and is not working for me sadly):

    ip route add $IP via $ROUTED_GW dev $ROUTED_DEV

  2. Felix Schwarz says:

    > [[Ignore this. TODO, verify; libvirt incorrect iptables rules, correct them with script.
    > Write about libvirt hook inadequacies for net-start, create script to compensate]]

    Can you explain this a bit?
    Is this similar to my issue on the libvirt mailing regarding automatically added firewall rules for nat/routed setups in 2009?

    • Kinslayer says:

      Hi, apologies, I missed this. These are just some notes to me for the future, unfortunately I don’t recall if it’s exactly your problem, but yeah the idea was to have a script that fixes up iptables on start. Unfortunately, there are no hooks for net-start, so the plan was to write a script that runs on (I think) every VM start and check if it needs to run any stuff 1st time for the network. Unfortunately I still haven’t gotten round to this.

  3. Felix Schwarz says:

    caveat: When libvirt is restarted (e.g. due to a package update), it will reset the iptables configuration. Restarting libvirt will not trigger the qemu hook!

  4. anagha says:

    a@a:~$ sudo virsh net-start routed
    error: Failed to start network routed
    error: internal error Child process (/usr/sbin/dnsmasq -u libvirt-dnsmasq –strict-order –bind-interfaces –pid-file=/var/run/libvirt/network/routed.pid –conf-file= –except-interface lo –listen-address 10.0.0.1) status unexpected: exit status 2

    error is coming .

    I have attached the eth0 interface to openvswitch bridge should i remove this ?
    and on host just use eth0 static .
    I want communication between vms of different vlans and communication of vm to outside world with their own ip not the NAT.

  5. anagha says:

    I removed the openvswitch bridge and just using the eth0 static address in n/w interface

    still unable to create network.

Leave a Reply

Your email address will not be published. Required fields are marked *