Routed (non-bridged) setup for KVM & libvirt (avoids port security) – Method 2: Routed subnet

3/19/2012 by gadicc in Tech, Coding, Sysadmin

Note: if you’re working with multiple individual IPs each in their own subnet, you’re better off referring to this post.

Introduction

Generally when hosting VMs on publically accessible IPs, we create a network bridge and add the host’s network interface and all guest interfaces to that bridge. This is efficient and works great, except in situations where out hosting provider has “port security” enabled on their switches. This limits each port on the switch to one MAC address (the port on the switch being the port which our network card is physically connected to by cable). In the default bridge setup, every virtual network card will create traffic with it’s own MAC address, which isn’t allowed by port security… usually traffic from these addresses will be silently ignored, or even worse, our entire switch port might be disconnected, preventing access to the host machine too.

There are two solutions, the one is a ‘proxy arp’ setup, where our host will masquerade all VM traffic using it’s own MAC address, but transparently modify traffic accordingly so all works intended. The second is a routed setup, where our host acts as another router between our VMs and our ISP. While these are essentially almost the same thing, the advantage of the latter approach is that we can still use DHCP to configure our clients (which is impossible with proxy_arp since we don’t know the VM’s actual MAC address on the host’s side of the bridge).

Note: this article discusses how to set up a block of IP addresses. If you’re being allocated individual non-contiguous IP addresses, unfortunately with libvirt+KVM the only solution involves a number of security sacrifices, which aren’t discussed here. (Update: this is now possible, see this post for details.) However, even if you aren’t assigned a big block, but have many IPs in the same subnet, you can still “cheat” and use the setup described below, as long as you’ll never need to access any IPs in that subnet that aren’t assigned to you (since all attempts to access those addresses will be routed to your VMs – although you could correct your hosts routing table as required). So, not ideal, but it works.

Creating the Routed Network

Ref: Network type info from libvirt.org

Prerequisites

1. Generate a UUID for our network

$ uuidgen 
0fdc9175-4800-4499-a320-9fadf87a5bd3

2. Create a MAC address for our the router interface. KVM MAC addresses should look like this 52:54:00:XX:XX:XX, where the XX are random hex digits. We can generate one like this:

$ echo 52:54:00:`openssl rand -hex 3 | sed 's/\(..\)/\1:/g; s/.$//'`
52:54:00:e0:50:66

3. Our last prerequisite is an IP address for the bridge. If you’ve been assigned a real CIDR block, use the correct router address for that block. Otherwise, it’s possible to “cheat” and use any IP in the subnet, even if you don’t own it, since it will never be used outside of your system. (See the note about cheating above.)

Network Setup

Create a file ‘routed.xml’ similar to the below, but with the information you generated/chose above. In our example we’ll use the 192.168.123.0 network, but obviously this works fine with public IPs.

<network>
  <name>routed</name><!-- ◀ -->
  <uuid>0fdc9175-4800-4499-a320-9fadf87a5bd3</uuid><!-- ◀ -->
  <forward mode='route'/>
  <bridge name='virbr1' dev="eth1" />
  <mac address='52:54:00:e0:50:66'/><!-- ◀ -->
  <ip address='192.168.123.1' netmask='255.255.255.0'><!-- ◀ -->
    <dhcp>
      <host ip='192.168.123.2' name='static-ip.domain.net' mac='xx:xx:xx:xx:xx:xx' />
      <range start='192.168.123.2' end='192.168.123.254' />
    </dhcp>
  </ip>
</network>

Activate

virsh net-define routed.xml
virsh net-start routed
# start automatically in future
virsh net-autostart routed

VM setup

In your VMs definition xml file, have the network section look similar to the following.

<interface type='network'>
      <mac address='00:16:36:7b:4e:64'/>
      <source network='routed'/>
      <forward mode='route'/>
      <model type='virtio'/>
</interface>

You’ll find this setup works great with port security and dhcp too. Any questions, ask in comments below.