Routed (non-bridged) setup for KVM & libvirt (avoids port security) – Method 2: Routed subnet

Note: if you’re working with multiple individual IPs each in their own subnet, you’re better off referring to this post.

Introduction

Generally when hosting VMs on publically accessible IPs, we create a network bridge and add the host’s network interface and all guest interfaces to that bridge. This is efficient and works great, except in situations where out hosting provider has “port security” enabled on their switches. This limits each port on the switch to one MAC address (the port on the switch being the port which our network card is physically connected to by cable). In the default bridge setup, every virtual network card will create traffic with it’s own MAC address, which isn’t allowed by port security… usually traffic from these addresses will be silently ignored, or even worse, our entire switch port might be disconnected, preventing access to the host machine too.

There are two solutions, the one is a ‘proxy arp’ setup, where our host will masquerade all VM traffic using it’s own MAC address, but transparently modify traffic accordingly so all works intended. The second is a routed setup, where our host acts as another router between our VMs and our ISP. While these are essentially almost the same thing, the advantage of the latter approach is that we can still use DHCP to configure our clients (which is impossible with proxy_arp since we don’t know the VM’s actual MAC address on the host’s side of the bridge).

Note: this article discusses how to set up a block of IP addresses. If you’re being allocated individual non-contiguous IP addresses, unfortunately with libvirt+KVM the only solution involves a number of security sacrifices, which aren’t discussed here.  (Update:  this is now possible, see this post for details.) However, even if you aren’t assigned a big block, but have many IPs in the same subnet, you can still “cheat” and use the setup described below, as long as you’ll never need to access any IPs in that subnet that aren’t assigned to you (since all attempts to access those addresses will be routed to your VMs – although you could correct your hosts routing table as required).  So, not ideal, but it works.

Creating the Routed Network

Ref: Network type info from libvirt.org

Prerequisites

1) Generate a UUID for our network

$ uuidgen 
0fdc9175-4800-4499-a320-9fadf87a5bd3

2) Create a MAC address for our the router interface. KVM MAC addresses should look like this 52:54:00:XX:XX:XX, where the XX are random hex digits. We can generate one like this:

$ echo 52:54:00:`openssl rand -hex 3 | sed 's/\(..\)/\1:/g; s/.$//'`
52:54:00:e0:50:66

3) Our last prerequisite is an IP address for the bridge. If you’ve been assigned a real CIDR block, use the correct router address for that block. Otherwise, it’s possible to “cheat” and use any IP in the subnet, even if you don’t own it, since it will never be used outside of your system. (See the note about cheating above.)

Network Setup

Create a file ‘routed.xml’ similar to the below, but with the information you generated/chose above. In our example we’ll use the 192.168.123.0 network, but obviously this works fine with public IPs.

<network>
  <name>routed</name>
  <uuid>0fdc9175-4800-4499-a320-9fadf87a5bd3</uuid>
  <forward mode='route'/>
  <bridge name='virbr1' dev="eth1" />
  <mac address='52:54:00:e0:50:66'/>
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <dhcp>
      <host ip='192.168.123.2' name='static-ip.domain.net' mac='xx:xx:xx:xx:xx:xx' />
      <range start='192.168.123.2' end='192.168.123.254' />
    </dhcp>
  </ip>
</network>

Activate

$ virsh net-define routed.xml
$ virsh net-start routed
$ # start automatically in future
$ virsh net-autostart routed

VM setup

In your VMs definition xml file, have the network section look similar to the following.

<interface type='network'>
      <mac address='00:16:36:7b:4e:64'/>
      <source network='routed'/>
      <forward mode='route'/>
      <model type='virtio'/>
</interface>

You’ll find this setup works great with port security and dhcp too. Any questions, ask in comments below.

This entry was posted in kvm, linux, networking and tagged , , , , , . Bookmark the permalink.

5 Responses to Routed (non-bridged) setup for KVM & libvirt (avoids port security) – Method 2: Routed subnet

  1. In the VM configuration is it possible to statically assign IP addresses rather than have DHCP dish them out?

    Nice write up though this looks to be along the lines of what im looking for.

    Kind Regards

    • Kinslayer says:

      Thanks, glad it’s been helpful.

      Yes, sure. There’s nothing you need to do in the guest definition file. Inside the guest machine, just statically assign the IP and routing setup as usual. e.g. in ifcfg-eth0 have BOOTPROTO=static, IPADDR=x.x.x.x, gateway, network, dns1, dns2, etc.

  2. lijunle says:

    I am using virt-manager and set the same configuation as you.
    But I can not ping external website from guest using Routed mode.
    It can when using NAT mode.

    I don’t know how to debug the network configuration,
    it there some info.

    Thanks

  3. serg says:

    What if I have Proxmox on the KVM-Host and two VMs.
    I have one main IP for the KVM-Host (its a dedicated server) [146.y.z.133] and two additional “single” IPs [146.x.y.151, 146.x.y.156] from the same subnet (/27) [i.e. “single” seems to be managed via VLAN tags].
    Does the schema from the post above suite to this problem?

  4. Karl says:

    Thanks – that was very helpful. I spent the best part of a day or two to figure out how to do something like this.

Leave a Reply

Your email address will not be published. Required fields are marked *