Foreword: This post is aimed at advanced users and expects you to already understand LVM and virtualization (although all commands needed are shown).
Summary: What follows is an explanation of how, and why, you should do what I call hybrid LVM partitioning for your virtual machines, where each logical volume on the host is an entire file system on the guest, with the exception of a tiny logical volume which will appear as the boot drive, and have just a boot petition on it. This offers numerous advantages, as explained below.
Rationale:
I started off with virtualization on Xen; there it was standard practice to have every partition in the guest tied to a logical volume in the host. Like this:
Host Guest --------------------- --------- /dev/vg0/gandalf-boot /dev/xvda1 /dev/vg0/gandalf-root /dev/xvda2 /dev/vg0/gandalf-swap /dev/xvda3
This is incredibly efficient and convenient for many reasons, not the least being easy backups and maintenance of those file systems from the host. But maintenance doesn’t just mean mounting, it means it’s incredibly easy to resize the partitions too, which is a common requirement when hosting many VMs.
In KVM, the standard practice is to have each logical volume on the host be an entire virtual disk in the guest, and have that partitioned appropriately. That means whereas before, a single logical volume would contain a single, easy to maintain file system, you’d now have a set up like this:
/dev/vg0/gandalf-disk + +-- partition table + +-- boot partition + +-- LVM partition + +-- swap partition +-- root partition
Currently, everyone seems to argue that this really isn’t such a big deal, since we can just use kpartx to map out these partitions on the host, and is at most just a few extra steps. And yes, that doesn’t seem so bad in the beginning. Until it comes time to resize the partitions, and then all hell breaks loose.
Look again at the diagram above, and think what’s involved in resizing all those file systems, logical volumes, partitions, and containing logical volume. Fortunately libvirt offers a tool called virt-resize which can automate a lot of this process, but owing to the inherent shortcomings of the above technique, you require twice again as much space as that first LVM partition… because essentially what the script does is create a new LVM partition, with a new partition table, and copies all the data across for you. While a lot more convenient for the user than doing all the insane steps by hand, this is still very inefficient, slow and simply unnecessary.
KVM virtual machines are generally set up this way because KVM guests aren’t paravirtualized, and the system needs a full (albeit virtual) hard drive to boot off, just like in a real system. But many seem to have overlooked the fact that while this is required for booting, other “drives” can easily host a full file system without the need to be partitioned first.
This leads us to what I call hybrid LVM partitioning. It may be a bit hard to understand just from the name what’s involved there, but basically, we have one small logical volume appear as an entire disk, with boot sector, partition table, and single small boot partition. All other partitions will be as described in the first setup, where each logical volume on the host is a device with just a full file system on it.
This let’s us boot in KVM, while still giving us all the convenience and other advantages of the set up we first described. And this is fine too, since we’ll most likely never need to resize the boot partition, and if we do, there really isn’t so much work involved. Everything else, we can resize as we’d normally expect for a logical volume. So now we have this:
Host Guest --------------------- --------- /dev/vg0/gandalf-boot /dev/vda + partition table + + boot partition ( /dev/vda1 ) /dev/vg0/gandalf-root /dev/xvda2 /dev/vg0/gandalf-swap /dev/xvda3
Here’s an example of how to create that first device on volume group vg0:
lvcreate -n gandalf-boot -L 256M /dev/vg0 fdisk /dev/vg0/gandalf-boot << __END__ n p 1 a 1 w __END__ kpartx -a /dev/vg0/gandalf-boot mkfs.ext2 /dev/mapper/vg0-gandalf--boot1 # 1st partition # possibly mount device, copy files, unmount, if desired kpartx -d /dev/vg0/gandalf-boot
What we just did: 1) Create the logical volume, 2) in fdisk: create a new primary partition 1, filling the full disk, make it bootable, write, and 3) map the drive and create the ext2 file system), unmap the drive. As for the other partitions, something like this:
lvcreate -n gandalf-root -L 8G /dev/vg0 mkfs.ext4 /dev/vg0/gandalf-root vcreate -n gandalf-swap -L 2G /dev/vg0 mkswap /dev/vg0/gandalf-swap
In your libvirt guest xml file, define the disks like this
<disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/vg0/gandalf-boot'/> <target dev='vda' bus='virtio'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/vg0/gandalf-root'/> <target dev='vdb' bus='virtio'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/vg0/gandalf-swap'/> <target dev='vdc' bus='virtio'/> </disk>
In your guest, the assignments will be as per the diagram above (gandalf-boot = vda, with the boot partition on vda1, gandalf-root = vdb, gandalf-swap = vdc) -- however, you'll probably want to use UUIDs or LABELs to find them in the guest (beyond the scope of this blog entry).
Good luck!
See also: