And here I run into a problem. XenServer doesn't want to work nicely with mirrored disks. It's expecting SAN or similar to provide the redundancy for disks. Now people have worked out options to convert a XenServer to a RAID disk, but I'm very very worried about how upgrades might break the OS partition. The XenServer upgrade process appears to recreate sda and I'm afraid of it either not working or breaking the mirror. Of course I can't tell until I actually have to do an upgrade, but I'm not sure I want to to risk it.

So... let's see how the world has moved on in the past two years.

Now XenServer can "export" backups of VMs in "xva" format. This is a proprietary format that's mostly a tar file with disk blocks. Fortunately you can convert these files into raw disk images.

Whatever I replace the XenServer with should be able to handle these image files. I don't expect the Linux images to be hard, but the Windows images... hmm.

CentOS 6.2 and Xen

I'm primarily a CentOS user, so my first thought would be using CentOS 6.2. Now the GUI installer for this handles RAID, after a fashion. I created a mirrored root, mirrored /boot and installed a minimal OS. After that completed I converted the rest of the disk (sda3/sdb3) into a mirror and LVMd it. So far, so normal.

Now I start to run into issues; RedHat uses KVM for virtualization, and not Xen. But there is a process. Basically, install a xen kernel, and replace libvirt with one that understands Xen. Not the easiest to do, but not hard and, in theory, a one-off. Although kernel and libvirt patching might become an issue down the line.

So, within an hour of rebuilding the machine I had a fully mirrored Xen system and had test installed a new Linux image using a logical volume for storage. So far so good. I then installed virt-manager on my Ubuntu desktop, and was able to use that to create and install an XP image using a datafile in /var/lib/libvirt/images. virt-manager is nowhere near as powerful as Citrix XenCenter, but it works for basic stuff. Hmm, the distinction between using libvirtd for management vs Xen's own tools seems to be a little confusing.

But then a problem... closing down a VM caused the whole machine to crash. Not every time, and not consistently, but I got 2 kernel crashes in one day. Not good. Not suitable.

At least following that method, CentOS 6.2+Xen is not stable enough for me.

CentOS 6.2 + KVM

I possibly should have tried this one first time around, since KVM is the native solution in CentOS. An important question will be whether the KVM emulated hardware is sufficiently close to the Xen hardware for the Windows sessions to work in it (assuming I can convert the disks, of course) and how much work will be needed to convert the Linux images, but.. first steps first; is KVM usable?

This time around I install all three virtualization groups from the CentOS installer. Also, rather than using virt-manager from my Ubuntu desktop I would ssh into the VM host and use the virt-manager from there (so RedHat^WCentos tools all the way. Needed "dejavu-lgc-sans-fonts" installed for words to display properly!). Hmm, this appears to add some niceness in building out the network devices and the storage pools, so the ISO images can be seen. I don't recall seeing those options in my previous test but I might have been blind!

Now here is where I might be stupid. Using virt-manager I did a basic CentOS install and it worked just fine... except the network didn't come up inside the guest. This might be a RedHat/CentOS oddity with Network Manager (I've seen messages similar on the mailing lists); setting the device to not use NM and to dhcp boot worked just fine. So then I tried another install, but this time _without_ any graphical console; using virt-manager I removed the VNC, the input device, the screen. So there was just a serial connection. It started up as normal...but then after selecting "install" from the CentOS menu it froze. Umm.

So I try it from the command line:
virt-install -n t2 --vcpus=1 -s 8 -r 512 -b br0 -l /OS_ISO_images/CentOS-6.2-x86_64-bin-DVD1.iso -f /var/lib/libvirt/images/t2.img --nographics

And I get exactly the same problem. If I don't use "--nographics" then I get a good install (but with the network config issue), whereas if I do specify it then the install freezes.

Hmm.

Debian 6 with Xen 4

(rant: "amd64" doesn't mean AMD in Debian land, apparently! Also the installer is crap; does some work and makes you think it's gonna run on its own, but then asks more questions later. On the plus side, the loader is better than the text-based CentOS one and works nicely with the existing mirrors.)

There are a quite few web pages that explain how to install Xen on Debian. It looks pretty simple. So let's give it a try!

After getting my base Debian install setup, I followed some of the setup on that page. Hmm, Debian Xen seems to be a little different to CentOS (and XenServer?). When a domain is shutdown it no longer appears in the 'xm list' output (so you can't just do "xm start name"; you have to do "xm create /etc/xen/name.cfg").

The "xen-create-image" command is funky, and a great way of creating new Debian images. If I wanted a Debian image!

Let's see if I can boot one of the older CentOS images I created in an earlier test... well, by setting the mac address and using pygrub then it worked to boot that.

bootloader = '/usr/lib/xen-default/bin/pygrub'
vcpus       = '1'
memory      = '512'
root        = '/dev/xvda1 ro'
disk        = [
                  'file:/VMs/t1.img,xvda,w',
              ]
name        = 't1old'
dhcp        = 'dhcp'
vif         = [ 'mac=52:54:00:01:72:1B' ]
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

# dd if=/dev/zero of=/VMs/xp.img bs=1 count=0 seek=8G
# cat /etc/xen/xp.cfg
import os, re
arch = os.uname()[4]
kernel = "/usr/lib/xen-default/boot/hvmloader"
builder='hvm'

memory = 512
shadow_memory = 8
name = "xp"
# Setting ne2k_pci is essential if you want the virtual NIC to work
vif = [ 'type=ioemu, bridge=eth0, model=ne2k_pci' ]

disk = [ 'file:/VMs/xp.img,xvda,w', 'file:/OS_ISO_images/XPSP3.iso,xvdc:cdrom,r' ]
device_model = '/usr/lib/xen-default/bin/qemu-dm'
boot="cd"

sdl=0
# Enable VNC
vnc=1
vncconsole=1
listen='0.0.0.0'
vncviewer=0
nographic=0
stdvga=0
serial='pty'
audio=0
localtime=1
acpi=0
apic=0

# xm create --vncviewer xp.cfg

Let's try the VM I created on CentOS.. hmm failed to start (it crashed and on restart brings up the 'fail' menu, but then fails again, even in safe mode). Hmm, OK, turning on acpi and apic and now the image boots! Neat. But, again, mouse coordination issues. Let's try an image all the way from the original XenServer. Hmm... it's running, but taking forever to get through the startup screen. Hmm. "xm list" always shows it in blocked state, but the indicator bar on the boot screen is moving. Hmm.

Hmm, after 11 hours it's still in the same state. We might be able to kludge things, but... hmm.

However, I'm not feeling "comfortable". And here we come to the core of the matter; the Xen install on Debian appears to be nice and stable. But it's not very friendly. It's all command-line, and (from what I've read), you need to have different config files for install vs run. Which seems odd. The CentOS libvirt method is a lot smoother and prettier... but doesn't appear to be stable.

I'm not sure either is suitable to replace the XenServer. If the CentOS one hadn't crashed (twice) in testing, then I might have continued further with it, but maybe not (considering the libvirt compatibility thing).

So this leads to the next test...

Debian 6 with Xen 4 and libvirt

Starting from a fresh install of Debian, we'll do an install of virt-manager and see how well that improves the user experience.
apt-get install virt-manager xen-qemu-dm xen-linux-system
This seems to pull in world+dog, but that's OK :-)
0 upgraded, 220 newly installed, 0 to remove and 0 not upgraded.
Need to get 177 MB of archives.
After this operation, 474 MB of additional disk space will be used.

Hmm, xend needed to be configured to listen to http requests on localhost:8000 before virt-manager could talk, but that's a minor issue.

Let's see what we can do with this version! OK, we can set up storage pools, so I can find the ISO images. But I can't set up network bridging via virt-manager. Maybe that's not important; let's create a VM and see what happens! OK, the "create a VM" has more options for different types of Linux. On the final screen I manually specify "eth0" as the bridge device.

And the build fails:
xend.err "Error creating domain: device model '/usr/lib64/xen/bin/qemu-dm' not found"
That's because it's in /usr/lib/xen-default/bin/qemu-dm (/usr/lib64 is a symlink to /usr/lib). Hmm. OK, let's symlink /usr/lib/xen -> xen-default and try again! Now I get the vnc based boot screen. Of course I only gave 512Mb to the VM and that's, apparently, not enough for a GUI install, but the TUI install ran. So far, so good. Except that on reboot it needed me to kludge the settings to boot into HD. Hmm. The resulting image also has the network-manager problem seen previously; I'm beginning to think this is a CentOS/RedHat issue. Heh.

Oh, hold on... that's doing a full virtualisation and not paravirt. Why? Oh, huh... "only URL or import installs work with paravirt". Hmm, seems an odd restriction, but OK... I can do that. Umm, OK, not via PXEboot, but via HTTP it allows it. OK, I can do that as well. Here we go, a second build using paravirt. I do not recall if virt-manager on CentOS had that restriction or not! I don't _think_ it did; I'm pretty sure I select Xen (paravirt), but I might be misremembering.

Hmm, if I try to install a second image then virt-manager keeps grey-ing out ("not responding" type indication in Ubuntu).

Interesting... the second image booted cleanly and brought up the network! Hmm.

Let's import that XP VM I just built in the previous test. Yup, that worked. And the virt-manager display keeps the mouse in sync properly. Although screen updates are a little slow and jerky.

Let's try with the VM that still didn't work after 12 hours :-) Oh, huh, in virt-manager I clearly get the choice of i686 or x86_64; I wonder if the previous import was the wrong CPU type. Hmm. No, it still does the same thing, irregardless.

Results

And, to be honest, I'm still not happy.

Modern OS management of Xen has definitely improved a lot over 2 years ago. But I don't get the feeling of "comfort" that I get from XenServer. For the time being, I'm gonna stick with that and the kludged up mirror solution, and hope that the update process doesn't break things!

... unless someone can point me at a better VM management solution!