CentOS virt - Oct 2007 - Xen/VMWare Server comparison and "best Xen practices"?

I'm wondering what others who have already used the Xen kernel for DomUs 
and the (free) VMWare Server can say about the comparison in actual 
day-to-day operation. I've only started playing with Xen on CentOS 5 and 
I've been running VMWare Server only on Win2k3 servers so far, so I'm 
missing direct comparibility. I find that VMWare is highly reliable and 
flexible, but needs a good portion of RAM and CPU for working nicely. Xen 
seems to be less reliable, but more responsive on low ressources. It's 
also more flexible when you want to move it around as it is all stored in 
a single file and the config file takes only a few options.

Xen is supposed to give "almost the same performance" as if not 
virtualized. I get confusing figures from the Virtual Machine Manager. 
On my test machine the Dom0 takes about 9% of the CPU with one VM when 
both are mostly idle. Top says that 3% of that is taken by the X-Server, 
most of the rest is shared by two python processes (which belong to Xen I 
suppose) and xenstored. (The test machine is on a single Athlon 2500+ or 
around that, not sure about the exact speed.) But regularly one of the 
python processes grabs another 10-15%, so the whole CPU utilization goes 
up to around 20%. Although the single VM is idle at that time and shows 
0.0x%. So, even when idle the whole CPU utilization zigzags between 10 and 
20%.
I found that when I close the VM console that drops to 2-3%, so that 
python process is obviously related to the VM console. The interesting 
thing is that when I then reopen the console from the VM manager it keeps 
going at about 3% and doesn't go up to the earlier 9%. But it still 
zigzags between 3 and 11% then. All the figures have been taken from the 
VM manager. The %us count in top seems to stay at 3% all the time.

With less reliable I refer to a filesystem problem that seems to occur 
sometimes after rebooting/shutdown. Sometimes after rebooting there either 
is no filesystem file anymore (which is not healable, of course) or the 
kernel panics with a filesystem problem on first boot or reboot, but may 
boot just fine after a second or third try. The problem that the 
filesystem file is just gone seems to happen only when I reboot/shutdown 
from within the console with the shutdown/reboot command. For instance it 
can happen if I let it do the first reboot after I installed the OS on the 
VM. When I "xm shutdown" or CRTL+ALT+DEL or hit the shutdown button on
the
Virtual Machine Manager it does not *seem* to happen (maybe I just didn't 
shutdown often enough to see it happen there as well.) I've already lost 
several testing VMs because of this.
I wonder if this problem might happen because I use the option of not 
allocating all space in the filesystem file right-away. I also wonder if 
performance might be better if it wouldn't need to grow.

So, are there recommended best practices for Xen VMs like "always use 
partitions", "always allocated whole space at once", "never
shutdown from
the console window", "never use virtual machine manager" or some
such?

Kai

-- 
Kai Sch?tzl, Berlin, Germany

On 10/15/07, Kai Schaetzl <maillists at conactive.com>
wrote:
> I'm wondering what others who have already used the Xen kernel for
DomUs
> and the (free) VMWare Server can say about the comparison in actual
> day-to-day operation. I've only started playing with Xen on CentOS 5
and
> I've been running VMWare Server only on Win2k3 servers so far, so
I'm
> missing direct comparibility. I find that VMWare is highly reliable and
> flexible, but needs a good portion of RAM and CPU for working nicely. Xen
> seems to be less reliable, but more responsive on low ressources. It's
> also more flexible when you want to move it around as it is all stored in
> a single file and the config file takes only a few options.
>
> Xen is supposed to give "almost the same performance" as if not
> virtualized. I get confusing figures from the Virtual Machine Manager.
> On my test machine the Dom0 takes about 9% of the CPU with one VM when
> both are mostly idle. Top says that 3% of that is taken by the X-Server,
> most of the rest is shared by two python processes (which belong to Xen I
> suppose) and xenstored. (The test machine is on a single Athlon 2500+ or
> around that, not sure about the exact speed.) But regularly one of the
> python processes grabs another 10-15%, so the whole CPU utilization goes
> up to around 20%. Although the single VM is idle at that time and shows
> 0.0x%. So, even when idle the whole CPU utilization zigzags between 10 and
> 20%.
> I found that when I close the VM console that drops to 2-3%, so that
> python process is obviously related to the VM console. The interesting
> thing is that when I then reopen the console from the VM manager it keeps
> going at about 3% and doesn't go up to the earlier 9%. But it still
> zigzags between 3 and 11% then. All the figures have been taken from the
> VM manager. The %us count in top seems to stay at 3% all the time.
>
> With less reliable I refer to a filesystem problem that seems to occur
> sometimes after rebooting/shutdown. Sometimes after rebooting there either
> is no filesystem file anymore (which is not healable, of course) or the
> kernel panics with a filesystem problem on first boot or reboot, but may
> boot just fine after a second or third try. The problem that the
> filesystem file is just gone seems to happen only when I reboot/shutdown
> from within the console with the shutdown/reboot command. For instance it
> can happen if I let it do the first reboot after I installed the OS on the
> VM. When I "xm shutdown" or CRTL+ALT+DEL or hit the shutdown
button on the
> Virtual Machine Manager it does not *seem* to happen (maybe I just
didn't
> shutdown often enough to see it happen there as well.) I've already
lost
> several testing VMs because of this.
> I wonder if this problem might happen because I use the option of not
> allocating all space in the filesystem file right-away. I also wonder if
> performance might be better if it wouldn't need to grow.
>
> So, are there recommended best practices for Xen VMs like "always use
> partitions", "always allocated whole space at once",
"never shutdown from
> the console window", "never use virtual machine manager" or
some such?
>
> Kai
Hi Kai,

I have read that Xen is supposed to only take a few % without
virtualization support in the CPU, in my experience this is not
entirely true. I have experienced good CPU performance with degraded
I/O performance on Xen compared to the physical environment although
sufficient hardware resources were provided. You can read more about
other tests done in this regard, URL:

http://lists.xensource.com/archives/html/xen-users/2007-04/msg00512.html

I've shutdown Xen 2 and 3 from within the console (shutdown -h now),
xm shutdown ... or xm destroy ... and I never got a crashed file
system, all environments on LVM partitions on Intel and AMD hardware.
I don't use the I use the option of not allocating all space in the
file system file right-away so it could be this or the image file, or
why not the combination? ;) Perhaps you can try out some combination
and let us know how this turns out..


Regards,
Nicolas

On Mon, 2007-10-15 at 13:32 +0200, Kai Schaetzl wrote:
> I'm wondering what others who have already used the Xen kernel for
DomUs
> and the (free) VMWare Server can say about the comparison in actual 
> day-to-day operation. I've only started playing with Xen on CentOS 5
and
> I've been running VMWare Server only on Win2k3 servers so far, so
I'm
> missing direct comparibility.
It depends on what you want to run as a domU. Paravirtualization is very
fast (e.g. for running domU CentOS 4/5 kernels). On the other hand, some
devices are very slow if you use Xen HVM for running systems that do not
have a kernel that functions as a Xen domU. As long as we don't have
paravirtualized network/display drivers for those systems,
network/graphic performance will not be very good.

So, what do you plan to run as a virtual machine?
> I found that when I close the VM console that drops to 2-3%, so that 
> python process is obviously related to the VM console. The interesting 
> thing is that when I then reopen the console from the VM manager it keeps 
> going at about 3% and doesn't go up to the earlier 9%. But it still 
> zigzags between 3 and 11% then. All the figures have been taken from the 
> VM manager. The %us count in top seems to stay at 3% all the time.
Did you try to connect to the VM virtual framebuffer with vncviewer,
rather than virt-manager? What loads do you get then?
> I wonder if this problem might happen because I use the option of not 
> allocating all space in the filesystem file right-away. I also wonder if 
> performance might be better if it wouldn't need to grow.
I have never seen this problem on production machines. But I don't use
virt-manager, so I am not sure at what point it rewrites the domain
configuration files after the installation.

-- Daniel