Xen users - Dec 2011 - Xen HVMs run VERY slowly on SAN box

We are running a 2-node cluster with Debian Squeeze (2.6.32-5-xen-amd64) 
and Xen 4.0.1 on the dom0 and Ubuntu lucid domUs. We are now using fully 
virtualized (HVM) vms. In our configuration the dom0 administers to 
domUs created and run on a SAN box accessed via private network 
(192.168...). A sample configuration file is pasted below.

We have had little or no IO problem with HVMs created and run locally on 
dom0. However, HVMs run on the SAN suffer from VERY feeble IO. Even a 
entry of a single character or entering return will cause the command 
line to freeze for 30 seconds or more.

We have had no IO problems running ''regular'' para-virtualised
Xen vms on
the SAN box. (We are choosing HVMs for separate reasons).

As I understand it, perhaps the problem lies with HVM architecture--IO 
having to travel into dom0 before it is handled by the hardware 
emulator(?). But I am not sure why this would slow IO so much.

Are there any parameters we should adjust to enable faster IO for HVMs 
administered on a SAN box?


Any ideas are greatly appreciated!

Ben Weaver



cfg file for an HVM:

[BEGINNING OF FILE]

import os, re
arch = os.uname()[4]
if re.search(''64'', arch):
         arch_libdir = ''lib64''
else:
         arch_libdir = ''lib''


kernel = "/usr/lib/xen-default/boot/hvmloader"

builder=''hvm''

memory = 512
#shadow_memory = 8
name = "lucidxentest3.phon"
vif = [ ''type=ioemu, mac=00:16:3e:6e:76:2b, bridge=eth0'' ]
disk = [ ''file:/ocfs2SAN/lucidxentest3.phon.img,hda,w'', 
''file:/ocfs2SAN/ubuntu-10.04.2-server-amd64-copy2.iso,hdc:cdrom,r''
]
#disk = [ ''file:/home/xen1.example.com.img,xvda,w'', 
''phy:/dev/cdrom,xvdc:cdrom,r'' ]
device_model = ''/usr/lib/xen-default/bin/qemu-dm''
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
#boot="dc"
boot = "c"

sdl=0
vnc=1
#vnclisten=''0.0.0.0''
vfb = [ "type=vnc,vncunused=1,vnclisten=0.0.0.0" ]
#vncviewer 127.0.0.1
#vncconsole=1
vncpasswd=''''
stdvga=0
serial=''pty''
usbdevice=''tablet''

on_poweroff = ''destroy''
on_reboot   = ''restart''
on_crash    = ''restart''


device_model = ''/usr/'' + arch_libdir +
''/xen/bin/quemu-dm''

bootloader = "/usr/bin/pygrub"

[END OF FILE]

On Wed, Dec 14, 2011 at 12:07 AM, Benjamin Weaver
<benjamin.weaver@phon.ox.ac.uk> wrote:
> We are running a 2-node cluster with Debian Squeeze (2.6.32-5-xen-amd64)
and
> Xen 4.0.1 on the dom0 and Ubuntu lucid domUs. We are now using fully
> virtualized (HVM) vms. In our configuration the dom0 administers to domUs
> created and run on a SAN box accessed via private network (192.168...). A
> sample configuration file is pasted below.
>
> We have had little or no IO problem with HVMs created and run locally on
> dom0. However, HVMs run on the SAN suffer from VERY feeble IO. Even a entry
> of a single character or entering return will cause the command line to
> freeze for 30 seconds or more.
>
> We have had no IO problems running ''regular''
para-virtualised Xen vms on the
> SAN box. (We are choosing HVMs for separate reasons).
is it the SAME SAN box serving both HVM and PV?
>
> As I understand it, perhaps the problem lies with HVM architecture--IO
> having to travel into dom0 before it is handled by the hardware
emulator(?).
> But I am not sure why this would slow IO so much.
It shouldn''t.
>
> Are there any parameters we should adjust to enable faster IO for HVMs
> administered on a SAN box?
I''d start by testing if it''s REALLY HVM problem. Something
like dd/fio
on the dom0, PV domU, and HVM domU to the same path/LUN.
After that, make sure you have PV drivers loaded (exact procedure
depends on what OS you have as domU).
> disk = [ ''file:/ocfs2SAN/lucidxentest3.phon.img,hda,w'',
ocfs2 (or any cluster filesystem) will inherently be slower than local
fs (e.g. ext4) or block device. You can try using tap:aio (or
tap:tapdisk:aio, or tap2:tapdisk:aio, whichever works on your version
of Xen) and see if it performs better.

-- 
Fajar