Xen users - Mar 2011 - Poor IO performance in hosts and VMs - XCP 1.0 b42052

Dear list,

I am battling with rather grave performance issues in the 42052 build
of XCP 1.0. Specifically, this concerns disk IO both on the host and
in the VMs.

I have tested the connectivity to the NFS/iSCSI server that backs the
whole platform (Nexenta, SSD ZIL/l2arc, striping over 9 5-disk
zraid1), and from a clean Ubuntu install, I am able to saturate the
network both over NFS and iSCSI before the storage starts struggling.

Furthermore, raw network has been tested to/from Ubuntu installs, XCP
hosts and VMs - I do lose the expected 3% pure network speed as soon
as a VM is involved, but apart from that, I get 930-950Mb/s, which I
am happy with.

However, as soon as I start testing from a XCP host, the iSCSI
performance drops from the 50-55 MB/s in Ubuntu to 25-30 in XCP.

What really bugs me, is the VM performance - on a good day, I get
10MB/s writes to the iSCSI root VHDs, with short peaks of 20-25. The
performance varies greatly, from those peaks of 20-25 to deep troughs
of 2-3MB/s. Writing to an NFS share from a VM is still fine, though -
in-VM NFS performance equals that of a "raw" Ubuntu install.

Furthermore, I have a few hosts with local SSDs meant for database
storage. These Pliant SSDs with on-paper sustained writes of 220MB/s
now look like a nightmare purchase - I get only 20-25MB/s when writing
to them from a VM. (And host performance is not great either - 40-50.
Ubuntu: 150MB/s)

The rather substantial drop as soon as I test iSCSI from a XCP host
leads me to think that something is amiss with the current XCP kernel
(2.6.32.12-0.7.1.xs1.0.0.311.170586xen). The VM performance is also a
bit of a puzzle - I can see tapdisk2 going mad when I start pushing
things a little in the VMs - 30-75% CPU ("top" eyeballing) is the
norm.

Sorry for the long post. Has anybody seen anything similar? Does
anybody have any tips on where to start looking for clues as to what
is going on? I am completely open to this being due to configuration
errors, but at the same time, I can''t remember anything trying to be
"clever" in this install.

Thanks in advance for any assistance!

Best regards
Jan

--
Jan-Aage Frydenbø-Bruvoll
Principal architech
architechs ltd

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users