Xen users - Feb 2009 - bad I/O performance with HP Smart Array RAID

Hi all,

I discovered lately that two of my XEN servers suffer from a really 
bad disk throughput - in domU as well as in dom0. All of them run 
Debian/Etch, Kernel 2.6.18-6-xen (orig. Debian) with XEN 3.2 (backported). 
The software versions seem to be ok for most machines:

  Lenovo A57 with S-ATA and (E8200, VT enabled): up to 90MB/s
  Shuttle, FB61 based, ATA-Disk (Celeron, no VT): up to 55MB/s
  HP ML350 G5, Smart-Array RAID-Controller (E5420, VT enabled): ~12MB/s
  IBM x3650, ServeRaid 8k RAID-Controller (E5405, VT enabled): ~140MB/s

The figures are not real benchmarks - just a dd of a file which is big
enough to get around cache effects in dom0.

Unfortunately, the HP machines with the problem are in production use so
that I cannot confirm, but I usually get normal performance out of HPs
with SmartArray controllers and non-Xen kernels. I''ll go for a
maintenance window in a few days to do some testing with a non-XEN
kernel on them.

I have seen other posts on the list having problems that smell similiar 
with other hardware - but no solution.

Is there anything I could try to find out what the problem is? Any hint
on what I could try? Even a "of course, the driver for SmartArray does not
perform well" would help.

Best regards,
Tom

-- 
------------------------------------------------------------------------
Thomas Aeby, Kirchweg 52, 1735 Giffers, Switzerland
Voice : (+41)26 4180040
Internet: aeby@graeff.com                       PGP public key available
------------------------------------------------------------------------

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

DOGUET Emmanuel wrote:
>  And what about the Write Cache on all your RAID card? 
>  On SmartArray P400i it''s an option :/
>  I have buy one for trying.
Well, I''d expect the RAID controller to deliver at least the throughput
of a ATA-133 controller without the battery buffered write cache.

But it seems that I might have been a little too fast blaming this on 
Xen. Googling for cciss performance issues is quite enlightening. 
However, I haven''t found a solution so far, though tuning a few driver 
parameters gave a few Megs/s more.

Best regards,
Tom
-- 
------------------------------------------------------------------------
Thomas Aeby, Kirchweg 52, 1735 Giffers, Switzerland
Voice : (+41)26 4180040
Internet: aeby@graeff.com                       PGP public key available
------------------------------------------------------------------------

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ok, sorry for having thought this might be a XEN issue.

Multiple Smart Array controllers are known for their really bad disk 
write performance. The controller on the machine in question is an 
E200i. I have seen many complaints from users out there that around 
10-12 MB/s write throughput is about what one can expect from this 
controller and even an inofficial citation from HP support classifying 
this problem as a "feature".

So far, there seems to be no real solution - a few users have 
successfully been able to improve performance by enabling the drive 
write caches (which of course is dangerous). My controller does not seem 
to support this option at all.

The only real solution seems to get rid of the controller and either use 
the disks with a normal SATA-controller with software RAID (where you 
can expect 50-100MB/s - a real performance boost) or buy a less trashy 
RAID controller.

I do not know yet what we are going to do. So far I know that we must 
avoid many HP so-called-servers with I/O-performance of a desktop PC of 
the mid-90s.

Best regards,
Tom
-- 
------------------------------------------------------------------------
Thomas Aeby, Kirchweg 52, 1735 Giffers, Switzerland
Voice : (+41)26 4180040
Internet: aeby@graeff.com                       PGP public key available
------------------------------------------------------------------------

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Feb 25, 2009 at 1:47 AM, Thomas Aeby <aeby@graeff.com>
wrote:
> Ok, sorry for having thought this might be a XEN issue.
Glad you solved it!
>
> Multiple Smart Array controllers are known for their really bad disk write
> performance. The controller on the machine in question is an E200i. I have
> seen many complaints from users out there that around 10-12 MB/s write
> throughput is about what one can expect from this controller and even an
> inofficial citation from HP support classifying this problem as a
"feature".
>
> So far, there seems to be no real solution - a few users have successfully
> been able to improve performance by enabling the drive write caches (which
> of course is dangerous). My controller does not seem to support this option
> at all.
>
> The only real solution seems to get rid of the controller and either use
the
> disks with a normal SATA-controller with software RAID (where you can
expect
I considered that option once, but the thing is some (if not most) HP
servers NEEDS RAID controller and doesn''t even offer plain SATA
controller. Yaiks.
> 50-100MB/s - a real performance boost) or buy a less trashy RAID
controller.
>
I usually stick to the one with 512MB battery-backed cache.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> Ok, sorry for having thought this might be a XEN issue.
> 
> Multiple Smart Array controllers are known for their really bad disk
> write performance. The controller on the machine in question is an
> E200i. I have seen many complaints from users out there that around
> 10-12 MB/s write throughput is about what one can expect from this
> controller and even an inofficial citation from HP support classifying
> this problem as a "feature".
> 
> So far, there seems to be no real solution - a few users have
> successfully been able to improve performance by enabling the drive
> write caches (which of course is dangerous). My controller does not
seem
> to support this option at all.
> 
> The only real solution seems to get rid of the controller and either
use
> the disks with a normal SATA-controller with software RAID (where you
> can expect 50-100MB/s - a real performance boost) or buy a less trashy
> RAID controller.
> 
> I do not know yet what we are going to do. So far I know that we must
> avoid many HP so-called-servers with I/O-performance of a desktop PC
of
> the mid-90s.
> 
I had an E200 RAID controller on SATA disks with RAID1, and the
performance was horrible. I have quantify ''horrible'' though:

. hdparm -tT showed some impressive figures (comparable to raw disk
speed)
. under light load it worked pretty well
. under heavy load things slowed down to a crawl. Literally.

My workload was doing a Symantec Backup Exec restore to a DomU from a
physical machine across a 1G network. Performance started off around
800Mbytes/minute and quickly dropped to around 100Mbytes/minute. Then I
switched to tap:aio from file: and it dropped to around 12Mbytes/minute.
It was like the more data I tried to push through it the slower it went.

I thought I had discovered the problem when I noticed that the network
adapter and the raid controller were sharing an interrupt - both devices
were going flat out when doing the restore - but changing that made
little difference.

Eventually I put the disks back onto the onboard SATA (it''s an ML115)
and did software RAID1 and restore performance went back up to
600-800Mbytes/second.

Write cache might have made a difference (I didn''t have any) but I
wasn''t about to fork out a few hundred dollars for what is essentially
a
dev/test box.

The E200 has a much much shorter io queue length than some of its bigger
siblings. I suspect that maybe the Linux driver just isn''t tuned for it
properly

We have other Xen servers with the higher end HP RAID controllers with
battery backed write cache and they work flawlessly.

James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Timh B wrote:
> We are actually running quite a few hp servers with their E200i/p and 400P
> controllers and overall im happy with the performance ( this is debian
> etch, not really xen related) though we do have the BBWC module on all the
> servers, and write up to 250-300MB,2k iops+ 
That''s great and it sounds promising. I''m not very eager to
install BBWC
since I saw quite a few posts stating that even with BBWC installed (and 
write cache enabled) they get a really poor throughput, i.e.:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1235589954133+28353475&threadId=1188496

Maybe this does not apply to SAS disks - I''m not sure. Are you running 
the controller with SAS disks?

Unfortunately I haven''t been able to enable the drives internal write 
cache so far.

Best regards,
Tom
-- 
------------------------------------------------------------------------
Thomas Aeby, Kirchweg 52, 1735 Giffers, Switzerland
Voice : (+41)26 4180040
Internet: aeby@graeff.com                       PGP public key available
------------------------------------------------------------------------

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Timh B wrote:
> Yes, we are almost exclusively running SAS disk, so comparing to SATA
> disks may not be fair.
It''s well probable that different experiences rely on the controller
not
supporting SATA well. We usually use SATA disks in preference just in
order to have another option for a possible disaster recovery scenario.
Normal SATA vs SAS performance is normally not an issue for us, the
100-150MB/s you get normally out of a RAID 10 4-drive SATA array has always
been sufficient - but of course not the HP e200i 10-15MB/s.
> Even though sub-30mb writes is really low, perhaps
> the cciss driver is not good enough or compatible with xen-kernels?
As far as I see (look at the thread I pointed you to) the people
complaining about horribly poor performance are running a wide range of
different kernel versions, most of them non-Xen. So, I assume it is not
a Xen issue. Also, I''ve seen quite a few complaints of Windows users, 
so the issue does not seem to be a Linux driver issue, either. It''s
most
probably an issue between the controller and the drives - that might be
a reason why people are complaining that enabling BBWC does not help at
all (well, a few megs of cache is filled in milliseconds), while enabling
the drives'' write cache seems to have solved the issue for many users.
> What were the exact measurements you used?
Oh, just the silly old "dd" to a large file and to an LVM volume - the
danger with "dd" is to get higher throughput figures because of
caching
effects, so it is ok for determining max figures. But of course I did
test after getting complaints about the performance and after seeing
myself a DRBD mirror syncing with only about 10MB/s, either (after
having suspected someone had plugged the server into a FastEthernet
port I was quite surprised to see that this was not a network issue at 
all).

Best regards,
Tom
-- 
------------------------------------------------------------------------
Thomas Aeby, Kirchweg 52, 1735 Giffers, Switzerland
Voice : (+41)26 4180040
Internet: aeby@graeff.com                       PGP public key available
------------------------------------------------------------------------

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

2009/2/25 Thomas Aeby <aeby@graeff.com>:
> I''m not very eager to install BBWC
> since I saw quite a few posts stating that even with BBWC installed (and
> write cache enabled) they get a really poor throughput,
Just to give you a single datapoint, on a smartarray P800 BBWC
controller using a RAID1+0 array of eight 15krpm 300GB SAS disks I
fully zeroed a 1TB file in just under an hour, 2.2gbit/sec sustained
isn''t so bad ;-)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users