Xen users - Sep 2009 - LVM Read/Write speed <10% drive''s normal speed

Hello all,

I''ve been searching through the archives and the internet for a while
now, and can''t seem to find anything that helps me out. I hope you
don''t
my posting this to both xen-users and linux-lvm simultaneously, but i
figured it''d help keep the solution in one place... even though
it''ll be
the same across two places... Anyways:

My setup is this:
        Xen 3.3 hypervisor (from the ubuntu repositories)
        Ubuntu 9.04 server
        Kernel and module listed here:
http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html

The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that
kernel. (I''m not sure if the kernel has something to do with it, so i
included it just in case.) All the domUs are set up to use LVM. The dom0
uses a straight ext3 fs on /dev/sda1. The domUs use ext3 fs''s within
LVMs (using LVM2) set up on /dev/sda5.

I have noticed a massive file I/O problem on all of my domUs. While I
can peak file operations around 100 MB/s within the dom0, I can''t get
anything more than 3 MB/s read OR write out of a file operation on the
domUs. I am using values reported by "iotop" to make this distinction.
I''m using both the deployment of a Jboss server and the "dd"
command to
benchmark this. The Jboss slowdown is how I found this. On the dom0, the
server takes 56s to come up. On a domU, the same operation takes 15
minutes.

It should also be noted that I attempted to change the filesystems to
ext2, because I noticed that ''kjournald'' was chugging away
taking up
most of the i/o percentage, but only writing at KB/s! Actually, when I
removed the journaling is when the process slowed from 7 minutes to 15.
Even without the kjournald there, the java process took forever to load
and never wrote more than 3 MB/s.

I have messed around with the ''blockdev'' and
''hdparm'' commands (the
second of which doesn''t even interact with any partition on my
computer,
lvm or no, instead simply failing with an "input/output error"
message),
but have had no performance increases after messing around with the read
ahead speed of the devices. I also tried the "lvchange -r" command to
set the readahead higher. I think there''s something else missing here,
though.

I''m fairly fresh to xen, this being only my second installation, and
I''m
extremely fresh to lvm. I''ve played with what seems like every option
within "xm" and the ''lv*'' commands. The LVMs
themselves I haven''t played
with all too much yet, as I don''t want to nuke a certain domU before it
needs to be used heavily tomorrow. I''ve already killed a test domU that
I brought up just for this.

-Stephen


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

what mount options do you use on the DomU ?
Under Debian there is xen-tools example with the "sync" mount options
:
if you use it, writes will be really slow.

Olivier

Stephen Hamer a écrit :
> Hello all,
>
> I''ve been searching through the archives and the internet for a
while
> now, and can''t seem to find anything that helps me out. I hope you
> don''t my posting this to both xen-users and linux-lvm
simultaneously,
> but i figured it''d help keep the solution in one place... even
though
> it''ll be the same across two places... Anyways:
>
> My setup is this:
>         Xen 3.3 hypervisor (from the ubuntu repositories)
>         Ubuntu 9.04 server
>         Kernel and module listed here: 
>
http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html
>
> The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that 
> kernel. (I''m not sure if the kernel has something to do with it,
so i
> included it just in case.) All the domUs are set up to use LVM. The 
> dom0 uses a straight ext3 fs on /dev/sda1. The domUs use ext3 fs''s
> within LVMs (using LVM2) set up on /dev/sda5.
>
> I have noticed a massive file I/O problem on all of my domUs. While I 
> can peak file operations around 100 MB/s within the dom0, I can''t
get
> anything more than 3 MB/s read OR write out of a file operation on the 
> domUs. I am using values reported by "iotop" to make this
distinction.
> I''m using both the deployment of a Jboss server and the
"dd" command
> to benchmark this. The Jboss slowdown is how I found this. On the 
> dom0, the server takes 56s to come up. On a domU, the same operation 
> takes 15 minutes.
>
> It should also be noted that I attempted to change the filesystems to 
> ext2, because I noticed that ''kjournald'' was chugging
away taking up
> most of the i/o percentage, but only writing at KB/s! Actually, when I 
> removed the journaling is when the process slowed from 7 minutes to 
> 15. Even without the kjournald there, the java process took forever to 
> load and never wrote more than 3 MB/s.
>
> I have messed around with the ''blockdev'' and
''hdparm'' commands (the
> second of which doesn''t even interact with any partition on my 
> computer, lvm or no, instead simply failing with an "input/output 
> error" message), but have had no performance increases after messing 
> around with the read ahead speed of the devices. I also tried the 
> "lvchange -r" command to set the readahead higher. I think
there''s
> something else missing here, though.
>
> I''m fairly fresh to xen, this being only my second installation,
and
> I''m extremely fresh to lvm. I''ve played with what seems
like every
> option within "xm" and the ''lv*'' commands. The
LVMs themselves I
> haven''t played with all too much yet, as I don''t want to
nuke a
> certain domU before it needs to be used heavily tomorrow. I''ve
already
> killed a test domU that I brought up just for this.
>
> -Stephen
> ------------------------------------------------------------------------
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, 2009-09-16 at 21:49 +0000, Stephen Hamer wrote:
> Hello all,
> 
> I''ve been searching through the archives and the internet for a
while
> now, and can''t seem to find anything that helps me out. I hope you
> don''t my posting this to both xen-users and linux-lvm
simultaneously,
> but i figured it''d help keep the solution in one place... even
though
> it''ll be the same across two places... Anyways:
> 
> My setup is this:
>         Xen 3.3 hypervisor (from the ubuntu repositories)
>         Ubuntu 9.04 server
>         Kernel and module listed here:
>
http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html
> 
> The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that
> kernel. (I''m not sure if the kernel has something to do with it,
so i
> included it just in case.) All the domUs are set up to use LVM. The
> dom0 uses a straight ext3 fs on /dev/sda1. The domUs use ext3 fs''s
> within LVMs (using LVM2) set up on /dev/sda5.
What is the configuration of the guest''s storage? I.e. what type of
virt
driver are you using to expose the dom0 storage to the domUs? This can
have a massive impact on the performance levels you can achieve.

You should also test the performance of these underlying devices in the
domUs - i.e. benchmark read (and if possible write) performance of
the /dev/sd*, /dev/hd* or /dev/xvd* devices seen in domU (or whatever
other type of virtual disks you are using).

Regards,
Bryn.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Your domU's--are they paravirt, or HVM?

 

From: xen-users-bounces@lists.xensource.com
[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Stephen Hamer
Sent: Wednesday, September 16, 2009 5:50 PM
To: xen-users@lists.xensource.com; linux-lvm@redhat.com
Subject: [Xen-users] [linux-lvm] LVM Read/Write speed <10% drive's
normalspeed

 

Hello all,

I've been searching through the archives and the internet for a while now,
and can't seem to find anything that helps me out. I hope you don't my
posting this to both xen-users and linux-lvm simultaneously, but i figured
it'd help keep the solution in one place... even though it'll be the
same across two places... Anyways:

My setup is this:
        Xen 3.3 hypervisor (from the ubuntu repositories)
        Ubuntu 9.04 server
        Kernel and module listed here:
http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html

The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that kernel.
(I'm not sure if the kernel has something to do with it, so i included it
just in case.) All the domUs are set up to use LVM. The dom0 uses a straight
ext3 fs on /dev/sda1. The domUs use ext3 fs's within LVMs (using LVM2) set
up on /dev/sda5.

I have noticed a massive file I/O problem on all of my domUs. While I can peak
file operations around 100 MB/s within the dom0, I can't get anything more
than 3 MB/s read OR write out of a file operation on the domUs. I am using
values reported by "iotop" to make this distinction. I'm using
both the deployment of a Jboss server and the "dd" command to
benchmark this. The Jboss slowdown is how I found this. On the dom0, the server
takes 56s to come up. On a domU, the same operation takes 15 minutes.

It should also be noted that I attempted to change the filesystems to ext2,
because I noticed that 'kjournald' was chugging away taking up most of
the i/o percentage, but only writing at KB/s! Actually, when I removed the
journaling is when the process slowed from 7 minutes to 15. Even without the
kjournald there, the java process took forever to load and never wrote more than
3 MB/s.

I have messed around with the 'blockdev' and 'hdparm' commands
(the second of which doesn't even interact with any partition on my
computer, lvm or no, instead simply failing with an "input/output
error" message), but have had no performance increases after messing around
with the read ahead speed of the devices. I also tried the "lvchange
-r" command to set the readahead higher. I think there's something else
missing here, though.

I'm fairly fresh to xen, this being only my second installation, and I'm
extremely fresh to lvm. I've played with what seems like every option within
"xm" and the 'lv*' commands. The LVMs themselves I haven't
played with all too much yet, as I don't want to nuke a certain domU before
it needs to be used heavily tomorrow. I've already killed a test domU that I
brought up just for this.

-Stephen 



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Sep 17, 2009 at 4:49 AM, Stephen Hamer <Stephen.Hamer@jhuapl.edu>
wrote:
> My setup is this:
>         Xen 3.3 hypervisor (from the ubuntu repositories)
>         Ubuntu 9.04 server
>         Kernel and module listed here:
>
http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html
>
> The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that kernel.
> (I''m not sure if the kernel has something to do with it, so i
included it
> just in case.)
I believe which kernel you use matters a lot in this case :)
> All the domUs are set up to use LVM. The dom0 uses a straight
> ext3 fs on /dev/sda1. The domUs use ext3 fs''s within LVMs (using
LVM2) set
> up on /dev/sda5.
>
> I have noticed a massive file I/O problem on all of my domUs. While I can
> peak file operations around 100 MB/s within the dom0, I can''t get
anything
> more than 3 MB/s read OR write out of a file operation on the domUs.
I suggest you try using official xen.org''s 2.6.18 kernel, or other
vendor-supported 2.6.18 kernel-xen (like RHEL5''s kernel-xen, or even
Etch''s kernel) for both domU and dom0 and see if the problem persists.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

WOW. Thank you all for the overwhelming response!!! I''m going to answer
you all in order of who replied first. 
(NOTE: the solution is in the 2nd section. Thanks Oliver!)
-Stephen



=================================On Thu, 2009-09-17 at 03:41 -0400, Robert
Dunkley wrote:
> >Hi Stephen,
> >Can you please post your DomU config file to the list. 
> >Rob

Sure, without any of the comments, here it is (without the true names or
IPs, of course):

kernel      = ''/boot/vmlinuz-2.6.26-2-xen-686''
ramdisk     = ''/boot/initrd.img-2.6.26-2-xen-686''
memory      = ''1024''
root        = ''/dev/hda2 ro''
disk        = [ ''phy:/dev/xenvg/testVM1-swap,hda1,w'',
                  ''phy:/dev/xenvg/testVM1-root,hda2,w'', ]

name        = ''testVM1''

vif = [ ''ip=192.168.0.100'' ]

on_poweroff = ''destroy''
on_reboot   = ''restart''
on_crash    = ''restart''

# Needs to be inserted to allow console to attach to VM
extra="console=hvc0 xencons=tty"


This was generated by xen-tools (after a little modification to
xen-tools xm.tmpl). I like how easy xen-tools is to expand upon. For
example, I used ''xen-update-image'' as a model to create my own
''xen-image-cmd'' which can pass a single command to multiple
offline
domUs. Works just like ''xen-update-image'', except for the fact
that you
can pass ANY command that the domU can execute AND you can force the
output to be redirected to a file INSIDE the chrooted image. I was sort
of proud of myself for that scripting voodoo. =D


=================================On Thu, 2009-09-17 at 03:45 -0400, Olivier B.
wrote:
> >Hi,
> 
> >what mount options do you use on the DomU ?
> >Under Debian there is xen-tools example with the "sync" mount
options :
> >if you use it, writes will be really slow.
> 
> >Olivier

THIS is where the money is. Once I removed the ''sync'' option
from the
fstab for my root partition, the I/O speed sailed. Makes sense, really.
This got the VM''s performance to actually BEAT the old bare metal
machine that we were running this server on before. 
Thanks for your help!


=================================On Thu, 2009-09-17 at 06:21 -0400, Bryn M.
Reeves wrote:
> What is the configuration of the guest''s storage? I.e. what type
of virt
> driver are you using to expose the dom0 storage to the domUs? This can
> have a massive impact on the performance levels you can achieve.
> 
> You should also test the performance of these underlying devices in the
> domUs - i.e. benchmark read (and if possible write) performance of
> the /dev/sd*, /dev/hd* or /dev/xvd* devices seen in domU (or whatever
> other type of virtual disks you are using).
> 
> Regards,
> Bryn.

I''m not PRECISELY sure what you mean here, but I think you''re
referring
to the xen driver used to pass the storage in as a block dev. to the
domU. If that is what you mean, I used the ''phy:'' command. The
performance of the /dev/hda* inside the domU is what I was mentioning
trying to benchmark simplistically with ''dd''. 
- A ''dd'' command in my dom0 on the true physical /dev/sda1 got
100
MB/s. 
- Before I turned off ''sync'', a ''dd'' command
on /dev/hda2 inside my domU
(physically a 10GB LVM on the physical device /dev/sda5) got 3 MB/s. 
I noted later that running a DD command directly to another LVM on that
same VG as the others got comparable performance to the 100 MB/s. That
was my first indicator that the domUs were doing something nasty. 
- After I turned off ''sync'', a ''dd'' command
on that same /dev/hda2
inside the same domU got 100 MB/s!




=================================On Thu, 2009-09-17 at 10:59 -0400, Jeff Sturm
wrote:
> Your domU''s--are they paravirt, or HVM?
I believe they''re paravirtualized, unless I''m doing something
silly.
They''re running off the modified kernel, and I''m not doing any
hardware
forwarding magic. See config file at the top.

=================================On Thu, 2009-09-17 at 11:16 -0400, Fajar A.
Nugraha wrote:
> I suggest you try using official xen.org''s 2.6.18 kernel, or other
> vendor-supported 2.6.18 kernel-xen (like RHEL5''s kernel-xen, or
even
> Etch''s kernel) for both domU and dom0 and see if the problem
persists.
I''ve noticed a couple of quirks in the startup sequence of my domUs, I
figured it had something interesting to do with the kernel. I''ll look
into using one of those kernels. Thanks!


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users