Xen devel - Jul 2008 - xenoprobes output re: I/O performance problem using LVM mirrors to back phy: devices

Recently I started using the ''mirror'' functionality of LVM
rather than
backing my LVM pv devices with md mirrors.  The odd thing is that this produces
acceptable performance when mounting the LVs from within the Dom0 directly,
but if you access them via xm block-attach in the Dom0 (or from within a DomU)
performance slows to a crawl.  Using --corelog solves the problem, but is
obviously suboptimal, as it requires that all disks are rebuilt every 
reboot.

Our hardware is as follows: intel q6600 CPU, 8Gb
unbuffered ECC, ICH7 sata (2x1TB disks)  

we are running CentOS 5.1, kernel 2.6.18-53.1.14.el5

We noted that the performance of mirrored logical volumes accessed
through xenblk was about 1/10th that of non-mirrored LVs, or of LVs
mirrored with the --corelog option.  Mirrored LVs performed fine when
accessed normally within the dom0, but performance dropped when
accessed via xm block-attach.  This was, to our minds, ridiculous.

First, we created two logical volumes in the volume group "test":
one with mirroring and a mirror log and one with the --corelog option.

 # lvcreate -m 1 -L 2G -n test_mirror test
 # lvcreate -m 1 --corelog -L 2G -n test_core test

Then we made filesystems and mounted them:

 # mke2fs -j /dev/test/test*
 # mkdir -p /mnt/test/mirror
 # mkdir -p /mnt/test/core
 # mount /dev/test/test_mirror /mnt/test/mirror

Next we started oprofile, instructing it to count BUS_IO_WAIT events:

 # opcontrol --start --event=BUS_IO_WAIT:500:0xc0
  --xen=/usr/lib/debug/boot/xen-syms-2.6.18-53.1.14.el5.debug
  --vmlinux=/usr/lib/debug/lib/modules/2.6.18-53.1.14.el5xen/vmlinux
  --separate=all

Then we ran bonnie on each device in sequence, stopping oprofile and
saving the output each time.

 # bonnie++ -d /mnt/test/mirror
 # opcontrol --stop
 # opcontrol --save=mirrorlog
 # opcontrol --reset


The LV with the corelog displayed negligible iowait, as expected.
However, the other experienced quite a bit:

# opreport -t 1 --symbols session:iowait_mirror
warning: /ahci could not be found.
CPU: Core 2, speed 2400.08 MHz (estimated)
Counted BUS_IO_WAIT events (IO requests waiting in the bus queue) with a unit
mask of 0xc0 (All cores) count 500
Processes with a thread ID of 0
Processes with a thread ID of 463
Processes with a thread ID of 14185
samples  %        samples  %        samples  %        app name                
symbol name
32       91.4286  15       93.7500  0              0 
xen-syms-2.6.18-53.1.14.el5.debug pit_read_counter
1         2.8571  0              0  0              0  ahci                    
(no symbols)
1         2.8571  0              0  0              0  vmlinux                 
bio_put
1         2.8571  0              0  0              0  vmlinux                 
hypercall_page

>From this, it seemed clear that the culprit was in the pit_read_counter 
function, but we don''t know where to go from here. 


Credit to Chris Takemura <chris@prgmr.com>  for replicating the problem
with
oprofile, and for the writeup


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Normally I configure my own servers such that the dom0 boots with a fixed
amount of ram (dom_mem on the xen command line)  then I set dom0-min-mem
to the same value as dom0_mem, thus disabling ballooning.  

the other day I was documenting this and looking at other ways to do the
same thing.

I noticed something odd.  In newer versions of Xen, 
(enable-dom0-ballooning no) will disable ballooning.  But that is only 
respected in newer versions of Xen.   Before the enable-dom0-ballooning 
option was enabled, setting dom0-min-mem to 0 would disable ballooning.  
After the enable-dom0-ballooning setting was added, though, setting 
dom0-min-mem to 0 would actually set the minimum to zero.

This seems dangerous to me, as I can''t think of any time where
I''d really
want to let the dom0 balloon to 0, and it''s pretty easy to make the
mistake
of setting dom0-min-mem to 0 if you come across older documentation.  

I would suggest changing xen so that dom0-min-mem 0 disabled ballooning as
it did before.  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 07/06/2009 03:08, "Luke S Crawford" <lsc@prgmr.com> wrote:
> I noticed something odd.  In newer versions of Xen,
> (enable-dom0-ballooning no) will disable ballooning.  But that is only
> respected in newer versions of Xen.   Before the enable-dom0-ballooning
> option was enabled, setting dom0-min-mem to 0 would disable ballooning.
> After the enable-dom0-ballooning setting was added, though, setting
> dom0-min-mem to 0 would actually set the minimum to zero.
> 
> This seems dangerous to me, as I can''t think of any time where
I''d really
> want to let the dom0 balloon to 0, and it''s pretty easy to make
the mistake
> of setting dom0-min-mem to 0 if you come across older documentation.
> 
> I would suggest changing xen so that dom0-min-mem 0 disabled ballooning as
> it did before.  
If you look at xen/xend/XendOptions.py:get_enable_dom0_ballooning(),
you''ll
see that dom0-min-mem=0 should still disable dom0 ballooning. At least in
theory!

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser <keir.fraser@eu.citrix.com> writes:
> If you look at xen/xend/XendOptions.py:get_enable_dom0_ballooning(),
you''ll
> see that dom0-min-mem=0 should still disable dom0 ballooning. At least in
> theory!
doesn''t seem to work for me... here is a test I just ran:


[root@cattle ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1024     1     r-----     22.4
rickey                                       1   128     1     -b----      0.1
[root@cattle ~]# xm mem-set 0 512
[root@cattle ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   512     1     r-----     23.0
rickey                                       1   128     1     -b----      0.1
[root@cattle ~]# grep dom0-min-mem /etc/xen/xend-config.sxp 
# dom0-min-mem is the lowest permissible memory level (in MB) for dom0.
(dom0-min-mem 0)
[root@cattle ~]# xm info |grep xen_changeset
xen_changeset          : Mon May 11 13:55:26 2009 +0100 18646:6e1e1b4dcad0



xen_major              : 3
xen_minor              : 3
xen_extra              : .2-rc1-pre

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel