Hi everyone.
I have a really severe performance issue on disk write on my domU''s PV
guest.
- Xen 4.1 fresh install on Debian Wheezy (kernel 3.2.0-4-amd64) via apt-get
- Simple disk (no raid) for Linux/Xen Dom0 installation
- LVM2 on top of raid5 disk raid5 disk for storing DomU
- Single LV for each DomU
The problem
While write speed (mesured via dd command) on Dom0 is near 163MB/s on
the mounted DomU LV , the same dd command provides 8MB/s if executed within
the running DomU (with "noop" I/O scheduler set)
Running "top" in DomU, i get high load average due to high %wa (no
high
CPU load)
DomU disk is submitted to Xen via configuration file with:
disk = [ ''phy:/dev/xenstore/test,xvda1,w'' ]
If the same DomU is deployed on file image ("test.img" cloned from
"test" LV), write speed rise up to 63MB/s
disk = [ ''file:/home/test.img,xvda1,w'' ]
Obviously I can provide further detailed configuration parameters
Does anyone faced to the same problem? Does anyone can point me in the
right direction?
I''m loosing my nights trying to solve this issue :( , please help!!
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
Adam Goryachev
2013-Aug-01 03:27 UTC
Re: Severe slow disk write issue on LVM-deployed DomU
On 01/08/13 12:56, g wrote:> Hi everyone. > I have a really severe performance issue on disk write on my domU''s PV > guest. > - Xen 4.1 fresh install on Debian Wheezy (kernel 3.2.0-4-amd64) via > apt-get > - Simple disk (no raid) for Linux/Xen Dom0 installation > - LVM2 on top of raid5 disk raid5 disk for storing DomU > - Single LV for each DomU > > The problem > While write speed (mesured via dd command) on Dom0 is near 163MB/s on > the mounted DomU LV , the same dd command provides 8MB/s if executed > within > the running DomU (with "noop" I/O scheduler set) > Running "top" in DomU, i get high load average due to high %wa (no > high CPU load) > > DomU disk is submitted to Xen via configuration file with: > disk = [ ''phy:/dev/xenstore/test,xvda1,w'' ] > > If the same DomU is deployed on file image ("test.img" cloned from > "test" LV), write speed rise up to 63MB/s > disk = [ ''file:/home/test.img,xvda1,w'' ] > > Obviously I can provide further detailed configuration parameters > > Does anyone faced to the same problem? Does anyone can point me in the > right direction?A couple of things to check. shutdown the VM, and check the read/write speed on the LV from the dom0 /dev/xenstore/test Most likely, you have one or more snapshots of the LV, and this totally kills performance. You might get away with a single snapshot and performance might still be acceptable. Also, try adjusting the value of /sys/block/md1/md/stripe_cache_size I ended up using 4096 which seemed to provide the best performance (I''m using 4 x SSD in RAID5). Verify performance on dom0 first, then compare it to the performance within the domu. Make sure you allocate sufficient CPU resources to dom0 as well, or performance of the domu can suffer. Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi Adam. Thanks for you quick answer. Am very happy to see that somebody is involved in this issue. I feel less alone :) Here below my reply, on line point by point: On 08/01/2013 05:27 AM, Adam Goryachev wrote:> On 01/08/13 12:56, g wrote: >> Hi everyone. >> I have a really severe performance issue on disk write on my domU''s PV >> guest. >> - Xen 4.1 fresh install on Debian Wheezy (kernel 3.2.0-4-amd64) via >> apt-get >> - Simple disk (no raid) for Linux/Xen Dom0 installation >> - LVM2 on top of raid5 disk raid5 disk for storing DomU >> - Single LV for each DomU >> >> The problem >> While write speed (mesured via dd command) on Dom0 is near 163MB/s on >> the mounted DomU LV , the same dd command provides 8MB/s if executed >> within >> the running DomU (with "noop" I/O scheduler set) >> Running "top" in DomU, i get high load average due to high %wa (no >> high CPU load) >> >> DomU disk is submitted to Xen via configuration file with: >> disk = [ ''phy:/dev/xenstore/test,xvda1,w'' ] >> >> If the same DomU is deployed on file image ("test.img" cloned from >> "test" LV), write speed rise up to 63MB/s >> disk = [ ''file:/home/test.img,xvda1,w'' ] >> >> Obviously I can provide further detailed configuration parameters >> >> Does anyone faced to the same problem? Does anyone can point me in the >> right direction? > > A couple of things to check. > > shutdown the VM, and check the read/write speed on the LV from the dom0 > /dev/xenstore/test >The speed on LVM /dev/xenstore/test when DomU is powered off is impressive: 163MB/s> Most likely, you have one or more snapshots of the LV, and this totally > kills performance. You might get away with a single snapshot and > performance might still be acceptable.No snapshots active (never run any snapshot on this new Dom0 server)> Also, try adjusting the value of /sys/block/md1/md/stripe_cache_size > I ended up using 4096 which seemed to provide the best performance (I''m > using 4 x SSD in RAID5).I''m not using software raid. Are you writing "md1" as generic block device? In my case which block device should I adjust?> Verify performance on dom0 first, then compare it to the performance > within the domu.Performance for disk write in Dom0 are great (> 20) if compared with DomU deployed on LVM volume. Disk write speed on DomU deployed on file-image disk are half the speed of Dom0, but it''s dangerous working with fileimage disk> Make sure you allocate sufficient CPU resources to dom0 as well, or > performance of the domu can suffer. >On this server (HP Proliant) I have 8GB RAM, 2x 8-core Intel CPUs, only one domU with 1 vCpu, 1GB RAM, no swap On another server (same model, old xen linux kernel 2.6.32-5) where the problems began, I have 24GB RAM, 2x 8-core Intel CPUs, DomU with 4 vCPUs, 4GB swap On both the server the problem is the same: 8MB/s disk write speed on DomU if deployed on an LVM2 volume (unpartioned, with ext3 or ext4 FS) Thanks to anyone can give help to this issue giuliano> Regards, > Adam > > -- > Adam Goryachev Website Managers www.websitemanagers.com.au > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >
Little info more: on the Server with xen kernel 2.6.32-5, even in Dom0 I have slow disk write (10MB/s) on LVM volumes normally mounted :( I think the issue, on both server, is clearly related on the interaction between XEN and LVM2 layer On 08/01/2013 12:16 PM, g wrote:> Hi Adam. Thanks for you quick answer. Am very happy to see that somebody > is involved in this issue. I feel less alone :) > Here below my reply, on line point by point: > > On 08/01/2013 05:27 AM, Adam Goryachev wrote: >> On 01/08/13 12:56, g wrote: >>> Hi everyone. >>> I have a really severe performance issue on disk write on my domU''s PV >>> guest. >>> - Xen 4.1 fresh install on Debian Wheezy (kernel 3.2.0-4-amd64) via >>> apt-get >>> - Simple disk (no raid) for Linux/Xen Dom0 installation >>> - LVM2 on top of raid5 disk raid5 disk for storing DomU >>> - Single LV for each DomU >>> >>> The problem >>> While write speed (mesured via dd command) on Dom0 is near 163MB/s on >>> the mounted DomU LV , the same dd command provides 8MB/s if executed >>> within >>> the running DomU (with "noop" I/O scheduler set) >>> Running "top" in DomU, i get high load average due to high %wa (no >>> high CPU load) >>> >>> DomU disk is submitted to Xen via configuration file with: >>> disk = [ ''phy:/dev/xenstore/test,xvda1,w'' ] >>> >>> If the same DomU is deployed on file image ("test.img" cloned from >>> "test" LV), write speed rise up to 63MB/s >>> disk = [ ''file:/home/test.img,xvda1,w'' ] >>> >>> Obviously I can provide further detailed configuration parameters >>> >>> Does anyone faced to the same problem? Does anyone can point me in the >>> right direction? >> >> A couple of things to check. >> >> shutdown the VM, and check the read/write speed on the LV from the dom0 >> /dev/xenstore/test >> > The speed on LVM /dev/xenstore/test when DomU is powered off is > impressive: 163MB/s > >> Most likely, you have one or more snapshots of the LV, and this totally >> kills performance. You might get away with a single snapshot and >> performance might still be acceptable. > > No snapshots active (never run any snapshot on this new Dom0 server) > >> Also, try adjusting the value of /sys/block/md1/md/stripe_cache_size >> I ended up using 4096 which seemed to provide the best performance (I''m >> using 4 x SSD in RAID5). > > I''m not using software raid. Are you writing "md1" as generic block > device? In my case which block device should I adjust? > >> Verify performance on dom0 first, then compare it to the performance >> within the domu. > > Performance for disk write in Dom0 are great (> 20) if compared with > DomU deployed on LVM volume. > Disk write speed on DomU deployed on file-image disk are half the > speed of Dom0, but it''s dangerous working with fileimage disk > >> Make sure you allocate sufficient CPU resources to dom0 as well, or >> performance of the domu can suffer. >> > > On this server (HP Proliant) I have 8GB RAM, 2x 8-core Intel CPUs, only > one domU with 1 vCpu, 1GB RAM, no swap > On another server (same model, old xen linux kernel 2.6.32-5) where the > problems began, I have 24GB RAM, 2x 8-core Intel CPUs, DomU with 4 > vCPUs, 4GB swap > > On both the server the problem is the same: 8MB/s disk write speed on > DomU if deployed on an LVM2 volume (unpartioned, with ext3 or ext4 FS) > > Thanks to anyone can give help to this issue > giuliano > > >> Regards, >> Adam >> >> -- >> Adam Goryachev Website Managers www.websitemanagers.com.au >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xen.org >> http://lists.xen.org/xen-users >>