benco
2010-Feb-28 23:04 UTC
[Xen-users] slow dom0 write operations with 2.6.32 pv-ops and Xen4.0.0-rc4
Hi, i did a simple write test on my system and found out that there is a big performance difference regarding to write operations between non-xen and xen kernels. HW is dual 4-core Xeon5570 with 32GB RAM and FC attached storage with SATA drives (RAID5). SW is linux 2.6.32 pv-ops with Xen4.0.0-rc4 on Debian Lenny. There were no VMs running. I did simple ''dd'' write test with 3GB file on LVM partition. dom0 with 2GB RAM and 2CPU ========================= linux2.6.32 pv-ops + Xen4.0.0-rc4 title Xen 4.0.0-rc4 / Debian GNU/Linux, kernel 2.6.32 root (hd0,0) kernel /xen-4.0.0-rc4.gz dom0_mem=2048M dom0_max_vcpus=2 module /vmlinuz-2.6.32 root=/dev/sda6 ro console=tty0 module /initrd.img-2.6.32 #xm vcpu-pin Domain-0 0 0 #xm vcpu-pin Domain-0 1 1 #time dd if=/dev/zero of=zerofile.tst bs=1k count=3145728 3145728+0 records in 3145728+0 records out 3221225472 bytes (3.2 GB) copied, 48.4788 s, 66.4 MB/s real 0m48.531s user 0m1.940s sys 0m12.745s dom0 with no memory and cpu limit ================================ linux2.6.32 pv-ops + Xen4.0.0-rc4 title Xen 4.0.0-rc4 / Debian GNU/Linux, kernel 2.6.32 root (hd0,0) kernel /xen-4.0.0-rc4.gz module /vmlinuz-2.6.32 root=/dev/sda6 ro console=tty0 module /initrd.img-2.6.32 #time dd if=/dev/zero of=zerofile.tst bs=1k count=3145728 3145728+0 records in 3145728+0 records out 3221225472 bytes (3.2 GB) copied, 15.7325 s, 205 MB/s real 0m15.735s user 0m2.312s sys 0m13.273s bare metal ========= linux2.6.32 pv-ops title Debian GNU/Linux, kernel 2.6.32 root (hd0,0) kernel /vmlinuz-2.6.32 root=/dev/sda6 ro initrd /initrd.img-2.6.32 #time dd if=/dev/zero of=zerofile.tst bs=1k count=3145728 3145728+0 records in 3145728+0 records out 3221225472 bytes (3.2 GB) copied, 6.42111 s, 502 MB/s real 0m6.422s user 0m0.304s sys 0m6.024s Any idea what to check/modify to improve? thanks, roman -- ---------------------------------------------------------------------- ,''''`. [benco] | mailto: benco@acid.sk | silc: /msg benco : :'' : ------------------------------------------------------------- `. `'' GPG publickey: http://www.acid.sk/pubkey.asc `- KF = 0DF6 0592 74D2 F17A DACF A5C3 1720 CB7C F54C F429 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Mar-01 07:40 UTC
Re: [Xen-users] slow dom0 write operations with 2.6.32 pv-ops and Xen4.0.0-rc4
On Mon, Mar 01, 2010 at 12:04:20AM +0100, benco wrote:> Hi, > > i did a simple write test on my system and found out that there is a big > performance difference regarding to write operations between non-xen and xen > kernels. > > HW is dual 4-core Xeon5570 with 32GB RAM and FC attached storage with SATA > drives (RAID5). > > SW is linux 2.6.32 pv-ops with Xen4.0.0-rc4 on Debian Lenny. There were no > VMs running. I did simple ''dd'' write test with 3GB file on LVM partition. > > > dom0 with 2GB RAM and 2CPU > =========================> > linux2.6.32 pv-ops + Xen4.0.0-rc4 > > title Xen 4.0.0-rc4 / Debian GNU/Linux, kernel 2.6.32 > root (hd0,0) > kernel /xen-4.0.0-rc4.gz dom0_mem=2048M dom0_max_vcpus=2 > module /vmlinuz-2.6.32 root=/dev/sda6 ro console=tty0 > module /initrd.img-2.6.32 > > #xm vcpu-pin Domain-0 0 0 > #xm vcpu-pin Domain-0 1 1 > #time dd if=/dev/zero of=zerofile.tst bs=1k count=3145728 > 3145728+0 records in > 3145728+0 records out > 3221225472 bytes (3.2 GB) copied, 48.4788 s, 66.4 MB/s > > real 0m48.531s > user 0m1.940s > sys 0m12.745s > > > > dom0 with no memory and cpu limit > ================================> > linux2.6.32 pv-ops + Xen4.0.0-rc4 > > title Xen 4.0.0-rc4 / Debian GNU/Linux, kernel 2.6.32 > root (hd0,0) > kernel /xen-4.0.0-rc4.gz > module /vmlinuz-2.6.32 root=/dev/sda6 ro console=tty0 > module /initrd.img-2.6.32 > > #time dd if=/dev/zero of=zerofile.tst bs=1k count=3145728 > 3145728+0 records in > 3145728+0 records out > 3221225472 bytes (3.2 GB) copied, 15.7325 s, 205 MB/s > > real 0m15.735s > user 0m2.312s > sys 0m13.273s > > > bare metal > =========> > linux2.6.32 pv-ops > > title Debian GNU/Linux, kernel 2.6.32 > root (hd0,0) > kernel /vmlinuz-2.6.32 root=/dev/sda6 ro > initrd /initrd.img-2.6.32 > > #time dd if=/dev/zero of=zerofile.tst bs=1k count=3145728 > 3145728+0 records in > 3145728+0 records out > 3221225472 bytes (3.2 GB) copied, 6.42111 s, 502 MB/s > > real 0m6.422s > user 0m0.304s > sys 0m6.024s > > Any idea what to check/modify to improve? >You''re measuring buffered (cached) performance, so when you limit the amount of cache to 2 GB the performance will be slower. Try adding "oflag=direct" do your dd commands and test again. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Mar-01 11:43 UTC
Re: [Xen-users] slow dom0 write operations with 2.6.32 pv-ops and Xen4.0.0-rc4
On Mon, Mar 01, 2010 at 01:07:10PM +0100, benco@acid.sk wrote:> > You''re measuring buffered (cached) performance, so when you limit > > the amount of cache to 2 GB the performance will be slower. > > > > Try adding "oflag=direct" do your dd commands and test again. > > > oflag=direct option produce almost the same results on xen and non-xen > systems: > > #dd if=/dev/zero of=zerofile.tst bs=1k count=1000000 oflag=direct > 1000000+0 records in > 1000000+0 records out > 1024000000 bytes (1.0 GB) copied, 616.219 s, 1.7 MB/s >Try bigger block size and smaller count, for example bs=64k. 1k block are really small and won''t give you good performance.> what is interesting for me is the fact that with no cpu/memory limit and with > caching the result is approx. 60% slowdown with xen kernel in comparison > with non-xen kernel. Is it normal? >Yeah, it depends on a lot of things. Try with DIRECT io (and big enough block size), to get some real/meaningful numbers. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
benco@acid.sk
2010-Mar-01 12:07 UTC
Re: [Xen-users] slow dom0 write operations with 2.6.32 pv-ops and Xen4.0.0-rc4
> You''re measuring buffered (cached) performance, so when you limit > the amount of cache to 2 GB the performance will be slower. > > Try adding "oflag=direct" do your dd commands and test again.oflag=direct option produce almost the same results on xen and non-xen systems: #dd if=/dev/zero of=zerofile.tst bs=1k count=1000000 oflag=direct 1000000+0 records in 1000000+0 records out 1024000000 bytes (1.0 GB) copied, 616.219 s, 1.7 MB/s what is interesting for me is the fact that with no cpu/memory limit and with caching the result is approx. 60% slowdown with xen kernel in comparison with non-xen kernel. Is it normal? thanks, roman <>> -- Pasi > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- ---------------------------------------------------------------------- ,''''`. [benco] | mailto: benco@acid.sk | silc: /msg benco : :'' : ------------------------------------------------------------- `. `'' GPG publickey: http://www.acid.sk/pubkey.asc `- KF = 0DF6 0592 74D2 F17A DACF A5C3 1720 CB7C F54C F429 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fantu
2010-Mar-04 14:02 UTC
Re: [Xen-users] slow dom0 write operations with 2.6.32 pv-ops and Xen4.0.0-rc4
on one server with xen 4 from hg unstable, kernel pv_ops xen/stable and performance of disk in dom0: dd if=/dev/zero of=/mnt/vm/disks/xpt2.disk1.xm bs=1M count=10000 10000+0 records in 10000+0 records out 10485760000 bytes (10 GB) copied, 146,118 s, 71,8 MB/s benco-2 wrote:> >> You''re measuring buffered (cached) performance, so when you limit >> the amount of cache to 2 GB the performance will be slower. >> >> Try adding "oflag=direct" do your dd commands and test again. > > > oflag=direct option produce almost the same results on xen and non-xen > systems: > > #dd if=/dev/zero of=zerofile.tst bs=1k count=1000000 oflag=direct > 1000000+0 records in > 1000000+0 records out > 1024000000 bytes (1.0 GB) copied, 616.219 s, 1.7 MB/s > > what is interesting for me is the fact that with no cpu/memory limit and > with > caching the result is approx. 60% slowdown with xen kernel in comparison > with non-xen kernel. Is it normal? > > thanks, > > roman > > > <> >> -- Pasi >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > > -- > ---------------------------------------------------------------------- > ,''''`. [benco] | mailto: benco@acid.sk | silc: /msg benco > : :'' : ------------------------------------------------------------- > `. `'' GPG publickey: http://www.acid.sk/pubkey.asc > `- KF = 0DF6 0592 74D2 F17A DACF A5C3 1720 CB7C F54C F429 > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >-- View this message in context: http://old.nabble.com/slow-dom0-write-operations-with-2.6.32-pv-ops-and-Xen4.0.0-rc4-tp27738409p27781144.html Sent from the Xen - User mailing list archive at Nabble.com. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fantu
2010-Mar-04 15:18 UTC
Re: [Xen-users] slow dom0 write operations with 2.6.32 pv-ops and Xen4.0.0-rc4
Dell T610 with standard disk sata2 benco-2 wrote:> > Hi, > > thanks for sharing, could you please post also some specs of your server > and > disk system? > > thanks, > > roman > >-- View this message in context: http://old.nabble.com/slow-dom0-write-operations-with-2.6.32-pv-ops-and-Xen4.0.0-rc4-tp27738409p27782116.html Sent from the Xen - User mailing list archive at Nabble.com. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
benco
2010-Mar-04 15:54 UTC
Re: [Xen-users] slow dom0 write operations with 2.6.32 pv-ops and Xen4.0.0-rc4
Hi, thanks for sharing, could you please post also some specs of your server and disk system? thanks, roman On Thu, Mar 04, 2010 at 06:02:24AM -0800, Fantu wrote:> > on one server with xen 4 from hg unstable, kernel pv_ops xen/stable and > performance of disk in dom0: > dd if=/dev/zero of=/mnt/vm/disks/xpt2.disk1.xm bs=1M count=10000 > 10000+0 records in > 10000+0 records out > 10485760000 bytes (10 GB) copied, 146,118 s, 71,8 MB/s > > > benco-2 wrote: > > > >> You''re measuring buffered (cached) performance, so when you limit > >> the amount of cache to 2 GB the performance will be slower. > >> > >> Try adding "oflag=direct" do your dd commands and test again. > > > > > > oflag=direct option produce almost the same results on xen and non-xen > > systems: > > > > #dd if=/dev/zero of=zerofile.tst bs=1k count=1000000 oflag=direct > > 1000000+0 records in > > 1000000+0 records out > > 1024000000 bytes (1.0 GB) copied, 616.219 s, 1.7 MB/s > > > > what is interesting for me is the fact that with no cpu/memory limit and > > with > > caching the result is approx. 60% slowdown with xen kernel in comparison > > with non-xen kernel. Is it normal? > > > > thanks, > > > > roman > > > > > > <> > >> -- Pasi > >> > >> > >> _______________________________________________ > >> Xen-users mailing list > >> Xen-users@lists.xensource.com > >> http://lists.xensource.com/xen-users > > > > -- > > ---------------------------------------------------------------------- > > ,''''`. [benco] | mailto: benco@acid.sk | silc: /msg benco > > : :'' : ------------------------------------------------------------- > > `. `'' GPG publickey: http://www.acid.sk/pubkey.asc > > `- KF = 0DF6 0592 74D2 F17A DACF A5C3 1720 CB7C F54C F429 > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > > > -- > View this message in context: http://old.nabble.com/slow-dom0-write-operations-with-2.6.32-pv-ops-and-Xen4.0.0-rc4-tp27738409p27781144.html > Sent from the Xen - User mailing list archive at Nabble.com. > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- ---------------------------------------------------------------------- ,''''`. [benco] | mailto: benco@acid.sk | silc: /msg benco : :'' : ------------------------------------------------------------- `. `'' GPG publickey: http://www.acid.sk/pubkey.asc `- KF = 0DF6 0592 74D2 F17A DACF A5C3 1720 CB7C F54C F429 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users