Roberto Scudeller
2011-Nov-09 14:58 UTC
[Xen-devel] why do I get bad disk write performance in the kernel 3.1?
Hi all, I''m testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre. I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv drivers, and receive an amazing result for reading, but the write numbers are very disappointing. I run the VM in local disk, and receive these numbers: Seq read: 244 MB/s 512K read: 239 MB/s 4K read: 27 MB/s 4K QD 32: 90 MB/s Seq write: 20 MB/s 512K: 20 MB/s 4K: 8 MB/s 4K QD 32: 12MB/s In older kernel, 2.6.32.x: Seq read: 189 MB/s 512K read: 169 MB/s 4K read: 11 MB/s 4K QD 32: 34 MB/s Seq write: 177 MB/s 512K: 166 MB/s 4K: 12 MB/s 4K QD 32: 36 MB/s Could anyone help me. Why is the new kernel faster in reading, but too slow writing ? -- Roberto Scudeller _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Nov-09 15:18 UTC
Re: [Xen-devel] why do I get bad disk write performance in the kernel 3.1?
On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote:> Hi all, > > I''m testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre. > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv > drivers, and receive an amazing result for reading, but the write numbers > are very disappointing. > > I run the VM in local disk, and receive these numbers: > Seq read: 244 MB/s > 512K read: 239 MB/s > 4K read: 27 MB/s > 4K QD 32: 90 MB/s > > Seq write: 20 MB/s > 512K: 20 MB/s > 4K: 8 MB/s > 4K QD 32: 12MB/s > > In older kernel, 2.6.32.x: > Seq read: 189 MB/s > 512K read: 169 MB/s > 4K read: 11 MB/s > 4K QD 32: 34 MB/s > > Seq write: 177 MB/s > 512K: 166 MB/s > 4K: 12 MB/s > 4K QD 32: 36 MB/s > > Could anyone help me. Why is the new kernel faster in reading, but too slow > writing ?It would help if you gave an idea of what the guest configuration is and what is the backend. It might be that the backend you are using in 3.1 is QEMU qdisk, not xen-blkback. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Roberto Scudeller
2011-Nov-09 16:52 UTC
Re: [Xen-devel] why do I get bad disk write performance in the kernel 3.1?
Konrad, Thanks for help. My windows.cfg is: name=''benchCM-windows-2003-64b-std-test'' kernel=''/usr/lib/xen/boot/hvmloader'' builder=''hvm'' memory=512 vcpus=2 pae=1 acpi=1 apic=1 disk=[ ''tap2:aio:/local-disk/benchCM-windows-2003-64b-std/xvda,xvda,w'' ] device_model=''/usr/lib/xen/bin/qemu-dm'' boot=''c'' sdl=0 vnc=1 vncunused=1 vnclisten=''0.0.0.0'' vncpasswd='''' stdvga=0 extra='''' on_reboot=''restart'' on_shutdown=''destroy'' ramdisk='''' image_name='''' on_crash=''destroy'' bootloader='''' root='''' platform=''xen'' network_mode=''tap'' usb = 1 usbdevice = ''tablet'' The xvda disk is a raw disk, created with dd. I use the windows.cfg in both kernel. 2011/11/9 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>> On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote: > > Hi all, > > > > I''m testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre. > > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv > > drivers, and receive an amazing result for reading, but the write numbers > > are very disappointing. > > > > I run the VM in local disk, and receive these numbers: > > Seq read: 244 MB/s > > 512K read: 239 MB/s > > 4K read: 27 MB/s > > 4K QD 32: 90 MB/s > > > > Seq write: 20 MB/s > > 512K: 20 MB/s > > 4K: 8 MB/s > > 4K QD 32: 12MB/s > > > > In older kernel, 2.6.32.x: > > Seq read: 189 MB/s > > 512K read: 169 MB/s > > 4K read: 11 MB/s > > 4K QD 32: 34 MB/s > > > > Seq write: 177 MB/s > > 512K: 166 MB/s > > 4K: 12 MB/s > > 4K QD 32: 36 MB/s > > > > Could anyone help me. Why is the new kernel faster in reading, but too > slow > > writing ? > > It would help if you gave an idea of what the guest configuration is and > what is the backend. It might be that the backend you are using in 3.1 is > QEMU qdisk, not xen-blkback. >-- Roberto Scudeller _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Roberto Scudeller
2011-Nov-09 17:14 UTC
Re: [Xen-devel] why do I get bad disk write performance in the kernel 3.1?
Konrad, # xenstore-ls /local/domain/6/device/vbd 51712 = "" backend = "/local/domain/0/backend/qdisk/6/51712" backend-id = "0" state = "4" virtual-device = "51712" device-type = "disk" ring-ref = "16383" event-channel = "7" 51728 = "" backend = "/local/domain/0/backend/qdisk/6/51728" backend-id = "0" state = "4" virtual-device = "51728" device-type = "disk" ring-ref = "16371" event-channel = "8" Thanks for help. 2011/11/9 Roberto Scudeller <beto.rvs@gmail.com>> Konrad, > > Thanks for help. > > My windows.cfg is: > name=''benchCM-windows-2003-64b-std-test'' > kernel=''/usr/lib/xen/boot/hvmloader'' > builder=''hvm'' > memory=512 > vcpus=2 > pae=1 > acpi=1 > apic=1 > disk=[ ''tap2:aio:/local-disk/benchCM-windows-2003-64b-std/xvda,xvda,w'' ] > device_model=''/usr/lib/xen/bin/qemu-dm'' > boot=''c'' > sdl=0 > vnc=1 > vncunused=1 > vnclisten=''0.0.0.0'' > vncpasswd='''' > stdvga=0 > extra='''' > on_reboot=''restart'' > on_shutdown=''destroy'' > ramdisk='''' > image_name='''' > on_crash=''destroy'' > bootloader='''' > root='''' > platform=''xen'' > network_mode=''tap'' > usb = 1 > usbdevice = ''tablet'' > > The xvda disk is a raw disk, created with dd. > I use the windows.cfg in both kernel. > > > > > 2011/11/9 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > >> On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote: >> > Hi all, >> > >> > I''m testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre. >> > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv >> > drivers, and receive an amazing result for reading, but the write >> numbers >> > are very disappointing. >> > >> > I run the VM in local disk, and receive these numbers: >> > Seq read: 244 MB/s >> > 512K read: 239 MB/s >> > 4K read: 27 MB/s >> > 4K QD 32: 90 MB/s >> > >> > Seq write: 20 MB/s >> > 512K: 20 MB/s >> > 4K: 8 MB/s >> > 4K QD 32: 12MB/s >> > >> > In older kernel, 2.6.32.x: >> > Seq read: 189 MB/s >> > 512K read: 169 MB/s >> > 4K read: 11 MB/s >> > 4K QD 32: 34 MB/s >> > >> > Seq write: 177 MB/s >> > 512K: 166 MB/s >> > 4K: 12 MB/s >> > 4K QD 32: 36 MB/s >> > >> > Could anyone help me. Why is the new kernel faster in reading, but too >> slow >> > writing ? >> >> It would help if you gave an idea of what the guest configuration is and >> what is the backend. It might be that the backend you are using in 3.1 is >> QEMU qdisk, not xen-blkback. >> > > > > -- > Roberto Scudeller > > >-- Roberto Scudeller _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Nov-10 03:42 UTC
Re: [Xen-devel] why do I get bad disk write performance in the kernel 3.1?
On Wed, Nov 09, 2011 at 02:52:59PM -0200, Roberto Scudeller wrote:> Konrad, > > Thanks for help.OK, so looks like you are using qdisk. Can you try creating an LVM and blasting the image onto that? And obviously then using ''phy:/dev/vg_guest/...'' ? You could also try the ''file:/local-disk'' which will setup a loopback device and use that. Try that as well.> > My windows.cfg is: > name=''benchCM-windows-2003-64b-std-test'' > kernel=''/usr/lib/xen/boot/hvmloader'' > builder=''hvm'' > memory=512 > vcpus=2 > pae=1 > acpi=1 > apic=1 > disk=[ ''tap2:aio:/local-disk/benchCM-windows-2003-64b-std/xvda,xvda,w'' ] > device_model=''/usr/lib/xen/bin/qemu-dm'' > boot=''c'' > sdl=0 > vnc=1 > vncunused=1 > vnclisten=''0.0.0.0'' > vncpasswd='''' > stdvga=0 > extra='''' > on_reboot=''restart'' > on_shutdown=''destroy'' > ramdisk='''' > image_name='''' > on_crash=''destroy'' > bootloader='''' > root='''' > platform=''xen'' > network_mode=''tap'' > usb = 1 > usbdevice = ''tablet'' > > The xvda disk is a raw disk, created with dd. > I use the windows.cfg in both kernel. > > > > 2011/11/9 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > > > On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote: > > > Hi all, > > > > > > I''m testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre. > > > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv > > > drivers, and receive an amazing result for reading, but the write numbers > > > are very disappointing. > > > > > > I run the VM in local disk, and receive these numbers: > > > Seq read: 244 MB/s > > > 512K read: 239 MB/s > > > 4K read: 27 MB/s > > > 4K QD 32: 90 MB/s > > > > > > Seq write: 20 MB/s > > > 512K: 20 MB/s > > > 4K: 8 MB/s > > > 4K QD 32: 12MB/s > > > > > > In older kernel, 2.6.32.x: > > > Seq read: 189 MB/s > > > 512K read: 169 MB/s > > > 4K read: 11 MB/s > > > 4K QD 32: 34 MB/s > > > > > > Seq write: 177 MB/s > > > 512K: 166 MB/s > > > 4K: 12 MB/s > > > 4K QD 32: 36 MB/s > > > > > > Could anyone help me. Why is the new kernel faster in reading, but too > > slow > > > writing ? > > > > It would help if you gave an idea of what the guest configuration is and > > what is the backend. It might be that the backend you are using in 3.1 is > > QEMU qdisk, not xen-blkback. > > > > > > -- > Roberto Scudeller_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Nov-10 09:24 UTC
Re: [Xen-devel] why do I get bad disk write performance in the kernel 3.1?
On Thu, 2011-11-10 at 03:42 +0000, Konrad Rzeszutek Wilk wrote:> > > OK, so looks like you are using qdisk. [...]That depends on whether Roberto is using xm/xend or xl, I think. Roberto? I''m not sure what xend does for ''tap2:aio'' if blktap isn''t available. I suspect it doesn''t fallback to qdisk. Roberto can you report the content of xenstore under /local/domain/0/backend/ (use xenstore-ls <path>).> You could also try the ''file:/local-disk'' which will setup a loopback device > and use that. Try that as well.This is true with xm/xend but with xl you will need to manually setup the looback and use phy:/dev/loop in order to perform this experiment. For xl file: turns into qdisk unless blktap is available. Ian.> > > > > My windows.cfg is: > > name=''benchCM-windows-2003-64b-std-test'' > > kernel=''/usr/lib/xen/boot/hvmloader'' > > builder=''hvm'' > > memory=512 > > vcpus=2 > > pae=1 > > acpi=1 > > apic=1 > > disk=[ ''tap2:aio:/local-disk/benchCM-windows-2003-64b-std/xvda,xvda,w'' ] > > device_model=''/usr/lib/xen/bin/qemu-dm'' > > boot=''c'' > > sdl=0 > > vnc=1 > > vncunused=1 > > vnclisten=''0.0.0.0'' > > vncpasswd='''' > > stdvga=0 > > extra='''' > > on_reboot=''restart'' > > on_shutdown=''destroy'' > > ramdisk='''' > > image_name='''' > > on_crash=''destroy'' > > bootloader='''' > > root='''' > > platform=''xen'' > > network_mode=''tap'' > > usb = 1 > > usbdevice = ''tablet'' > > > > The xvda disk is a raw disk, created with dd. > > I use the windows.cfg in both kernel. > > > > > > > > 2011/11/9 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > > > > > On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote: > > > > Hi all, > > > > > > > > I''m testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre. > > > > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv > > > > drivers, and receive an amazing result for reading, but the write numbers > > > > are very disappointing. > > > > > > > > I run the VM in local disk, and receive these numbers: > > > > Seq read: 244 MB/s > > > > 512K read: 239 MB/s > > > > 4K read: 27 MB/s > > > > 4K QD 32: 90 MB/s > > > > > > > > Seq write: 20 MB/s > > > > 512K: 20 MB/s > > > > 4K: 8 MB/s > > > > 4K QD 32: 12MB/s > > > > > > > > In older kernel, 2.6.32.x: > > > > Seq read: 189 MB/s > > > > 512K read: 169 MB/s > > > > 4K read: 11 MB/s > > > > 4K QD 32: 34 MB/s > > > > > > > > Seq write: 177 MB/s > > > > 512K: 166 MB/s > > > > 4K: 12 MB/s > > > > 4K QD 32: 36 MB/s > > > > > > > > Could anyone help me. Why is the new kernel faster in reading, but too > > > slow > > > > writing ? > > > > > > It would help if you gave an idea of what the guest configuration is and > > > what is the backend. It might be that the backend you are using in 3.1 is > > > QEMU qdisk, not xen-blkback. > > > > > > > > > > > -- > > Roberto Scudeller > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Roberto Scudeller
2011-Nov-10 14:25 UTC
Re: [Xen-devel] why do I get bad disk write performance in the kernel 3.1?
Ian,> That depends on whether Roberto is using xm/xend or xl, I think. > Roberto? >I''m using "xl create windows.cfg"> > I''m not sure what xend does for ''tap2:aio'' if blktap isn''t available. I > suspect it doesn''t fallback to qdisk. Roberto can you report the content > of xenstore under /local/domain/0/backend/ (use xenstore-ls <path>). > > # xenstore-ls /local/domain/0/backendqdisk = "" 10 = "" 51712 = "" frontend = "/local/domain/10/device/vbd/51712" params = "aio:/local-disk/benchCM-windows-2003-64b-std/xvda" frontend-id = "10" online = "1" removable = "0" bootable = "1" state = "4" dev = "xvda" type = "tap" mode = "w" feature-barrier = "1" info = "0" sector-size = "512" sectors = "67108864" hotplug-status = "connected" 51728 = "" frontend = "/local/domain/10/device/vbd/51728" params = "aio:/local-disk/benchCM-windows-2003-64b-std/xvdb" frontend-id = "10" online = "1" removable = "0" bootable = "1" state = "4" dev = "xvdb" type = "tap" mode = "w" feature-barrier = "1" info = "0" sector-size = "512" sectors = "614400" hotplug-status = "connected" console = "" 10 = "" 0 = "" frontend = "/local/domain/10/console" frontend-id = "10" online = "1" state = "1" domain = "benchCM-windows-2003-64b-std-test" protocol = "vt100"> > You could also try the ''file:/local-disk'' which will setup a loopback > device > > and use that. Try that as well. > > This is true with xm/xend but with xl you will need to manually setup > the looback and use phy:/dev/loop in order to perform this experiment. > For xl file: turns into qdisk unless blktap is available. > > I''m execute with file:/, but in xenstore-list, continue with qdisk. Look:device = "" suspend = "" event-channel = "6" vbd = "" 51712 = "" backend = "/local/domain/0/backend/qdisk/10/51712" backend-id = "0" state = "4" virtual-device = "51712" device-type = "disk" ring-ref = "16383" event-channel = "7" 51728 = "" backend = "/local/domain/0/backend/qdisk/10/51728" backend-id = "0" state = "4" virtual-device = "51728" device-type = "disk" ring-ref = "16371" event-channel = "8" Konrad, I try with "file://" and result was qdisk again. Do you think is LVM test need? I testing my windows.cfg in xen 4.1.2-rc1-pre with kernel 2.6.32.43, the result of xenstore-ls: # xenstore-ls /local/domain/0/backend vbd = "" 2 = "" 51712 = "" frontend = "/local/domain/2/device/vbd/51712" tapdisk-params = "aio:/disk-local/benchCM-windows-2003-64b-std/xvda" params = "/dev/xen/blktap-2/tapdev2" physical-device = "fd:2" frontend-id = "2" online = "1" removable = "0" bootable = "1" state = "4" dev = "xvda" type = "phy" mode = "w" feature-barrier = "1" sectors = "67108864" info = "0" sector-size = "512" 51728 = "" frontend = "/local/domain/2/device/vbd/51728" tapdisk-params = "aio:/local-disk/benchCM-windows-2003-64b-std/xvdb" params = "/dev/xen/blktap-2/tapdev3" physical-device = "fd:3" frontend-id = "2" online = "1" removable = "0" bootable = "1" state = "4" dev = "xvdb" type = "phy" mode = "w" feature-barrier = "1" sectors = "614400" info = "0" sector-size = "512" console = "" 2 = "" 0 = "" frontend = "/local/domain/2/console" frontend-id = "2" online = "1" state = "1" domain = "benchCM-windows-2003-64b-std-test" protocol = "vt100" Why did kernel 3.1 use the qdisk? I want to use blktap backend (''tap2:aio:/disk-local/xvda,xvda,w''). Thanks for help. -- Roberto Scudeller _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Nov-10 14:29 UTC
Re: [Xen-devel] why do I get bad disk write performance in the kernel 3.1?
On Thu, 2011-11-10 at 14:25 +0000, Roberto Scudeller wrote:> Ian, > > > That depends on whether Roberto is using xm/xend or xl, I > think. > Roberto? > > I''m using "xl create windows.cfg"OK, so you will be getting qdisk (and your logs confirm this).> > > You could also try the ''file:/local-disk'' which will setup a > loopback device > > and use that. Try that as well. > > > This is true with xm/xend but with xl you will need to > manually setup > the looback and use phy:/dev/loop in order to perform this > experiment. > For xl file: turns into qdisk unless blktap is available. > > I''m execute with file:/, but in xenstore-list, continue with qdisk.Please try using losetup to bind your file to /dev/loopN and specifying phy:/dev/loopN in your config. That should get you blkback. [...]> I try with "file://" and result was qdisk again. Do you think is LVM > test need?That would also be interesting.> Why did kernel 3.1 use the qdisk? I want to use blktap backend > (''tap2:aio:/disk-local/xvda,xvda,w'').blktap is not available in 3.1. Unfortunately it is not upstreamable so it is unlikely to become available until someone does a pile of work to make it a userspace only thing (see the list archives for various explanations of this). Since you are using a raw image (rather than e.g. vhd or qcow) there is no benefit to blktap. Ian.> > > > Thanks for help. > > -- > Roberto Scudeller > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel