Xen devel - Jun 2011 - bad write performance with qdisk with larger files in pv-domU

On Fri, Jun 03, 2011 at 11:49:47PM +0000, Ronny Hegewald
wrote:
> Im using the following 32-bit setup:
> 
> - xen 4.1.0 
> - upstream linux-kernel 2.6.39 as dom0
> - linux 2.6.32 pv-domU that has several ext3 partitions mounted with qdisk 
> (same behaviour with a 2.6.39 kernel, so i continued the investigation with
> the 2.6.32er kernel)
> 
> Die read performance is good (ca. 60 MB/s)
> 
> For smaller files (< 30-40 MB) the write-speed is ok.
> 
> But if i copy a larger file (ca > 40 MB), the write speed decreases to
ca. 0,5
> MB/s, after the first ca. 40 MBs are written.
> 
> One reason for the bad performance might be that qdisk doesnt use AIO. For 
> testing purposes i activated AIO in hw/xen_disk.c (i set use_aio=1), but
the
> domU freezed shortly after the domU-kernel started.
You could also use this patch:
http://darnok.org/xen/qdisk_vs_blkback_v3.1/qemu-enable-aio.patch

But why not use the 3.0-rc1 with the xen-blkback? Or if you want to use 2.6.39
you could use the 

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git stable/2.6.39.x
tree

> 
> Is this performance-impact expected when no AIO is used? 
Yeah, it is slow.
> 
> I compared the raw-block implementation from xen-qemu 4.1.0 and current 
> upstream, in case xen-qemu has some missing bugfixes and found the
following
> patch that looks a bit interesting
> 
> 	commit 4899d10d142e97eea8f64141a3507b2ee1a64f52
> 	Author: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
> 	Date:   Mon Apr 19 13:34:11 2010 +0100
>     	raw-posix: Use pread/pwrite instead of lseek+read/write
> 
> 	This patch combines the lseek+read/write calls to use pread/pwrite
>     	instead.  This will result in fewer system calls and is already used
by
>     	AIO.
> 
> 
> From the first look the patch cannot be backported 1:1, so i havent tried
it
> yet, because i doubt that it can make such a huge difference. Or would it
be
> worth a try?
> 
> Any other ideas how/what to investigate this issue further, in case the
write-
> speed should be better also without AIO? I know that the qdisk
implementation
> is expected to be slower, but i would expect at least lets say 5 MB/s.
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Im using the following 32-bit setup:

- xen 4.1.0 
- upstream linux-kernel 2.6.39 as dom0
- linux 2.6.32 pv-domU that has several ext3 partitions mounted with qdisk 
(same behaviour with a 2.6.39 kernel, so i continued the investigation with 
the 2.6.32er kernel)

Die read performance is good (ca. 60 MB/s)

For smaller files (< 30-40 MB) the write-speed is ok.

But if i copy a larger file (ca > 40 MB), the write speed decreases to ca.
0,5
MB/s, after the first ca. 40 MBs are written.

One reason for the bad performance might be that qdisk doesnt use AIO. For 
testing purposes i activated AIO in hw/xen_disk.c (i set use_aio=1), but the 
domU freezed shortly after the domU-kernel started.

Is this performance-impact expected when no AIO is used? 

I compared the raw-block implementation from xen-qemu 4.1.0 and current 
upstream, in case xen-qemu has some missing bugfixes and found the following 
patch that looks a bit interesting

	commit 4899d10d142e97eea8f64141a3507b2ee1a64f52
	Author: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
	Date:   Mon Apr 19 13:34:11 2010 +0100
    	raw-posix: Use pread/pwrite instead of lseek+read/write

	This patch combines the lseek+read/write calls to use pread/pwrite
    	instead.  This will result in fewer system calls and is already used by
    	AIO.


From the first look the patch cannot be backported 1:1, so i havent tried it 
yet, because i doubt that it can make such a huge difference. Or would it be 
worth a try?

Any other ideas how/what to investigate this issue further, in case the write-
speed should be better also without AIO? I know that the qdisk implementation 
is expected to be slower, but i would expect at least lets say 5 MB/s.




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> You could also use this patch:
>, http://darnok.org/xen/qdisk_vs_blkback_v3.1/qemu-enable-aio.patch
Wasn''t aware of that patch, i will try it out.
>
> But why not use the 3.0-rc1 with the xen-blkback? Or if you want to use
> 2.6.39 you could use the
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> stable/2.6.39.x tree
I planned to try out the qdisk/qemu blktap implementation a bit. The missing 
xen-blkback in officiall 2.6.39 was just a good opportunity for this.

And from what i have read the future of blktap might be more in qemu then in 
the kernel, so thats another reason to get used to it.
> > Is this performance-impact expected when no AIO is used?
>
> Yeah, it is slow.
Ok, then no reason to dig deeper into that. So my current impression is that 
its better not use qdisk/qemu-blktap until xen can use upstream qemu?



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, 4 Jun 2011, Ronny Hegewald wrote:
> > > Is this performance-impact expected when no AIO is used?
> >
> > Yeah, it is slow.
> 
> Ok, then no reason to dig deeper into that. So my current impression is
that
> its better not use qdisk/qemu-blktap until xen can use upstream qemu?
Yes, indeed.
Also bear in mind that blkback is a very good option for LVM or physical
partitions, but it is not very fast for raw files.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel