Keith Coleman
2010-Feb-19 22:41 UTC
[Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
I am posting this to xen-devel instead of -users because it paints an incomplete picture that shouldn''t be the basis for deciding how to run production systems. This graph shows the performance under a webserver disk IO workload at different queue depths. It compares the 4 main IO methods for windows guests that will be available in the upcoming xen 4.0.0 and 3.4.3 releases: pure HVM, stub domains, gplpv drivers, and xcp winpv drivers. The gplpv and xcp winpv drivers have comparable performance with gplpv being slightly faster. Both pv drivers are considerably faster than pure hvm or stub domains. Stub domain performance was about even with HVM which is lower than we were expecting. We tried a different cpu pinning in "Stubdom B" with little impact. Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Feb-20 00:08 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Fri, 2010-02-19 at 17:41 -0500, Keith Coleman wrote:> This graph shows the performance under a webserver disk IO workload at > different queue depths. It compares the 4 main IO methods for windows > guests that will be available in the upcoming xen 4.0.0 and 3.4.3 > releases: pure HVM, stub domains, gplpv drivers, and xcp winpv > drivers.Cool, thanks. If I may ask, what exactly did you run?> The gplpv and xcp winpv drivers have comparable performance with gplpv > being slightly faster. Both pv drivers are considerably faster than > pure hvm or stub domains. Stub domain performance was about even with > HVM which is lower than we were expecting. We tried a different cpu > pinning in "Stubdom B" with little impact.Is this an SMP dom0? A single guest? Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keith Coleman
2010-Feb-20 00:50 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Fri, Feb 19, 2010 at 7:08 PM, Daniel Stodden <daniel.stodden@citrix.com> wrote:> On Fri, 2010-02-19 at 17:41 -0500, Keith Coleman wrote: > >> This graph shows the performance under a webserver disk IO workload at >> different queue depths. It compares the 4 main IO methods for windows >> guests that will be available in the upcoming xen 4.0.0 and 3.4.3 >> releases: pure HVM, stub domains, gplpv drivers, and xcp winpv >> drivers. > > Cool, thanks. If I may ask, what exactly did you run?iometer>> The gplpv and xcp winpv drivers have comparable performance with gplpv >> being slightly faster. Both pv drivers are considerably faster than >> pure hvm or stub domains. Stub domain performance was about even with >> HVM which is lower than we were expecting. We tried a different cpu >> pinning in "Stubdom B" with little impact. > > Is this an SMP dom0? A single guest?Dual core server with dom0 pinned to core 0 and a single domU pinned to core 1. Stubdom was pinned to core 0 then core 1. Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Feb-20 01:46 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Fri, 2010-02-19 at 19:50 -0500, Keith Coleman wrote:> On Fri, Feb 19, 2010 at 7:08 PM, Daniel Stodden > <daniel.stodden@citrix.com> wrote: > > On Fri, 2010-02-19 at 17:41 -0500, Keith Coleman wrote: > > > >> This graph shows the performance under a webserver disk IO workload at > >> different queue depths. It compares the 4 main IO methods for windows > >> guests that will be available in the upcoming xen 4.0.0 and 3.4.3 > >> releases: pure HVM, stub domains, gplpv drivers, and xcp winpv > >> drivers. > > > > Cool, thanks. If I may ask, what exactly did you run? > > iometer > > >> The gplpv and xcp winpv drivers have comparable performance with gplpv > >> being slightly faster. Both pv drivers are considerably faster than > >> pure hvm or stub domains. Stub domain performance was about even with > >> HVM which is lower than we were expecting. We tried a different cpu > >> pinning in "Stubdom B" with little impact. > > > > Is this an SMP dom0? A single guest? > > Dual core server with dom0 pinned to core 0 and a single domU pinned > to core 1. Stubdom was pinned to core 0 then core 1.I don''t see why stubdom would be faster in either configuration. Once you''re through DM emulation, there''s plenty of cycles to spend waiting for I/O completion. So dom0 won''t mind spending them on qemu either. Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2010-Feb-22 16:33 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Fri, 19 Feb 2010, Keith Coleman wrote:> I am posting this to xen-devel instead of -users because it paints an > incomplete picture that shouldn''t be the basis for deciding how to run > production systems. > > This graph shows the performance under a webserver disk IO workload at > different queue depths. It compares the 4 main IO methods for windows > guests that will be available in the upcoming xen 4.0.0 and 3.4.3 > releases: pure HVM, stub domains, gplpv drivers, and xcp winpv > drivers. > > The gplpv and xcp winpv drivers have comparable performance with gplpv > being slightly faster. Both pv drivers are considerably faster than > pure hvm or stub domains. Stub domain performance was about even with > HVM which is lower than we were expecting. We tried a different cpu > pinning in "Stubdom B" with little impact. >What disk backend are you using? If you are using a raw file, it is worth trying with an lvm volume as well. Also are using blktap or blktap2? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keith Coleman
2010-Feb-22 17:14 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On 2/22/10, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Fri, 19 Feb 2010, Keith Coleman wrote: >> I am posting this to xen-devel instead of -users because it paints an >> incomplete picture that shouldn''t be the basis for deciding how to run >> production systems. >> >> This graph shows the performance under a webserver disk IO workload at >> different queue depths. It compares the 4 main IO methods for windows >> guests that will be available in the upcoming xen 4.0.0 and 3.4.3 >> releases: pure HVM, stub domains, gplpv drivers, and xcp winpv >> drivers. >> >> The gplpv and xcp winpv drivers have comparable performance with gplpv >> being slightly faster. Both pv drivers are considerably faster than >> pure hvm or stub domains. Stub domain performance was about even with >> HVM which is lower than we were expecting. We tried a different cpu >> pinning in "Stubdom B" with little impact. >> > > What disk backend are you using?phy, LV Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2010-Feb-22 17:27 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Mon, 22 Feb 2010, Keith Coleman wrote:> On 2/22/10, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote: > > On Fri, 19 Feb 2010, Keith Coleman wrote: > >> I am posting this to xen-devel instead of -users because it paints an > >> incomplete picture that shouldn''t be the basis for deciding how to run > >> production systems. > >> > >> This graph shows the performance under a webserver disk IO workload at > >> different queue depths. It compares the 4 main IO methods for windows > >> guests that will be available in the upcoming xen 4.0.0 and 3.4.3 > >> releases: pure HVM, stub domains, gplpv drivers, and xcp winpv > >> drivers. > >> > >> The gplpv and xcp winpv drivers have comparable performance with gplpv > >> being slightly faster. Both pv drivers are considerably faster than > >> pure hvm or stub domains. Stub domain performance was about even with > >> HVM which is lower than we were expecting. We tried a different cpu > >> pinning in "Stubdom B" with little impact. > >> > > > > What disk backend are you using? > > phy, LV >That is strange because in that configuration I get a far better disk bandwidth with stubdoms compared to qemu running in dom0. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keith Coleman
2010-Feb-22 21:13 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Mon, Feb 22, 2010 at 12:27 PM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Mon, 22 Feb 2010, Keith Coleman wrote: >> On 2/22/10, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote: >> > On Fri, 19 Feb 2010, Keith Coleman wrote: >> >> I am posting this to xen-devel instead of -users because it paints an >> >> incomplete picture that shouldn''t be the basis for deciding how to run >> >> production systems. >> >> >> >> This graph shows the performance under a webserver disk IO workload at >> >> different queue depths. It compares the 4 main IO methods for windows >> >> guests that will be available in the upcoming xen 4.0.0 and 3.4.3 >> >> releases: pure HVM, stub domains, gplpv drivers, and xcp winpv >> >> drivers. >> >> >> >> The gplpv and xcp winpv drivers have comparable performance with gplpv >> >> being slightly faster. Both pv drivers are considerably faster than >> >> pure hvm or stub domains. Stub domain performance was about even with >> >> HVM which is lower than we were expecting. We tried a different cpu >> >> pinning in "Stubdom B" with little impact. >> >> >> > >> > What disk backend are you using? >> >> phy, LV >> > > That is strange because in that configuration I get a far better > disk bandwidth with stubdoms compared to qemu running in dom0. >What type of test are you doing? Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2010-Feb-23 13:14 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Mon, 22 Feb 2010, Keith Coleman wrote:> On Mon, Feb 22, 2010 at 12:27 PM, Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > On Mon, 22 Feb 2010, Keith Coleman wrote: > >> On 2/22/10, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote: > >> > On Fri, 19 Feb 2010, Keith Coleman wrote: > >> >> I am posting this to xen-devel instead of -users because it paints an > >> >> incomplete picture that shouldn''t be the basis for deciding how to run > >> >> production systems. > >> >> > >> >> This graph shows the performance under a webserver disk IO workload at > >> >> different queue depths. It compares the 4 main IO methods for windows > >> >> guests that will be available in the upcoming xen 4.0.0 and 3.4.3 > >> >> releases: pure HVM, stub domains, gplpv drivers, and xcp winpv > >> >> drivers. > >> >> > >> >> The gplpv and xcp winpv drivers have comparable performance with gplpv > >> >> being slightly faster. Both pv drivers are considerably faster than > >> >> pure hvm or stub domains. Stub domain performance was about even with > >> >> HVM which is lower than we were expecting. We tried a different cpu > >> >> pinning in "Stubdom B" with little impact. > >> >> > >> > > >> > What disk backend are you using? > >> > >> phy, LV > >> > > > > That is strange because in that configuration I get a far better > > disk bandwidth with stubdoms compared to qemu running in dom0. > > > > What type of test are you doing? >these are the results I got a while ago running a simple "dd if=/dev/zero of=file" for 10 seconds: qemu in dom0: 25.1 MB/s qemu in a stubdom: 56.7 MB/s I have just run just now "tiobench with --size 256 --numruns 4 --threads 4" using a raw file as a backend: qemu in dom0, using blktap2, best run: File Blk Num Avg Maximum Lat% Lat% CPU Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 256 4096 4 85.82 108.6% 0.615 1534.10 0.00000 0.00000 79 qemu in a stubdom, using phy on a loop device, best run: File Blk Num Avg Maximum Lat% Lat% CPU Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 256 4096 4 130.49 163.8% 0.345 1459.94 0.00000 0.00000 80 These results as for the "sequential reads" test and rate is in megabytes per second. If I use phy on a loop device with qemu in dom0 unexpectedly I get much worse results. Same thing happens if I use tap:aio with qemu in a stubdom, but this is kind of expected since blktap is never going to be as fast as blkback. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Feb-23 14:44 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
> > > That is strange because in that configuration I get a far better > > > disk bandwidth with stubdoms compared to qemu running in dom0. > > > > > > > What type of test are you doing? > > > > these are the results I got a while ago running a simple "dd if=/dev/zero > of=file" for 10 seconds:Keep in mind that iometer (both the Windows a Linux version) by default do random seak of 50% reads and 50% writes. They do have some set of templates - "web server", "file server", "database server" that change the read/write ratio, size of blocks, and the queue length.(fyi, you can use fio to set the same values, if you can''t get dynamo to compile on your Linux box). Keith, which workload did you choose? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Feb-23 19:38 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Tue, Feb 23, 2010 at 01:14:47PM +0000, Stefano Stabellini wrote:> On Mon, 22 Feb 2010, Keith Coleman wrote: > > On Mon, Feb 22, 2010 at 12:27 PM, Stefano Stabellini > > <stefano.stabellini@eu.citrix.com> wrote: > > > On Mon, 22 Feb 2010, Keith Coleman wrote: > > >> On 2/22/10, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote: > > >> > On Fri, 19 Feb 2010, Keith Coleman wrote: > > >> >> I am posting this to xen-devel instead of -users because it paints an > > >> >> incomplete picture that shouldn''t be the basis for deciding how to run > > >> >> production systems. > > >> >> > > >> >> This graph shows the performance under a webserver disk IO workload at > > >> >> different queue depths. It compares the 4 main IO methods for windows > > >> >> guests that will be available in the upcoming xen 4.0.0 and 3.4.3 > > >> >> releases: pure HVM, stub domains, gplpv drivers, and xcp winpv > > >> >> drivers. > > >> >> > > >> >> The gplpv and xcp winpv drivers have comparable performance with gplpv > > >> >> being slightly faster. Both pv drivers are considerably faster than > > >> >> pure hvm or stub domains. Stub domain performance was about even with > > >> >> HVM which is lower than we were expecting. We tried a different cpu > > >> >> pinning in "Stubdom B" with little impact. > > >> >> > > >> > > > >> > What disk backend are you using? > > >> > > >> phy, LV > > >> > > > > > > That is strange because in that configuration I get a far better > > > disk bandwidth with stubdoms compared to qemu running in dom0. > > > > > > > What type of test are you doing? > > > > these are the results I got a while ago running a simple "dd if=/dev/zero > of=file" for 10 seconds: > > qemu in dom0: 25.1 MB/s > qemu in a stubdom: 56.7 MB/s >For dd tests you might want to use "oflag=direct" to make it use direct IO, and not domU kernel cached.. also longer test would be good.> > > I have just run just now "tiobench with --size 256 --numruns 4 --threads > 4" using a raw file as a backend: > > > qemu in dom0, using blktap2, best run: > > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff > ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- > 256 4096 4 85.82 108.6% 0.615 1534.10 0.00000 0.00000 79 > > qemu in a stubdom, using phy on a loop device, best run: > > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff > ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- > 256 4096 4 130.49 163.8% 0.345 1459.94 0.00000 0.00000 80 > > > These results as for the "sequential reads" test and rate is in > megabytes per second. > If I use phy on a loop device with qemu in dom0 unexpectedly I get much > worse results. > Same thing happens if I use tap:aio with qemu in a stubdom, but this is > kind of expected since blktap is never going to be as fast as blkback. >Hmm... what''s the overall cpu usage difference, measured from the hypervisor? "xm top" or so.. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Feb-23 19:39 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Tue, Feb 23, 2010 at 09:44:24AM -0500, Konrad Rzeszutek Wilk wrote:> > > > That is strange because in that configuration I get a far better > > > > disk bandwidth with stubdoms compared to qemu running in dom0. > > > > > > > > > > What type of test are you doing? > > > > > > > these are the results I got a while ago running a simple "dd if=/dev/zero > > of=file" for 10 seconds: > > Keep in mind that iometer (both the Windows a Linux version) by default > do random seak of 50% reads and 50% writes. They do have some set of > templates - "web server", "file server", "database server" that change > the read/write ratio, size of blocks, and the queue length.(fyi, you can > use fio to set the same values, if you can''t get dynamo to compile on > your Linux box). >Does iometer run correctly on Linux nowadays? I remember it having problems with more than 1 outstanding IO.. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Feb-23 20:03 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Tue, Feb 23, 2010 at 09:39:26PM +0200, Pasi Kärkkäinen wrote:> On Tue, Feb 23, 2010 at 09:44:24AM -0500, Konrad Rzeszutek Wilk wrote: > > > > > That is strange because in that configuration I get a far better > > > > > disk bandwidth with stubdoms compared to qemu running in dom0. > > > > > > > > > > > > > What type of test are you doing? > > > > > > > > > > these are the results I got a while ago running a simple "dd if=/dev/zero > > > of=file" for 10 seconds: > > > > Keep in mind that iometer (both the Windows a Linux version) by default > > do random seak of 50% reads and 50% writes. They do have some set of > > templates - "web server", "file server", "database server" that change > > the read/write ratio, size of blocks, and the queue length.(fyi, you can > > use fio to set the same values, if you can''t get dynamo to compile on > > your Linux box). > > > > Does iometer run correctly on Linux nowadays? I remember it having problems > with more than 1 outstanding IO..It seems to work for me. Thought if you download it from their web-site expect to hack a bit of their module to make it work with new kernels. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keith Coleman
2010-Feb-23 20:11 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Tue, Feb 23, 2010 at 9:44 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> > > That is strange because in that configuration I get a far better >> > > disk bandwidth with stubdoms compared to qemu running in dom0. >> > > >> > >> > What type of test are you doing? >> > >> >> these are the results I got a while ago running a simple "dd if=/dev/zero >> of=file" for 10 seconds: > > Keep in mind that iometer (both the Windows a Linux version) by default > do random seak of 50% reads and 50% writes. They do have some set of > templates - "web server", "file server", "database server" that change > the read/write ratio, size of blocks, and the queue length.(fyi, you can > use fio to set the same values, if you can''t get dynamo to compile on > your Linux box). > > Keith, which workload did you choose? >Web server workload was used in my graph. Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Marco Sinhoreli
2010-Feb-23 20:47 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
Hi Keith, Do you have some KVM graph? It will be interesting to compare both. Cheers On Tue, Feb 23, 2010 at 4:39 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> On Tue, Feb 23, 2010 at 09:44:24AM -0500, Konrad Rzeszutek Wilk wrote: > > > > > That is strange because in that configuration I get a far better > > > > > disk bandwidth with stubdoms compared to qemu running in dom0. > > > > > > > > > > > > > What type of test are you doing? > > > > > > > > > > these are the results I got a while ago running a simple "dd > if=/dev/zero > > > of=file" for 10 seconds: > > > > Keep in mind that iometer (both the Windows a Linux version) by default > > do random seak of 50% reads and 50% writes. They do have some set of > > templates - "web server", "file server", "database server" that change > > the read/write ratio, size of blocks, and the queue length.(fyi, you can > > use fio to set the same values, if you can''t get dynamo to compile on > > your Linux box). > > > > Does iometer run correctly on Linux nowadays? I remember it having problems > with more than 1 outstanding IO.. > > -- Pasi > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >-- Marco Sinhoreli _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keith Coleman
2010-Feb-23 21:06 UTC
Re: [Xen-devel] windows domU disk performance graph comparing hvm vs stubdom vs pv drivers
On Tue, Feb 23, 2010 at 3:47 PM, Marco Sinhoreli <msinhore@gmail.com> wrote:> Hi Keith, > Do you have some KVM graph? It will be interesting to compare both. >I have not compared to KVM. Keith Coleman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel