Xen devel - Oct 2006 - Does Xen hypervisor overwrite O_DIRECT setting of Linux 2.6 kernel?

Hi,

I''m doing some I/O performance analysis on Xen now. I noticed some
weird
thing when comparing the I/O performance data of Xen domain0 and Linux 
native. For sequential READ performance (no matter small packet or large 
packet), Linux native shows better performance than Xen domain0 (this is 
expected). However, for sequential write of small packet (512B and 1KB 
packets), Xen domain0 always outperforms the Linux native a lot. I ran the 
performance for several times and the results are pretty consistent.

The performance data is collected on 8 SAS drives (used as physical drives) 
and IOMeter is used as the benchmark tool. The latest IOMeter version used 
O_DIRECT. We know, Linux 2.6 kernel starts supporting O_DIRECT which makes 
all I/O requests work around buffer cache. The good thing for O_DIRECT is it 
reduces the CPU utilization and cache pollution. The bad thing is O_DIRECT 
not only forces all I/O requests become synchronous and no I/O coalescing 
will happen. Thus sequential write of small packets will be impacted most. 
For Xen, however, I believe Xen hypervisor overwrites this O_DIRECT setting 
and maybe it favors better performance over CPU and FSB utilization. Thus 
Xen domain0 can have better write performance than Linux native.

Is this correct?


Thanks,


Liang




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> The performance data is collected on 8 SAS drives (used as physical
drives)
> and IOMeter is used as the benchmark tool. The latest IOMeter version
used
> O_DIRECT. We know, Linux 2.6 kernel starts supporting O_DIRECT which
makes
> all I/O requests work around buffer cache. The good thing for O_DIRECT
is
> it
> reduces the CPU utilization and cache pollution. The bad thing is
O_DIRECT
> not only forces all I/O requests become synchronous and no I/O
coalescing
> will happen. Thus sequential write of small packets will be impacted
most.
> For Xen, however, I believe Xen hypervisor overwrites this O_DIRECT
setting
> and maybe it favors better performance over CPU and FSB utilization.
Thus
> Xen domain0 can have better write performance than Linux native.
Are you sure you''re comparing identical native and dom0 kernel
versions?
Same drivers and settings?
Xen does not disable O_DIRECT. 

Ian 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yes, all the testing configurations are the exactly same (both uses Linux 
kernel 2.6.16.29) and I used Xen 3.0.3. I did several indepedent runs for my 
testing and the results are consistent, e.g. the performance of small packet 
sequential write under Xen Domain0 outperform Linux native by 10~20%.

Some people told me Xen hypervisor adds addtional layer beyond Linux I/O 
stack. So O_DIRECT under Xen hypervisor is still valid, but I/O coalescing 
for small packets are enabled by Xen hypervisor. However, Linux native 
kernel would not do any I/O coalescing when using O_DIRECT.

I hope Xen gurus can give me more explanations about this.

Thanks,

Liang

----- Original Message ----- 
From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
To: "Liang Yang" <multisyncfe991@hotmail.com>; 
<xen-devel@lists.xensource.com>
Cc: <ian.pratt@cl.cam.ac.uk>
Sent: Saturday, October 28, 2006 2:34 PM
Subject: RE: [Xen-devel] Does Xen hypervisor overwrite O_DIRECT setting of 
Linux2.6 kernel?

> The performance data is collected on 8 SAS drives (used as physical
drives)
> and IOMeter is used as the benchmark tool. The latest IOMeter version
used
> O_DIRECT. We know, Linux 2.6 kernel starts supporting O_DIRECT which
makes
> all I/O requests work around buffer cache. The good thing for O_DIRECT
is
> it
> reduces the CPU utilization and cache pollution. The bad thing is
O_DIRECT
> not only forces all I/O requests become synchronous and no I/O
coalescing
> will happen. Thus sequential write of small packets will be impacted
most.
> For Xen, however, I believe Xen hypervisor overwrites this O_DIRECT
setting
> and maybe it favors better performance over CPU and FSB utilization.
Thus
> Xen domain0 can have better write performance than Linux native.
Are you sure you''re comparing identical native and dom0 kernel
versions?
Same drivers and settings?
Xen does not disable O_DIRECT.

Ian


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Yes, all the testing configurations are the exactly same (both uses
Linux
> kernel 2.6.16.29) and I used Xen 3.0.3. I did several indepedent runs
for
> my testing and the results are consistent, e.g. the performance of
small
> packet
> sequential write under Xen Domain0 outperform Linux native by 10~20%.
> 
> Some people told me Xen hypervisor adds addtional layer beyond Linux
I/O
> stack. So O_DIRECT under Xen hypervisor is still valid, but I/O
coalescing
> for small packets are enabled by Xen hypervisor. However, Linux native
> kernel would not do any I/O coalescing when using O_DIRECT.
For IO from dom0 there is no additional layer. There will be no
difference in the coalescing behaviour.

Are you sure you''re writing to the same part of the disk in both cases?
You know about disk zoning, right?


Ian
> I hope Xen gurus can give me more explanations about this.
> 
> Thanks,
> 
> Liang
> 
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "Liang Yang" <multisyncfe991@hotmail.com>;
> <xen-devel@lists.xensource.com>
> Cc: <ian.pratt@cl.cam.ac.uk>
> Sent: Saturday, October 28, 2006 2:34 PM
> Subject: RE: [Xen-devel] Does Xen hypervisor overwrite O_DIRECT
setting of
> Linux2.6 kernel?
> 
> 
> > The performance data is collected on 8 SAS drives (used as physical
> drives)
> > and IOMeter is used as the benchmark tool. The latest IOMeter
version
> used
> > O_DIRECT. We know, Linux 2.6 kernel starts supporting O_DIRECT which
> makes
> > all I/O requests work around buffer cache. The good thing for
O_DIRECT
> is
> > it
> > reduces the CPU utilization and cache pollution. The bad thing is
> O_DIRECT
> > not only forces all I/O requests become synchronous and no I/O
> coalescing
> > will happen. Thus sequential write of small packets will be impacted
> most.
> > For Xen, however, I believe Xen hypervisor overwrites this O_DIRECT
> setting
> > and maybe it favors better performance over CPU and FSB utilization.
> Thus
> > Xen domain0 can have better write performance than Linux native.
> 
> Are you sure you''re comparing identical native and dom0 kernel
versions?
> Same drivers and settings?
> Xen does not disable O_DIRECT.
> 
> Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Ian,

Thanks a lot for your prompt reply.

I''m doing physcial drive performance testing for SAS drives. All the
SAS
drives under test are indeed raw drives (no filesystems on it). Each SAS 
drive is assigned maximum 32 outstanding I/Os. Linux, Xen and benchmark tool 
IOMeter are installed on a separate SATA drive. I don''t know how disk
zoning
could be related to this issue.

Only for sequential write of small packets (512B and 1KB), Xen domain0 
outperform Linux native. For sequential/random read (512B, 1K, 4K, 8K, 64K, 
128K and 256K), Linux native outperform Xen domain0(this is expected). For 
random write(4K and 8K), both Xen domain0 and Linux native have almost 
identical performance.

Liang

----- Original Message ----- 
From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
To: "Liang Yang" <multisyncfe991@hotmail.com>; 
<xen-devel@lists.xensource.com>
Cc: <ian.pratt@cl.cam.ac.uk>
Sent: Saturday, October 28, 2006 3:35 PM
Subject: RE: [Xen-devel] Does Xen hypervisor overwrite O_DIRECT setting of 
Linux2.6 kernel?


> Yes, all the testing configurations are the exactly same (both uses
Linux
> kernel 2.6.16.29) and I used Xen 3.0.3. I did several indepedent runs
for
> my testing and the results are consistent, e.g. the performance of
small
> packet
> sequential write under Xen Domain0 outperform Linux native by 10~20%.
>
> Some people told me Xen hypervisor adds addtional layer beyond Linux
I/O
> stack. So O_DIRECT under Xen hypervisor is still valid, but I/O
coalescing
> for small packets are enabled by Xen hypervisor. However, Linux native
> kernel would not do any I/O coalescing when using O_DIRECT.
For IO from dom0 there is no additional layer. There will be no
difference in the coalescing behaviour.

Are you sure you''re writing to the same part of the disk in both cases?
You know about disk zoning, right?


Ian
> I hope Xen gurus can give me more explanations about this.
>
> Thanks,
>
> Liang
>
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "Liang Yang" <multisyncfe991@hotmail.com>;
> <xen-devel@lists.xensource.com>
> Cc: <ian.pratt@cl.cam.ac.uk>
> Sent: Saturday, October 28, 2006 2:34 PM
> Subject: RE: [Xen-devel] Does Xen hypervisor overwrite O_DIRECT
setting of
> Linux2.6 kernel?
>
>
> > The performance data is collected on 8 SAS drives (used as physical
> drives)
> > and IOMeter is used as the benchmark tool. The latest IOMeter
version
> used
> > O_DIRECT. We know, Linux 2.6 kernel starts supporting O_DIRECT which
> makes
> > all I/O requests work around buffer cache. The good thing for
O_DIRECT
> is
> > it
> > reduces the CPU utilization and cache pollution. The bad thing is
> O_DIRECT
> > not only forces all I/O requests become synchronous and no I/O
> coalescing
> > will happen. Thus sequential write of small packets will be impacted
> most.
> > For Xen, however, I believe Xen hypervisor overwrites this O_DIRECT
> setting
> > and maybe it favors better performance over CPU and FSB utilization.
> Thus
> > Xen domain0 can have better write performance than Linux native.
>
> Are you sure you''re comparing identical native and dom0 kernel
versions?
> Same drivers and settings?
> Xen does not disable O_DIRECT.
>
> Ian


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Ian,

Thanks a lot for your prompt reply.

I''m doing physical drive performance testing for SAS drives. All the
SAS
drives under test are indeed raw drives (no filesystems on it). Each SAS
drive is assigned maximum 32 outstanding I/Os. Linux, Xen and benchmark tool
IOMeter are installed on a separate SATA drive. I don''t know how disk
zoning
could be related to this issue.

Only for sequential write of small packets (512B and 1KB), Xen domain0
outperform Linux native. For sequential/random read (512B, 1K, 4K, 8K, 64K,
128K and 256K), Linux native outperform Xen domain0(this is expected). For
random write(4K and 8K), both Xen domain0 and Linux native have almost
identical performance.

Liang

----- Original Message ----- 
From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
To: "Liang Yang" <multisyncfe991@hotmail.com>; 
<xen-devel@lists.xensource.com>
Cc: <ian.pratt@cl.cam.ac.uk>
Sent: Saturday, October 28, 2006 3:35 PM
Subject: RE: [Xen-devel] Does Xen hypervisor overwrite O_DIRECT setting of 
Linux2.6 kernel?


> Yes, all the testing configurations are the exactly same (both uses
Linux
> kernel 2.6.16.29) and I used Xen 3.0.3. I did several indepedent runs
for
> my testing and the results are consistent, e.g. the performance of
small
> packet
> sequential write under Xen Domain0 outperform Linux native by 10~20%.
>
> Some people told me Xen hypervisor adds addtional layer beyond Linux
I/O
> stack. So O_DIRECT under Xen hypervisor is still valid, but I/O
coalescing
> for small packets are enabled by Xen hypervisor. However, Linux native
> kernel would not do any I/O coalescing when using O_DIRECT.
For IO from dom0 there is no additional layer. There will be no
difference in the coalescing behaviour.

Are you sure you''re writing to the same part of the disk in both cases?
You know about disk zoning, right?


Ian
> I hope Xen gurus can give me more explanations about this.
>
> Thanks,
>
> Liang
>
> ----- Original Message -----
> From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
> To: "Liang Yang" <multisyncfe991@hotmail.com>;
> <xen-devel@lists.xensource.com>
> Cc: <ian.pratt@cl.cam.ac.uk>
> Sent: Saturday, October 28, 2006 2:34 PM
> Subject: RE: [Xen-devel] Does Xen hypervisor overwrite O_DIRECT
setting of
> Linux2.6 kernel?
>
>
> > The performance data is collected on 8 SAS drives (used as physical
> drives)
> > and IOMeter is used as the benchmark tool. The latest IOMeter
version
> used
> > O_DIRECT. We know, Linux 2.6 kernel starts supporting O_DIRECT which
> makes
> > all I/O requests work around buffer cache. The good thing for
O_DIRECT
> is
> > it
> > reduces the CPU utilization and cache pollution. The bad thing is
> O_DIRECT
> > not only forces all I/O requests become synchronous and no I/O
> coalescing
> > will happen. Thus sequential write of small packets will be impacted
> most.
> > For Xen, however, I believe Xen hypervisor overwrites this O_DIRECT
> setting
> > and maybe it favors better performance over CPU and FSB utilization.
> Thus
> > Xen domain0 can have better write performance than Linux native.
>
> Are you sure you''re comparing identical native and dom0 kernel
versions?
> Same drivers and settings?
> Xen does not disable O_DIRECT.
>
> Ian


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel