Xen devel - Apr 2012 - Benchmark Xen writes with sync - Xen ignores fsync, O_SYNC?

Hi there Xen community,

I am trying to benchmark and compare I/O in Xen/domU to native performance.

In order to do this I started trying to benchmark writes so as to avoid
caching effects that surely turn up when performing reads due to the page
cache et al.
However, I have quickly run into a problem: Xen domU reports that a 128MB
file is written at close to 300MB/s, while the disk''s performance peaks
at
about 80MB/s (I observed this on a dom0 and on a bare-metal kernel with no
hypervisor).
Please note that I fsync() after all writes in hopes to avoid the effect of
write buffers. I have tried with O_SYNC as well, observing a similar
outcome.
I can confirm this writing a simple program, and verified exactly same
results running bonnie++ with the fsync() option turned on.

I am surprised to see writes reaching a throughput as high as 300MB/s, as
the disk surely isn''t physically capable of reaching that bandwidth,
meaning that writes are not being really synced to disk.

Is this a bug in Xen, or is there a way to make Xen not ignore fsync,
fdatasync, O_SYNC, etc..?
How would I proceed to measure and compare real read/write speeds on a Xen
domU ?

My disk drivers are specified with "file:/path/to/image.img,xvda,1,w"
(I
could not get the tapdisk driver to work properly, I tried with vanilla 3.2
and 3.0.0 ubuntu kernels)
Xen is version 4.1.1 and is running Oneiric domUs (kernel 3.0.0)
For the dom0 I have a 3.2 vanilla kernel and a ubuntu (oneiric) 3.0.0 kernel

Thank you in advance,

Pablo


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 6 April 2012 08:30, Pablo Llopis <pllopis@arcos.inf.uc3m.es>
wrote:
> Hi there Xen community,
>
> I am trying to benchmark and compare I/O in Xen/domU to native performance.
>
> In order to do this I started trying to benchmark writes so as to avoid
> caching effects that surely turn up when performing reads due to the page
> cache et al.
> However, I have quickly run into a problem: Xen domU reports that a 128MB
> file is written at close to 300MB/s, while the disk''s performance
peaks at
> about 80MB/s (I observed this on a dom0 and on a bare-metal kernel with no
> hypervisor).
> Please note that I fsync() after all writes in hopes to avoid the effect of
> write buffers. I have tried with O_SYNC as well, observing a similar
> outcome.
> I can confirm this writing a simple program, and verified exactly same
> results running bonnie++ with the fsync() option turned on.
>
> I am surprised to see writes reaching a throughput as high as 300MB/s, as
> the disk surely isn''t physically capable of reaching that
bandwidth, meaning
> that writes are not being really synced to disk.
>
> Is this a bug in Xen, or is there a way to make Xen not ignore fsync,
> fdatasync, O_SYNC, etc..?
> How would I proceed to measure and compare real read/write speeds on a Xen
> domU ?
>
> My disk drivers are specified with
"file:/path/to/image.img,xvda,1,w" (I
> could not get the tapdisk driver to work properly, I tried with vanilla 3.2
> and 3.0.0 ubuntu kernels)
To measure real disk speed I recommend using the phy:/ handler with a
block device.
You can use a partition, an LVM logical volume or an entire disk.
Use of files on filesytems for domain block devices is discouraged and
should be avoided at all costs.

For testing IO throughput actively I wouldn''t use bonnie++ as it is a
filesystem level tool and not overly accurate at that.
Use the Flexible IO Tester (aka fio) by Jens Axobe.
You can install it on Ubuntu with:
apt-get install fio

I would also recommend using the libaio engine with fio if you are
using an SSD and need to drive the queue depth up to get a better idea
of the IOPs the device can deliver.
You will need to install libaio with:
apt-get install libaio
> Xen is version 4.1.1 and is running Oneiric domUs (kernel 3.0.0)
> For the dom0 I have a 3.2 vanilla kernel and a ubuntu (oneiric) 3.0.0
kernel
>
> Thank you in advance,
>
> Pablo
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
Joseph.

-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846

thank you a lot for your advice, I will try to repeat the experiments with
the phy driver (and that will hopefully sync properly), and check out fio
for benchmarking :)

On Fri, Apr 6, 2012 at 1:04 AM, Joseph Glanville <
joseph.glanville@orionvm.com.au> wrote:
> On 6 April 2012 08:30, Pablo Llopis <pllopis@arcos.inf.uc3m.es>
wrote:
> > Hi there Xen community,
> >
> > I am trying to benchmark and compare I/O in Xen/domU to native
> performance.
> >
> > In order to do this I started trying to benchmark writes so as to
avoid
> > caching effects that surely turn up when performing reads due to the
page
> > cache et al.
> > However, I have quickly run into a problem: Xen domU reports that a
128MB
> > file is written at close to 300MB/s, while the disk''s
performance peaks
> at
> > about 80MB/s (I observed this on a dom0 and on a bare-metal kernel
with
> no
> > hypervisor).
> > Please note that I fsync() after all writes in hopes to avoid the
effect
> of
> > write buffers. I have tried with O_SYNC as well, observing a similar
> > outcome.
> > I can confirm this writing a simple program, and verified exactly same
> > results running bonnie++ with the fsync() option turned on.
> >
> > I am surprised to see writes reaching a throughput as high as 300MB/s,
as
> > the disk surely isn''t physically capable of reaching that
bandwidth,
> meaning
> > that writes are not being really synced to disk.
> >
> > Is this a bug in Xen, or is there a way to make Xen not ignore fsync,
> > fdatasync, O_SYNC, etc..?
> > How would I proceed to measure and compare real read/write speeds on a
> Xen
> > domU ?
> >
> > My disk drivers are specified with
"file:/path/to/image.img,xvda,1,w" (I
> > could not get the tapdisk driver to work properly, I tried with
vanilla
> 3.2
> > and 3.0.0 ubuntu kernels)
>
> To measure real disk speed I recommend using the phy:/ handler with a
> block device.
> You can use a partition, an LVM logical volume or an entire disk.
> Use of files on filesytems for domain block devices is discouraged and
> should be avoided at all costs.
>
> For testing IO throughput actively I wouldn''t use bonnie++ as it
is a
> filesystem level tool and not overly accurate at that.
> Use the Flexible IO Tester (aka fio) by Jens Axobe.
> You can install it on Ubuntu with:
> apt-get install fio
>
> I would also recommend using the libaio engine with fio if you are
> using an SSD and need to drive the queue depth up to get a better idea
> of the IOPs the device can deliver.
> You will need to install libaio with:
> apt-get install libaio
>
> > Xen is version 4.1.1 and is running Oneiric domUs (kernel 3.0.0)
> > For the dom0 I have a 3.2 vanilla kernel and a ubuntu (oneiric) 3.0.0
> kernel
> >
> > Thank you in advance,
> >
> > Pablo
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> >
>
> Joseph.
>
> --
> Founder | Director | VP Research
> Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
> 99 52 | Mobile: 0428 754 846
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, 5 Apr 2012, Pablo Llopis wrote:
> Hi there Xen community,
> I am trying to benchmark and compare I/O in Xen/domU to native performance.
> 
> In order to do this I started trying to benchmark writes so as to avoid
caching effects that surely turn up when
> performing reads due to the page cache et al.
> However, I have quickly run into a problem: Xen domU reports that a 128MB
file is written at close to 300MB/s, while the
> disk''s performance peaks at about 80MB/s (I observed this on a
dom0 and on a bare-metal kernel with no hypervisor).
> Please note that I fsync() after all writes in hopes to avoid the effect of
write buffers. I have tried with O_SYNC as
> well, observing a similar outcome.
> I can confirm this writing a simple program, and verified exactly same
results running bonnie++ with the fsync() option
> turned on.
> 
> I am surprised to see writes reaching a throughput as high as 300MB/s, as
the disk surely isn''t physically capable of
> reaching that bandwidth, meaning that writes are not being really synced to
disk. 
> 
> Is this a bug in Xen, or is there a way to make Xen not ignore fsync,
fdatasync, O_SYNC, etc..? 
> How would I proceed to measure and compare real read/write speeds on a Xen
domU ?
> 
> My disk drivers are specified with
"file:/path/to/image.img,xvda,1,w" (I could not get the tapdisk driver
to work
> properly, I tried with vanilla 3.2 and 3.0.0 ubuntu kernels)
> Xen is version 4.1.1 and is running Oneiric domUs (kernel 3.0.0)
> For the dom0 I have a 3.2 vanilla kernel and a ubuntu (oneiric) 3.0.0
kernel
Are you using xend/xm to create the guest? In that case you are using
a loop device with blkback behind the scenes, that might go through the
disk cache.
--8323329-503233747-1334071560=:15151
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

--8323329-503233747-1334071560=:15151--

Yes, I am using "xm create" to instance the guest (xen-tools to
actually
create the guest image). And yes, I am pretty sure I am using blkback in
combination with a loopback device (I''m using file:/ after all).
But I thought the xend backend systems would somehow, at least, force a
fsync() for that file. It seems that when using file:/ sync operations
don''t really ensure that contents are being synced to disk, but only to
the
loopback-mounted file, which is probably still affected by the pagecache
(but that''s only a guess).

For the record, I tried with phy:/ and a LVM volume and everything works as
expected (i.e. sync and fsync operations sync data to disk).

Thanks!

On Tue, Apr 10, 2012 at 5:25 PM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 5 Apr 2012, Pablo Llopis wrote:
> > Hi there Xen community,
> > I am trying to benchmark and compare I/O in Xen/domU to native
> performance.
> >
> > In order to do this I started trying to benchmark writes so as to
avoid
> caching effects that surely turn up when
> > performing reads due to the page cache et al.
> > However, I have quickly run into a problem: Xen domU reports that a
> 128MB file is written at close to 300MB/s, while the
> > disk''s performance peaks at about 80MB/s (I observed this on
a dom0 and
> on a bare-metal kernel with no hypervisor).
> > Please note that I fsync() after all writes in hopes to avoid the
effect
> of write buffers. I have tried with O_SYNC as
> > well, observing a similar outcome.
> > I can confirm this writing a simple program, and verified exactly same
> results running bonnie++ with the fsync() option
> > turned on.
> >
> > I am surprised to see writes reaching a throughput as high as 300MB/s,
> as the disk surely isn''t physically capable of
> > reaching that bandwidth, meaning that writes are not being really
synced
> to disk.
> >
> > Is this a bug in Xen, or is there a way to make Xen not ignore fsync,
> fdatasync, O_SYNC, etc..?
> > How would I proceed to measure and compare real read/write speeds on a
> Xen domU ?
> >
> > My disk drivers are specified with
"file:/path/to/image.img,xvda,1,w" (I
> could not get the tapdisk driver to work
> > properly, I tried with vanilla 3.2 and 3.0.0 ubuntu kernels)
> > Xen is version 4.1.1 and is running Oneiric domUs (kernel 3.0.0)
> > For the dom0 I have a 3.2 vanilla kernel and a ubuntu (oneiric) 3.0.0
> kernel
>
> Are you using xend/xm to create the guest? In that case you are using
> a loop device with blkback behind the scenes, that might go through the
> disk cache.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, 11 Apr 2012, Pablo Llopis wrote:
> Yes, I am using "xm create" to instance the guest (xen-tools to
actually create the guest image). And yes, I am pretty
> sure I am using blkback in combination with a loopback device (I''m
using file:/ after all).  But I thought the xend
> backend systems would somehow, at least, force a fsync() for that file. It
seems that when using file:/ sync operations
> don''t really ensure that contents are being synced to disk, but
only to the loopback-mounted file, which is probably still
> affected by the pagecache (but that''s only a guess). 
> 
> For the record, I tried with phy:/ and a LVM volume and everything works as
expected (i.e. sync and fsync operations sync
> data to disk). 
That would be the best solution with Xen 4.1 and it is also the fastest.
--8323329-112930903-1334143015=:15151
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

--8323329-112930903-1334143015=:15151--