Xen users - Jan 2009 - xen 3.3 -> tap:aio performance over nfs

I''ve been getting disappointing performance for file-backed disks
mounted using tap:aio over an NFS mount.
For example, bonnie++ reports 3 MB/s for block writes in the domU to
that
disk, but in dom0, bonnie++ reports 38 MB/s for that same mount. The
same domU can get 40 MB/s to the same filer using a phy: block device
over iSCSI .

It seems that 
1. tap:aio causes synchronous writes to the nfs mount (6 writes, wait for ack,
write 6 more etc) as opposed to NFSv3 async write
2. Each synchronous write writes very small blocks (4K bytes vs max of 32K
bytes)

Is there any configuration option to tweak this behavior to get
better-performing async writes to the filer?

Thanks
--
CV

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> I''ve been getting disappointing performance for file-backed disks
mounted
> using tap:aio over an NFS mount.
> For example, bonnie++ reports 3 MB/s for block writes in the domU to
> that
> disk, but in dom0, bonnie++ reports 38 MB/s for that same mount. The
> same domU can get 40 MB/s to the same filer using a phy: block device
> over iSCSI .
> 
> It seems that
> 1. tap:aio causes synchronous writes to the nfs mount (6 writes, wait
for
> ack, write 6 more etc) as opposed to NFSv3 async write
> 2. Each synchronous write writes very small blocks (4K bytes vs max of
32K
> bytes)
> 
> Is there any configuration option to tweak this behavior to get
better-
> performing async writes to the filer?
> 
I am assuming here that you have two physical servers, one with your
file-backed disk images on it, and a second which runs Xen and accesses
those files over NFS. If I got that the wrong way around then ignore the
rest of this email :)

I think that using tap:aio over NFS is not the correct solution to the
problem. I can think of a whole load of ways it could go wrong...

You might like to try file: but you would be sacrificing performance for
integrity.

But I think the correct way to do it would be to use iSCSI or AoE. Using
vblade you could set up AoE very very quickly and do some testing,
assuming that you are on the same broadcast segment (AoE can''t cross a
router).

James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

C V wrote:
> It seems that 
> 1. tap:aio causes synchronous writes to the nfs mount (6 writes, wait for
ack, write 6 more etc) as opposed to NFSv3 async write
> 2. Each synchronous write writes very small blocks (4K bytes vs max of 32K
bytes)
Then increase it ;)
> Is there any configuration option to tweak this behavior to get
better-performing async writes to the filer?
This is my big comparison table:

http://xen.bot.nu/benchmarks/virtueel.html


Stefan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I am mounting file-based disk images (tap:aio) ) for VM from a nfs server
(SATA, RAID 5) too. The link to the nfs server is a 1 G ethernet.
The read and write performance is also bad, all around 10M/s. The write size
for the nfs client is set to 8K (I tested with different sizes, like 16K and
32K, similar performance).
Is there any way to optimize the disk-based VM performance? Currently there
are 10+ VMs mounted to the nfs server and they include webservers, OLTP, and
Java application servers.

Thanks.

On Fri, Jan 30, 2009 at 9:04 PM, C V <rayvittal-lists@yahoo.com> wrote:
> I''ve been getting disappointing performance for file-backed disks
mounted
> using tap:aio over an NFS mount.
> For example, bonnie++ reports 3 MB/s for block writes in the domU to
> that
> disk, but in dom0, bonnie++ reports 38 MB/s for that same mount. The
> same domU can get 40 MB/s to the same filer using a phy: block device
> over iSCSI .
>
> It seems that
> 1. tap:aio causes synchronous writes to the nfs mount (6 writes, wait for
> ack, write 6 more etc) as opposed to NFSv3 async write
> 2. Each synchronous write writes very small blocks (4K bytes vs max of 32K
> bytes)
>
> Is there any configuration option to tweak this behavior to get
> better-performing async writes to the filer?
>
> Thanks
> --
> CV
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> I am mounting file-based disk images (tap:aio) ) for VM from a nfs
server
> (SATA, RAID 5) too. The link to the nfs server is a 1 G ethernet.
> The read and write performance is also bad, all around 10M/s. The
write
> size for the nfs client is set to 8K (I tested with different sizes,
like
> 16K and 32K, similar performance).
> Is there any way to optimize the disk-based VM performance? Currently
> there are 10+ VMs mounted to the nfs server and they include
webservers,
> OLTP, and Java application servers.
> 
To elaborate on my previous reply to the parent poster, do it a
different way. AoE and iSCSI are both protocols which are designed to
share a block device (or a file acting as a block device) between two
servers. It is designed with the sorts of things a server is going to
expect of a block device.

NFS is a filesystem, it is designed for exporting files, and takes care
of things like file locking and concurrency. It isn''t designed to do
what you want to do with it, and so probably isn''t going to perform
well.

AoE at least is really really easy to set up. Run vblade on your file
server to export the disk image, and use the aoe tools on your xen
server, and use tap:aio on the aoe disk device. If performance improves
then maybe look at iSCSI too, which is harder to set up but may give you
better performance still.

James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

When you say "increase it", are you referring to the nfs block size
(wsize )? It is set to 32K.
The 4k numbers are the actual blocks sent (despite the wsize setting) (I got
that from from tcpdump)

Your table is a little cryptic, but the closest to my config is I''m
guessing is the Netapp-NFS-IMG where you get 83M.
or Solaris-NFS-IMG where you get 13M.



----- Original Message ----
From: Stefan de Konink <stefan@konink.de>
To: C V <rayvittal-lists@yahoo.com>
Cc: xen-users@lists.xensource.com
Sent: Friday, January 30, 2009 6:48:27 PM
Subject: Re: [Xen-users] xen 3.3 -> tap:aio performance over nfs

C V wrote:
> It seems that 1. tap:aio causes synchronous writes to the nfs mount (6
writes, wait for ack, write 6 more etc) as opposed to NFSv3 async write
> 2. Each synchronous write writes very small blocks (4K bytes vs max of 32K
bytes)
Then increase it ;)
> Is there any configuration option to tweak this behavior to get
better-performing async writes to the filer?
This is my big comparison table:

http://xen.bot.nu/benchmarks/virtueel.html


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users