Xen users - Feb 2006 - Xen, NFS performance, rsize, wsize and MTU

We have a Xen 3.0 / Linux kernel 2.6.15 machine with the domU''s
configured for shared /home directories.  One of the domU''s is an NFS
server, exporting /home, and the other domU''s all mount this.

It all functions fine, but it''s pretty slow.

I tried untarring a recent Linux kernel tarball.  On the NFS server
domU this takes about 25 seconds.  I didn''t wait long enough to find
out how long it was going to take on the NFS client domU''s - I killed
it after 15 minutes.

I was going to try compiling Linux kernels on the various domU''s to
test for stability, but that''s excruciatingly slow too.

Following the instructions here:

http://nfs.sourceforge.net/nfs-howto/performance.html

I''ve been playing with rsize, wsize and sync mount options.  As far as
I can see, any rsize/wsize other than the default is slower than the
default choice (I couldn''t find out what it is, but it _seems_ to be
1K/1K).  The ''sync'' mount option has further catastrophic
effects on
write performance.

I also notice that the MTU chosen on the eth0 virtual interfaces is
1500 bytes.  This seems to make no sense, because the purpose behind
the MTU is to do with the physical characteristics of ethernet itself,
but here we''ve got an entirely virtual bridge setup.  Surely I should
choose an MTU as large as possible (ie. 64K)?

Before I start to look into changing MTU and further modifying
rsize/wsize, has anyone got any quick tips on how to make my NFS
config faster?

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Feb 17, 2006 at 02:41:12PM +0000, Richard Jones
wrote:
> We have a Xen 3.0 / Linux kernel 2.6.15 machine with the domU''s
> configured for shared /home directories.  One of the domU''s is an
NFS
> server, exporting /home, and the other domU''s all mount this.
> It all functions fine, but it''s pretty slow.
> I tried untarring a recent Linux kernel tarball.  On the NFS server
> domU this takes about 25 seconds.  I didn''t wait long enough to
find
> out how long it was going to take on the NFS client domU''s - I
killed
> it after 15 minutes.
How many CPUs does this server have? If you''ve got a single CPU (sans
hyperthreading) machine then, there''ll be the overhead of at least ~4
context
switches per request to deal with (the domU sends the initial request packet,
which
is bridged / routed via dom0, which sends it to the NFS server domU,
and, then back agan). 

However, if you''ve a multiprocessor / hyperthreaded machine, then
I''d
imagine performance would be significantly better, because then you''d
be
able to, for example, have dom0 pinned to one CPU, and leave the domUs
to compete for the other CPU.

Unfortunately, I don''t really have any numbers to back this up, as
I''ve
not had a chance to play with Xen on anything but a single-threaded
machine. I''m sure others would be also able to advise, though.

Cheers.
-- 
Ceri Storey <cez@necrofish.org.uk>
''What I really want is "apt-get smite"'' 
    --Rob Partington <http://rjp.frottage.org>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Feb 17, 2006 at 08:34:00PM +0000, Ceri Storey
wrote:
> On Fri, Feb 17, 2006 at 02:41:12PM +0000, Richard Jones wrote:
> > We have a Xen 3.0 / Linux kernel 2.6.15 machine with the
domU''s
> > configured for shared /home directories.  One of the domU''s
is an NFS
> > server, exporting /home, and the other domU''s all mount this.
> 
> > It all functions fine, but it''s pretty slow.
> 
> > I tried untarring a recent Linux kernel tarball.  On the NFS server
> > domU this takes about 25 seconds.  I didn''t wait long enough
to find
> > out how long it was going to take on the NFS client domU''s -
I killed
> > it after 15 minutes.
> 
> How many CPUs does this server have? If you''ve got a single CPU
(sans
> hyperthreading) machine then, there''ll be the overhead of at least
~4 context
> switches per request to deal with (the domU sends the initial request
packet, which
> is bridged / routed via dom0, which sends it to the NFS server domU,
> and, then back agan). 
You''re right and it is a single processor machine.  I''m not
expecting
blazing performance (we far prefer reliability / predictability).
It''s just really is quite slow at the moment and I''m sure that
it''s
down to some sort of configuration mistake.

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 18/02/06, Richard Jones <rich@annexia.org>
wrote:
>
>
> You''re right and it is a single processor machine.  I''m
not expecting
> blazing performance (we far prefer reliability / predictability).
> It''s just really is quite slow at the moment and I''m sure
that it''s
> down to some sort of configuration mistake.

I thought the same thing, but two processors didn''t seem to solve this
for
me when I was running something similar with 2.0. Moving the NFS to host0
seemed to be the solution. I didn''t try running the NFS server with a 
dom0
kernel. That might be worth trying.

http://stateless.geek.nz/2005/08/29/xen-disk-performance/


--
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

Am Freitag, 17. Februar 2006 15:41 schrieb Richard
Jones:
> We have a Xen 3.0 / Linux kernel 2.6.15 machine with the domU''s
> configured for shared /home directories.  One of the domU''s is an
NFS
> server, exporting /home, and the other domU''s all mount this.
> It all functions fine, but it''s pretty slow.
well, perhaps because all the network traffic has to go from domU (nfs-server) 
over dom0 (network bridge or routing) to domU (nfs-client).

What says xm top? How much is dom0 loaded in this scenario?


-- 
greetings

eMHa


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 19/02/06, Markus Hochholdinger <Markus@hochholdinger.net>
wrote:
> well, perhaps because all the network traffic has to go from domU
(nfs-server)
> over dom0 (network bridge or routing) to domU (nfs-client).
No, because when a domU NFS server provides NFS to another physical
machine, usually there are no problems.  Its maybe not quite as fast
as normal, but very workable.

--
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> http://stateless.geek.nz/2005/08/29/xen-disk-performance/
thanks for the insight Nicholas.

So, would you say that any file sharing (smb for example) might be
better on dom0?

Chris.

On 2/18/06, Nicholas Lee <emptysands@gmail.com>
wrote:
> On 18/02/06, Richard Jones <rich@annexia.org> wrote:
> >
> > You''re right and it is a single processor machine. 
I''m not expecting
> > blazing performance (we far prefer reliability / predictability).
> > It''s just really is quite slow at the moment and I''m
sure that it''s
> > down to some sort of configuration mistake.
>
> I thought the same thing, but two processors didn''t seem to solve
this for
> me when I was running something similar with 2.0. Moving the NFS to host0
> seemed to be the solution. I didn''t try running the NFS server
with a  dom0
> kernel. That might be worth trying.
>
> http://stateless.geek.nz/2005/08/29/xen-disk-performance/
>
>
> --
> Nicholas Lee
> http://stateless.geek.nz
> gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Before I start to look into changing MTU and further modifying
> rsize/wsize, has anyone got any quick tips on how to make my NFS
> config faster?
Rich,

I got a huge speedup by using NFS over TCP.
Unfortunately, I do not know why it''s so much faster...

Regards, Arie Kraai

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Feb 20, 2006 at 09:32:51AM +0100, Arie Kraai
wrote:
> I got a huge speedup by using NFS over TCP.
> Unfortunately, I do not know why it''s so much faster...
Thanks - NFS mounting over TCP made my kernel compile about 2.5 times
faster.

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users