CentOS - Jun 2010 - Increasing NFS Performance

Greetings all-

I have a CentOS 5 (Final) system that is serving up content to several other
hosts via NFS. The amount of data transferred is rather small as most of the
files are under 100kb and each export has maybe 100 files that are accessed
regularly. I'm finding that as I add more hosts accessing the NFS server,
the performance seems to be getting poorer. There are obvious delays when doing
simple 'ls' on an NFS mounted directory.

The mounts are all done on the local LAN (100mbit FDX), all on the same
broadcast domain, no routing. One of the clients is a physical CentOS 5 node,
the rest (5-6 total) are OpenVZ containers also running CentOS 5. The NFS server
is a Dell PE2950 loaded with 7.2K SATA drives. While the disks aren't
exactly performance grade, the bottleneck does not seem to be the disks as
access on the NFS server itself is 'normal'.

Here are the items I've found so far that are told to increase performance.
Since this is a production system, I have yet to try these:

1. Increase the number of instances of NFS running. (As found in
/etc/sysconfig/nfs)
2. Try sync vs async behavior in mount parameters on clients
3. rmem_default and wmem_default parameters 
4. rsize and wsize parameters (Dependent on MTU. Currently, mine is default at
1500)

These are the items I'm planning to try but before I dive in (especially
during a late night maintenance period...), I was hoping the list members
brighter than me could give some comments on the above items and/or suggest some
other things I might try.

Thank you!

--Tim

On Mon, Jun 21, 2010 at 06:08:05PM -0500, Tim Nelson
wrote:
> Greetings all-
> 
> I have a CentOS 5 (Final) system that is serving up content to
> several other hosts via NFS. The amount of data transferred is rather
> small as most of the files are under 100kb and each export has maybe
> 100 files that are accessed regularly. I'm finding that as I add more
> hosts accessing the NFS server, the performance seems to be getting
> poorer. There are obvious delays when doing simple 'ls' on an NFS
> mounted directory.
> 
> The mounts are all done on the local LAN (100mbit FDX), all on the
> same broadcast domain, no routing. One of the clients is a physical
> CentOS 5 node, the rest (5-6 total) are OpenVZ containers also
> running CentOS 5. The NFS server is a Dell PE2950 loaded with 7.2K
> SATA drives. While the disks aren't exactly performance grade, the
> bottleneck does not seem to be the disks as access on the NFS server
> itself is 'normal'.
> 
> Here are the items I've found so far that are told to increase
> performance. Since this is a production system, I have yet to try
> these:
> 
> 1. Increase the number of instances of NFS running. (As found in
/etc/sysconfig/nfs)
> 2. Try sync vs async behavior in mount parameters on clients
> 3. rmem_default and wmem_default parameters 
> 4. rsize and wsize parameters (Dependent on MTU. Currently, mine is default
at 1500)
> 
> These are the items I'm planning to try but before I dive in
> (especially during a late night maintenance period...), I was hoping
> the list members brighter than me could give some comments on the
> above items and/or suggest some other things I might try.
> 
> Thank you!
Depends on the type of client access, but wouldn't surprise me if async
gives you a big performance boost.

"Right" way to fix that is to configure your storage to cache sync
writes more quickly (eg nice RAID card with lots of cache).  async mode
may speed things up, but you open yourself up to potential corruption
in a power loss situation or confused clients needing to be rebooted.

Ray

What kind of local(on the box themselves) read/write speeds are you
observing.
What is the specs on your server: memory, # cpus, cpu speeds, any raid?
What kind of remote access read/write are your observing? Try doing a dd
or something to get a ball park idea.

To first order in NFS performance tuning:
1. rsize,wsize are the most important. Generally for linux this is 32k.
2. The number of nfsd threads can't help also.

but this is after you have obtain some numbers for your performce(like
also "top" and/or iostat when your server seems real slow  and when
your
performance is good...you have to quantify your observations to really get
meaningful help.

~Lawrence
> Greetings all-
>
> I have a CentOS 5 (Final) system that is serving up content to several
> other hosts via NFS. The amount of data transferred is rather small as
> most of the files are under 100kb and each export has maybe 100 files that
> are accessed regularly. I'm finding that as I add more hosts accessing
the
> NFS server, the performance seems to be getting poorer. There are obvious
> delays when doing simple 'ls' on an NFS mounted directory.
>
> The mounts are all done on the local LAN (100mbit FDX), all on the same
> broadcast domain, no routing. One of the clients is a physical CentOS 5
> node, the rest (5-6 total) are OpenVZ containers also running CentOS 5.
> The NFS server is a Dell PE2950 loaded with 7.2K SATA drives. While the
> disks aren't exactly performance grade, the bottleneck does not seem to
be
> the disks as access on the NFS server itself is 'normal'.
>
> Here are the items I've found so far that are told to increase
> performance. Since this is a production system, I have yet to try these:
>
> 1. Increase the number of instances of NFS running. (As found in
> /etc/sysconfig/nfs)
> 2. Try sync vs async behavior in mount parameters on clients
> 3. rmem_default and wmem_default parameters
> 4. rsize and wsize parameters (Dependent on MTU. Currently, mine is
> default at 1500)
>
> These are the items I'm planning to try but before I dive in
(especially
> during a late night maintenance period...), I was hoping the list members
> brighter than me could give some comments on the above items and/or
> suggest some other things I might try.
>
> Thank you!
>
> --Tim
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>

Tim Nelson writes:
> Greetings all-
> 
> I have a CentOS 5 (Final) system that is serving up content to several
other
> hosts via NFS. The amount of data transferred is rather small as most of
the
> files are under 100kb and each export has maybe 100 files that are accessed
> regularly. I'm finding that as I add more hosts accessing the NFS
server, the
> performance seems to be getting poorer. There are obvious delays when doing
> simple 'ls' on an NFS mounted directory.
 
 So, this is CentOS 5.0 or 5.1. You should upgrade to at least CentOS 5.2. The
 5.2 kernel fixes serious performance problems with NFS.

 I think this is the related kernel changelog. Our docs show that we had
 machines running 5.1 with kernel 2.6.18-53.1.21.el5.bz321111 to work around
 the problem.

* Tue Feb 05 2008 Don Zickus <dzickus at redhat.com> [2.6.18-78.el5]
...
- [nfs] reduce number of wire RPC ops, increase perf (Peter Staubach ) [321111]



---------------------------------------------------------------
This message and any attachments may contain Cypress (or its
subsidiaries) confidential information. If it has been received
in error, please advise the sender and immediately delete this
message.
---------------------------------------------------------------