CentOS - Apr 2010 - Performance problems with XFS on Centos 5.4

Hi!

During the last weeks I experienced some performance problems with a
large file-system on XFS basis. Sometimes for instance ls is
painfully. Immidiatly afterwards ls on the same directory is
immidiate. I used strace on this ls and found that during the first ls
the lstat-calls need approx 0.02s each while during the second ls the
are two orders of magnitude faster.

Googling around I stumbled upon some messages similar like this

http://www.opensubscriber.com/message/linux-xfs at oss.sgi.com/1355060.html

which have in common
a) they're from around 2006
b) they suggest to increase a mount-option ihashsize. This mount
   option is listed as deprecated in the current kernel-doc

So my question: does anyone have experience with that kind of
performance problem? Do you think it is a XFS problem or are there
some other tuning parameters in the kernel that could be modified for
instance via /proc?

The reason why I'm asking here is that it is a production file-system
so I would be very unpopular if I experiment too much (a couple of
reboots is OK ;) )

Bernhard

PS: the situation got worse during the last weeks when the file-system
increased in size, so the option that some kind of buffer now is too
small and I'm experiencing some kind of thrashing seems very likely to
me

On Apr 9, 2010, at 9:59 AM, Bernhard Gschaider <bgschaid_lists at ice- 
sf.at> wrote:
>
> Hi!
>
> During the last weeks I experienced some performance problems with a
> large file-system on XFS basis. Sometimes for instance ls is
> painfully. Immidiatly afterwards ls on the same directory is
> immidiate. I used strace on this ls and found that during the first ls
> the lstat-calls need approx 0.02s each while during the second ls the
> are two orders of magnitude faster.
>
> Googling around I stumbled upon some messages similar like this
>
> http://www.opensubscriber.com/message/linux-xfs at oss.sgi.com/1355060.html
>
> which have in common
> a) they're from around 2006
> b) they suggest to increase a mount-option ihashsize. This mount
>   option is listed as deprecated in the current kernel-doc
>
> So my question: does anyone have experience with that kind of
> performance problem? Do you think it is a XFS problem or are there
> some other tuning parameters in the kernel that could be modified for
> instance via /proc?
>
> The reason why I'm asking here is that it is a production file-system
> so I would be very unpopular if I experiment too much (a couple of
> reboots is OK ;) )
>
> Bernhard
>
> PS: the situation got worse during the last weeks when the file-system
> increased in size, so the option that some kind of buffer now is too
> small and I'm experiencing some kind of thrashing seems very likely to
> me
Are you defragging the file system regularly?

How much memory do you have in the system and how big is the file  
system?

What are the XFS parameters for the file system?

What is the storage setup?

Need the info.

-Ross