Gluster users - Sep 2009 - speeding up getdents()/stat() and/or general perf tuning...

Hi

I've been noticing some long delays on doing a simple `ls' in 
directories that haven't been recently accessed on a test glusterfs
system we've put together.  The system(s) consists of a 4 node 
DHT + AFR (x1) setup, running 2.0.6 all with 10GbE connectivity 
between the nodes (and no there is no network bottleneck here as
iperf proves that the throughput between the machines is ~9Gbps).

stat("usr/bin", {st_mode=S_IFDIR|0755, st_size=28672, ...}) = 0
<1.659084>
open("usr/bin", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
<0.001231>
fstat(3, {st_mode=S_IFDIR|0755, st_size=28672, ...}) = 0 <0.000007>
fcntl(3, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
<0.000005>
getdents(3, /* 75 entries */, 4096)     = 2256 <0.033347>
getdents(3, /* 75 entries */, 4096)     = 2280 <0.001858>
getdents(3, /* 41 entries */, 4096)     = 1256 <0.000961>
getdents(3, /* 73 entries */, 4096)     = 2216 <0.034120>
getdents(3, /* 73 entries */, 4096)     = 2296 <0.001393>
getdents(3, /* 11 entries */, 4096)     = 344 <0.000481>
getdents(3, /* 72 entries */, 4096)     = 2264 <0.028436>
mmap(NULL, 147456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f061c9b5000 <0.000005>
getdents(3, /* 74 entries */, 4096)     = 2304 <0.001711>
getdents(3, /* 42 entries */, 4096)     = 1344 <0.001304>
getdents(3, /* 74 entries */, 4096)     = 2200 <3.176572> <- SLOW
OPERATION
getdents(3, /* 74 entries */, 4096)     = 2288 <0.012123>
getdents(3, /* 10 entries */, 4096)     = 280 <0.000376>
getdents(3, /* 0 entries */, 4096)      = 0 <0.000243>
close(3)                                = 0 <0.000011>

The client configs are simple (stripped down for simplicity sakes):

brick 1 tcp 10.10.10.11
brick 2 tcp 10.10.10.12
...
brick 8 tcp 10.10.10.18
replicate1 brick1 brick2
replicate2 brick3 brick5
replicate3 brick4 brick7
replicate4 brick6 brick8
distribute replicate1 replicate2 replicate3 replicate4
io-threads thread-count 16
readahead page-count 16
io-cache cache-size 512MB
write-behind cache-size 1MB

The storage nodes are even simpler yet (trimmed):

posix background-unlink yes
posix-locks
io-threads thread-count 16
transport-type tcp

The client nodes are running an unpatched fuse 2.7.4 userland (debian 
lenny) and the default kernel 2.6.31.1 fuse module and I'm curious if 
the delay in the getdents/stat calls are because of fuse or something 
else config wise that I've managed to miss?

Anyone have any recommendations or thoughts as to how to improve the 
performance?

Thanks.

Mohan

On 09/29/2009 07:14 PM, mki-glusterfs at mozone.net
wrote:
> Hi
>
> I've been noticing some long delays on doing a simple `ls' in
> directories that haven't been recently accessed on a test glusterfs
> system we've put together.  The system(s) consists of a 4 node
> DHT + AFR (x1) setup, running 2.0.6 all with 10GbE connectivity
> between the nodes (and no there is no network bottleneck here as
> iperf proves that the throughput between the machines is ~9Gbps).
>    

I would suggest following advice in another thread:
>  
http://www.gluster.com/community/documentation/index.php/Translators/cluster/distribute
> >
> >  It seems to suggest that 'lookup-unhashed' says that the
default is 'on'.
> >
> >  Perhaps try turning it 'off'?
>    
> Wei,
>     There are two things we would like you to try. First is what Mark
> has just pointed, the 'option lookup-unhashed off' in distribute.
The
> second is 'option transport.socket.nodelay on' in each of your
> protocol/client_and_  protocol/server volumes. Do let us know what
> influence these changes have on your performance.
>
> Avati
>    
The TCP_NODELAY seems particularly relevant to me if many small requests 
are being issued in sequence as a /bin/ls is likely to do?

The lookup-unhashed might be relevant to stat() calls issued as a part 
of the /bin/ls process.

I've been hitting 'ls' problems myself on another system with NFS
and
AutoFS where we have a directory with many symlinks in it. The 'ls' does
a stat() on each of the symlinks, and in the case of auto-mount - it can 
take a while... :-)

There is a stat-prefetch module that I do not see documentation for. I 
wish there was more comments. A quick skim of it suggests that it 
*might* be designed to improve /bin/ls performance. That it's not 
documented may mean it is for 2.1 or later?

Cheers,
mark


-- 

Mark Mielke<mark at mielke.cc>