Gluster users - Jun 2010 - gluster client hang when using iozone

I just upgraded from 2.x to 3.0.4, as I experienced some random client 
hangs with 2.x.

Unfortunately, the issue is not solved for me - I can reliably reproduce 
the hang with such iozone command line (it usually hangs every 2-3 times):

# iozone -R -l 5 -u 5 -r 4k -s 100m


When I look at the traffic, I can see it still flows between the client 
and gluster servers - but at a very low speed, around 10 kB/s (with 1 
Gbit link).

Any access to the gluster filesystem "hangs".

Killing "glusterfs" process and mounting the fs again makes the thing 
recover (until at least I try to start iozone 2-3 more times).

Any issue what can be wrong here? Neither the client nor the servers 
produce anything in logs when it happens (I didn't wait for more than 10 
minutes though).



Gluster client uses this config file:

volume remote1
   type protocol/client
   option transport-type tcp
   option remote-host gluster1
   option remote-subvolume brick
   option ping-timeout 5
end-volume

volume remote2
   type protocol/client
   option transport-type tcp
   option remote-host gluster2
   option remote-subvolume brick
   option ping-timeout 5
end-volume

volume replicate1
   type cluster/replicate
   subvolumes remote1 remote2
end-volume

volume distribute
   type cluster/distribute
   subvolumes replicate1
end-volume

volume writebehind
   type performance/write-behind
   option cache-size 32MB
   subvolumes distribute
end-volume

volume cache
   type performance/io-cache
   option cache-size 512MB
   subvolumes writebehind
end-volume

-- 
Tomasz Chmielewski
http://wpkg.org

On 06/07/2010 01:50 PM, Tomasz Chmielewski wrote:
> Unfortunately, the issue is not solved for me - I can reliably reproduce
> the hang with such iozone command line (it usually hangs every 2-3 times):
>
> # iozone -R -l 5 -u 5 -r 4k -s 100m
>
>
> When I look at the traffic, I can see it still flows between the client
> and gluster servers - but at a very low speed, around 10 kB/s (with 1
> Gbit link).
>
> Any access to the gluster filesystem "hangs".
>
> Killing "glusterfs" process and mounting the fs again makes the
thing
> recover (until at least I try to start iozone 2-3 more times).
>
> Any issue what can be wrong here? Neither the client nor the servers
> produce anything in logs when it happens (I didn't wait for more than
10
> minutes though).
What distro ?  What kernel version ?  Hardware specs ?

As a counter-point, a few months ago i evaluated glfs 3.x for one of our 
internal systems, and ran iozone against a simple three-node client-side 
replication setup (such as the one described by your configs) with no 
problems such as those your described.

-- 
Daniel Maher <dma+gluster AT witbe DOT net>