[Gluster-users] Unexpected behaviour during replication heal

Marco Agostini
2011-06-28T20:23:59Z

2011/6/28 Darren Austin <darren-lists at
widgit.com>:
>
> Also, when one of the servers disconnects, is it notmal that the client
"stalls" the write until the keepalive time expires and the online
servers notice one has vanished?
>
You can modify the parameter network.ping-timeout from 46sec to 5 or
10 second to reduce the "time stalls" of client.

> Finally, during my testing I encountered a replicable hard lock up of the
client... here's the situation:
> ?Server1 and Server2 in the cluster, sharing 'data-volume' (which
is /data on both servers).
> ?Client mounts server1:data-volume as /mnt.
> ?Client begins to write a large (1 or 2 GB) file to /mnt ?(I just used
random data).
> ?Server1 goes down part way through the write (I simulated this by iptables
-j DROP'ing everything from relevant IPs).
> ?Client "stalls" writes until the keepalive timeout, and then
continues to send data to Server2.
> ?Server1 comes back online shortly after the keepalive timeout - but BEFORE
the Client has written all the data toServer2.
> ?Server1 and Server2 reconnect and the writes on the Client completely
hang.
>

I have similar problem with a file that I'm using with KVM for storage
virtual disk

> The mounted directory on the client becomes completely in-accessible when
the two servers reconnect.
>
actualy is normal :-|

> I had to kill -9 the dd process doing the write (along with the glusterfs
process on the client) in order to release the mountpoint.
>
If you don't kill the process and wait that all node are syncronized
all the system should return ready.

To force a syncronization of all volume you can type these command on
the client:
find <gluster-mount> -noleaf -print0 | xargs --null stat >/dev/null

... and wait

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Triggering_Self-Heal_on_Replicate


Craig Carl said me, three days ago:
------------------------------------------------------
?that happens because Gluster's self heal is a blocking operation. We
are working on a non-blocking self heal, we are hoping to ship it in
early September.
------------------------------------------------------

You can verify that directly from your client log... you can read that:
[2011-06-28 13:28:17.484646] I
[client-lk.c:617:decrement_reopen_fd_count] 0-data-volume-client-0:
last fd open'd/lock-self-heal'd - notifying CHILD-UP

Marco