Gluster users - Mar 2016 - Question on the rename atomicity behavior in gluster

Hi, We had some questions with respect to expectations of atomicity of
rename in gluster.

To elaborate :

We have setup with two machines (lets call them M1 and M2) which
essentially access a file (F) on a gluster volume (mounted by M1 and M2)
A program does the following steps sequentially on each of the two machines
(M1 & M2) in an infinite loop
 1) Opens the file F in O_RDWR|O_EXCL mode, reads some data and closes (F)
 2)  Renames some other file F' => F

Periodically either M1 or M2 sees a "Stalefile handle error" when it
tries
to read the file (step (1)) after opening the file in O_RDWR|O_EXCL (the
open is successful)

The specific error reported the client volume logs
 (/var/log/glusterfs/mnt-repos-volume1.log)
[2016-03-21 16:53:17.897902] I [dht-rename.c:1344:dht_rename]
0-volume1-dht: renaming master.lock
(hash=volume1-replicate-0/cache=volume1-replicate-0) => master
(hash=volume1-replicate-0/cache=<nul>)
[2016-03-21 16:53:18.735090] W [client-rpc-fops.c:504:client3_3_stat_cbk]
0-volume1-client-0: remote operation failed: Stale file handle

We see no error when: have two processes of the above program running on
the same machine (say on M1) accessing the file F on the gluster volume,
for which we want to understand the expectations of atomicity in gluster
specifically specifically for rename, and if the above is a bug.

Also  glusterfs --version => glusterfs 3.6.9 built on Mar  2 2016 18:21:14

We also would like to know if there any parameter in the one translators
 that we can tweak to prevent this problem

Any help or insights here is appreciated

Thanks
Rama
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160322/64a1786d/attachment.html>

On 03/23/2016 04:10 AM, Rama Shenai wrote:
> Hi, We had some questions with respect to expectations of atomicity of
> rename in gluster.
>
> To elaborate : 
>
> We have setup with two machines (lets call them M1 and M2) which
> essentially access a file (F) on a gluster volume (mounted by M1 and M2)
> A program does the following steps sequentially on each of the two
> machines (M1 & M2) in an infinite loop
>  1) Opens the file F in O_RDWR|O_EXCL mode, reads some data and closes (F)
>  2)  Renames some other file F' => F
>  
> Periodically either M1 or M2 sees a "Stalefile handle error" when
it
> tries to read the file (step (1)) after opening the file in
> O_RDWR|O_EXCL (the open is successful)
>
> The specific error reported the client volume logs
>  (/var/log/glusterfs/mnt-repos-volume1.log)
> [2016-03-21 16:53:17.897902] I [dht-rename.c:1344:dht_rename]
> 0-volume1-dht: renaming master.lock
> (hash=volume1-replicate-0/cache=volume1-replicate-0) => master
> (hash=volume1-replicate-0/cache=<nul>)
> [2016-03-21 16:53:18.735090] W
> [client-rpc-fops.c:504:client3_3_stat_cbk] 0-volume1-client-0: remote
> operation failed: Stale file handle
Hi Rama,

ESTALE error in rename normally generated when either the source file is
not resolvable (deleted or inaccessible) or when the parent of
destination is not resolvable. It can happen when let's say file F' was
present when your application did a lookup before rename, but if it is
got renamed by Node M1 before M2 could rename it. Basically a race
between two rename on the same file can result in ESTALE for either of one.

To confirm this, Can you please paste the log message from brick
"0-volume1-client-0". You can find out the brick name from the graph.

Also if you can share the program or snippet that used to reproduce this
issue, that would be great.

Rafi KC


>
> We see no error when: have two processes of the above program running
> on the same machine (say on M1) accessing the file F on the gluster
> volume, for which we want to understand the expectations of atomicity
> in gluster specifically specifically for rename, and if the above is a
> bug.
>
> Also  glusterfs --version => glusterfs 3.6.9 built on Mar  2 2016
18:21:14
>
> We also would like to know if there any parameter in the one
> translators  that we can tweak to prevent this problem
>
> Any help or insights here is appreciated
>
> Thanks
> Rama
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160323/a8011d84/attachment.html>