Gluster users - Jan 2018 - stale file handle on gluster NFS client when trying to remove a directory

Hi,


After running rm -rf on a directory, the files under it got deleted, but
the directory was not deleted and was showing stale file handle error.
After 18 minutes, I'm able to delete the directory. So could anyone help me
in knowing what could have happened or when in general I get such errors.


The following is NFS log:


[2017-12-21 13:56:01.592256] I [MSGID: 108019]
[afr-transaction.c:1903:afr_post_blocking_entrylk_cbk]
0-g_sitework2-replicate-5: Blocking entrylks failed.

[2017-12-21 13:56:01.594350] W [MSGID: 108019]
[afr-lk-common.c:1064:afr_log_entry_locks_failure]
0-g_sitework2-replicate-4: Unable to obtain sufficient blocking entry locks
on at least one child while attempting RMDIR on
{pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c, name:csrc}.

[2017-12-21 13:56:01.594648] I [MSGID: 108019]
[afr-transaction.c:1903:afr_post_blocking_entrylk_cbk]
0-g_sitework2-replicate-4: Blocking entrylks failed.

[2017-12-21 13:56:01.594790] W [MSGID: 112032]
[nfs3.c:3713:nfs3svc_rmdir_cbk] 0-nfs: df521f4d:
<gfid:23558c59-87e5-4e90-a610-8a47ec08b27c>/csrc => -1 (Stale file
handle)
[Stale file handle]

[2017-12-21 13:56:01.594816] W [MSGID: 112199]
[nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3:
<gfid:23558c59-87e5-4e90-a610-8a47ec08b27c>/csrc => (XID: df521f4d,
RMDIR:
NFS: 70(Invalid file handle), POSIX: 116(Stale file handle))

[2017-12-21 13:56:01.590522] W [MSGID: 108019]
[afr-lk-common.c:1064:afr_log_entry_locks_failure]
0-g_sitework2-replicate-2: Unable to obtain sufficient blocking entry locks
on at least one child while attempting RMDIR on
{pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c, name:csrc}.

[2017-12-21 13:56:01.590569] W [MSGID: 108019]
[afr-lk-common.c:1064:afr_log_entry_locks_failure]
0-g_sitework2-replicate-1: Unable to obtain sufficient blocking entry locks
on at least one child while attempting RMDIR on
{pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c, name:csrc}.


Regards,

Jeevan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171221/3563ce0e/attachment.html>

Hi all,

I haven't found any root cause or workaround for this yet. Can any one help
me in underatanding the issue?

Regards,
Jeevan.

On Dec 21, 2017 8:20 PM, "Jeevan Patnaik" <g1patnaik at
gmail.com> wrote:
> Hi,
>
>
> After running rm -rf on a directory, the files under it got deleted, but
> the directory was not deleted and was showing stale file handle error.
> After 18 minutes, I'm able to delete the directory. So could anyone
help me
> in knowing what could have happened or when in general I get such errors.
>
>
> The following is NFS log:
>
>
> [2017-12-21 13:56:01.592256] I [MSGID: 108019]
[afr-transaction.c:1903:afr_post_blocking_entrylk_cbk]
> 0-g_sitework2-replicate-5: Blocking entrylks failed.
>
> [2017-12-21 13:56:01.594350] W [MSGID: 108019]
> [afr-lk-common.c:1064:afr_log_entry_locks_failure]
> 0-g_sitework2-replicate-4: Unable to obtain sufficient blocking entry locks
> on at least one child while attempting RMDIR on
{pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c,
> name:csrc}.
>
> [2017-12-21 13:56:01.594648] I [MSGID: 108019]
[afr-transaction.c:1903:afr_post_blocking_entrylk_cbk]
> 0-g_sitework2-replicate-4: Blocking entrylks failed.
>
> [2017-12-21 13:56:01.594790] W [MSGID: 112032]
[nfs3.c:3713:nfs3svc_rmdir_cbk]
> 0-nfs: df521f4d: <gfid:23558c59-87e5-4e90-a610-8a47ec08b27c>/csrc
=> -1
> (Stale file handle) [Stale file handle]
>
> [2017-12-21 13:56:01.594816] W [MSGID: 112199]
> [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3:
> <gfid:23558c59-87e5-4e90-a610-8a47ec08b27c>/csrc => (XID:
df521f4d,
> RMDIR: NFS: 70(Invalid file handle), POSIX: 116(Stale file handle))
>
> [2017-12-21 13:56:01.590522] W [MSGID: 108019]
> [afr-lk-common.c:1064:afr_log_entry_locks_failure]
> 0-g_sitework2-replicate-2: Unable to obtain sufficient blocking entry locks
> on at least one child while attempting RMDIR on
{pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c,
> name:csrc}.
>
> [2017-12-21 13:56:01.590569] W [MSGID: 108019]
> [afr-lk-common.c:1064:afr_log_entry_locks_failure]
> 0-g_sitework2-replicate-1: Unable to obtain sufficient blocking entry locks
> on at least one child while attempting RMDIR on
{pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c,
> name:csrc}.
>
>
> Regards,
>
> Jeevan.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180103/21b822ce/attachment.html>

An ESTALE error usually means the gfid could not be found. Does repeating
the "rm -rf" delete the directory?

Regards,
Nithya

On 3 January 2018 at 12:16, Jeevan Patnaik <g1patnaik at gmail.com> wrote:
> Hi all,
>
> I haven't found any root cause or workaround for this yet. Can any one
> help me in underatanding the issue?
>
> Regards,
> Jeevan.
>
> On Dec 21, 2017 8:20 PM, "Jeevan Patnaik" <g1patnaik at
gmail.com> wrote:
>
>> Hi,
>>
>>
>> After running rm -rf on a directory, the files under it got deleted,
but
>> the directory was not deleted and was showing stale file handle error.
>> After 18 minutes, I'm able to delete the directory. So could anyone
help me
>> in knowing what could have happened or when in general I get such
errors.
>>
>>
>> The following is NFS log:
>>
>>
>> [2017-12-21 13:56:01.592256] I [MSGID: 108019]
>> [afr-transaction.c:1903:afr_post_blocking_entrylk_cbk]
>> 0-g_sitework2-replicate-5: Blocking entrylks failed.
>>
>> [2017-12-21 13:56:01.594350] W [MSGID: 108019]
>> [afr-lk-common.c:1064:afr_log_entry_locks_failure]
>> 0-g_sitework2-replicate-4: Unable to obtain sufficient blocking entry
locks
>> on at least one child while attempting RMDIR on
>> {pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c, name:csrc}.
>>
>> [2017-12-21 13:56:01.594648] I [MSGID: 108019]
>> [afr-transaction.c:1903:afr_post_blocking_entrylk_cbk]
>> 0-g_sitework2-replicate-4: Blocking entrylks failed.
>>
>> [2017-12-21 13:56:01.594790] W [MSGID: 112032]
>> [nfs3.c:3713:nfs3svc_rmdir_cbk] 0-nfs: df521f4d:
>> <gfid:23558c59-87e5-4e90-a610-8a47ec08b27c>/csrc => -1 (Stale
file
>> handle) [Stale file handle]
>>
>> [2017-12-21 13:56:01.594816] W [MSGID: 112199]
>> [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3:
>> <gfid:23558c59-87e5-4e90-a610-8a47ec08b27c>/csrc => (XID:
df521f4d,
>> RMDIR: NFS: 70(Invalid file handle), POSIX: 116(Stale file handle))
>>
>> [2017-12-21 13:56:01.590522] W [MSGID: 108019]
>> [afr-lk-common.c:1064:afr_log_entry_locks_failure]
>> 0-g_sitework2-replicate-2: Unable to obtain sufficient blocking entry
locks
>> on at least one child while attempting RMDIR on
>> {pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c, name:csrc}.
>>
>> [2017-12-21 13:56:01.590569] W [MSGID: 108019]
>> [afr-lk-common.c:1064:afr_log_entry_locks_failure]
>> 0-g_sitework2-replicate-1: Unable to obtain sufficient blocking entry
locks
>> on at least one child while attempting RMDIR on
>> {pgfid:23558c59-87e5-4e90-a610-8a47ec08b27c, name:csrc}.
>>
>>
>> Regards,
>>
>> Jeevan.
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180103/3d46b9b8/attachment.html>