thr3ads.net - Gluster users - [Gluster-users] blocking process on FUSE mount in directory which is using quota [Aug 2018]

If this information is useful, please help other people find it:
Share via:

mabi

2018-Aug-09 12:54 UTC

[Gluster-users] blocking process on FUSE mount in directory which is using quota

Hello,

I recently upgraded my GlusterFS replica 2+1 (aribter) to version 3.12.12 and
now I see a weird behaviour on my client (using FUSE mount) where I have
processes (PHP 5.6 FPM) trying to access a specific directory and then the
process blocks. I can't kill the process either, not even with kill -9. I
need to reboot the machine in order to get rid of these blocked processes.

This directory has one particularity compared to the other directories it is
that it has reached it's quota soft-limit as you can see here in the output
of gluster volume quota list:

                  Path                   Hard-limit  Soft-limit      Used 
Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/directory                          100.0GB     80%(80.0GB)   90.5GB   9.5GB    
Yes                   No

That does not mean that it is the quota's fault but it might be a hint where
to start looking for... And by the way can someone explain me what the
soft-limit does? or does it not do anything special?

Here is an the linux stack of a blocking process on that directory which
happened with a simple "ls -la":

[Thu Aug  9 14:21:07 2018] INFO: task ls:2272 blocked for more than 120 seconds.
[Thu Aug  9 14:21:07 2018]       Not tainted 3.16.0-4-amd64 #1
[Thu Aug  9 14:21:07 2018] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Aug  9 14:21:07 2018] ls              D ffff88017ef93200     0  2272   2268
0x00000004
[Thu Aug  9 14:21:07 2018]  ffff88017653f490 0000000000000286 0000000000013200
ffff880174d7bfd8
[Thu Aug  9 14:21:07 2018]  0000000000013200 ffff88017653f490 ffff8800eeb3d5f0
ffff8800fefac800
[Thu Aug  9 14:21:07 2018]  ffff880174d7bbe0 ffff8800eeb3d6d0 ffff8800fefac800
ffff8800ffe1e1c0
[Thu Aug  9 14:21:07 2018] Call Trace:
[Thu Aug  9 14:21:07 2018]  [<ffffffffa00dc50d>] ?
__fuse_request_send+0xbd/0x270 [fuse]
[Thu Aug  9 14:21:07 2018]  [<ffffffff810abce0>] ?
prepare_to_wait_event+0xf0/0xf0
[Thu Aug  9 14:21:07 2018]  [<ffffffffa00e0791>] ?
fuse_dentry_revalidate+0x181/0x300 [fuse]
[Thu Aug  9 14:21:07 2018]  [<ffffffff811b944e>] ? lookup_fast+0x25e/0x2b0
[Thu Aug  9 14:21:07 2018]  [<ffffffff811bacc5>] ?
path_lookupat+0x155/0x780
[Thu Aug  9 14:21:07 2018]  [<ffffffff81195715>] ?
kmem_cache_alloc+0x75/0x480
[Thu Aug  9 14:21:07 2018]  [<ffffffffa00dfca9>] ?
fuse_getxattr+0xe9/0x150 [fuse]
[Thu Aug  9 14:21:07 2018]  [<ffffffff811bb316>] ?
filename_lookup+0x26/0xc0
[Thu Aug  9 14:21:07 2018]  [<ffffffff811bf594>] ?
user_path_at_empty+0x54/0x90
[Thu Aug  9 14:21:07 2018]  [<ffffffff81193e08>] ?
kmem_cache_free+0xd8/0x210
[Thu Aug  9 14:21:07 2018]  [<ffffffff811bf59f>] ?
user_path_at_empty+0x5f/0x90
[Thu Aug  9 14:21:07 2018]  [<ffffffff811b3d46>] ? vfs_fstatat+0x46/0x90
[Thu Aug  9 14:21:07 2018]  [<ffffffff811b421d>] ? SYSC_newlstat+0x1d/0x40
[Thu Aug  9 14:21:07 2018]  [<ffffffff811d34b8>] ? SyS_lgetxattr+0x58/0x80
[Thu Aug  9 14:21:07 2018]  [<ffffffff81525d0d>] ?
system_call_fast_compare_end+0x10/0x15


My 3 gluster nodes are all Debian 9 and my client Debian 8.

Let me know if you need more information.

Best regards,
Mabi

Nithya Balachandran

2018-Aug-09 13:10 UTC

head link

[Gluster-users] blocking process on FUSE mount in directory which is using quota

Hi,

Please provide the following:

   1. gluster volume info
   2. statedump of the fuse process when it hangs


Thanks,
Nithya

On 9 August 2018 at 18:24, mabi <mabi at protonmail.ch> wrote:
> Hello,
>
> I recently upgraded my GlusterFS replica 2+1 (aribter) to version 3.12.12
> and now I see a weird behaviour on my client (using FUSE mount) where I
> have processes (PHP 5.6 FPM) trying to access a specific directory and then
> the process blocks. I can't kill the process either, not even with kill
-9.
> I need to reboot the machine in order to get rid of these blocked
processes.
>
> This directory has one particularity compared to the other directories it
> is that it has reached it's quota soft-limit as you can see here in the
> output of gluster volume quota list:
>
>                   Path                   Hard-limit  Soft-limit      Used
> Available  Soft-limit exceeded? Hard-limit exceeded?
> ------------------------------------------------------------
> -------------------------------------------------------------------
> /directory                          100.0GB     80%(80.0GB)   90.5GB
>  9.5GB             Yes                   No
>
> That does not mean that it is the quota's fault but it might be a hint
> where to start looking for... And by the way can someone explain me what
> the soft-limit does? or does it not do anything special?
>
> Here is an the linux stack of a blocking process on that directory which
> happened with a simple "ls -la":
>
> [Thu Aug  9 14:21:07 2018] INFO: task ls:2272 blocked for more than 120
> seconds.
> [Thu Aug  9 14:21:07 2018]       Not tainted 3.16.0-4-amd64 #1
> [Thu Aug  9 14:21:07 2018] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [Thu Aug  9 14:21:07 2018] ls              D ffff88017ef93200     0  2272
>  2268 0x00000004
> [Thu Aug  9 14:21:07 2018]  ffff88017653f490 0000000000000286
> 0000000000013200 ffff880174d7bfd8
> [Thu Aug  9 14:21:07 2018]  0000000000013200 ffff88017653f490
> ffff8800eeb3d5f0 ffff8800fefac800
> [Thu Aug  9 14:21:07 2018]  ffff880174d7bbe0 ffff8800eeb3d6d0
> ffff8800fefac800 ffff8800ffe1e1c0
> [Thu Aug  9 14:21:07 2018] Call Trace:
> [Thu Aug  9 14:21:07 2018]  [<ffffffffa00dc50d>] ?
> __fuse_request_send+0xbd/0x270 [fuse]
> [Thu Aug  9 14:21:07 2018]  [<ffffffff810abce0>] ?
> prepare_to_wait_event+0xf0/0xf0
> [Thu Aug  9 14:21:07 2018]  [<ffffffffa00e0791>] ?
> fuse_dentry_revalidate+0x181/0x300 [fuse]
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811b944e>] ?
lookup_fast+0x25e/0x2b0
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811bacc5>] ?
> path_lookupat+0x155/0x780
> [Thu Aug  9 14:21:07 2018]  [<ffffffff81195715>] ?
> kmem_cache_alloc+0x75/0x480
> [Thu Aug  9 14:21:07 2018]  [<ffffffffa00dfca9>] ?
> fuse_getxattr+0xe9/0x150 [fuse]
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811bb316>] ?
> filename_lookup+0x26/0xc0
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811bf594>] ?
> user_path_at_empty+0x54/0x90
> [Thu Aug  9 14:21:07 2018]  [<ffffffff81193e08>] ?
> kmem_cache_free+0xd8/0x210
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811bf59f>] ?
> user_path_at_empty+0x5f/0x90
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811b3d46>] ?
vfs_fstatat+0x46/0x90
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811b421d>] ?
SYSC_newlstat+0x1d/0x40
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811d34b8>] ?
SyS_lgetxattr+0x58/0x80
> [Thu Aug  9 14:21:07 2018]  [<ffffffff81525d0d>] ?
> system_call_fast_compare_end+0x10/0x15
>
>
> My 3 gluster nodes are all Debian 9 and my client Debian 8.
>
> Let me know if you need more information.
>
> Best regards,
> Mabi
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180809/4e9c0625/attachment.html>

Gluster users - Aug 2018 - blocking process on FUSE mount in directory which is using quota

[Gluster-users] blocking process on FUSE mount in directory which is using quota

[Gluster-users] blocking process on FUSE mount in directory which is using quota