Hu Bert
2023-Apr-28 04:28 UTC
[Gluster-users] gluster 10.3: task glfs_fusenoti blocked for more than 120 seconds
Good morning, we've recently had some strange message in /var/log/syslog. System: debian bullseye, kernel 5.10.0-21-amd64 and 5.10.0-22-amd64 gluster 10.3 The message look like: Apr 27 13:30:18 piggy kernel: [24287.715229] INFO: task glfs_fusenoti:2787 blocked for more than 120 seconds. Apr 27 13:30:18 piggy kernel: [24287.715327] Not tainted 5.10.0-22-amd64 #1 Debian 5.10.178-3 Apr 27 13:30:18 piggy kernel: [24287.715419] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 27 13:30:18 piggy kernel: [24287.715575] task:glfs_fusenoti state:D stack: 0 pid: 2787 ppid: 1 flags:0x00000000 Apr 27 13:30:18 piggy kernel: [24287.715734] Call Trace: Apr 27 13:30:18 piggy kernel: [24287.715847] __schedule+0x282/0x870 Apr 27 13:30:18 piggy kernel: [24287.715959] schedule+0x46/0xb0 Apr 27 13:30:18 piggy kernel: [24287.716073] rwsem_down_write_slowpath+0x257/0x4d0 Apr 27 13:30:18 piggy kernel: [24287.716194] fuse_reverse_inval_entry+0x3b/0x1e0 [fuse] etc. Full excerpt here: https://pastebin.com/6gDHgh16 This may cause our application to "hang". In the latest release i read something about fuse: https://docs.gluster.org/en/latest/release-notes/10.4/ "Fix fuse concurrency problems" but i checked the tickets and wasn't able to find something. However, i upgraded gluster client 10.3 -> 10.4, hoping that this fixes the hang-issue. Has anyone seen these messages before? Best regards, Hubert
Mohit Agrawal
2023-May-02 12:28 UTC
[Gluster-users] [Gluster-devel] gluster 10.3: task glfs_fusenoti blocked for more than 120 seconds
I don't think the issue is on gluster side, it seems the issue is on kernel side (possible deadlock in fuse_reverse_inval_entry) https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bda9a71980e083699a0360963c0135657b73f47a On Tue, May 2, 2023 at 5:48?PM Hu Bert <revirii at googlemail.com> wrote:> Good morning, > > we've recently had some strange message in /var/log/syslog. > > System: > debian bullseye, kernel 5.10.0-21-amd64 and 5.10.0-22-amd64 > gluster 10.3 > > The message look like: > > Apr 27 13:30:18 piggy kernel: [24287.715229] INFO: task > glfs_fusenoti:2787 blocked for more than 120 seconds. > Apr 27 13:30:18 piggy kernel: [24287.715327] Not tainted > 5.10.0-22-amd64 #1 Debian 5.10.178-3 > Apr 27 13:30:18 piggy kernel: [24287.715419] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Apr 27 13:30:18 piggy kernel: [24287.715575] task:glfs_fusenoti > state:D stack: 0 pid: 2787 ppid: 1 flags:0x00000000 > Apr 27 13:30:18 piggy kernel: [24287.715734] Call Trace: > Apr 27 13:30:18 piggy kernel: [24287.715847] __schedule+0x282/0x870 > Apr 27 13:30:18 piggy kernel: [24287.715959] schedule+0x46/0xb0 > Apr 27 13:30:18 piggy kernel: [24287.716073] > rwsem_down_write_slowpath+0x257/0x4d0 > Apr 27 13:30:18 piggy kernel: [24287.716194] > fuse_reverse_inval_entry+0x3b/0x1e0 [fuse] > > etc. Full excerpt here: https://pastebin.com/6gDHgh16 > > This may cause our application to "hang". In the latest release i read > something about fuse: > https://docs.gluster.org/en/latest/release-notes/10.4/ > "Fix fuse concurrency problems" > > but i checked the tickets and wasn't able to find something. > > However, i upgraded gluster client 10.3 -> 10.4, hoping that this > fixes the hang-issue. > > Has anyone seen these messages before? > > > Best regards, > Hubert > ------- > > Community Meeting Calendar: > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20230502/f748ed4c/attachment.html>