Ilan Schwarts
2017-Aug-07 14:18 UTC
[Gluster-users] gluster stuck when trying to list a successful mount
Hi all, My infrastructure is GlusterFS with 2 Nodes: L137B-GlusterFS-Node1.L137B-root.com L137B-GlusterFS-Node2.L137B-root.com I have followed this guide: http://www.itzgeek.com/how-tos/linux/centos-how-tos/install-and-configure-glusterfs-on-centos-7-rhel-7.html/2 I have created a device, formatted as ext4, mounted it. The next step was to install gluster FS. I installed glusterfs 3.10.3, created a volume and started it [root at L137B-GlusterFS-Node2 someuser]# gluster volume info Volume Name: gv0 Type: Replicate Volume ID: a606f77b-c0df-427c-99ec-cee98b3ecd71 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: L137B-GlusterFS-Node1.L137B-root.com:/mnt/gluster/gv0 Brick2: L137B-GlusterFS-Node2.L137B-root.com:/mnt/gluster/gv0 Options Reconfigured: transport.address-family: inet nfs.disable: on Now when I try to mount to Node1 from Node2, with the command: mount -t glusterfs L137B-GlusterFS-Node2.L137B-root.com:/gv0 /mnt L137B-GlusterFS-Node2.L137B-root.com:/gv0 on /mnt type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) it returns ok, no errors, but when i do: ls -la /mnt the console get stucked. on dmesg -T i get (from a duplicate session on another console): [Mon Aug 7 17:05:58 2017] fuse init (API version 7.22) [Mon Aug 7 17:09:32 2017] INFO: task glusterfsd:3273 blocked for more than 120 seconds. [Mon Aug 7 17:09:32 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Aug 7 17:09:32 2017] glusterfsd D ffff880092e380b0 0 3273 1 0x00000080 [Mon Aug 7 17:09:32 2017] ffff8800a1e3bb70 0000000000000086 ffff880139b7af10 ffff8800a1e3bfd8 [Mon Aug 7 17:09:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8 ffff880139b7af10 ffff880092e380a8 [Mon Aug 7 17:09:32 2017] ffff880092e380ac ffff880139b7af10 00000000ffffffff ffff880092e380b0 [Mon Aug 7 17:09:32 2017] Call Trace: [Mon Aug 7 17:09:32 2017] [<ffffffff8168d259>] schedule_preempt_disabled+0x29/0x70 [Mon Aug 7 17:09:32 2017] [<ffffffff8168aeb5>] __mutex_lock_slowpath+0xc5/0x1c0 [Mon Aug 7 17:09:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140 [Mon Aug 7 17:09:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f [Mon Aug 7 17:09:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7 [Mon Aug 7 17:09:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0 [Mon Aug 7 17:09:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0 [Mon Aug 7 17:09:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0 [Mon Aug 7 17:09:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280 [Mon Aug 7 17:09:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0 [Mon Aug 7 17:09:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0 [Mon Aug 7 17:09:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0 [Mon Aug 7 17:09:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0 [Mon Aug 7 17:09:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160 [Mon Aug 7 17:09:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20 [Mon Aug 7 17:09:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0 [Mon Aug 7 17:09:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60 [Mon Aug 7 17:09:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180 [Mon Aug 7 17:09:32 2017] [<ffffffff8111f426>] ? __audit_syscall_exit+0x1e6/0x280 [Mon Aug 7 17:09:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10 [Mon Aug 7 17:09:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b [Mon Aug 7 17:11:32 2017] INFO: task glusterfsd:3273 blocked for more than 120 seconds. [Mon Aug 7 17:11:32 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Aug 7 17:11:32 2017] glusterfsd D ffff880092e380b0 0 3273 1 0x00000080 [Mon Aug 7 17:11:32 2017] ffff8800a1e3bb70 0000000000000086 ffff880139b7af10 ffff8800a1e3bfd8 [Mon Aug 7 17:11:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8 ffff880139b7af10 ffff880092e380a8 [Mon Aug 7 17:11:32 2017] ffff880092e380ac ffff880139b7af10 00000000ffffffff ffff880092e380b0 [Mon Aug 7 17:11:32 2017] Call Trace: [Mon Aug 7 17:11:32 2017] [<ffffffff8168d259>] schedule_preempt_disabled+0x29/0x70 [Mon Aug 7 17:11:32 2017] [<ffffffff8168aeb5>] __mutex_lock_slowpath+0xc5/0x1c0 [Mon Aug 7 17:11:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140 [Mon Aug 7 17:11:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f [Mon Aug 7 17:11:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7 [Mon Aug 7 17:11:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0 [Mon Aug 7 17:11:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0 [Mon Aug 7 17:11:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0 [Mon Aug 7 17:11:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280 [Mon Aug 7 17:11:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0 [Mon Aug 7 17:11:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0 [Mon Aug 7 17:11:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0 [Mon Aug 7 17:11:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0 [Mon Aug 7 17:11:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160 [Mon Aug 7 17:11:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20 [Mon Aug 7 17:11:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0 [Mon Aug 7 17:11:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60 [Mon Aug 7 17:11:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180 [Mon Aug 7 17:11:32 2017] [<ffffffff8111f426>] ? __audit_syscall_exit+0x1e6/0x280 [Mon Aug 7 17:11:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10 [Mon Aug 7 17:11:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b [Mon Aug 7 17:13:32 2017] INFO: task glusterfsd:3273 blocked for more than 120 seconds. [Mon Aug 7 17:13:32 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Aug 7 17:13:32 2017] glusterfsd D ffff880092e380b0 0 3273 1 0x00000080 [Mon Aug 7 17:13:32 2017] ffff8800a1e3bb70 0000000000000086 ffff880139b7af10 ffff8800a1e3bfd8 [Mon Aug 7 17:13:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8 ffff880139b7af10 ffff880092e380a8 [Mon Aug 7 17:13:32 2017] ffff880092e380ac ffff880139b7af10 00000000ffffffff ffff880092e380b0 [Mon Aug 7 17:13:32 2017] Call Trace: [Mon Aug 7 17:13:32 2017] [<ffffffff8168d259>] schedule_preempt_disabled+0x29/0x70 [Mon Aug 7 17:13:32 2017] [<ffffffff8168aeb5>] __mutex_lock_slowpath+0xc5/0x1c0 [Mon Aug 7 17:13:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140 [Mon Aug 7 17:13:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f [Mon Aug 7 17:13:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7 [Mon Aug 7 17:13:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0 [Mon Aug 7 17:13:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0 [Mon Aug 7 17:13:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0 [Mon Aug 7 17:13:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280 [Mon Aug 7 17:13:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0 [Mon Aug 7 17:13:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0 [Mon Aug 7 17:13:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0 [Mon Aug 7 17:13:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0 [Mon Aug 7 17:13:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160 [Mon Aug 7 17:13:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20 [Mon Aug 7 17:13:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0 [Mon Aug 7 17:13:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60 [Mon Aug 7 17:13:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180 [Mon Aug 7 17:13:32 2017] [<ffffffff8111f426>] ? __audit_syscall_exit+0x1e6/0x280 [Mon Aug 7 17:13:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10 [Mon Aug 7 17:13:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b [Mon Aug 7 17:15:32 2017] INFO: task glusterfsd:3273 blocked for more than 120 seconds. [Mon Aug 7 17:15:32 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Aug 7 17:15:32 2017] glusterfsd D ffff880092e380b0 0 3273 1 0x00000080 [Mon Aug 7 17:15:32 2017] ffff8800a1e3bb70 0000000000000086 ffff880139b7af10 ffff8800a1e3bfd8 [Mon Aug 7 17:15:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8 ffff880139b7af10 ffff880092e380a8 [Mon Aug 7 17:15:32 2017] ffff880092e380ac ffff880139b7af10 00000000ffffffff ffff880092e380b0 [Mon Aug 7 17:15:32 2017] Call Trace: [Mon Aug 7 17:15:32 2017] [<ffffffff8168d259>] schedule_preempt_disabled+0x29/0x70 [Mon Aug 7 17:15:32 2017] [<ffffffff8168aeb5>] __mutex_lock_slowpath+0xc5/0x1c0 [Mon Aug 7 17:15:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140 [Mon Aug 7 17:15:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f [Mon Aug 7 17:15:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7 [Mon Aug 7 17:15:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0 [Mon Aug 7 17:15:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0 [Mon Aug 7 17:15:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0 [Mon Aug 7 17:15:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280 [Mon Aug 7 17:15:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0 [Mon Aug 7 17:15:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0 [Mon Aug 7 17:15:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0 [Mon Aug 7 17:15:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0 [Mon Aug 7 17:15:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160 [Mon Aug 7 17:15:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20 [Mon Aug 7 17:15:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0 [Mon Aug 7 17:15:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60 [Mon Aug 7 17:15:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180 [Mon Aug 7 17:15:32 2017] [<ffffffff8111f426>] ? __audit_syscall_exit+0x1e6/0x280 [Mon Aug 7 17:15:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10 [Mon Aug 7 17:15:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b
Ilan Schwarts
2017-Aug-08 08:22 UTC
[Gluster-users] gluster stuck when trying to list a successful mount
OK. The issue was bad disk/mount on one of the machines. When I tried to: df -h from Node1 it stucked. So removed volume, peer, remounted the device. Now its ok. On Mon, Aug 7, 2017 at 5:18 PM, Ilan Schwarts <ilan84 at gmail.com> wrote:> Hi all, > My infrastructure is GlusterFS with 2 Nodes: > L137B-GlusterFS-Node1.L137B-root.com > L137B-GlusterFS-Node2.L137B-root.com > > > I have followed this guide: > http://www.itzgeek.com/how-tos/linux/centos-how-tos/install-and-configure-glusterfs-on-centos-7-rhel-7.html/2 > > I have created a device, formatted as ext4, mounted it. > The next step was to install gluster FS. > I installed glusterfs 3.10.3, created a volume and started it > [root at L137B-GlusterFS-Node2 someuser]# gluster volume info > > Volume Name: gv0 > Type: Replicate > Volume ID: a606f77b-c0df-427c-99ec-cee98b3ecd71 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: L137B-GlusterFS-Node1.L137B-root.com:/mnt/gluster/gv0 > Brick2: L137B-GlusterFS-Node2.L137B-root.com:/mnt/gluster/gv0 > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > > > Now when I try to mount to Node1 from Node2, with the command: > mount -t glusterfs L137B-GlusterFS-Node2.L137B-root.com:/gv0 /mnt > > L137B-GlusterFS-Node2.L137B-root.com:/gv0 on /mnt type fuse.glusterfs > (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) > > > > it returns ok, no errors, but when i do: ls -la /mnt the console get stucked. > on dmesg -T i get (from a duplicate session on another console): > [Mon Aug 7 17:05:58 2017] fuse init (API version 7.22) > [Mon Aug 7 17:09:32 2017] INFO: task glusterfsd:3273 blocked for more > than 120 seconds. > [Mon Aug 7 17:09:32 2017] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [Mon Aug 7 17:09:32 2017] glusterfsd D ffff880092e380b0 0 > 3273 1 0x00000080 > [Mon Aug 7 17:09:32 2017] ffff8800a1e3bb70 0000000000000086 > ffff880139b7af10 ffff8800a1e3bfd8 > [Mon Aug 7 17:09:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8 > ffff880139b7af10 ffff880092e380a8 > [Mon Aug 7 17:09:32 2017] ffff880092e380ac ffff880139b7af10 > 00000000ffffffff ffff880092e380b0 > [Mon Aug 7 17:09:32 2017] Call Trace: > [Mon Aug 7 17:09:32 2017] [<ffffffff8168d259>] > schedule_preempt_disabled+0x29/0x70 > [Mon Aug 7 17:09:32 2017] [<ffffffff8168aeb5>] > __mutex_lock_slowpath+0xc5/0x1c0 > [Mon Aug 7 17:09:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140 > [Mon Aug 7 17:09:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f > [Mon Aug 7 17:09:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7 > [Mon Aug 7 17:09:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0 > [Mon Aug 7 17:09:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0 > [Mon Aug 7 17:09:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0 > [Mon Aug 7 17:09:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280 > [Mon Aug 7 17:09:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0 > [Mon Aug 7 17:09:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0 > [Mon Aug 7 17:09:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0 > [Mon Aug 7 17:09:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0 > [Mon Aug 7 17:09:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160 > [Mon Aug 7 17:09:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20 > [Mon Aug 7 17:09:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0 > [Mon Aug 7 17:09:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60 > [Mon Aug 7 17:09:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180 > [Mon Aug 7 17:09:32 2017] [<ffffffff8111f426>] ? > __audit_syscall_exit+0x1e6/0x280 > [Mon Aug 7 17:09:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10 > [Mon Aug 7 17:09:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b > [Mon Aug 7 17:11:32 2017] INFO: task glusterfsd:3273 blocked for more > than 120 seconds. > [Mon Aug 7 17:11:32 2017] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [Mon Aug 7 17:11:32 2017] glusterfsd D ffff880092e380b0 0 > 3273 1 0x00000080 > [Mon Aug 7 17:11:32 2017] ffff8800a1e3bb70 0000000000000086 > ffff880139b7af10 ffff8800a1e3bfd8 > [Mon Aug 7 17:11:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8 > ffff880139b7af10 ffff880092e380a8 > [Mon Aug 7 17:11:32 2017] ffff880092e380ac ffff880139b7af10 > 00000000ffffffff ffff880092e380b0 > [Mon Aug 7 17:11:32 2017] Call Trace: > [Mon Aug 7 17:11:32 2017] [<ffffffff8168d259>] > schedule_preempt_disabled+0x29/0x70 > [Mon Aug 7 17:11:32 2017] [<ffffffff8168aeb5>] > __mutex_lock_slowpath+0xc5/0x1c0 > [Mon Aug 7 17:11:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140 > [Mon Aug 7 17:11:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f > [Mon Aug 7 17:11:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7 > [Mon Aug 7 17:11:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0 > [Mon Aug 7 17:11:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0 > [Mon Aug 7 17:11:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0 > [Mon Aug 7 17:11:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280 > [Mon Aug 7 17:11:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0 > [Mon Aug 7 17:11:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0 > [Mon Aug 7 17:11:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0 > [Mon Aug 7 17:11:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0 > [Mon Aug 7 17:11:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160 > [Mon Aug 7 17:11:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20 > [Mon Aug 7 17:11:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0 > [Mon Aug 7 17:11:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60 > [Mon Aug 7 17:11:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180 > [Mon Aug 7 17:11:32 2017] [<ffffffff8111f426>] ? > __audit_syscall_exit+0x1e6/0x280 > [Mon Aug 7 17:11:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10 > [Mon Aug 7 17:11:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b > [Mon Aug 7 17:13:32 2017] INFO: task glusterfsd:3273 blocked for more > than 120 seconds. > [Mon Aug 7 17:13:32 2017] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [Mon Aug 7 17:13:32 2017] glusterfsd D ffff880092e380b0 0 > 3273 1 0x00000080 > [Mon Aug 7 17:13:32 2017] ffff8800a1e3bb70 0000000000000086 > ffff880139b7af10 ffff8800a1e3bfd8 > [Mon Aug 7 17:13:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8 > ffff880139b7af10 ffff880092e380a8 > [Mon Aug 7 17:13:32 2017] ffff880092e380ac ffff880139b7af10 > 00000000ffffffff ffff880092e380b0 > [Mon Aug 7 17:13:32 2017] Call Trace: > [Mon Aug 7 17:13:32 2017] [<ffffffff8168d259>] > schedule_preempt_disabled+0x29/0x70 > [Mon Aug 7 17:13:32 2017] [<ffffffff8168aeb5>] > __mutex_lock_slowpath+0xc5/0x1c0 > [Mon Aug 7 17:13:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140 > [Mon Aug 7 17:13:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f > [Mon Aug 7 17:13:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7 > [Mon Aug 7 17:13:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0 > [Mon Aug 7 17:13:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0 > [Mon Aug 7 17:13:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0 > [Mon Aug 7 17:13:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280 > [Mon Aug 7 17:13:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0 > [Mon Aug 7 17:13:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0 > [Mon Aug 7 17:13:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0 > [Mon Aug 7 17:13:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0 > [Mon Aug 7 17:13:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160 > [Mon Aug 7 17:13:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20 > [Mon Aug 7 17:13:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0 > [Mon Aug 7 17:13:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60 > [Mon Aug 7 17:13:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180 > [Mon Aug 7 17:13:32 2017] [<ffffffff8111f426>] ? > __audit_syscall_exit+0x1e6/0x280 > [Mon Aug 7 17:13:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10 > [Mon Aug 7 17:13:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b > [Mon Aug 7 17:15:32 2017] INFO: task glusterfsd:3273 blocked for more > than 120 seconds. > [Mon Aug 7 17:15:32 2017] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [Mon Aug 7 17:15:32 2017] glusterfsd D ffff880092e380b0 0 > 3273 1 0x00000080 > [Mon Aug 7 17:15:32 2017] ffff8800a1e3bb70 0000000000000086 > ffff880139b7af10 ffff8800a1e3bfd8 > [Mon Aug 7 17:15:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8 > ffff880139b7af10 ffff880092e380a8 > [Mon Aug 7 17:15:32 2017] ffff880092e380ac ffff880139b7af10 > 00000000ffffffff ffff880092e380b0 > [Mon Aug 7 17:15:32 2017] Call Trace: > [Mon Aug 7 17:15:32 2017] [<ffffffff8168d259>] > schedule_preempt_disabled+0x29/0x70 > [Mon Aug 7 17:15:32 2017] [<ffffffff8168aeb5>] > __mutex_lock_slowpath+0xc5/0x1c0 > [Mon Aug 7 17:15:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140 > [Mon Aug 7 17:15:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f > [Mon Aug 7 17:15:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7 > [Mon Aug 7 17:15:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0 > [Mon Aug 7 17:15:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0 > [Mon Aug 7 17:15:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0 > [Mon Aug 7 17:15:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280 > [Mon Aug 7 17:15:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0 > [Mon Aug 7 17:15:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0 > [Mon Aug 7 17:15:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0 > [Mon Aug 7 17:15:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0 > [Mon Aug 7 17:15:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160 > [Mon Aug 7 17:15:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20 > [Mon Aug 7 17:15:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0 > [Mon Aug 7 17:15:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60 > [Mon Aug 7 17:15:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180 > [Mon Aug 7 17:15:32 2017] [<ffffffff8111f426>] ? > __audit_syscall_exit+0x1e6/0x280 > [Mon Aug 7 17:15:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10 > [Mon Aug 7 17:15:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b-- - Ilan Schwarts