Could you provide this information from the server (also the client/server
logs):
cat /proc/<pid>/task/*/stack
ls -l /proc/<pid>/task/*/fd
<pid> == process ID of glusterfsd
On Fri, Feb 7, 2014 at 6:14 PM, Barry Stetler <barry at hivelocity.net>
wrote:
> We started using gluster in a custom solution recently. Basically we
> setup a gluster volume that we mount at /home. Then we give users their own
> home directory with a gluster quota. They have FTP, SFTP and SSHMOUNT
> access. I first used vsftpd and notice that ftp connections will not close
> and the load goes up ever couple of days the funny thing is the load does
> not seem to slow down the server. The load went up to 20 in the last two
> months but the server was not slow at all... At first I thought it was
> vsftpd so I changed to proftpd then the issue came back. If I reboot the
> server it goes away for a few days depending on how many ftp connections. I
> noticed in the dmesg log this making me think the issue was some kind of
> leak in gluster-fuse
>
> This is on CentOS 6.5 64 bit.
>
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task proftpd:7831 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
> proftpd D 0000000000000000 0 7831 4384 0x00000004
> ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
> ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
> ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
> Call Trace:
> [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
> [<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
> [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
> [<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
> [<ffffffff8117e0bc>] filp_close+0x3c/0x90
> [<ffffffff8117e1b5>] sys_close+0xa5/0x100
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
>
>
> I appreciate any help I can get on this...
>
> Thank you
>
> --
> Barry Stetler
> Hivelocity Hosting
> http://hivelocity.net
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140212/c5e91174/attachment.html>