Gluster users - Mar 2017 - hanging httpd processes.

We are running gluster 3.8.9-1 on Centos 7.3.1611 for the servers and on 
the clients 3.7.11-2 on Centos 6.8

We are seeing httpd processes hang in fuse_request_send or sync_page.

These calls are from PHP 5.3.3-48 scripts

I am attaching  a tgz file that contains the process dump from 
glusterfsd and the hung pids along with the offending pid's stacks from 
/proc/{pid}/stack.

This has been a low level annoyance for a while but it has become a much 
bigger issue because the number of hung processes went from a few a week 
to a few hundred a day.


-- 
Alvin Starr                   ||   voice: (905)513-7688
Netvel Inc.                   ||   Cell:  (416)806-0133
alvin at netvel.net              ||

-------------- next part --------------
A non-text attachment was scrubbed...
Name: gdump.tgz
Type: application/x-compressed
Size: 89667 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170328/fc584940/attachment.bin>

Hi Alvin,

Thanks for the dump output. It helped a bit.

For now, recommend turning off open-behind and read-ahead performance
translators for you to get rid of this situation, As I noticed hung FLUSH
operations from these translators.

-Amar

On Wed, Mar 29, 2017 at 6:56 AM, Alvin Starr <alvin at netvel.net> wrote:
> We are running gluster 3.8.9-1 on Centos 7.3.1611 for the servers and on
> the clients 3.7.11-2 on Centos 6.8
>
> We are seeing httpd processes hang in fuse_request_send or sync_page.
>
> These calls are from PHP 5.3.3-48 scripts
>
> I am attaching  a tgz file that contains the process dump from glusterfsd
> and the hung pids along with the offending pid's stacks from
> /proc/{pid}/stack.
>
> This has been a low level annoyance for a while but it has become a much
> bigger issue because the number of hung processes went from a few a week to
> a few hundred a day.
>
>
> --
> Alvin Starr                   ||   voice: (905)513-7688
> Netvel Inc.                   ||   Cell:  (416)806-0133
> alvin at netvel.net              ||
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/854b1f8f/attachment.html>