Gluster users - Dec 2014 - glusterfs and glusterfsd process utilization extremely high

This is an extenuation of a problem that I posted about last month that I
am still experiencing.  The original post with more detail can be found at
http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019587.html.
To sum up my problem, I have a freshly created 3 node replicated cluster.
It contains roughly 135 GB of files, many of which are small.  It is home
to several web sites hosted with Apache.  I am using Gluster version
3.6.1-1 installed from RPMs mounted from the server via the Fuse client (I
have tried NFS but it makes no difference).

When I posted last about the problem of extreme processor utilization, the
solution I was given by Parnith was to utilize another file system other
than EXT4 and to turn off cluster.entry-self-heal.  I am now using XFS
and cluster.entry-self-heal is turned off and I even turned off
cluster.self-heal-daemon but it made absolutely no difference.  All is fine
during the entire time the cluster is loaded via rsync however the minute I
point Apache traffic at the sites hosted on the cluster, glusterfs and
glusterfsd begin to climb to levels so high that in a matter of minutes it
is not even possible to log on to the system.  No modification have been
made to any of the other Gluster settings.

Any additional help resolving this matter would be greatly appreciated.

-- 
Regards,

Kyle
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141217/d5cd5a28/attachment.html>

----- Original Message -----
> From: "Kyle Harris" <kyle.harris98 at gmail.com>
> To: gluster-users at gluster.org
> Sent: Thursday, December 18, 2014 4:47:35 AM
> Subject: [Gluster-users] glusterfs and glusterfsd process utilization
> extremely high
> This is an extenuation of a problem that I posted about last month that I
am
> still experiencing. The original post with more detail can be found at
>
http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019587.html
> . To sum up my problem, I have a freshly created 3 node replicated cluster.
> It contains roughly 135 GB of files, many of which are small. It is home to
> several web sites hosted with Apache. I am using Gluster version 3.6.1-1
> installed from RPMs mounted from the server via the Fuse client (I have
> tried NFS but it makes no difference).
> When I posted last about the problem of extreme processor utilization, the
> solution I was given by Parnith was to utilize another file system other
> than EXT4 and to turn off cluster.entry-self-heal. I am now using XFS and
> cluster.entry-self-heal is turned off and I even turned off
> cluster.self-heal-daemon but it made absolutely no difference. All is fine
> during the entire time the cluster is loaded via rsync however the minute I
> point Apache traffic at the sites hosted on the cluster, glusterfs and
> glusterfsd begin to climb to levels so high that in a matter of minutes it
> is not even possible to log on to the system. No modification have been
made
> to any of the other Gluster settings.
> Any additional help resolving this matter would be greatly appreciated.
Hello, 

First of all, do the logs suggest anything useful? 

Could you perform the following steps while the I/O is going on (this is
assuming the nodes are not thrashed to the extent that it is impossible to
execute these commands):

1) On the shell, on one of the nodes in the cluster, execute `gluster volume
profile <volname> start`
Wait for a minute or two. And then execute `gluster volume profile
<volname> info` and collect its output.
Wait for another minute or so. And execute `gluster volume profile
<volname> info` and collect its output too, and share them?
You can stop the profiling once you are done using `gluster volume profile
<volname> stop`.
2) Assuming it is the brick processes (glusterfsd) that are showing high CPU
utilisation, is it possible to get the core of the processes when this is
happening?

-Krutika 
> --
> Regards,
> Kyle
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141217/37e04684/attachment.html>