Gluster users - Jun 2014 - glusterfsd process spinning

Hi

I've been suffering from continual problems with my gluster filesystem
slowing down due to what I thought was congestion on a single brick
being caused by a problem with the underlying filesystem running slow
but I've just noticed that the glusterfsd process for that particular
brick is running at 100%+, even when the filesystem is almost idle.

I've done a couple of straces of the brick and another on the same
server, does the high number of futex errors give any clues as to what
might be wrong?

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
45.58    0.027554           0    191665     20772 futex
28.26    0.017084           0    137133           readv
26.04    0.015743           0     66259           epoll_wait
  0.13    0.000077           3        23           writev
  0.00    0.000000           0         1           epoll_ctl
------ ----------- ----------- --------- --------- ----------------
100.00    0.060458                395081     20772 total

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
99.25    0.334020         133      2516           epoll_wait
  0.40    0.001347           0      4090        26 futex
  0.35    0.001192           0      5064           readv
  0.00    0.000000           0        20           writev
------ ----------- ----------- --------- --------- ----------------
100.00    0.336559                 11690        26 total



Cheers,

Franco,
    Could you do the following to get more information:

"gluster volume profile <volname> start"

Wait for some time, this will start gathering what operations are coming to all
the bricks"
Now execute "gluster volume profile <volname> info" >
/file/you/should/reply/to/this/mail/with

Then execute:
gluster volume profile <volname> stop

Lets see if this throws any light on the problem at hand

Pranith
----- Original Message -----
> From: "Franco Broi" <franco.broi at iongeo.com>
> To: gluster-users at gluster.org
> Sent: Sunday, June 1, 2014 9:02:48 AM
> Subject: [Gluster-users] glusterfsd process spinning
> 
> Hi
> 
> I've been suffering from continual problems with my gluster filesystem
> slowing down due to what I thought was congestion on a single brick
> being caused by a problem with the underlying filesystem running slow
> but I've just noticed that the glusterfsd process for that particular
> brick is running at 100%+, even when the filesystem is almost idle.
> 
> I've done a couple of straces of the brick and another on the same
> server, does the high number of futex errors give any clues as to what
> might be wrong?
> 
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 45.58    0.027554           0    191665     20772 futex
> 28.26    0.017084           0    137133           readv
> 26.04    0.015743           0     66259           epoll_wait
>   0.13    0.000077           3        23           writev
>   0.00    0.000000           0         1           epoll_ctl
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.060458                395081     20772 total
> 
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 99.25    0.334020         133      2516           epoll_wait
>   0.40    0.001347           0      4090        26 futex
>   0.35    0.001192           0      5064           readv
>   0.00    0.000000           0        20           writev
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.336559                 11690        26 total
> 
> 
> 
> Cheers,
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>