thr3ads.net - Gluster users - [Gluster-users] high CPU load on all bricks [Feb 2013]

If this information is useful, please help other people find it:
Share via:

Michael Colonno

2013-Feb-01 20:46 UTC

[Gluster-users] high CPU load on all bricks

Gluster gurus ~

 

            I've deployed and 8-brick (2x replicate) Gluster 3.3.1 volume on
CentOS 6.3 with tcp transport. I was able to build, start, mount, and use
the volume. On each system contributing a brick, however, my CPU usage
(glusterfsd) is hovering around 20% (virtually zero memory usage
thankfully). These are brand new, fairly beefy servers so 20% CPU load is
quite a bit. The deployment is pretty plain with each brick mounting the
volume to itself via a glusterfs mount. I assume this type of CPU usage is
atypically high; is there anything I can do to investigate what's soaking up
CPU and minimize it? Total usable volume size is only about 22 TB (about 45
TB total with 2x replicate). 

 

            Thanks,

            ~Mike C. 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130201/404dacf2/attachment.html>

Michael Colonno

2013-Feb-01 20:53 UTC

head link

[Gluster-users] high CPU load on all bricks

Forgot to mention: on a client system (not a brick) the
glusterfs process is consuming ~ 68% CPU continuously. This is a much less
powerful desktop system so the CPU load can't be compared 1:1 with the
systems comprising the bricks but still very high. So the issue seems to
exist with both glusterfsd and glusterfs processes. 

 

            Thanks,

            ~Mike C. 

 

From: gluster-users-bounces at gluster.org
[mailto:gluster-users-bounces at gluster.org] On Behalf Of Michael Colonno
Sent: Friday, February 01, 2013 12:46 PM
To: gluster-users at gluster.org
Subject: [Gluster-users] high CPU load on all bricks

 

            Gluster gurus ~

 

            I've deployed and 8-brick (2x replicate) Gluster 3.3.1 volume on
CentOS 6.3 with tcp transport. I was able to build, start, mount, and use
the volume. On each system contributing a brick, however, my CPU usage
(glusterfsd) is hovering around 20% (virtually zero memory usage
thankfully). These are brand new, fairly beefy servers so 20% CPU load is
quite a bit. The deployment is pretty plain with each brick mounting the
volume to itself via a glusterfs mount. I assume this type of CPU usage is
atypically high; is there anything I can do to investigate what's soaking up
CPU and minimize it? Total usable volume size is only about 22 TB (about 45
TB total with 2x replicate). 

 

            Thanks,

            ~Mike C. 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130201/a8848ca0/attachment.html>

Robert Hajime Lanning

2013-Feb-01 21:48 UTC

head link

[Gluster-users] high CPU load on all bricks

On 02/01/13 12:46, Michael Colonno wrote:> Gluster gurus ~
>
> I?ve deployed and 8-brick (2x replicate) Gluster 3.3.1 volume on CentOS
> 6.3 with tcp transport. I was able to build, start, mount, and use the
> volume. On each system contributing a brick, however, my CPU usage
> (glusterfsd) is hovering around 20% (virtually zero memory usage
> thankfully). These are brand new, fairly beefy servers so 20% CPU load
> is quite a bit. The deployment is pretty plain with each brick mounting
> the volume to itself via a glusterfs mount. I assume this type of CPU
> usage is atypically high; is there anything I can do to investigate
> what?s soaking up CPU and minimize it? Total usable volume size is only
> about 22 TB (about 45 TB total with 2x replicate).
Where are you reading the 20% from?

If it is from the per process line in top, then that 20% is of a single 
core/hyperthread.  It is not of the whole system.

If you have an otherwise quiescent server and the "Cpu(s)" summary
line
at the top says 20%us, then you can extrapolate to under 20% of the 
total system.

-- 
Mr. Flibble
King of the Potato People

Michael Colonno

2013-Feb-03 19:22 UTC

head link

[Gluster-users] high CPU load on all bricks

Having taken a lot more data it does seem the glusterfsd and glusterd processes
(along with several ksoftirqd) spike up to near 100% on both client and brick
servers during any file transport across the mount. Thankfully this is
short-lived for the most part but I?m wondering if this is expected behavior or
what others have experienced(?) I?m a little surprised such a large CPU load
would be required to move small files and / or use an application within a
Gluster mount point.

 

            I wanted to test this against an NFS mount of the same Gluster
volume. I managed to get rstatd installed and running but my attempts to mount
the volume via NFS are met with:

 

            mount.nfs: requested NFS version or transport protocol is not
supported

 

            Relevant line in /etc/fstab:

 

            node1:/volume    /volume    nfs    
defaults,_netdev,vers=3,mountproto=tcp        0 0

 

It looks like CentOS 6.x has NFS version 4 built into everything. So a few
questions:

 

-       Has anyone else noted significant performance differences between a
glusterfs mount and NFS mount for volumes of 8+ bricks?

-       Is there a straightforward way to make the newer versions of CentOS play
nice with NFS version 3 + Gluster?

-       Are there any general performance tuning guidelines I can follow to
improve CPU performance? I found a few references to the cache settings but
nothing solid.

 

If the consensus is that NFS will not gain anything then I won?t waste the time
setting it all up.

 

Thanks,

~Mike C. 

 

 

From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at
gluster.org] On Behalf Of Michael Colonno
Sent: Friday, February 01, 2013 4:46 PM
To: gluster-users at gluster.org
Subject: Re: [Gluster-users] high CPU load on all bricks

 

            Update: after a few hours the CPU usage seems to have dropped down
to a small value. I did not change anything with respect to the configuration or
unmount / stop anything as I wanted to see if this would persist for a long
period of time. Both the client and the self-mounted bricks are now showing CPU
< 1% (as reported by top). Prior to the larger CPU loads I installed a bunch
of software into the volume (~ 5 GB total). Is this kind a transient behavior ?
by which I mean larger CPU loads after a lot of filesystem activity in short
time ? typical? This is not a problem in my deployment; I just want to know what
to expect in the future and to complete this thread for future users. If this is
expected behavior we can wrap up this thread. If not then I?ll do more digging
into the logs on the client and brick sides.

 

            Thanks,

            ~Mike C. 

 

From: Joe Julian [mailto:joe at julianfamily.org] 
Sent: Friday, February 01, 2013 2:08 PM
To: Michael Colonno; gluster-users at gluster.org
Subject: Re: [Gluster-users] high CPU load on all bricks

 

Check the client log(s). 

Michael Colonno <mcolonno at stanford.edu> wrote:

            Forgot to mention: on a client system (not a brick) the glusterfs
process is consuming ~ 68% CPU continuously. This is a much less powerful
desktop system so the CPU load can?t be compared 1:1 with the systems comprising
the bricks but still very high. So the issue seems to exist with both glusterfsd
and glusterfs processes.

 

            Thanks,

            ~Mike C. 

 

From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at
gluster.org] On Behalf Of Michael Colonno
Sent: Friday, February 01, 2013 12:46 PM
To: gluster-users at gluster.org
Subject: [Gluster-users] high CPU load on all bricks

 

            Gluster gurus ~

 

            I?ve deployed and 8-brick (2x replicate) Gluster 3.3.1 volume on
CentOS 6.3 with tcp transport. I was able to build, start, mount, and use the
volume. On each system contributing a brick, however, my CPU usage (glusterfsd)
is hovering around 20% (virtually zero memory usage thankfully). These are brand
new, fairly beefy servers so 20% CPU load is quite a bit. The deployment is
pretty plain with each brick mounting the volume to itself via a glusterfs
mount. I assume this type of CPU usage is atypically high; is there anything I
can do to investigate what?s soaking up CPU and minimize it? Total usable volume
size is only about 22 TB (about 45 TB total with 2x replicate).

 

            Thanks,

            ~Mike C. 

 


  _____  


Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130203/fd6f6622/attachment.html>

Gluster users - Feb 2013 - high CPU load on all bricks

[Gluster-users] high CPU load on all bricks

[Gluster-users] high CPU load on all bricks

[Gluster-users] high CPU load on all bricks

[Gluster-users] high CPU load on all bricks