Gluster users - Feb 2015 - High CPU Usage - Glusterfsd

Dear all,

I have implemented glusterfs storage on my company - 2 servers with replicate.
But glustherfsd shows more than 100% CPU utilization most of the time. So it is
so slow to access the gluster volume. My setup is two glusterfs servers with
replication. The gluster volume (almost 10TB of data) is mounted on another
server (glusterfs native client) and using samba share for the network users to
access those files. Is there any way to reduce the processor usage on these
servers? Please give a solution ASAP since the users are complaining about the
poor performance. I am using glusterfs version 3.6.

Regards

Renchu Mathew  |  Sr. IT Administrator

[Description: Description: cid:image002.jpg at 01CF53EA.2A4680F0]

CRACKNELL  DUBAI   |  P.O. Box 66231  |   United Arab Emirates  |  T +971 4
3445417  |  F +971 4 3493675 |  M +971 50 7386484
ABU DHABI | DUBAI | LONDON | MUSCAT | DOHA | JEDDAH
EMAIL renchu at cracknell.com<mailto:renchu at cracknell.com> | WEB
www.cracknell.com

This email, its content and any files transmitted with it are intended solely
for the addressee(s) and may be legally privileged and/or confidential. If you
are not the intended recipient please let us know by email reply and delete it
from the system. Please note that any views or opinions presented in this email
do not necessarily represent those of the company. Email transmissions cannot be
guaranteed to be secure or error-free as information could be intercepted,
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
company therefore does not accept liability for any errors or omissions in the
contents of this message which arise as a result of email transmission.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150222/b4c9fc51/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1893 bytes
Desc: image001.jpg
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150222/b4c9fc51/attachment.jpg>

Renchu, 

I didn't see anything about average file size and read/write mix.  One
example of how to observe both of these, as well as latency and throughput - on
server run these commands:

# gluster volume profile your-volume start
# gluster volume profile your-volume info > /tmp/dontcare
# sleep 60
# gluster volume profile your-volume info > profile-for-last-minute.log

There is also a "gluster volume top" command that may be of use to you
in understanding what your users are doing with Gluster.

Also you may want to run "top -H" and see whether any threads in
either glusterfsd or smbd are at or near 100% CPU - if so, you really are
hitting a CPU bottleneck.  Looking at process CPU utilization can be deceptive,
since a process may include multiple threads.  "sar -n DEV 2" will
show you network utilization, and "iostat -mdx /dev/sd? 2" on your
server will show block device queue depth (latter two tools require sysstat
rpm).  Together these can help you to understand what kind of bottleneck you are
seeing.

I don't see how many "bricks" are in your Gluster volume but it
sounds like you have only one glusterfsd/server.   If you have idle cores on
your servers, you can harness more CPU power by using multiple bricks/server,
which results in multiple glusterfsd processes on each server, allowing greater
parallelism.    For example, you can do this by presenting individual disk
drives as bricks rather than RAID volumes.

Let us know if these suggestions helped

-ben england

----- Original Message -----
> From: "Renchu Mathew" <renchu at cracknell.com>
> To: gluster-users at gluster.org
> Cc: gluster-devel at gluster.org
> Sent: Sunday, February 22, 2015 7:09:09 AM
> Subject: [Gluster-devel] High CPU Usage - Glusterfsd
> 
> 
> 
> Dear all,
> 
> 
> 
> I have implemented glusterfs storage on my company ? 2 servers with
> replicate. But glustherfsd shows more than 100% CPU utilization most of the
> time. So it is so slow to access the gluster volume. My setup is two
> glusterfs servers with replication. The gluster volume (almost 10TB of
data)
> is mounted on another server (glusterfs native client) and using samba
share
> for the network users to access those files. Is there any way to reduce the
> processor usage on these servers? Please give a solution ASAP since the
> users are complaining about the poor performance. I am using glusterfs
> version 3.6.
> 
> 
> 
> Regards
> 
> 
> 
> Renchu Mathew | Sr. IT Administrator
> 
> 
> 
> 
> 
> 
> 
> CRACKNELL DUBAI | P.O. Box 66231 | United Arab Emirates | T +971 4 3445417
|
> F +971 4 3493675 | M +971 50 7386484
> 
> ABU DHABI | DUBAI | LONDON | MUSCAT | DOHA | JEDDAH
> 
> EMAIL renchu at cracknell.com | WEB www.cracknell.com
> 
> 
> 
> This email, its content and any files transmitted with it are intended
solely
> for the addressee(s) and may be legally privileged and/or confidential. If
> you are not the intended recipient please let us know by email reply and
> delete it from the system. Please note that any views or opinions presented
> in this email do not necessarily represent those of the company. Email
> transmissions cannot be guaranteed to be secure or error-free as
information
> could be intercepted, corrupted, lost, destroyed, arrive late or
incomplete,
> or contain viruses. The company therefore does not accept liability for any
> errors or omissions in the contents of this message which arise as a result
> of email transmission.
> 
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>