[Gluster-users] Very odd performance issue

David Miller
Thu, 04 May 2017 18:48:38 -0000

Background:  4 identical gluster servers with 15 TB each in 2x2 setup.
CentOS Linux release 7.3.1611 (Core)
glusterfs-server-3.9.1-1.el7.x86_64
client systems are using:
glusterfs-client	 3.5.2-2+deb8u3

The cluster has ~12 TB in use with 21 million files.  Lots of jpgs.  About 12
clients are mounting gluster volumes.

Network load is light: iftop shows each server has 10-15 Mbit reads and about
half that in writes.

What I?m seeing that concerns me is that one box, gluster4, has roughly twice
the CPU utilization and twice or more the load average of the other three
servers.  gluster4 has a 24 hour average of about 30% CPU utilization, something
that seems to me to be way out of line for a couple MB/sec of traffic.

In running volume top, the odd thing I see is that for gluster1-3 I get latency
summaries like this:
Brick: gluster1.publicinteractive.com:/gluster/drupal_prod
?????????????????????????????
%-latency  Avg-latency  Min-Latency  Max-Latency   No. of calls       Fop
 --------  -----------  -----------  -----------   ------------      ----

 9.96     675.07 us      15.00 us 1067793.00 us         205060     INODELK
15.85    3414.20 us      16.00 us  773621.00 us          64494        READ
51.35    2235.96 us      12.00 us 1093609.00 us         319120      LOOKUP

? but my problem server has far more inodelk latency:

12.01    4712.03 us      17.00 us 1773590.00 us          47214        READ
27.50    2390.27 us      14.00 us 1877571.00 us         213121     INODELK
28.70    1643.65 us      12.00 us 1837696.00 us         323407      LOOKUP

The servers are intended to be identical, and are indeed identical hardware.

Suggestions on where to look or which FM to RT ver welcome indeed.

Thanks,

David




-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170504/64821ce5/attachment.html>