This seems to have cleared itself. For future reference though, what
kinds of things should I look at do diagnose an issue like this?
Thanks
On 04/14/2017 01:16 PM, Pat Haley wrote:>
> Hi,
>
> Today we suddenly experienced a performance drop in gluster: e.g.
> doing an "ls" of a directory with about 20 files takes about 5
> minutes. This is way beyond (and seem separate from) some previous
> concerns we had.
>
> Our gluster filesystem is two bricks hosted on a single server.
> Logging onto that server and doing "top" shows a load average of
~30.
> In general, no process is showing significant CPU usage except an
> occasional flash a 3300% from glusterfsd. The rest of our system is
> not doing any exceptional data demands on the file system (i.e. we
> aren't suddenly running more jobs than we were yesterday).
>
> Any thoughts on how we can proceed with debugging this will be greatly
> appreciated.
>
> Some additional information:
>
> glusterfs 3.7.11 built on Apr 27 2016 14:09:22
> CentOS release 6.8 (Final)
>
>
> [root at mseas-data2 ~]# gluster volume status data-volume
> Status of volume: data-volume
> Gluster process TCP Port RDMA Port
> Online Pid
>
------------------------------------------------------------------------------
>
> Brick mseas-data2:/mnt/brick1 49154 0 Y 5021
> Brick mseas-data2:/mnt/brick2 49155 0 Y 5026
>
> Task Status of Volume data-volume
>
------------------------------------------------------------------------------
>
> Task : Rebalance
> ID : 892d9e3a-b38c-4971-b96a-8e4a496685ba
> Status : completed
>
>
> [root at mseas-data2 ~]# gluster volume info data-volume
>
> Volume Name: data-volume
> Type: Distribute
> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: mseas-data2:/mnt/brick1
> Brick2: mseas-data2:/mnt/brick2
> Options Reconfigured:
> diagnostics.brick-sys-log-level: WARNING
> performance.readdir-ahead: on
> nfs.disable: on
> nfs.export-volumes: off
>
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301