Thank you for the acknowledgement.
On Thu, Aug 31, 2017 at 8:30 PM, mohammad kashif <kashif.alig at
gmail.com>
wrote:
> Hi Atin
>
> Thanks, I was not running any script or gluster command. But now gluster
> status command started working. CPU usage also came down and looking at the
> ganglia graph, cpu usage is strongly correlated with network activity.
>
> It may be due to high load that status command failed. I will keep an eye
> and see whether it happens again.
>
> Cheers
>
> Kashif
>
> On Thu, Aug 31, 2017 at 2:40 AM, Atin Mukherjee <amukherj at
redhat.com>
> wrote:
>
>>
>> On Wed, 30 Aug 2017 at 20:55, mohammad kashif <kashif.alig at
gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I am running a 400TB five node purely distributed gluster setup. I
am
>>> troubleshooting an issue where some times files creation fails. I
found
>>> that volume status is not working
>>>
>>> gluster volume status
>>> Another transaction is in progress for atlasglust. Please try again
>>> after sometime.
>>>
>>> When I tried from other node then it seems two nodes have Locking
issue
>>>
>>> gluster volume status
>>> Locking failed on pplxgluster01... Please check log file for
details.
>>> Locking failed on pplxgluster04... Please check log file for
details.
>>>
>>
>> This suggests that there are concurrent gluster cli operations been
>> performed on the same volume. Are you monitoring the cluster through
nagios
>> or you have a script on all the nodes which checks for volume's
health in a
>> period of interval? Please note glusterd will process one cli operation
on
>> a volume at one time, the rest all transactions on the same volume will
be
>> failed.
>>
>>
>>
>>>
>>> Also noticed that glusterfsd process is using around 1000% cpu
usage.
>>> It is a decent server with 16 core and 64GB RAM.
>>>
>>> Gluster version is 3.11.2-1
>>>
>>> Can you please suggest that how to troubleshoot further?
>>>
>>> Thanks
>>>
>>> Kashif
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> --
>> - Atin (atinm)
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170831/34cb0b1d/attachment.html>
