Hi I am running a 400TB five node purely distributed gluster setup. I am troubleshooting an issue where some times files creation fails. I found that volume status is not working gluster volume status Another transaction is in progress for atlasglust. Please try again after sometime. When I tried from other node then it seems two nodes have Locking issue gluster volume status Locking failed on pplxgluster01... Please check log file for details. Locking failed on pplxgluster04... Please check log file for details. Also noticed that glusterfsd process is using around 1000% cpu usage. It is a decent server with 16 core and 64GB RAM. Gluster version is 3.11.2-1 Can you please suggest that how to troubleshoot further? Thanks Kashif -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170830/caa59de7/attachment.html>
On 30 August 2017 at 20:54, mohammad kashif <kashif.alig at gmail.com> wrote:> Hi > > I am running a 400TB five node purely distributed gluster setup. I am > troubleshooting an issue where some times files creation fails. I found > that volume status is not working > > gluster volume status > Another transaction is in progress for atlasglust. Please try again after > sometime. > > When I tried from other node then it seems two nodes have Locking issue > > gluster volume status > Locking failed on pplxgluster01... Please check log file for details. > Locking failed on pplxgluster04... Please check log file for details. > > Also noticed that glusterfsd process is using around 1000% cpu usage. It > is a decent server with 16 core and 64GB RAM. > > Gluster version is 3.11.2-1 > > Can you please suggest that how to troubleshoot further? >Please check the mount log files to see if there are any errors for the files that were not created. Gluster logs are present in /var/log/glusterfs. If you are using FUSE mount, say /mnt/fuse1, the log will be called mnt-fuse1.log.> > Thanks > > Kashif > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170830/f8c0bc8b/attachment.html>
On Wed, 30 Aug 2017 at 20:55, mohammad kashif <kashif.alig at gmail.com> wrote:> Hi > > I am running a 400TB five node purely distributed gluster setup. I am > troubleshooting an issue where some times files creation fails. I found > that volume status is not working > > gluster volume status > Another transaction is in progress for atlasglust. Please try again after > sometime. > > When I tried from other node then it seems two nodes have Locking issue > > gluster volume status > Locking failed on pplxgluster01... Please check log file for details. > Locking failed on pplxgluster04... Please check log file for details. >This suggests that there are concurrent gluster cli operations been performed on the same volume. Are you monitoring the cluster through nagios or you have a script on all the nodes which checks for volume's health in a period of interval? Please note glusterd will process one cli operation on a volume at one time, the rest all transactions on the same volume will be failed.> > Also noticed that glusterfsd process is using around 1000% cpu usage. It > is a decent server with 16 core and 64GB RAM. > > Gluster version is 3.11.2-1 > > Can you please suggest that how to troubleshoot further? > > Thanks > > Kashif > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-- - Atin (atinm) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170831/10c3a639/attachment.html>