Kotresh Hiremath Ravishankar
2017-Apr-24 09:37 UTC
[Gluster-users] High load on glusterfsd process
Hi Abhishek, Bitrot requires versioning of files to be down on writes. This was being done irrespective of whether bitrot is enabled or not. This takes considerable CPU. With the fix https://review.gluster.org/#/c/14442/, it is made optional and is enabled only with bitrot. If bitrot is not enabled, then you won't see any setxattr/getxattrs related to bitrot. The fix would be available in 3.11. Thanks and Regards, Kotresh H R ----- Original Message -----> From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> > To: "Pranith Kumar Karampuri" <pkarampu at redhat.com> > Cc: "Gluster Devel" <gluster-devel at gluster.org>, "gluster-users" <gluster-users at gluster.org>, "Kotresh Hiremath > Ravishankar" <khiremat at redhat.com> > Sent: Monday, April 24, 2017 11:30:57 AM > Subject: Re: [Gluster-users] High load on glusterfsd process > > Hi Kotresh, > > Could you please update me on this? > > Regards, > Abhishek > > On Sat, Apr 22, 2017 at 12:31 PM, Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > > > +Kotresh who seems to have worked on the bug you mentioned. > > > > On Fri, Apr 21, 2017 at 12:21 PM, ABHISHEK PALIWAL < > > abhishpaliwal at gmail.com> wrote: > > > >> > >> If the patch provided in that case will resolve my bug as well then > >> please provide the patch so that I will backport it on 3.7.6 > >> > >> On Fri, Apr 21, 2017 at 11:30 AM, ABHISHEK PALIWAL < > >> abhishpaliwal at gmail.com> wrote: > >> > >>> Hi Team, > >>> > >>> I have noticed that there are so many glusterfsd threads are running in > >>> my system and we observed some of those thread consuming more cpu. I > >>> did ?strace? on two such threads (before the problem disappeared by > >>> itself) > >>> and found that there is a continuous activity like below: > >>> > >>> lstat("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92f8-4 > >>> dfe-9a7f-246e901cbdf1/002700/firewall_-J208482-4000025_20170126T113552+0000.log.gz", > >>> {st_mode=S_IFREG|0670, st_size=1995, ...}) = 0 > >>> lgetxattr("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92 > >>> f8-4dfe-9a7f-246e901cbdf1/002700/firewall_-J208482-4000025_20170126T113552+0000.log.gz", > >>> "trusted.bit-rot.bad-file", 0x3fff81f58550, 255) = -1 ENODATA (No data > >>> available) > >>> lgetxattr("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92 > >>> f8-4dfe-9a7f-246e901cbdf1/002700/firewall_-J208482-4000025_20170126T113552+0000.log.gz", > >>> "trusted.bit-rot.signature", 0x3fff81f58550, 255) = -1 ENODATA (No data > >>> available) > >>> lstat("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92f8-4 > >>> dfe-9a7f-246e901cbdf1/002700/tcli_-J208482-4000025_20170123T180550+0000.log.gz", > >>> {st_mode=S_IFREG|0670, st_size=169, ...}) = 0 > >>> lgetxattr("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92 > >>> f8-4dfe-9a7f-246e901cbdf1/002700/tcli_-J208482-4000025_20170123T180550+0000.log.gz", > >>> "trusted.bit-rot.bad-file", 0x3fff81f58550, 255) = -1 ENODATA (No data > >>> available) > >>> lgetxattr("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92 > >>> f8-4dfe-9a7f-246e901cbdf1/002700/tcli_-J208482-4000025_20170123T180550+0000.log.gz", > >>> "trusted.bit-rot.signature", 0x3fff81f58550, 255) = -1 ENODATA (No data > >>> available) > >>> > >>> I have found the below existing issue which is very similar to my > >>> scenario. > >>> > >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298258 > >>> > >>> We are using the gluster-3.7.6 and it seems that the issue is fixed in > >>> 3.8.4 version. > >>> > >>> Could you please let me know why it showing the number of above logs and > >>> reason behind it as it is not explained in the above bug. > >>> > >>> Regards, > >>> Abhishek > >>> > >>> -- > >>> > >>> > >>> > >>> > >>> Regards > >>> Abhishek Paliwal > >>> > >> > >> > >> > >> -- > >> > >> > >> > >> > >> Regards > >> Abhishek Paliwal > >> > >> _______________________________________________ > >> Gluster-users mailing list > >> Gluster-users at gluster.org > >> http://lists.gluster.org/mailman/listinfo/gluster-users > >> > > > > > > > > -- > > Pranith > > > > > > -- > > > > > Regards > Abhishek Paliwal >
Hi Kotresh, I have seen the patch available on the link which you shared. It seems we don't have some files in gluser 3.7.6 which you modified in the patch. Is there any possibility to provide the patch for Gluster 3.7.6? Regards, Abhishek On Mon, Apr 24, 2017 at 3:07 PM, Kotresh Hiremath Ravishankar < khiremat at redhat.com> wrote:> Hi Abhishek, > > Bitrot requires versioning of files to be down on writes. > This was being done irrespective of whether bitrot is > enabled or not. This takes considerable CPU. With the > fix https://review.gluster.org/#/c/14442/, it is made > optional and is enabled only with bitrot. If bitrot > is not enabled, then you won't see any setxattr/getxattrs > related to bitrot. > > The fix would be available in 3.11. > > > Thanks and Regards, > Kotresh H R > > ----- Original Message ----- > > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> > > To: "Pranith Kumar Karampuri" <pkarampu at redhat.com> > > Cc: "Gluster Devel" <gluster-devel at gluster.org>, "gluster-users" < > gluster-users at gluster.org>, "Kotresh Hiremath > > Ravishankar" <khiremat at redhat.com> > > Sent: Monday, April 24, 2017 11:30:57 AM > > Subject: Re: [Gluster-users] High load on glusterfsd process > > > > Hi Kotresh, > > > > Could you please update me on this? > > > > Regards, > > Abhishek > > > > On Sat, Apr 22, 2017 at 12:31 PM, Pranith Kumar Karampuri < > > pkarampu at redhat.com> wrote: > > > > > +Kotresh who seems to have worked on the bug you mentioned. > > > > > > On Fri, Apr 21, 2017 at 12:21 PM, ABHISHEK PALIWAL < > > > abhishpaliwal at gmail.com> wrote: > > > > > >> > > >> If the patch provided in that case will resolve my bug as well then > > >> please provide the patch so that I will backport it on 3.7.6 > > >> > > >> On Fri, Apr 21, 2017 at 11:30 AM, ABHISHEK PALIWAL < > > >> abhishpaliwal at gmail.com> wrote: > > >> > > >>> Hi Team, > > >>> > > >>> I have noticed that there are so many glusterfsd threads are running > in > > >>> my system and we observed some of those thread consuming more cpu. I > > >>> did ?strace? on two such threads (before the problem disappeared by > > >>> itself) > > >>> and found that there is a continuous activity like below: > > >>> > > >>> lstat("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92f8-4 > > >>> dfe-9a7f-246e901cbdf1/002700/firewall_-J208482-4000025_ > 20170126T113552+0000.log.gz", > > >>> {st_mode=S_IFREG|0670, st_size=1995, ...}) = 0 > > >>> lgetxattr("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92 > > >>> f8-4dfe-9a7f-246e901cbdf1/002700/firewall_-J208482- > 4000025_20170126T113552+0000.log.gz", > > >>> "trusted.bit-rot.bad-file", 0x3fff81f58550, 255) = -1 ENODATA (No > data > > >>> available) > > >>> lgetxattr("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92 > > >>> f8-4dfe-9a7f-246e901cbdf1/002700/firewall_-J208482- > 4000025_20170126T113552+0000.log.gz", > > >>> "trusted.bit-rot.signature", 0x3fff81f58550, 255) = -1 ENODATA (No > data > > >>> available) > > >>> lstat("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92f8-4 > > >>> dfe-9a7f-246e901cbdf1/002700/tcli_-J208482-4000025_ > 20170123T180550+0000.log.gz", > > >>> {st_mode=S_IFREG|0670, st_size=169, ...}) = 0 > > >>> lgetxattr("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92 > > >>> f8-4dfe-9a7f-246e901cbdf1/002700/tcli_-J208482-4000025_ > 20170123T180550+0000.log.gz", > > >>> "trusted.bit-rot.bad-file", 0x3fff81f58550, 255) = -1 ENODATA (No > data > > >>> available) > > >>> lgetxattr("/opt/lvmdir/c2/brick/.glusterfs/e7/7d/e77d12b3-92 > > >>> f8-4dfe-9a7f-246e901cbdf1/002700/tcli_-J208482-4000025_ > 20170123T180550+0000.log.gz", > > >>> "trusted.bit-rot.signature", 0x3fff81f58550, 255) = -1 ENODATA (No > data > > >>> available) > > >>> > > >>> I have found the below existing issue which is very similar to my > > >>> scenario. > > >>> > > >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298258 > > >>> > > >>> We are using the gluster-3.7.6 and it seems that the issue is fixed > in > > >>> 3.8.4 version. > > >>> > > >>> Could you please let me know why it showing the number of above logs > and > > >>> reason behind it as it is not explained in the above bug. > > >>> > > >>> Regards, > > >>> Abhishek > > >>> > > >>> -- > > >>> > > >>> > > >>> > > >>> > > >>> Regards > > >>> Abhishek Paliwal > > >>> > > >> > > >> > > >> > > >> -- > > >> > > >> > > >> > > >> > > >> Regards > > >> Abhishek Paliwal > > >> > > >> _______________________________________________ > > >> Gluster-users mailing list > > >> Gluster-users at gluster.org > > >> http://lists.gluster.org/mailman/listinfo/gluster-users > > >> > > > > > > > > > > > > -- > > > Pranith > > > > > > > > > > > -- > > > > > > > > > > Regards > > Abhishek Paliwal > > >-- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170424/a11ae2c1/attachment.html>