I remember we have fixed 2 issues where such kind of error messages were coming and also we were seeing issues on mount. In one of the case the problem was in dht. Unfortunately, I don't remember the BZ's for those issues. As glusterfs 3.10.1 is an old version, I would request you to please upgrade it to latest one. I am sure this would have fix . ---- Ashish ----- Original Message ----- From: "Nithya Balachandran" <nbalacha at redhat.com> To: "Amudhan P" <amudhan83 at gmail.com>, "Ashish Pandey" <aspandey at redhat.com> Cc: "Amudhan Pandian" <amudh_an at hotmail.com>, gluster-users at gluster.org Sent: Monday, November 13, 2017 7:14:30 PM Subject: Re: [Gluster-users] Error logged in fuse-mount log file Adding Ashish . Hi Amudhan, Can you check the gfids for every dir in that heirarchy? Maybe one of the parent dirs has a gfid mismatch. Regards, Nithya On 13 November 2017 at 17:39, Amudhan P < amudhan83 at gmail.com > wrote: Hi Nithya, I have checked gfid in all the bricks in disperse set for the folder. it all same there is no difference. regards Amudhan P On Fri, Nov 10, 2017 at 9:02 AM, Nithya Balachandran < nbalacha at redhat.com > wrote: <blockquote> Hi, Comments inline. Regards, Nithya On 9 November 2017 at 15:05, Amudhan Pandian < amudh_an at hotmail.com > wrote: <blockquote> resending mail from another id, doubt on whether mail reaches mailing list. ---------- Forwarded message ---------- From: Amudhan P < amudhan83 at gmail.com > Date: Tue, Nov 7, 2017 at 6:43 PM Subject: error logged in fuse-mount log file To: Gluster Users < gluster-users at gluster.org > Hi, I am using glusterfs 3.10.1 and i am seeing below msg in fuse-mount log file. what does this error mean? should i worry about this and how do i resolve this? [2017-11-07 11:59:17.218973] W [MSGID: 109005] [dht-selfheal.c:2113:dht_selfheal_directory] 0-glustervol-dht: Directory selfheal fail ed : 1 subvolumes have unrecoverable errors. path = /fol1/fol2/fol3/fol4/fol5, gfid =3f856ab3-f538-43ee-b408-53dd3da617fb [2017-11-07 11:59:17.218935] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glustervol-dht: Found anomalies in /fol1/fol2/fol3/fol4/fol5 (gfid = 3f856ab3-f538-43ee-b408-53dd3da617fb). Holes=1 overlaps=0 DHT found a problem in the layout of this directory so it tries to recreate and set the layout again. <blockquote> [2017-11-07 11:59:17.199917] W [MSGID: 122019] [ec-helpers.c:413:ec_loc_gfid_check] 0-glustervol-disperse-5: Mismatching GFID's in loc </blockquote> There appears to be a GFID mismatch for this directory on one of the bricks of this disperse set. Can you check the gfid xattr for this directory on all the bricks of this set? <blockquote> [2017-11-07 11:03:08.999769] W [MSGID: 109005] [dht-selfheal.c:2113:dht_selfheal_directory] 0-glustervol-dht: Directory selfheal fail ed : 1 subvolumes have unrecoverable errors. path = /sec_fol1/sec_fol2/sec_fol3/sec_fol4/sec_fol5, gfid = 59b9762e-f419-4d56-9fa2-eea9ebc055b2 [2017-11-07 11:03:08.999749] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-glustervol-dht: Found anomalies in /sec_fol1/sec_fol2/sec_fol3/sec_fol4/sec_fol5 (gfid = 59b9762e-f419-4d56-9fa2-eea9ebc055b2). Holes=1 overlaps=0 [2017-11-07 11:03:08.980275] W [MSGID: 122019] [ec-helpers.c:413:ec_loc_gfid_check] 0-glustervol-disperse-7: Mismatching GFID's in loc [2017-11-07 12:58:43.569801] I [MSGID: 109069] [dht-common.c:1324:dht_lookup_unlink_cbk] 0-glustervol-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /thi_fol1/thi_fol2/thi_fol3/thi_fol4/thi_fol5/thi_fol6/thi_file1 [2017-11-07 12:58:43.528844] I [MSGID: 109045] [dht-common.c:2012:dht_lookup_everywhere_cbk] 0-glustervol-dht: attempting deletion of stale linkfile /thi_fol1/thi_fol2/thi_fol3/thi_fol4/thi_fol5/thi_fol6/thi_file1 on glustervol-disperse-77 (hashed subvol is glustervol-disperse-106) </blockquote> These can be ignored. <blockquote> regards Amudhan P _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users </blockquote> _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users </blockquote> </blockquote> _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171113/d2937e60/attachment.html>
Nithya Balachandran
2017-Nov-14 04:08 UTC
[Gluster-users] Error logged in fuse-mount log file
On 14 November 2017 at 08:36, Ashish Pandey <aspandey at redhat.com> wrote:> > I remember we have fixed 2 issues where such kind of error messages were > coming and also we were seeing issues on mount. > In one of the case the problem was in dht. Unfortunately, I don't > remember the BZ's for those issues. >I think the DHT BZ you are referring to 1438423 <https://bugzilla.redhat.com/show_bug.cgi?id=1438423>. That has been fixed in 3.10.2 so an upgrade will help with this issue.> As glusterfs 3.10.1 is an old version, I would request you to please > upgrade it to latest one. I am sure this > would have fix . > > ---- > Ashish > > > > > ------------------------------ > *From: *"Nithya Balachandran" <nbalacha at redhat.com> > *To: *"Amudhan P" <amudhan83 at gmail.com>, "Ashish Pandey" < > aspandey at redhat.com> > *Cc: *"Amudhan Pandian" <amudh_an at hotmail.com>, gluster-users at gluster.org > *Sent: *Monday, November 13, 2017 7:14:30 PM > *Subject: *Re: [Gluster-users] Error logged in fuse-mount log file > > > Adding Ashish . > > Hi Amudhan, > > Can you check the gfids for every dir in that heirarchy? Maybe one of the > parent dirs has a gfid mismatch. > > Regards, > Nithya > > On 13 November 2017 at 17:39, Amudhan P <amudhan83 at gmail.com> wrote: > >> Hi Nithya, >> >> I have checked gfid in all the bricks in disperse set for the folder. it >> all same there is no difference. >> >> regards >> Amudhan P >> >> >> >> >> On Fri, Nov 10, 2017 at 9:02 AM, Nithya Balachandran <nbalacha at redhat.com >> > wrote: >> >>> Hi, >>> >>> Comments inline. >>> >>> Regards, >>> Nithya >>> >>> On 9 November 2017 at 15:05, Amudhan Pandian <amudh_an at hotmail.com> >>> wrote: >>> >>>> resending mail from another id, doubt on whether mail reaches mailing >>>> list. >>>> >>>> >>>> ---------- Forwarded message ---------- >>>> From: *Amudhan P* <amudhan83 at gmail.com> >>>> Date: Tue, Nov 7, 2017 at 6:43 PM >>>> Subject: error logged in fuse-mount log file >>>> To: Gluster Users <gluster-users at gluster.org> >>>> >>>> >>>> Hi, >>>> >>>> I am using glusterfs 3.10.1 and i am seeing below msg in fuse-mount log >>>> file. >>>> >>>> what does this error mean? should i worry about this and how do i >>>> resolve this? >>>> >>>> [2017-11-07 11:59:17.218973] W [MSGID: 109005] [dht-selfheal.c:2113:dht_selfheal_directory] >>>> 0-glustervol-dht: Directory selfheal fail >>>> ed : 1 subvolumes have unrecoverable errors. path >>>> /fol1/fol2/fol3/fol4/fol5, gfid =3f856ab3-f538-43ee-b408-53dd3da617fb >>>> [2017-11-07 11:59:17.218935] I [MSGID: 109063] >>>> [dht-layout.c:713:dht_layout_normalize] 0-glustervol-dht: Found >>>> anomalies in /fol1/fol2/fol3/fol4/fol5 (gfid = 3f856ab3-f538-43ee-b408-53dd3da617fb). >>>> Holes=1 overlaps=0 >>>> >>> >>> >>> DHT found a problem in the layout of this directory so it tries to >>> recreate and set the layout again. >>> >>> >>> >>>> [2017-11-07 11:59:17.199917] W [MSGID: 122019] >>>> [ec-helpers.c:413:ec_loc_gfid_check] 0-glustervol-disperse-5: >>>> Mismatching GFID's in loc >>>> >>>> >>> There appears to be a GFID mismatch for this directory on one of the >>> bricks of this disperse set. Can you check the gfid xattr for this >>> directory on all the bricks of this set? >>> >>> >>> >>>> >>>> [2017-11-07 11:03:08.999769] W [MSGID: 109005] [dht-selfheal.c:2113:dht_selfheal_directory] >>>> 0-glustervol-dht: Directory selfheal fail >>>> ed : 1 subvolumes have unrecoverable errors. path >>>> /sec_fol1/sec_fol2/sec_fol3/sec_fol4/sec_fol5, gfid >>>> 59b9762e-f419-4d56-9fa2-eea9ebc055b2 >>>> [2017-11-07 11:03:08.999749] I [MSGID: 109063] >>>> [dht-layout.c:713:dht_layout_normalize] 0-glustervol-dht: Found >>>> anomalies in /sec_fol1/sec_fol2/sec_fol3/sec_fol4/sec_fol5 (gfid >>>> 59b9762e-f419-4d56-9fa2-eea9ebc055b2). Holes=1 overlaps=0 >>>> [2017-11-07 11:03:08.980275] W [MSGID: 122019] >>>> [ec-helpers.c:413:ec_loc_gfid_check] 0-glustervol-disperse-7: >>>> Mismatching GFID's in loc >>>> >>>> >>>> [2017-11-07 12:58:43.569801] I [MSGID: 109069] >>>> [dht-common.c:1324:dht_lookup_unlink_cbk] 0-glustervol-dht: >>>> lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for >>>> /thi_fol1/thi_fol2/thi_fol3/thi_fol4/thi_fol5/thi_fol6/thi_file1 >>>> [2017-11-07 12:58:43.528844] I [MSGID: 109045] >>>> [dht-common.c:2012:dht_lookup_everywhere_cbk] 0-glustervol-dht: >>>> attempting deletion of stale linkfile /thi_fol1/thi_fol2/thi_fol3/ >>>> thi_fol4/thi_fol5/thi_fol6/thi_file1 on glustervol-disperse-77 (hashed >>>> subvol is glustervol-disperse-106) >>>> >>>> >>> These can be ignored. >>> >>> >>> regards >>>> Amudhan P >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171114/c161c78a/attachment.html>
okay, I will schedule to update the latest version and let you know if the same issue persists. On Tue, Nov 14, 2017 at 9:38 AM, Nithya Balachandran <nbalacha at redhat.com> wrote:> > > On 14 November 2017 at 08:36, Ashish Pandey <aspandey at redhat.com> wrote: > >> >> I remember we have fixed 2 issues where such kind of error messages were >> coming and also we were seeing issues on mount. >> In one of the case the problem was in dht. Unfortunately, I don't >> remember the BZ's for those issues. >> > > I think the DHT BZ you are referring to 1438423 > <https://bugzilla.redhat.com/show_bug.cgi?id=1438423>. That has been > fixed in 3.10.2 so an upgrade will help with this issue. > > >> As glusterfs 3.10.1 is an old version, I would request you to please >> upgrade it to latest one. I am sure this >> would have fix . >> >> ---- >> Ashish >> >> >> >> >> ------------------------------ >> *From: *"Nithya Balachandran" <nbalacha at redhat.com> >> *To: *"Amudhan P" <amudhan83 at gmail.com>, "Ashish Pandey" < >> aspandey at redhat.com> >> *Cc: *"Amudhan Pandian" <amudh_an at hotmail.com>, gluster-users at gluster.org >> *Sent: *Monday, November 13, 2017 7:14:30 PM >> *Subject: *Re: [Gluster-users] Error logged in fuse-mount log file >> >> >> Adding Ashish . >> >> Hi Amudhan, >> >> Can you check the gfids for every dir in that heirarchy? Maybe one of the >> parent dirs has a gfid mismatch. >> >> Regards, >> Nithya >> >> On 13 November 2017 at 17:39, Amudhan P <amudhan83 at gmail.com> wrote: >> >>> Hi Nithya, >>> >>> I have checked gfid in all the bricks in disperse set for the folder. it >>> all same there is no difference. >>> >>> regards >>> Amudhan P >>> >>> >>> >>> >>> On Fri, Nov 10, 2017 at 9:02 AM, Nithya Balachandran < >>> nbalacha at redhat.com> wrote: >>> >>>> Hi, >>>> >>>> Comments inline. >>>> >>>> Regards, >>>> Nithya >>>> >>>> On 9 November 2017 at 15:05, Amudhan Pandian <amudh_an at hotmail.com> >>>> wrote: >>>> >>>>> resending mail from another id, doubt on whether mail reaches mailing >>>>> list. >>>>> >>>>> >>>>> ---------- Forwarded message ---------- >>>>> From: *Amudhan P* <amudhan83 at gmail.com> >>>>> Date: Tue, Nov 7, 2017 at 6:43 PM >>>>> Subject: error logged in fuse-mount log file >>>>> To: Gluster Users <gluster-users at gluster.org> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> I am using glusterfs 3.10.1 and i am seeing below msg in fuse-mount >>>>> log file. >>>>> >>>>> what does this error mean? should i worry about this and how do i >>>>> resolve this? >>>>> >>>>> [2017-11-07 11:59:17.218973] W [MSGID: 109005] >>>>> [dht-selfheal.c:2113:dht_selfheal_directory] 0-glustervol-dht: >>>>> Directory selfheal fail >>>>> ed : 1 subvolumes have unrecoverable errors. path >>>>> /fol1/fol2/fol3/fol4/fol5, gfid =3f856ab3-f538-43ee-b408-53dd3da617fb >>>>> [2017-11-07 11:59:17.218935] I [MSGID: 109063] >>>>> [dht-layout.c:713:dht_layout_normalize] 0-glustervol-dht: Found >>>>> anomalies in /fol1/fol2/fol3/fol4/fol5 (gfid >>>>> 3f856ab3-f538-43ee-b408-53dd3da617fb). Holes=1 overlaps=0 >>>>> >>>> >>>> >>>> DHT found a problem in the layout of this directory so it tries to >>>> recreate and set the layout again. >>>> >>>> >>>> >>>>> [2017-11-07 11:59:17.199917] W [MSGID: 122019] >>>>> [ec-helpers.c:413:ec_loc_gfid_check] 0-glustervol-disperse-5: >>>>> Mismatching GFID's in loc >>>>> >>>>> >>>> There appears to be a GFID mismatch for this directory on one of the >>>> bricks of this disperse set. Can you check the gfid xattr for this >>>> directory on all the bricks of this set? >>>> >>>> >>>> >>>>> >>>>> [2017-11-07 11:03:08.999769] W [MSGID: 109005] >>>>> [dht-selfheal.c:2113:dht_selfheal_directory] 0-glustervol-dht: >>>>> Directory selfheal fail >>>>> ed : 1 subvolumes have unrecoverable errors. path >>>>> /sec_fol1/sec_fol2/sec_fol3/sec_fol4/sec_fol5, gfid >>>>> 59b9762e-f419-4d56-9fa2-eea9ebc055b2 >>>>> [2017-11-07 11:03:08.999749] I [MSGID: 109063] >>>>> [dht-layout.c:713:dht_layout_normalize] 0-glustervol-dht: Found >>>>> anomalies in /sec_fol1/sec_fol2/sec_fol3/sec_fol4/sec_fol5 (gfid >>>>> 59b9762e-f419-4d56-9fa2-eea9ebc055b2). Holes=1 overlaps=0 >>>>> [2017-11-07 11:03:08.980275] W [MSGID: 122019] >>>>> [ec-helpers.c:413:ec_loc_gfid_check] 0-glustervol-disperse-7: >>>>> Mismatching GFID's in loc >>>>> >>>>> >>>>> [2017-11-07 12:58:43.569801] I [MSGID: 109069] >>>>> [dht-common.c:1324:dht_lookup_unlink_cbk] 0-glustervol-dht: >>>>> lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for >>>>> /thi_fol1/thi_fol2/thi_fol3/thi_fol4/thi_fol5/thi_fol6/thi_file1 >>>>> [2017-11-07 12:58:43.528844] I [MSGID: 109045] >>>>> [dht-common.c:2012:dht_lookup_everywhere_cbk] 0-glustervol-dht: >>>>> attempting deletion of stale linkfile /thi_fol1/thi_fol2/thi_fol3/th >>>>> i_fol4/thi_fol5/thi_fol6/thi_file1 on glustervol-disperse-77 (hashed >>>>> subvol is glustervol-disperse-106) >>>>> >>>>> >>>> These can be ignored. >>>> >>>> >>>> regards >>>>> Amudhan P >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >>> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171114/fc1a032a/attachment.html>