ABHISHEK PALIWAL
2016-Mar-10 05:52 UTC
[Gluster-users] gluster volume heal info split brain command not showing files in split-brain
Hi Anuradha, Please find the glusterfs and glusterd logs directory as an attachment. Regards, Abhishek On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <abhishpaliwal at gmail.com> wrote:> Hi Anuradha, > > Sorry for late reply. > > Please find the requested logs below: > > Remote: 10.32.0.48 > Local : 10.32.1.144 > > Local: > #gluster volume heal c_glusterfs info split-brain > Brick 10.32.1.144:/opt/lvmdir/c2/brick > Number of entries in split-brain: 0 > > Brick 10.32.0.48:/opt/lvmdir/c2/brick > Number of entries in split-brain: 0 > > Remote: > #gluster volume heal c_glusterfs info split-brain > Brick 10.32.1.144:/opt/lvmdir/c2/brick > Number of entries in split-brain: 0 > > Brick 10.32.0.48:/opt/lvmdir/c2/brick > Number of entries in split-brain: 0 > > auto-sync.sh. > Here you can see that i/o error is detected. Below is the required meta > data from both the bricks. > > 1) > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': > Input/output error > Remote: > > getfattr -d -m . -e hex > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > trusted.afr.dirty=0x000000000000000000000000 > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee > > Local: > > getfattr -d -m . -e hex > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd > > > 2) > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error > Remote: > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity > # file: opt/lvmdir/c2/brick/security/corbasecurity > trusted.afr.dirty=0x000000000000000000000000 > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013 > > Local: > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity > # file: opt/lvmdir/c2/brick/security/corbasecurity > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 > trusted.gfid=0x890df0f706184b52803fac3242a2f15b > > I observed that getfattr command output doesn't show all the fields all > the times. > > Here you can check that gluster split-brain command hasn't reported any > split-brains but resulted in IO errors when accessed few files. > > Could you please tell me if "split-brain" command doesn't reported any > entry as output, then is there any way through which we can find out that > the files are in split-brain if we are getting the IO error on those file. > > > Regards, > Abhishek > > > > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <atalur at redhat.com> wrote: > >> >> >> ----- Original Message ----- >> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >> > To: gluster-users at gluster.org, gluster-devel at gluster.org >> > Sent: Thursday, March 3, 2016 12:10:42 PM >> > Subject: [Gluster-users] gluster volume heal info split brain command >> not showing files in split-brain >> > >> > >> > Hello, >> > >> > In gluster, we use the command "gluster volume heal c_glusterfs info >> > split-brain" to find the files that are in split-brain scenario. >> > We run heal script (developed by Windriver prime team) on the files >> reported >> > by above command to resolve split-brain issue. >> > >> > But we observed that the above command is not showing all files that >> are in >> > split-brain, >> > even though split brain scenario actually exists on the node. >> > >> > Now a days this issue is seen more often and IO errors are reported when >> > tried to access these files under split-brain. >> > >> > Can you please check why this gluster command is not showing files under >> > split-brain? >> > We can provide you required logs and support to resolve this issue. >> Hi, >> >> Could you paste the output of getfattr -m. -de hex >> <path-to-files-in-split-brain> from all the bricks that the files lie in? >> >> > >> > Please reply on this because I am not getting any reply from the >> community. >> > >> > -- >> > >> > Regards >> > Abhishek Paliwal >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> >> -- >> Thanks, >> Anuradha. >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160310/64fa0e11/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterd.tar Type: application/x-tar Size: 61440 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160310/64fa0e11/attachment.tar> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterfs.tar Type: application/x-tar Size: 71680 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160310/64fa0e11/attachment-0001.tar>
ABHISHEK PALIWAL
2016-Mar-14 05:01 UTC
[Gluster-users] gluster volume heal info split brain command not showing files in split-brain
Hi Anuradha, Please reply on this issue. Regards, Abhishek On Thu, Mar 10, 2016 at 11:22 AM, ABHISHEK PALIWAL <abhishpaliwal at gmail.com> wrote:> Hi Anuradha, > > Please find the glusterfs and glusterd logs directory as an attachment. > > Regards, > Abhishek > > > > > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <abhishpaliwal at gmail.com> > wrote: > >> Hi Anuradha, >> >> Sorry for late reply. >> >> Please find the requested logs below: >> >> Remote: 10.32.0.48 >> Local : 10.32.1.144 >> >> Local: >> #gluster volume heal c_glusterfs info split-brain >> Brick 10.32.1.144:/opt/lvmdir/c2/brick >> Number of entries in split-brain: 0 >> >> Brick 10.32.0.48:/opt/lvmdir/c2/brick >> Number of entries in split-brain: 0 >> >> Remote: >> #gluster volume heal c_glusterfs info split-brain >> Brick 10.32.1.144:/opt/lvmdir/c2/brick >> Number of entries in split-brain: 0 >> >> Brick 10.32.0.48:/opt/lvmdir/c2/brick >> Number of entries in split-brain: 0 >> >> auto-sync.sh. >> Here you can see that i/o error is detected. Below is the required meta >> data from both the bricks. >> >> 1) >> stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': >> Input/output error >> Remote: >> >> getfattr -d -m . -e hex >> opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior >> # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior >> trusted.afr.dirty=0x000000000000000000000000 >> trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f >> trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee >> >> Local: >> >> getfattr -d -m . -e hex >> opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior >> # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior >> trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 >> trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd >> >> >> 2) >> stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error >> Remote: >> >> getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity >> # file: opt/lvmdir/c2/brick/security/corbasecurity >> trusted.afr.dirty=0x000000000000000000000000 >> trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f >> trusted.gfid=0xd298b7a0c8834f3e99abb39741363013 >> >> Local: >> >> getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity >> # file: opt/lvmdir/c2/brick/security/corbasecurity >> trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 >> trusted.gfid=0x890df0f706184b52803fac3242a2f15b >> >> I observed that getfattr command output doesn't show all the fields all >> the times. >> >> Here you can check that gluster split-brain command hasn't reported any >> split-brains but resulted in IO errors when accessed few files. >> >> Could you please tell me if "split-brain" command doesn't reported any >> entry as output, then is there any way through which we can find out that >> the files are in split-brain if we are getting the IO error on those file. >> >> >> Regards, >> Abhishek >> >> >> >> On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <atalur at redhat.com> wrote: >> >>> >>> >>> ----- Original Message ----- >>> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >>> > To: gluster-users at gluster.org, gluster-devel at gluster.org >>> > Sent: Thursday, March 3, 2016 12:10:42 PM >>> > Subject: [Gluster-users] gluster volume heal info split brain command >>> not showing files in split-brain >>> > >>> > >>> > Hello, >>> > >>> > In gluster, we use the command "gluster volume heal c_glusterfs info >>> > split-brain" to find the files that are in split-brain scenario. >>> > We run heal script (developed by Windriver prime team) on the files >>> reported >>> > by above command to resolve split-brain issue. >>> > >>> > But we observed that the above command is not showing all files that >>> are in >>> > split-brain, >>> > even though split brain scenario actually exists on the node. >>> > >>> > Now a days this issue is seen more often and IO errors are reported >>> when >>> > tried to access these files under split-brain. >>> > >>> > Can you please check why this gluster command is not showing files >>> under >>> > split-brain? >>> > We can provide you required logs and support to resolve this issue. >>> Hi, >>> >>> Could you paste the output of getfattr -m. -de hex >>> <path-to-files-in-split-brain> from all the bricks that the files lie in? >>> >>> > >>> > Please reply on this because I am not getting any reply from the >>> community. >>> > >>> > -- >>> > >>> > Regards >>> > Abhishek Paliwal >>> > >>> > _______________________________________________ >>> > Gluster-users mailing list >>> > Gluster-users at gluster.org >>> > http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> -- >>> Thanks, >>> Anuradha. >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160314/f68c6e73/attachment.html>
Anuradha Talur
2016-Mar-16 10:50 UTC
[Gluster-users] gluster volume heal info split brain command not showing files in split-brain
Hi Abhishek, The files that are reporting i/o error have gfid-mismatch. This situation is called directory or entry split-brain. You can find steps to resolve this kind of split brain here : https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ . Ideally, the parent directories of these files have to be listed in heal info split-brain output. Can you please get extended attributes of parent directories of the files that show i/o error (Same getfattr command that you previously used.) ? ----- Original Message -----> From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> > To: "Anuradha Talur" <atalur at redhat.com> > Cc: gluster-users at gluster.org, gluster-devel at gluster.org > Sent: Thursday, March 10, 2016 11:22:35 AM > Subject: Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain > > Hi Anuradha, > > Please find the glusterfs and glusterd logs directory as an attachment. > > Regards, > Abhishek > > > > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <abhishpaliwal at gmail.com> > wrote: > > > Hi Anuradha, > > > > Sorry for late reply. > > > > Please find the requested logs below: > > > > Remote: 10.32.0.48 > > Local : 10.32.1.144 > > > > Local: > > #gluster volume heal c_glusterfs info split-brain > > Brick 10.32.1.144:/opt/lvmdir/c2/brick > > Number of entries in split-brain: 0 > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick > > Number of entries in split-brain: 0 > > > > Remote: > > #gluster volume heal c_glusterfs info split-brain > > Brick 10.32.1.144:/opt/lvmdir/c2/brick > > Number of entries in split-brain: 0 > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick > > Number of entries in split-brain: 0 > > > > auto-sync.sh. > > Here you can see that i/o error is detected. Below is the required meta > > data from both the bricks. > > > > 1) > > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': > > Input/output error > > Remote: > > > > getfattr -d -m . -e hex > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > trusted.afr.dirty=0x000000000000000000000000 > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f > > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee > > > > Local: > > > > getfattr -d -m . -e hex > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 > > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd > > > > > > 2) > > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error > > Remote: > > > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity > > # file: opt/lvmdir/c2/brick/security/corbasecurity > > trusted.afr.dirty=0x000000000000000000000000 > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f > > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013 > > > > Local: > > > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity > > # file: opt/lvmdir/c2/brick/security/corbasecurity > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 > > trusted.gfid=0x890df0f706184b52803fac3242a2f15b > > > > I observed that getfattr command output doesn't show all the fields all > > the times. > > > > Here you can check that gluster split-brain command hasn't reported any > > split-brains but resulted in IO errors when accessed few files. > > > > Could you please tell me if "split-brain" command doesn't reported any > > entry as output, then is there any way through which we can find out that > > the files are in split-brain if we are getting the IO error on those file. > > > > > > Regards, > > Abhishek > > > > > > > > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <atalur at redhat.com> wrote: > > > >> > >> > >> ----- Original Message ----- > >> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> > >> > To: gluster-users at gluster.org, gluster-devel at gluster.org > >> > Sent: Thursday, March 3, 2016 12:10:42 PM > >> > Subject: [Gluster-users] gluster volume heal info split brain command > >> not showing files in split-brain > >> > > >> > > >> > Hello, > >> > > >> > In gluster, we use the command "gluster volume heal c_glusterfs info > >> > split-brain" to find the files that are in split-brain scenario. > >> > We run heal script (developed by Windriver prime team) on the files > >> reported > >> > by above command to resolve split-brain issue. > >> > > >> > But we observed that the above command is not showing all files that > >> are in > >> > split-brain, > >> > even though split brain scenario actually exists on the node. > >> > > >> > Now a days this issue is seen more often and IO errors are reported when > >> > tried to access these files under split-brain. > >> > > >> > Can you please check why this gluster command is not showing files under > >> > split-brain? > >> > We can provide you required logs and support to resolve this issue. > >> Hi, > >> > >> Could you paste the output of getfattr -m. -de hex > >> <path-to-files-in-split-brain> from all the bricks that the files lie in? > >> > >> > > >> > Please reply on this because I am not getting any reply from the > >> community. > >> > > >> > -- > >> > > >> > Regards > >> > Abhishek Paliwal > >> > > >> > _______________________________________________ > >> > Gluster-users mailing list > >> > Gluster-users at gluster.org > >> > http://www.gluster.org/mailman/listinfo/gluster-users > >> > >> -- > >> Thanks, > >> Anuradha. > >> > > >-- Thanks, Anuradha.