ABHISHEK PALIWAL
2016-Feb-19 11:51 UTC
[Gluster-users] Issue in Adding/Removing the gluster node
Hi Gaurav, Both are the board connect through the backplane using ethernet. Even this inconsistency also occurs when I am trying to bringing back the node in slot. Means some time add-brick executes without failure but some time following error occurs. volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Another transaction is in progress for c_glusterfs. Please try again after sometime. You can also see the attached logs for add-brick failure scenario. Please let me know if you need more logs. Regards, Abhishek On Fri, Feb 19, 2016 at 5:03 PM, Gaurav Garg <ggarg at redhat.com> wrote:> Hi Abhishek, > > How are you connecting two board, and how are you removing it manually > that need to know because if you are removing your 2nd board from the > cluster (abrupt shutdown) then you can't perform remove brick operation in > 2nd node from first node and its happening successfully in your case. could > you ensure your network connection once again while removing and bringing > back your node again. > > Thanks, > Gaurav > > ------------------------------ > *From: *"ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> > *To: *"Gaurav Garg" <ggarg at redhat.com> > *Cc: *gluster-users at gluster.org > *Sent: *Friday, February 19, 2016 3:36:21 PM > > *Subject: *Re: [Gluster-users] Issue in Adding/Removing the gluster node > > Hi Gaurav, > > Thanks for reply > > 1. Here, I removed the board manually here but this time it works fine > > [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 > 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS > [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS > > Yes this time board is reachable but how? don't know because board is > detached. > > 2. Here, I attached the board this time its works fine in add-bricks > > 2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS > [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 > 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS > > 3.Here, again I removed the board this time failed occur > > [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 > 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick > 10.32.1.144:/opt > /lvmdir/c2/brick for volume c_glusterfs > > but here board is not reachable. > > why this inconsistency is there while doing the same step multiple time. > > Hope you are getting my point. > > Regards, > Abhishek > > On Fri, Feb 19, 2016 at 3:25 PM, Gaurav Garg <ggarg at redhat.com> wrote: > >> Abhishek, >> >> when sometime its working fine means 2nd board network connection is >> reachable to first node. you can conform this by executing same #gluster >> peer status command. >> >> Thanks, >> Gaurav >> >> ----- Original Message ----- >> From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >> To: "Gaurav Garg" <ggarg at redhat.com> >> Cc: gluster-users at gluster.org >> Sent: Friday, February 19, 2016 3:12:22 PM >> Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node >> >> Hi Gaurav, >> >> Yes, you are right actually I am force fully detaching the node from the >> slave and when we removed the board it disconnected from the another >> board. >> >> but my question is I am doing this process multiple time some time it >> works >> fine but some time it gave these errors. >> >> >> you can see the following logs from cmd_history.log file >> >> [2016-02-18 10:03:34.497996] : volume set c_glusterfs nfs.disable on : >> SUCCESS >> [2016-02-18 10:03:34.915036] : volume start c_glusterfs force : SUCCESS >> [2016-02-18 10:03:40.250326] : volume status : SUCCESS >> [2016-02-18 10:03:40.273275] : volume status : SUCCESS >> [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS >> [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS >> [2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS >> [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS >> [2016-02-18 10:30:53.297415] : volume status : SUCCESS >> [2016-02-18 10:30:53.313096] : volume status : SUCCESS >> [2016-02-18 10:37:02.748714] : volume status : SUCCESS >> [2016-02-18 10:37:02.762091] : volume status : SUCCESS >> [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 >> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick >> 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs >> >> >> On Fri, Feb 19, 2016 at 3:05 PM, Gaurav Garg <ggarg at redhat.com> wrote: >> >> > Hi Abhishek, >> > >> > Seems your peer 10.32.1.144 have disconnected while doing remove brick. >> > see the below logs in glusterd: >> > >> > [2016-02-18 10:37:02.816009] E [MSGID: 106256] >> > [glusterd-brick-ops.c:1047:__glusterd_handle_remove_brick] 0-management: >> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs >> > [Invalid argument] >> > [2016-02-18 10:37:02.816061] E [MSGID: 106265] >> > [glusterd-brick-ops.c:1088:__glusterd_handle_remove_brick] 0-management: >> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs >> > The message "I [MSGID: 106004] >> > [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer >> > <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer >> in >> > Cluster>, has disconnected from glusterd." repeated 25 times between >> > [2016-02-18 10:35:43.131945] and [2016-02-18 10:36:58.160458] >> > >> > >> > >> > If you are facing the same issue now, could you paste your # gluster >> peer >> > status command output here. >> > >> > Thanks, >> > ~Gaurav >> > >> > ----- Original Message ----- >> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >> > To: gluster-users at gluster.org >> > Sent: Friday, February 19, 2016 2:46:35 PM >> > Subject: [Gluster-users] Issue in Adding/Removing the gluster node >> > >> > Hi, >> > >> > >> > I am working on two board setup connecting to each other. Gluster >> version >> > 3.7.6 is running and added two bricks in replica 2 mode but when I >> manually >> > removed (detach) the one board from the setup I am getting the following >> > error. >> > >> > volume remove-brick c_glusterfs replica 1 10.32.1.144: >> /opt/lvmdir/c2/brick >> > force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for >> > volume c_glusterfs >> > >> > Please find the logs file as an attachment. >> > >> > >> > Regards, >> > Abhishek >> > >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> > >> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> > > > > -- > > > > > Regards > Abhishek Paliwal > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: cmd_history.log Type: text/x-log Size: 21439 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0003.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: cli.log Type: text/x-log Size: 71308 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0004.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: etc-glusterfs-glusterd.vol.log Type: text/x-log Size: 111061 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0005.bin>
ABHISHEK PALIWAL
2016-Feb-19 11:57 UTC
[Gluster-users] Issue in Adding/Removing the gluster node
Hi Gaurav, After the failure of add-brick following is outcome "gluster peer status" command Number of Peers: 2 Hostname: 10.32.1.144 Uuid: bbe2a458-ad3d-406d-b233-b6027c12174e State: Peer in Cluster (Connected) Hostname: 10.32.1.144 Uuid: bbe2a458-ad3d-406d-b233-b6027c12174e State: Peer in Cluster (Connected) Regards, Abhishek On Fri, Feb 19, 2016 at 5:21 PM, ABHISHEK PALIWAL <abhishpaliwal at gmail.com> wrote:> Hi Gaurav, > > Both are the board connect through the backplane using ethernet. > > Even this inconsistency also occurs when I am trying to bringing back the > node in slot. Means some time add-brick executes without failure but some > time following error occurs. > > volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick > force : FAILED : Another transaction is in progress for c_glusterfs. Please > try again after sometime. > > > You can also see the attached logs for add-brick failure scenario. > > Please let me know if you need more logs. > > Regards, > Abhishek > > > On Fri, Feb 19, 2016 at 5:03 PM, Gaurav Garg <ggarg at redhat.com> wrote: > >> Hi Abhishek, >> >> How are you connecting two board, and how are you removing it manually >> that need to know because if you are removing your 2nd board from the >> cluster (abrupt shutdown) then you can't perform remove brick operation in >> 2nd node from first node and its happening successfully in your case. could >> you ensure your network connection once again while removing and bringing >> back your node again. >> >> Thanks, >> Gaurav >> >> ------------------------------ >> *From: *"ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >> *To: *"Gaurav Garg" <ggarg at redhat.com> >> *Cc: *gluster-users at gluster.org >> *Sent: *Friday, February 19, 2016 3:36:21 PM >> >> *Subject: *Re: [Gluster-users] Issue in Adding/Removing the gluster node >> >> Hi Gaurav, >> >> Thanks for reply >> >> 1. Here, I removed the board manually here but this time it works fine >> >> [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS >> [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS >> >> Yes this time board is reachable but how? don't know because board is >> detached. >> >> 2. Here, I attached the board this time its works fine in add-bricks >> >> 2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS >> [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS >> >> 3.Here, again I removed the board this time failed occur >> >> [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 >> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick >> 10.32.1.144:/opt >> /lvmdir/c2/brick for volume c_glusterfs >> >> but here board is not reachable. >> >> why this inconsistency is there while doing the same step multiple time. >> >> Hope you are getting my point. >> >> Regards, >> Abhishek >> >> On Fri, Feb 19, 2016 at 3:25 PM, Gaurav Garg <ggarg at redhat.com> wrote: >> >>> Abhishek, >>> >>> when sometime its working fine means 2nd board network connection is >>> reachable to first node. you can conform this by executing same #gluster >>> peer status command. >>> >>> Thanks, >>> Gaurav >>> >>> ----- Original Message ----- >>> From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >>> To: "Gaurav Garg" <ggarg at redhat.com> >>> Cc: gluster-users at gluster.org >>> Sent: Friday, February 19, 2016 3:12:22 PM >>> Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node >>> >>> Hi Gaurav, >>> >>> Yes, you are right actually I am force fully detaching the node from the >>> slave and when we removed the board it disconnected from the another >>> board. >>> >>> but my question is I am doing this process multiple time some time it >>> works >>> fine but some time it gave these errors. >>> >>> >>> you can see the following logs from cmd_history.log file >>> >>> [2016-02-18 10:03:34.497996] : volume set c_glusterfs nfs.disable on : >>> SUCCESS >>> [2016-02-18 10:03:34.915036] : volume start c_glusterfs force : SUCCESS >>> [2016-02-18 10:03:40.250326] : volume status : SUCCESS >>> [2016-02-18 10:03:40.273275] : volume status : SUCCESS >>> [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 >>> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS >>> [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS >>> [2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS >>> [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 >>> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS >>> [2016-02-18 10:30:53.297415] : volume status : SUCCESS >>> [2016-02-18 10:30:53.313096] : volume status : SUCCESS >>> [2016-02-18 10:37:02.748714] : volume status : SUCCESS >>> [2016-02-18 10:37:02.762091] : volume status : SUCCESS >>> [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 >>> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick >>> 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs >>> >>> >>> On Fri, Feb 19, 2016 at 3:05 PM, Gaurav Garg <ggarg at redhat.com> wrote: >>> >>> > Hi Abhishek, >>> > >>> > Seems your peer 10.32.1.144 have disconnected while doing remove brick. >>> > see the below logs in glusterd: >>> > >>> > [2016-02-18 10:37:02.816009] E [MSGID: 106256] >>> > [glusterd-brick-ops.c:1047:__glusterd_handle_remove_brick] >>> 0-management: >>> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume >>> c_glusterfs >>> > [Invalid argument] >>> > [2016-02-18 10:37:02.816061] E [MSGID: 106265] >>> > [glusterd-brick-ops.c:1088:__glusterd_handle_remove_brick] >>> 0-management: >>> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume >>> c_glusterfs >>> > The message "I [MSGID: 106004] >>> > [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer >>> > <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer >>> in >>> > Cluster>, has disconnected from glusterd." repeated 25 times between >>> > [2016-02-18 10:35:43.131945] and [2016-02-18 10:36:58.160458] >>> > >>> > >>> > >>> > If you are facing the same issue now, could you paste your # gluster >>> peer >>> > status command output here. >>> > >>> > Thanks, >>> > ~Gaurav >>> > >>> > ----- Original Message ----- >>> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >>> > To: gluster-users at gluster.org >>> > Sent: Friday, February 19, 2016 2:46:35 PM >>> > Subject: [Gluster-users] Issue in Adding/Removing the gluster node >>> > >>> > Hi, >>> > >>> > >>> > I am working on two board setup connecting to each other. Gluster >>> version >>> > 3.7.6 is running and added two bricks in replica 2 mode but when I >>> manually >>> > removed (detach) the one board from the setup I am getting the >>> following >>> > error. >>> > >>> > volume remove-brick c_glusterfs replica 1 10.32.1.144: >>> /opt/lvmdir/c2/brick >>> > force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for >>> > volume c_glusterfs >>> > >>> > Please find the logs file as an attachment. >>> > >>> > >>> > Regards, >>> > Abhishek >>> > >>> > >>> > _______________________________________________ >>> > Gluster-users mailing list >>> > Gluster-users at gluster.org >>> > http://www.gluster.org/mailman/listinfo/gluster-users >>> > >>> >>> >>> >>> -- >>> >>> >>> >>> >>> Regards >>> Abhishek Paliwal >>> >> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> >> > > > >-- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/3a729fe9/attachment.html>