Gaurav Garg
2016-Feb-19 11:33 UTC
[Gluster-users] Issue in Adding/Removing the gluster node
Hi Abhishek, How are you connecting two board, and how are you removing it manually that need to know because if you are removing your 2nd board from the cluster (abrupt shutdown) then you can't perform remove brick operation in 2nd node from first node and its happening successfully in your case. could you ensure your network connection once again while removing and bringing back your node again. Thanks, Gaurav ----- Original Message ----- From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> To: "Gaurav Garg" <ggarg at redhat.com> Cc: gluster-users at gluster.org Sent: Friday, February 19, 2016 3:36:21 PM Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node Hi Gaurav, Thanks for reply 1. Here, I removed the board manually here but this time it works fine [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS Yes this time board is reachable but how? don't know because board is detached. 2. Here, I attached the board this time its works fine in add-bricks 2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS 3.Here, again I removed the board this time failed occur [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick 10.32.1.144:/opt /lvmdir/c2/brick for volume c_glusterfs but here board is not reachable. why this inconsistency is there while doing the same step multiple time. Hope you are getting my point. Regards, Abhishek On Fri, Feb 19, 2016 at 3:25 PM, Gaurav Garg < ggarg at redhat.com > wrote: Abhishek, when sometime its working fine means 2nd board network connection is reachable to first node. you can conform this by executing same #gluster peer status command. Thanks, Gaurav ----- Original Message ----- From: "ABHISHEK PALIWAL" < abhishpaliwal at gmail.com > To: "Gaurav Garg" < ggarg at redhat.com > Cc: gluster-users at gluster.org Sent: Friday, February 19, 2016 3:12:22 PM Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node Hi Gaurav, Yes, you are right actually I am force fully detaching the node from the slave and when we removed the board it disconnected from the another board. but my question is I am doing this process multiple time some time it works fine but some time it gave these errors. you can see the following logs from cmd_history.log file [2016-02-18 10:03:34.497996] : volume set c_glusterfs nfs.disable on : SUCCESS [2016-02-18 10:03:34.915036] : volume start c_glusterfs force : SUCCESS [2016-02-18 10:03:40.250326] : volume status : SUCCESS [2016-02-18 10:03:40.273275] : volume status : SUCCESS [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS [2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS [2016-02-18 10:30:53.297415] : volume status : SUCCESS [2016-02-18 10:30:53.313096] : volume status : SUCCESS [2016-02-18 10:37:02.748714] : volume status : SUCCESS [2016-02-18 10:37:02.762091] : volume status : SUCCESS [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs On Fri, Feb 19, 2016 at 3:05 PM, Gaurav Garg < ggarg at redhat.com > wrote:> Hi Abhishek, > > Seems your peer 10.32.1.144 have disconnected while doing remove brick. > see the below logs in glusterd: > > [2016-02-18 10:37:02.816009] E [MSGID: 106256] > [glusterd-brick-ops.c:1047:__glusterd_handle_remove_brick] 0-management: > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs > [Invalid argument] > [2016-02-18 10:37:02.816061] E [MSGID: 106265] > [glusterd-brick-ops.c:1088:__glusterd_handle_remove_brick] 0-management: > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs > The message "I [MSGID: 106004] > [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer > <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer in > Cluster>, has disconnected from glusterd." repeated 25 times between > [2016-02-18 10:35:43.131945] and [2016-02-18 10:36:58.160458] > > > > If you are facing the same issue now, could you paste your # gluster peer > status command output here. > > Thanks, > ~Gaurav > > ----- Original Message ----- > From: "ABHISHEK PALIWAL" < abhishpaliwal at gmail.com > > To: gluster-users at gluster.org > Sent: Friday, February 19, 2016 2:46:35 PM > Subject: [Gluster-users] Issue in Adding/Removing the gluster node > > Hi, > > > I am working on two board setup connecting to each other. Gluster version > 3.7.6 is running and added two bricks in replica 2 mode but when I manually > removed (detach) the one board from the setup I am getting the following > error. > > volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick > force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for > volume c_glusterfs > > Please find the logs file as an attachment. > > > Regards, > Abhishek > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-- Regards Abhishek Paliwal -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/bab3815d/attachment.html>
ABHISHEK PALIWAL
2016-Feb-19 11:51 UTC
[Gluster-users] Issue in Adding/Removing the gluster node
Hi Gaurav, Both are the board connect through the backplane using ethernet. Even this inconsistency also occurs when I am trying to bringing back the node in slot. Means some time add-brick executes without failure but some time following error occurs. volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Another transaction is in progress for c_glusterfs. Please try again after sometime. You can also see the attached logs for add-brick failure scenario. Please let me know if you need more logs. Regards, Abhishek On Fri, Feb 19, 2016 at 5:03 PM, Gaurav Garg <ggarg at redhat.com> wrote:> Hi Abhishek, > > How are you connecting two board, and how are you removing it manually > that need to know because if you are removing your 2nd board from the > cluster (abrupt shutdown) then you can't perform remove brick operation in > 2nd node from first node and its happening successfully in your case. could > you ensure your network connection once again while removing and bringing > back your node again. > > Thanks, > Gaurav > > ------------------------------ > *From: *"ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> > *To: *"Gaurav Garg" <ggarg at redhat.com> > *Cc: *gluster-users at gluster.org > *Sent: *Friday, February 19, 2016 3:36:21 PM > > *Subject: *Re: [Gluster-users] Issue in Adding/Removing the gluster node > > Hi Gaurav, > > Thanks for reply > > 1. Here, I removed the board manually here but this time it works fine > > [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 > 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS > [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS > > Yes this time board is reachable but how? don't know because board is > detached. > > 2. Here, I attached the board this time its works fine in add-bricks > > 2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS > [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 > 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS > > 3.Here, again I removed the board this time failed occur > > [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 > 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick > 10.32.1.144:/opt > /lvmdir/c2/brick for volume c_glusterfs > > but here board is not reachable. > > why this inconsistency is there while doing the same step multiple time. > > Hope you are getting my point. > > Regards, > Abhishek > > On Fri, Feb 19, 2016 at 3:25 PM, Gaurav Garg <ggarg at redhat.com> wrote: > >> Abhishek, >> >> when sometime its working fine means 2nd board network connection is >> reachable to first node. you can conform this by executing same #gluster >> peer status command. >> >> Thanks, >> Gaurav >> >> ----- Original Message ----- >> From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >> To: "Gaurav Garg" <ggarg at redhat.com> >> Cc: gluster-users at gluster.org >> Sent: Friday, February 19, 2016 3:12:22 PM >> Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node >> >> Hi Gaurav, >> >> Yes, you are right actually I am force fully detaching the node from the >> slave and when we removed the board it disconnected from the another >> board. >> >> but my question is I am doing this process multiple time some time it >> works >> fine but some time it gave these errors. >> >> >> you can see the following logs from cmd_history.log file >> >> [2016-02-18 10:03:34.497996] : volume set c_glusterfs nfs.disable on : >> SUCCESS >> [2016-02-18 10:03:34.915036] : volume start c_glusterfs force : SUCCESS >> [2016-02-18 10:03:40.250326] : volume status : SUCCESS >> [2016-02-18 10:03:40.273275] : volume status : SUCCESS >> [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS >> [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS >> [2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS >> [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS >> [2016-02-18 10:30:53.297415] : volume status : SUCCESS >> [2016-02-18 10:30:53.313096] : volume status : SUCCESS >> [2016-02-18 10:37:02.748714] : volume status : SUCCESS >> [2016-02-18 10:37:02.762091] : volume status : SUCCESS >> [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 >> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick >> 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs >> >> >> On Fri, Feb 19, 2016 at 3:05 PM, Gaurav Garg <ggarg at redhat.com> wrote: >> >> > Hi Abhishek, >> > >> > Seems your peer 10.32.1.144 have disconnected while doing remove brick. >> > see the below logs in glusterd: >> > >> > [2016-02-18 10:37:02.816009] E [MSGID: 106256] >> > [glusterd-brick-ops.c:1047:__glusterd_handle_remove_brick] 0-management: >> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs >> > [Invalid argument] >> > [2016-02-18 10:37:02.816061] E [MSGID: 106265] >> > [glusterd-brick-ops.c:1088:__glusterd_handle_remove_brick] 0-management: >> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs >> > The message "I [MSGID: 106004] >> > [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer >> > <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer >> in >> > Cluster>, has disconnected from glusterd." repeated 25 times between >> > [2016-02-18 10:35:43.131945] and [2016-02-18 10:36:58.160458] >> > >> > >> > >> > If you are facing the same issue now, could you paste your # gluster >> peer >> > status command output here. >> > >> > Thanks, >> > ~Gaurav >> > >> > ----- Original Message ----- >> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> >> > To: gluster-users at gluster.org >> > Sent: Friday, February 19, 2016 2:46:35 PM >> > Subject: [Gluster-users] Issue in Adding/Removing the gluster node >> > >> > Hi, >> > >> > >> > I am working on two board setup connecting to each other. Gluster >> version >> > 3.7.6 is running and added two bricks in replica 2 mode but when I >> manually >> > removed (detach) the one board from the setup I am getting the following >> > error. >> > >> > volume remove-brick c_glusterfs replica 1 10.32.1.144: >> /opt/lvmdir/c2/brick >> > force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for >> > volume c_glusterfs >> > >> > Please find the logs file as an attachment. >> > >> > >> > Regards, >> > Abhishek >> > >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> > >> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> > > > > -- > > > > > Regards > Abhishek Paliwal > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: cmd_history.log Type: text/x-log Size: 21439 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0003.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: cli.log Type: text/x-log Size: 71308 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0004.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: etc-glusterfs-glusterd.vol.log Type: text/x-log Size: 111061 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0005.bin>