Hi Friends, Please note that, when replace-brick operation was tried for one of the bad brick present in distributed disperse EC volume, the command actually failed but the brick daemon of new replaced brick came online. Please help to understand in what situations this issue may arise and proposed solution if possible ? : glusterd.log : [2018-12-11 11:04:43.774120] I [MSGID: 106503] [glusterd-replace-brick.c:147:__glusterd_handle_replace_brick] 0-management: Received replace-brick commit force request. [2018-12-11 11:04:44.784578] I [MSGID: 106504] [glusterd-utils.c:13079:rb_update_dstbrick_port] 0-glusterd: adding dst-brick port no 0 ... [2018-12-11 11:04:46.457537] E [MSGID: 106029] [glusterd-utils.c:7981:glusterd_brick_signal] 0-glusterd: Unable to open pidfile: /var/run/gluster/vols/AM6_HyperScale/am6sv0004sds.saipemnet.saipem.intranet-ws-disk3-ws_brick.pid [No such file or directory] [2018-12-11 11:04:53.089810] I [glusterd-utils.c:5876:glusterd_brick_start] 0-management: starting a fresh brick process for brick /ws/disk15/ws_brick ... [2018-12-11 11:04:53.117935] W [socket.c:595:__socket_rwv] 0-socket.management: writev on 127.0.0.1:864 failed (Broken pipe) [2018-12-11 11:04:54.014023] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2018-12-11 11:04:54.273190] I [MSGID: 106005] [glusterd-handler.c:6120:__glusterd_brick_rpc_notify] 0-management: Brick am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick has disconnected from glusterd. [2018-12-11 11:04:54.297603] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Commit failed on am6sv0006sds.saipemnet.saipem.intranet. Please check log file for details. [2018-12-11 11:04:54.350666] I [MSGID: 106143] [glusterd-pmap.c:278:pmap_registry_bind] 0-pmap: adding brick /ws/disk15/ws_brick on port 49164 [2018-12-11 11:05:01.137449] E [MSGID: 106123] [glusterd-mgmt.c:1519:glusterd_mgmt_v3_commit] 0-management: Commit failed on peers [2018-12-11 11:05:01.137496] E [MSGID: 106123] [glusterd-replace-brick.c:660:glusterd_mgmt_v3_initiate_replace_brick_cmd_phases] 0-management: Commit Op Failed [2018-12-11 11:06:12.275867] I [MSGID: 106499] [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: Received status volume req for volume AM6_HyperScale [2018-12-11 13:35:51.529365] I [MSGID: 106499] [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: Received status volume req for volume AM6_HyperScale gluster volume replace-brick AM6_HyperScale am6sv0004sds.saipemnet.saipem.intranet:/ws/disk3/ws_brick am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick commit force Replace brick failure, brick [/ws/disk3], volume [AM6_HyperScale] "gluster volume status" now shows a new disk active /ws/disk15 The replacement appears to be successful, looks like healing started [cid:image001.png at 01D49944.02DC60D0] Thanks and Regards, --Anand ********************************Legal Disclaimer******************************** "This communication may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message by mistake, please advise the sender by reply email and delete the message. We may process information in the email header of business emails sent and received by us (including the names of recipient and sender, date and time of the email) for the purposes of evaluating our existing or prospective business relationship. The lawful basis we rely on for this processing is our legitimate interests. For more information about how we use personal information please read our privacy policy https://www.commvault.com/privacy-policy. Thank you." ******************************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181221/60c76d6c/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 2727 bytes Desc: image001.png URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181221/60c76d6c/attachment.png>
On Fri, 21 Dec 2018 at 15:54, Anand Malagi <amalagi at commvault.com> wrote:> Hi Friends, > > > > Please note that, when replace-brick operation was tried for one of the > bad brick present in distributed disperse EC volume, the command actually > failed but the brick daemon of new replaced brick came online. >This is probably because the new brick was already added before a step in the transaction failed and unfortunately there?s no rollback mechanism in place to handle such situation. Please help to understand in what situations this issue may arise and> proposed solution if possible ? : > > > > *glusterd.log :* > > > > [2018-12-11 11:04:43.774120] I [MSGID: 106503] > [glusterd-replace-brick.c:147:__glusterd_handle_replace_brick] > 0-management: Received replace-brick commit force request. > > [2018-12-11 11:04:44.784578] I [MSGID: 106504] > [glusterd-utils.c:13079:rb_update_dstbrick_port] 0-glusterd: adding > dst-brick port no 0 > > ? > > [2018-12-11 11:04:46.457537] E [MSGID: 106029] > [glusterd-utils.c:7981:glusterd_brick_signal] 0-glusterd: Unable to open > pidfile: > /var/run/gluster/vols/AM6_HyperScale/am6sv0004sds.saipemnet.saipem.intranet-ws-disk3-ws_brick.pid > [No such file or directory] > > [2018-12-11 11:04:53.089810] I > [glusterd-utils.c:5876:glusterd_brick_start] 0-management: starting a fresh > brick process for brick /ws/disk15/ws_brick > > ? > > [2018-12-11 11:04:53.117935] W [socket.c:595:__socket_rwv] > 0-socket.management: writev on 127.0.0.1:864 failed (Broken pipe) > > [2018-12-11 11:04:54.014023] I [socket.c:2465:socket_event_handler] > 0-transport: EPOLLERR - disconnecting now > > [2018-12-11 11:04:54.273190] I [MSGID: 106005] > [glusterd-handler.c:6120:__glusterd_brick_rpc_notify] 0-management: Brick > am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick has disconnected > from glusterd. > > [2018-12-11 11:04:54.297603] E [MSGID: 106116] > [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Commit failed > on am6sv0006sds.saipemnet.saipem.intranet. Please check log file for > details. > > [2018-12-11 11:04:54.350666] I [MSGID: 106143] > [glusterd-pmap.c:278:pmap_registry_bind] 0-pmap: adding brick > /ws/disk15/ws_brick on port 49164 > > [2018-12-11 11:05:01.137449] E [MSGID: 106123] > [glusterd-mgmt.c:1519:glusterd_mgmt_v3_commit] 0-management: Commit failed > on peers >I?m assuming that this operation actually failed on non local node, did you check the glusterd log of the node where this operation failed? [2018-12-11 11:05:01.137496] E [MSGID: 106123]> [glusterd-replace-brick.c:660:glusterd_mgmt_v3_initiate_replace_brick_cmd_phases] > 0-management: Commit Op Failed > > [2018-12-11 11:06:12.275867] I [MSGID: 106499] > [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume AM6_HyperScale > > [2018-12-11 13:35:51.529365] I [MSGID: 106499] > [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume AM6_HyperScale > > > > > > gluster volume replace-brick AM6_HyperScale > am6sv0004sds.saipemnet.saipem.intranet:/ws/disk3/ws_brick > am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick commit force > > Replace brick failure, brick [/ws/disk3], volume [AM6_HyperScale] > > > > ?gluster volume status? now shows a new disk active /ws/disk15 > > The replacement appears to be successful, looks like healing started > > > > > > > > Thanks and Regards, > > --Anand > ********************************Legal > Disclaimer******************************** > "This communication may contain confidential and privileged material for > the > sole use of the intended recipient. Any unauthorized review, use or > distribution > by others is strictly prohibited. If you have received the message by > mistake, > please advise the sender by reply email and delete the message. We may > process > information in the email header of business emails sent and received by us > (including the names of recipient and sender, date and time of the email) > for > the purposes of evaluating our existing or prospective business > relationship. > The lawful basis we rely on for this processing is our legitimate > interests. For > more information about how we use personal information please read our > privacy > policy https://www.commvault.com/privacy-policy. Thank you." > > ******************************************************************************** > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-- - Atin (atinm) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181224/008cf233/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 2727 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181224/008cf233/attachment.png>
Can someone please help here ?? From: gluster-users-bounces at gluster.org <gluster-users-bounces at gluster.org> On Behalf Of Anand Malagi Sent: Friday, December 21, 2018 3:44 PM To: gluster-users at gluster.org Subject: [Gluster-users] replace-brick operation issue... Hi Friends, Please note that, when replace-brick operation was tried for one of the bad brick present in distributed disperse EC volume, the command actually failed but the brick daemon of new replaced brick came online. Please help to understand in what situations this issue may arise and proposed solution if possible ? : glusterd.log : [2018-12-11 11:04:43.774120] I [MSGID: 106503] [glusterd-replace-brick.c:147:__glusterd_handle_replace_brick] 0-management: Received replace-brick commit force request. [2018-12-11 11:04:44.784578] I [MSGID: 106504] [glusterd-utils.c:13079:rb_update_dstbrick_port] 0-glusterd: adding dst-brick port no 0 ... [2018-12-11 11:04:46.457537] E [MSGID: 106029] [glusterd-utils.c:7981:glusterd_brick_signal] 0-glusterd: Unable to open pidfile: /var/run/gluster/vols/AM6_HyperScale/am6sv0004sds.saipemnet.saipem.intranet-ws-disk3-ws_brick.pid [No such file or directory] [2018-12-11 11:04:53.089810] I [glusterd-utils.c:5876:glusterd_brick_start] 0-management: starting a fresh brick process for brick /ws/disk15/ws_brick ... [2018-12-11 11:04:53.117935] W [socket.c:595:__socket_rwv] 0-socket.management: writev on 127.0.0.1:864 failed (Broken pipe) [2018-12-11 11:04:54.014023] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2018-12-11 11:04:54.273190] I [MSGID: 106005] [glusterd-handler.c:6120:__glusterd_brick_rpc_notify] 0-management: Brick am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick has disconnected from glusterd. [2018-12-11 11:04:54.297603] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Commit failed on am6sv0006sds.saipemnet.saipem.intranet. Please check log file for details. [2018-12-11 11:04:54.350666] I [MSGID: 106143] [glusterd-pmap.c:278:pmap_registry_bind] 0-pmap: adding brick /ws/disk15/ws_brick on port 49164 [2018-12-11 11:05:01.137449] E [MSGID: 106123] [glusterd-mgmt.c:1519:glusterd_mgmt_v3_commit] 0-management: Commit failed on peers [2018-12-11 11:05:01.137496] E [MSGID: 106123] [glusterd-replace-brick.c:660:glusterd_mgmt_v3_initiate_replace_brick_cmd_phases] 0-management: Commit Op Failed [2018-12-11 11:06:12.275867] I [MSGID: 106499] [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: Received status volume req for volume AM6_HyperScale [2018-12-11 13:35:51.529365] I [MSGID: 106499] [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: Received status volume req for volume AM6_HyperScale gluster volume replace-brick AM6_HyperScale am6sv0004sds.saipemnet.saipem.intranet:/ws/disk3/ws_brick am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick commit force Replace brick failure, brick [/ws/disk3], volume [AM6_HyperScale] "gluster volume status" now shows a new disk active /ws/disk15 The replacement appears to be successful, looks like healing started [cid:image001.png at 01D4A10E.2C4DFE70] Thanks and Regards, --Anand ********************************Legal Disclaimer******************************** "This communication may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message by mistake, please advise the sender by reply email and delete the message. We may process information in the email header of business emails sent and received by us (including the names of recipient and sender, date and time of the email) for the purposes of evaluating our existing or prospective business relationship. The lawful basis we rely on for this processing is our legitimate interests. For more information about how we use personal information please read our privacy policy https://www.commvault.com/privacy-policy. Thank you." ******************************************************************************** ********************************Legal Disclaimer******************************** "This communication may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message by mistake, please advise the sender by reply email and delete the message. We may process information in the email header of business emails sent and received by us (including the names of recipient and sender, date and time of the email) for the purposes of evaluating our existing or prospective business relationship. The lawful basis we rely on for this processing is our legitimate interests. For more information about how we use personal information please read our privacy policy https://www.commvault.com/privacy-policy. Thank you." ******************************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181231/99ef7100/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 2727 bytes Desc: image001.png URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181231/99ef7100/attachment.png>
Can I please get some help in understanding the issue mentioned ? From: Anand Malagi Sent: Monday, December 31, 2018 1:39 PM To: 'Anand Malagi' <amalagi at commvault.com>; gluster-users at gluster.org Subject: RE: replace-brick operation issue... Can someone please help here ?? From: gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org> <gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org>> On Behalf Of Anand Malagi Sent: Friday, December 21, 2018 3:44 PM To: gluster-users at gluster.org<mailto:gluster-users at gluster.org> Subject: [Gluster-users] replace-brick operation issue... Hi Friends, Please note that, when replace-brick operation was tried for one of the bad brick present in distributed disperse EC volume, the command actually failed but the brick daemon of new replaced brick came online. Please help to understand in what situations this issue may arise and proposed solution if possible ? : glusterd.log : [2018-12-11 11:04:43.774120] I [MSGID: 106503] [glusterd-replace-brick.c:147:__glusterd_handle_replace_brick] 0-management: Received replace-brick commit force request. [2018-12-11 11:04:44.784578] I [MSGID: 106504] [glusterd-utils.c:13079:rb_update_dstbrick_port] 0-glusterd: adding dst-brick port no 0 ... [2018-12-11 11:04:46.457537] E [MSGID: 106029] [glusterd-utils.c:7981:glusterd_brick_signal] 0-glusterd: Unable to open pidfile: /var/run/gluster/vols/AM6_HyperScale/am6sv0004sds.saipemnet.saipem.intranet-ws-disk3-ws_brick.pid [No such file or directory] [2018-12-11 11:04:53.089810] I [glusterd-utils.c:5876:glusterd_brick_start] 0-management: starting a fresh brick process for brick /ws/disk15/ws_brick ... [2018-12-11 11:04:53.117935] W [socket.c:595:__socket_rwv] 0-socket.management: writev on 127.0.0.1:864 failed (Broken pipe) [2018-12-11 11:04:54.014023] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2018-12-11 11:04:54.273190] I [MSGID: 106005] [glusterd-handler.c:6120:__glusterd_brick_rpc_notify] 0-management: Brick am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick has disconnected from glusterd. [2018-12-11 11:04:54.297603] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Commit failed on am6sv0006sds.saipemnet.saipem.intranet. Please check log file for details. [2018-12-11 11:04:54.350666] I [MSGID: 106143] [glusterd-pmap.c:278:pmap_registry_bind] 0-pmap: adding brick /ws/disk15/ws_brick on port 49164 [2018-12-11 11:05:01.137449] E [MSGID: 106123] [glusterd-mgmt.c:1519:glusterd_mgmt_v3_commit] 0-management: Commit failed on peers [2018-12-11 11:05:01.137496] E [MSGID: 106123] [glusterd-replace-brick.c:660:glusterd_mgmt_v3_initiate_replace_brick_cmd_phases] 0-management: Commit Op Failed [2018-12-11 11:06:12.275867] I [MSGID: 106499] [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: Received status volume req for volume AM6_HyperScale [2018-12-11 13:35:51.529365] I [MSGID: 106499] [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: Received status volume req for volume AM6_HyperScale gluster volume replace-brick AM6_HyperScale am6sv0004sds.saipemnet.saipem.intranet:/ws/disk3/ws_brick am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick commit force Replace brick failure, brick [/ws/disk3], volume [AM6_HyperScale] "gluster volume status" now shows a new disk active /ws/disk15 The replacement appears to be successful, looks like healing started [cid:image001.png at 01D4A84C.A15F8680] Thanks and Regards, --Anand ********************************Legal Disclaimer******************************** "This communication may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message by mistake, please advise the sender by reply email and delete the message. We may process information in the email header of business emails sent and received by us (including the names of recipient and sender, date and time of the email) for the purposes of evaluating our existing or prospective business relationship. The lawful basis we rely on for this processing is our legitimate interests. For more information about how we use personal information please read our privacy policy https://www.commvault.com/privacy-policy. Thank you." ******************************************************************************** ********************************Legal Disclaimer******************************** "This communication may contain confidential and privileged material for the sole use of the intended recipient. Any unauthorized review, use or distribution by others is strictly prohibited. If you have received the message by mistake, please advise the sender by reply email and delete the message. We may process information in the email header of business emails sent and received by us (including the names of recipient and sender, date and time of the email) for the purposes of evaluating our existing or prospective business relationship. The lawful basis we rely on for this processing is our legitimate interests. For more information about how we use personal information please read our privacy policy https://www.commvault.com/privacy-policy. Thank you." ******************************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190109/a0a9fba2/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 2727 bytes Desc: image001.png URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190109/a0a9fba2/attachment.png>