Thomas Bätzler
2015-Oct-30 11:26 UTC
[Gluster-users] Replacing a node in a 4x2 distributed/replicated setup
Hi, can somebody help me with fixing our 8 node gluster please? Setup is as follows: root at glucfshead2:~# gluster volume info Volume Name: archive Type: Distributed-Replicate Volume ID: d888b302-2a35-4559-9bb0-4e182f49f9c6 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: glucfshead1:/data/glusterfs/archive/brick1 Brick2: glucfshead5:/data/glusterfs/archive/brick1 Brick3: glucfshead2:/data/glusterfs/archive/brick1 Brick4: glucfshead6:/data/glusterfs/archive/brick1 Brick5: glucfshead3:/data/glusterfs/archive/brick1 Brick6: glucfshead7:/data/glusterfs/archive/brick1 Brick7: glucfshead4:/data/glusterfs/archive/brick1 Brick8: glucfshead8:/data/glusterfs/archive/brick1 Options Reconfigured: cluster.data-self-heal: off cluster.entry-self-heal: off cluster.metadata-self-heal: off features.lock-heal: on cluster.readdir-optimize: on performance.flush-behind: off performance.io-thread-count: 16 features.quota: off performance.quick-read: on performance.stat-prefetch: off performance.io-cache: on performance.cache-refresh-timeout: 1 nfs.disable: on performance.cache-max-file-size: 200kb performance.cache-size: 2GB performance.write-behind-window-size: 4MB performance.read-ahead: off storage.linux-aio: off diagnostics.brick-sys-log-level: WARNING cluster.self-heal-daemon: off Volume Name: archive2 Type: Distributed-Replicate Volume ID: 0fe86e42-e67f-46d8-8ed0-d0e34f539d69 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: glucfshead1:/data/glusterfs/archive2/brick1 Brick2: glucfshead5:/data/glusterfs/archive2/brick1 Brick3: glucfshead2:/data/glusterfs/archive2/brick1 Brick4: glucfshead6:/data/glusterfs/archive2/brick1 Brick5: glucfshead3:/data/glusterfs/archive2/brick1 Brick6: glucfshead7:/data/glusterfs/archive2/brick1 Brick7: glucfshead4:/data/glusterfs/archive2/brick1 Brick8: glucfshead8:/data/glusterfs/archive2/brick1 Options Reconfigured: cluster.metadata-self-heal: off cluster.entry-self-heal: off cluster.data-self-heal: off diagnostics.count-fop-hits: on diagnostics.latency-measurement: on features.lock-heal: on diagnostics.brick-sys-log-level: WARNING storage.linux-aio: off performance.read-ahead: off performance.write-behind-window-size: 4MB performance.cache-size: 2GB performance.cache-max-file-size: 200kb nfs.disable: on performance.cache-refresh-timeout: 1 performance.io-cache: on performance.stat-prefetch: off performance.quick-read: on features.quota: off performance.io-thread-count: 16 performance.flush-behind: off auth.allow: 172.16.15.* cluster.readdir-optimize: on cluster.self-heal-daemon: off Some time ago node, glucfshead1 broke down. After some fiddling it was decided not to deal with that immediately because the gluster was in production and a rebuild on 3.4 would basically render the gluster unusable. Recently it was felt that we needed to deal with the situation and we hired some experts to deal with the problem. So we reinstalled the broken node and gave it a new name/ip and upgraded all systems to 3.6.4. The plan was to probe the "new" node into the gluster and then do a brick-replace on it. However that did not go as expected. The node that we removed is now listed as "Peer Rejected": root at glucfshead2:~# gluster peer status Number of Peers: 7 Hostname: glucfshead1 Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf State: Peer Rejected (Disconnected) Hostname: glucfshead3 Uuid: a17ae95d-4598-4cd7-9ae7-808af10fedb5 State: Peer in Cluster (Connected) Hostname: glucfshead4 Uuid: 8547dadd-96bf-45fe-b49d-bab8f995c928 State: Peer in Cluster (Connected) Hostname: glucfshead5 Uuid: 249da8ea-fda6-47ff-98e0-dbff99dcb3f2 State: Peer in Cluster (Connected) Hostname: glucfshead6 Uuid: a0229511-978c-4904-87ae-7e1b32ac2c72 State: Peer in Cluster (Connected) Hostname: glucfshead7 Uuid: 548ec75a-0131-4c92-aaa9-7c6ee7b47a63 State: Peer in Cluster (Connected) Hostname: glucfshead8 Uuid: 5e54cbc1-482c-460b-ac38-00c4b71c50b9 State: Peer in Cluster (Connected) If I probe the replacement node (glucfshead9) it only ever shows up on one of my running nodes and it's in state "Rejected Peer (Connected)". How can we fix this - preferably without losing data? TIA, Thomas --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren gepr?ft. https://www.avast.com/antivirus
Gmail
2015-Oct-30 16:47 UTC
[Gluster-users] Replacing a node in a 4x2 distributed/replicated setup
try to restart glusterd on all the storage nodes. -Bishoy> On Oct 30, 2015, at 4:26 AM, Thomas B?tzler <t.baetzler at bringe.com> wrote: > > Hi, > > can somebody help me with fixing our 8 node gluster please? > > Setup is as follows: > > root at glucfshead2:~# gluster volume info > > Volume Name: archive > Type: Distributed-Replicate > Volume ID: d888b302-2a35-4559-9bb0-4e182f49f9c6 > Status: Started > Number of Bricks: 4 x 2 = 8 > Transport-type: tcp > Bricks: > Brick1: glucfshead1:/data/glusterfs/archive/brick1 > Brick2: glucfshead5:/data/glusterfs/archive/brick1 > Brick3: glucfshead2:/data/glusterfs/archive/brick1 > Brick4: glucfshead6:/data/glusterfs/archive/brick1 > Brick5: glucfshead3:/data/glusterfs/archive/brick1 > Brick6: glucfshead7:/data/glusterfs/archive/brick1 > Brick7: glucfshead4:/data/glusterfs/archive/brick1 > Brick8: glucfshead8:/data/glusterfs/archive/brick1 > Options Reconfigured: > cluster.data-self-heal: off > cluster.entry-self-heal: off > cluster.metadata-self-heal: off > features.lock-heal: on > cluster.readdir-optimize: on > performance.flush-behind: off > performance.io-thread-count: 16 > features.quota: off > performance.quick-read: on > performance.stat-prefetch: off > performance.io-cache: on > performance.cache-refresh-timeout: 1 > nfs.disable: on > performance.cache-max-file-size: 200kb > performance.cache-size: 2GB > performance.write-behind-window-size: 4MB > performance.read-ahead: off > storage.linux-aio: off > diagnostics.brick-sys-log-level: WARNING > cluster.self-heal-daemon: off > > Volume Name: archive2 > Type: Distributed-Replicate > Volume ID: 0fe86e42-e67f-46d8-8ed0-d0e34f539d69 > Status: Started > Number of Bricks: 4 x 2 = 8 > Transport-type: tcp > Bricks: > Brick1: glucfshead1:/data/glusterfs/archive2/brick1 > Brick2: glucfshead5:/data/glusterfs/archive2/brick1 > Brick3: glucfshead2:/data/glusterfs/archive2/brick1 > Brick4: glucfshead6:/data/glusterfs/archive2/brick1 > Brick5: glucfshead3:/data/glusterfs/archive2/brick1 > Brick6: glucfshead7:/data/glusterfs/archive2/brick1 > Brick7: glucfshead4:/data/glusterfs/archive2/brick1 > Brick8: glucfshead8:/data/glusterfs/archive2/brick1 > Options Reconfigured: > cluster.metadata-self-heal: off > cluster.entry-self-heal: off > cluster.data-self-heal: off > diagnostics.count-fop-hits: on > diagnostics.latency-measurement: on > features.lock-heal: on > diagnostics.brick-sys-log-level: WARNING > storage.linux-aio: off > performance.read-ahead: off > performance.write-behind-window-size: 4MB > performance.cache-size: 2GB > performance.cache-max-file-size: 200kb > nfs.disable: on > performance.cache-refresh-timeout: 1 > performance.io-cache: on > performance.stat-prefetch: off > performance.quick-read: on > features.quota: off > performance.io-thread-count: 16 > performance.flush-behind: off > auth.allow: 172.16.15.* > cluster.readdir-optimize: on > cluster.self-heal-daemon: off > > Some time ago node, glucfshead1 broke down. After some fiddling it was > decided not to deal with that immediately because the gluster was in > production and a rebuild on 3.4 would basically render the gluster unusable. > > Recently it was felt that we needed to deal with the situation and we > hired some experts to deal with the problem. So we reinstalled the > broken node and gave it a new name/ip and upgraded all systems to 3.6.4. > > The plan was to probe the "new" node into the gluster and then do a > brick-replace on it. However that did not go as expected. > > The node that we removed is now listed as "Peer Rejected": > > root at glucfshead2:~# gluster peer status > Number of Peers: 7 > > Hostname: glucfshead1 > Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf > State: Peer Rejected (Disconnected) > > Hostname: glucfshead3 > Uuid: a17ae95d-4598-4cd7-9ae7-808af10fedb5 > State: Peer in Cluster (Connected) > > Hostname: glucfshead4 > Uuid: 8547dadd-96bf-45fe-b49d-bab8f995c928 > State: Peer in Cluster (Connected) > > Hostname: glucfshead5 > Uuid: 249da8ea-fda6-47ff-98e0-dbff99dcb3f2 > State: Peer in Cluster (Connected) > > Hostname: glucfshead6 > Uuid: a0229511-978c-4904-87ae-7e1b32ac2c72 > State: Peer in Cluster (Connected) > > Hostname: glucfshead7 > Uuid: 548ec75a-0131-4c92-aaa9-7c6ee7b47a63 > State: Peer in Cluster (Connected) > > Hostname: glucfshead8 > Uuid: 5e54cbc1-482c-460b-ac38-00c4b71c50b9 > State: Peer in Cluster (Connected) > > If I probe the replacement node (glucfshead9) it only ever shows up on > one of my running nodes and it's in state "Rejected Peer (Connected)". > > How can we fix this - preferably without losing data? > > TIA, > Thomas > > --- > Diese E-Mail wurde von Avast Antivirus-Software auf Viren gepr?ft. > https://www.avast.com/antivirus > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Atin Mukherjee
2015-Oct-30 16:56 UTC
[Gluster-users] Replacing a node in a 4x2 distributed/replicated setup
This could very well be related to op-version. Could you look at the faulty node's glusterd log and see the error log entries, that would give us the exact reason of failure. -Atin Sent from one plus one On Oct 30, 2015 5:35 PM, "Thomas B?tzler" <t.baetzler at bringe.com> wrote:> Hi, > > can somebody help me with fixing our 8 node gluster please? > > Setup is as follows: > > root at glucfshead2:~# gluster volume info > > Volume Name: archive > Type: Distributed-Replicate > Volume ID: d888b302-2a35-4559-9bb0-4e182f49f9c6 > Status: Started > Number of Bricks: 4 x 2 = 8 > Transport-type: tcp > Bricks: > Brick1: glucfshead1:/data/glusterfs/archive/brick1 > Brick2: glucfshead5:/data/glusterfs/archive/brick1 > Brick3: glucfshead2:/data/glusterfs/archive/brick1 > Brick4: glucfshead6:/data/glusterfs/archive/brick1 > Brick5: glucfshead3:/data/glusterfs/archive/brick1 > Brick6: glucfshead7:/data/glusterfs/archive/brick1 > Brick7: glucfshead4:/data/glusterfs/archive/brick1 > Brick8: glucfshead8:/data/glusterfs/archive/brick1 > Options Reconfigured: > cluster.data-self-heal: off > cluster.entry-self-heal: off > cluster.metadata-self-heal: off > features.lock-heal: on > cluster.readdir-optimize: on > performance.flush-behind: off > performance.io-thread-count: 16 > features.quota: off > performance.quick-read: on > performance.stat-prefetch: off > performance.io-cache: on > performance.cache-refresh-timeout: 1 > nfs.disable: on > performance.cache-max-file-size: 200kb > performance.cache-size: 2GB > performance.write-behind-window-size: 4MB > performance.read-ahead: off > storage.linux-aio: off > diagnostics.brick-sys-log-level: WARNING > cluster.self-heal-daemon: off > > Volume Name: archive2 > Type: Distributed-Replicate > Volume ID: 0fe86e42-e67f-46d8-8ed0-d0e34f539d69 > Status: Started > Number of Bricks: 4 x 2 = 8 > Transport-type: tcp > Bricks: > Brick1: glucfshead1:/data/glusterfs/archive2/brick1 > Brick2: glucfshead5:/data/glusterfs/archive2/brick1 > Brick3: glucfshead2:/data/glusterfs/archive2/brick1 > Brick4: glucfshead6:/data/glusterfs/archive2/brick1 > Brick5: glucfshead3:/data/glusterfs/archive2/brick1 > Brick6: glucfshead7:/data/glusterfs/archive2/brick1 > Brick7: glucfshead4:/data/glusterfs/archive2/brick1 > Brick8: glucfshead8:/data/glusterfs/archive2/brick1 > Options Reconfigured: > cluster.metadata-self-heal: off > cluster.entry-self-heal: off > cluster.data-self-heal: off > diagnostics.count-fop-hits: on > diagnostics.latency-measurement: on > features.lock-heal: on > diagnostics.brick-sys-log-level: WARNING > storage.linux-aio: off > performance.read-ahead: off > performance.write-behind-window-size: 4MB > performance.cache-size: 2GB > performance.cache-max-file-size: 200kb > nfs.disable: on > performance.cache-refresh-timeout: 1 > performance.io-cache: on > performance.stat-prefetch: off > performance.quick-read: on > features.quota: off > performance.io-thread-count: 16 > performance.flush-behind: off > auth.allow: 172.16.15.* > cluster.readdir-optimize: on > cluster.self-heal-daemon: off > > Some time ago node, glucfshead1 broke down. After some fiddling it was > decided not to deal with that immediately because the gluster was in > production and a rebuild on 3.4 would basically render the gluster > unusable. > > Recently it was felt that we needed to deal with the situation and we > hired some experts to deal with the problem. So we reinstalled the > broken node and gave it a new name/ip and upgraded all systems to 3.6.4. > > The plan was to probe the "new" node into the gluster and then do a > brick-replace on it. However that did not go as expected. > > The node that we removed is now listed as "Peer Rejected": > > root at glucfshead2:~# gluster peer status > Number of Peers: 7 > > Hostname: glucfshead1 > Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf > State: Peer Rejected (Disconnected) > > Hostname: glucfshead3 > Uuid: a17ae95d-4598-4cd7-9ae7-808af10fedb5 > State: Peer in Cluster (Connected) > > Hostname: glucfshead4 > Uuid: 8547dadd-96bf-45fe-b49d-bab8f995c928 > State: Peer in Cluster (Connected) > > Hostname: glucfshead5 > Uuid: 249da8ea-fda6-47ff-98e0-dbff99dcb3f2 > State: Peer in Cluster (Connected) > > Hostname: glucfshead6 > Uuid: a0229511-978c-4904-87ae-7e1b32ac2c72 > State: Peer in Cluster (Connected) > > Hostname: glucfshead7 > Uuid: 548ec75a-0131-4c92-aaa9-7c6ee7b47a63 > State: Peer in Cluster (Connected) > > Hostname: glucfshead8 > Uuid: 5e54cbc1-482c-460b-ac38-00c4b71c50b9 > State: Peer in Cluster (Connected) > > If I probe the replacement node (glucfshead9) it only ever shows up on > one of my running nodes and it's in state "Rejected Peer (Connected)". > > How can we fix this - preferably without losing data? > > TIA, > Thomas > > --- > Diese E-Mail wurde von Avast Antivirus-Software auf Viren gepr?ft. > https://www.avast.com/antivirus > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151030/741a16d1/attachment.html>
Thomas Bätzler
2015-Nov-03 14:07 UTC
[Gluster-users] Replacing a node in a 4x2 distributed/replicated setup
Hi, Atin Mukherjee wrote:> This could very well be related to op-version. Could you look at the > faulty node's glusterd log and see the error log entries, that would > give us the exact reason of failure.op-version is 1 across all the nodes. I've made some progress: by persistently wiping /var/lib/glusterd except fpr glusterd.info and restarting glusterd on the new node, I've progressed to a state where all nodes agree that my replacement node is part of the gluster: root at glucfshead2:~# for i in `seq 2 9`; do echo "glucfshead$i:"; ssh glucfshead$i "gluster peer status" | grep -A2 glucfshead9 ; done glucfshead2: Hostname: glucfshead9 Uuid: 040e61dd-fd02-4957-8833-cf5708b837f0 State: Peer in Cluster (Connected) glucfshead3: Hostname: glucfshead9 Uuid: 040e61dd-fd02-4957-8833-cf5708b837f0 State: Peer in Cluster (Connected) glucfshead4: Hostname: glucfshead9 Uuid: 040e61dd-fd02-4957-8833-cf5708b837f0 State: Peer in Cluster (Connected) glucfshead5: Hostname: glucfshead9 Uuid: 040e61dd-fd02-4957-8833-cf5708b837f0 State: Peer in Cluster (Connected) glucfshead6: Hostname: glucfshead9 Uuid: 040e61dd-fd02-4957-8833-cf5708b837f0 State: Peer in Cluster (Connected) glucfshead7: Hostname: glucfshead9 Uuid: 040e61dd-fd02-4957-8833-cf5708b837f0 State: Peer in Cluster (Connected) glucfshead8: Hostname: glucfshead9 Uuid: 040e61dd-fd02-4957-8833-cf5708b837f0 State: Peer in Cluster (Connected) The new node sees all of the other nodes: root at glucfshead9:~# gluster peer status Number of Peers: 7 Hostname: glucfshead4.bo.rz.pixum.net Uuid: 8547dadd-96bf-45fe-b49d-bab8f995c928 State: Peer in Cluster (Connected) Hostname: glucfshead2 Uuid: 73596f88-13ae-47d7-ba05-da7c347f6141 State: Peer in Cluster (Connected) Hostname: glucfshead3 Uuid: a17ae95d-4598-4cd7-9ae7-808af10fedb5 State: Peer in Cluster (Connected) Hostname: glucfshead5.bo.rz.pixum.net Uuid: 249da8ea-fda6-47ff-98e0-dbff99dcb3f2 State: Peer in Cluster (Connected) Hostname: glucfshead6 Uuid: a0229511-978c-4904-87ae-7e1b32ac2c72 State: Peer in Cluster (Connected) Hostname: glucfshead7 Uuid: 548ec75a-0131-4c92-aaa9-7c6ee7b47a63 State: Peer in Cluster (Connected) Hostname: glucfshead8 Uuid: 5e54cbc1-482c-460b-ac38-00c4b71c50b9 State: Peer in Cluster (Connected) The old nodes all agree that the to-be-replaced node is offline: root at glucfshead2:~# for i in `seq 2 9`; do echo "glucfshead$i:"; ssh glucfshead$i "gluster peer status" | grep -B2 Rej ; done glucfshead2: Hostname: glucfshead1 Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf State: Peer Rejected (Disconnected) glucfshead3: Hostname: glucfshead1 Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf State: Peer Rejected (Disconnected) glucfshead4: Hostname: glucfshead1 Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf State: Peer Rejected (Disconnected) glucfshead5: Hostname: glucfshead1 Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf State: Peer Rejected (Disconnected) glucfshead6: Hostname: glucfshead1 Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf State: Peer Rejected (Disconnected) glucfshead7: Hostname: glucfshead1 Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf State: Peer Rejected (Disconnected) glucfshead8: Hostname: glucfshead1 Uuid: 09ed9a29-c923-4dc5-957a-e0d3e8032daf State: Peer Rejected (Disconnected) glucfshead9: If I try to replace the downed brick with my new brick it says it's successful: root at glucfshead2:~# gluster volume replace-brick archive glucfshead1:/data/glusterfs/archive/brick1 glucfshead9:/data/glusterfs/archive/brick1/brick commit force volume replace-brick: success: replace-brick commit successful However on checking the broken brick is still show as online: root at glucfshead2:~# gluster volume info Volume Name: archive Type: Distributed-Replicate Volume ID: d888b302-2a35-4559-9bb0-4e182f49f9c6 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: glucfshead1:/data/glusterfs/archive/brick1 Brick2: glucfshead5:/data/glusterfs/archive/brick1 Brick3: glucfshead2:/data/glusterfs/archive/brick1 Brick4: glucfshead6:/data/glusterfs/archive/brick1 Brick5: glucfshead3:/data/glusterfs/archive/brick1 Brick6: glucfshead7:/data/glusterfs/archive/brick1 Brick7: glucfshead4:/data/glusterfs/archive/brick1 Brick8: glucfshead8:/data/glusterfs/archive/brick1 Options Reconfigured: cluster.data-self-heal: off cluster.entry-self-heal: off cluster.metadata-self-heal: off features.lock-heal: on cluster.readdir-optimize: on auth.allow: 172.16.15.* performance.flush-behind: off performance.io-thread-count: 16 features.quota: off performance.quick-read: on performance.stat-prefetch: off performance.io-cache: on performance.cache-refresh-timeout: 1 nfs.disable: on performance.cache-max-file-size: 200kb performance.cache-size: 2GB performance.write-behind-window-size: 4MB performance.read-ahead: off storage.linux-aio: off diagnostics.brick-sys-log-level: INFO server.statedump-path: /var/tmp cluster.self-heal-daemon: off All of the old bricks complain loudly that they can't connect to glucfshead1: [2015-11-03 13:54:59.422135] I [MSGID: 106004] [glusterd-handler.c:4398:__glusterd_peer_rpc_notify] 0-management: Peer 09ed9a29-c923-4dc5-957a-e0d3e8032daf, in Peer Rejected state, has disconnected from glusterd. [2015-11-03 13:56:24.996215] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req [2015-11-03 13:56:24.996283] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick commit-force request [2015-11-03 13:56:25.016345] E [glusterd-rpc-ops.c:1087:__glusterd_stage_op_cbk] 0-management: Received stage RJT from uuid: 040e61dd-fd02-4957-8833-cf5708b837f0 The new server only logs "Stage failed". [2015-11-03 13:56:25.015942] E [glusterd-op-sm.c:4585:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Replace brick', Status : -1 I tried to detach glucfshead1 since it's no longer online, but I only get a message that I can't do it since that server is still part of a volume. Any further ideas ideas that I could try? TIA, Thomas --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren gepr?ft. https://www.avast.com/antivirus