Davy Croonen
2015-Sep-11 09:54 UTC
[Gluster-users] Gluster 3.6.4 peer rejected while doing probe
Hi all We have a production cluster with 2 nodes (gfs01a and gfs01b) in a distributed replicate setup with glusterfs 3.6.4. We want to expand the volume with 2 extra nodes (gfs02a and gfs02b) because we are running out of diskspace. Therefor we deployed 2 extra nodes with glusterfs 3.6.4. Now, while probing the 2 new nodes from a node in the existing cluster we got the following error: root at gfs01a-dcg:~# gluster peer probe gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be> peer probe: success. root at gfs01a-dcg:~# gluster peer status Number of Peers: 2 Hostname: gfs01b-dcg.intnet.be<http://gfs01b-dcg.intnet.be> Uuid: cfc83cf2-b719-40c7-afea-b23accc714c3 State: Peer in Cluster (Connected) Hostname: gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be> Uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f State: Peer Rejected (Connected) In the log file /var/log/glusterfs/etc-glusterfs-glusterd.vol.log the following entries are written: [2015-09-11 09:37:49.405906] I [glusterd-handler.c:1031:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be> 24007 [2015-09-11 09:37:49.428630] I [glusterd-handler.c:3198:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be> (24007) [2015-09-11 09:37:49.438636] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-09-11 09:37:49.440513] I [glusterd-handler.c:3131:glusterd_friend_add] 0-management: connect returned 0 [2015-09-11 09:37:49.474316] I [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f, host: gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be> [2015-09-11 09:37:49.481801] I [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2015-09-11 09:37:51.650265] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f, host: gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be>, port: 0 [2015-09-11 09:37:51.665861] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30603 [2015-09-11 09:37:51.690170] I [glusterd-handler.c:2543:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f [2015-09-11 09:37:51.692652] I [glusterd-handler.c:2595:__glusterd_handle_probe_query] 0-glusterd: Responded to gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be>, op_ret: 0, op_errno: 0, ret: 0 [2015-09-11 09:37:51.706203] I [glusterd-handler.c:2232:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f [2015-09-11 09:37:51.708909] E [MSGID: 106010] [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of Cksums public differ. local cksum = 1932535021, remote cksum = 2474653383 on peer gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be> [2015-09-11 09:37:51.709026] I [glusterd-handler.c:3367:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to gfs02a-dcg.intnet.be<http://gfs02a-dcg.intnet.be> (0), ret: 0 [2015-09-11 09:37:55.537231] I [glusterd-handler.c:1241:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req The exact same error appears while probing the second node (gfs02b). Anyone any idea how to solve this? Thanks in advance. Kind regards Davy -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150911/e4743765/attachment.html>
Mohammed Rafi K C
2015-Sep-11 10:39 UTC
[Gluster-users] Gluster 3.6.4 peer rejected while doing probe
Can you check the checksum of the volume "public" in both of the current nodes. Checksums are located in (/var/lib/glusterd/vols/public/cksum). Regards Rafi KC On 09/11/2015 03:24 PM, Davy Croonen wrote:> Hi all > > We have a production cluster with 2 nodes (gfs01a and gfs01b) in a > distributed replicate setup with glusterfs 3.6.4. We want to expand > the volume with 2 extra nodes (gfs02a and gfs02b) because we are > running out of diskspace. Therefor we deployed 2 extra nodes with > glusterfs 3.6.4. > > Now, while probing the 2 new nodes from a node in the existing cluster > we got the following error: > > root at gfs01a-dcg:~# gluster peer probe gfs02a-dcg.intnet.be > <http://gfs02a-dcg.intnet.be> > peer probe: success. > root at gfs01a-dcg:~# gluster peer status > Number of Peers: 2 > > Hostname: gfs01b-dcg.intnet.be <http://gfs01b-dcg.intnet.be> > Uuid: cfc83cf2-b719-40c7-afea-b23accc714c3 > State: Peer in Cluster (Connected) > > Hostname: gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be> > Uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f > *State: Peer Rejected (Connected)* > > In the log file /var/log/glusterfs/etc-glusterfs-glusterd.vol.log the > following entries are written: > > [2015-09-11 09:37:49.405906] I > [glusterd-handler.c:1031:__glusterd_handle_cli_probe] 0-glusterd: > Received CLI probe req gfs02a-dcg.intnet.be > <http://gfs02a-dcg.intnet.be> 24007 > [2015-09-11 09:37:49.428630] I > [glusterd-handler.c:3198:glusterd_probe_begin] 0-glusterd: Unable to > find peerinfo for host: gfs02a-dcg.intnet.be > <http://gfs02a-dcg.intnet.be> (24007) > [2015-09-11 09:37:49.438636] I > [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting > frame-timeout to 600 > [2015-09-11 09:37:49.440513] I > [glusterd-handler.c:3131:glusterd_friend_add] 0-management: connect > returned 0 > [2015-09-11 09:37:49.474316] I > [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received > probe resp from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f, host: > gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be> > [2015-09-11 09:37:49.481801] I > [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received > resp to probe req > [2015-09-11 09:37:51.650265] I > [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: > Received ACC from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f, host: > gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be>, port: 0 > [2015-09-11 09:37:51.665861] I > [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] > 0-management: using the op-version 30603 > [2015-09-11 09:37:51.690170] I > [glusterd-handler.c:2543:__glusterd_handle_probe_query] 0-glusterd: > Received probe from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f > [2015-09-11 09:37:51.692652] I > [glusterd-handler.c:2595:__glusterd_handle_probe_query] 0-glusterd: > Responded to gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be>, > op_ret: 0, op_errno: 0, ret: 0 > [2015-09-11 09:37:51.706203] I > [glusterd-handler.c:2232:__glusterd_handle_incoming_friend_req] > 0-glusterd: Received probe from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f > *[2015-09-11 09:37:51.708909] E [MSGID: 106010] > [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: > Version of Cksums public differ. local cksum = 1932535021, remote > cksum = 2474653383 on peer gfs02a-dcg.intnet.be > <http://gfs02a-dcg.intnet.be>* > [2015-09-11 09:37:51.709026] I > [glusterd-handler.c:3367:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be> (0), > ret: 0 > [2015-09-11 09:37:55.537231] I > [glusterd-handler.c:1241:__glusterd_handle_cli_list_friends] > 0-glusterd: Received cli list req > > The exact same error appears while probing the second node (gfs02b). > > Anyone any idea how to solve this? > > Thanks in advance. > > Kind regards > Davy > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150911/07d8fb59/attachment.html>