[Gluster-users] Gluster 3.6.4 peer rejected while doing probe

Davy Croonen
2015-09-11T12:11:53Z

Atin

Please see the requested attachments.

KR
Davy

> On 11 Sep 2015, at 14:03, Atin Mukherjee <amukherj at redhat.com>
wrote:
>
> Could you attach the contents of /var/lib/glusterd/vol/<volname>/info
> file from both the nodes?
>
> ~Atin
>
> On 09/11/2015 04:50 PM, Davy Croonen wrote:
>> Thanks for your quick respons.
>>
>> As reported in the log the checksums are indeed not the same. On
>> gfs01a-dcg it is 'info=1266454712? and on gfs02a-dcg it is
>> 'info=2613085848?. Of course my next question is how can I fix
this?
>>
>> I already tried by stopping the gluster daemon on gfs02a-dcg, deleting
>> the entire vols directory and starting the gluster daemon again. On the
>> gfs01a-dcg host I now did a gluster peer status which shows:
>>
>> Hostname: gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be>
>> Uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f
>> State: Peer in Cluster (Connected)
>>
>> But, the checksum of the public volume is still not the same on
>> gfs01a-dcg and gfs02a-dcg and also running a gluster peer status on
>> gfs01b-dcg (the replica of gfs01a-dcg) gives me:
>>
>> Hostname: gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be>
>> Uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f
>> State: Peer Rejected (Connected)
>>
>> So my question remains any way to fix this?
>>
>> Kind regards
>>
>> Davy
>>
>>> On 11 Sep 2015, at 12:39, Mohammed Rafi K C <rkavunga at
redhat.com
>>> <mailto:rkavunga at redhat.com>> wrote:
>>>
>>> Can you check the checksum of the volume "public" in both
of the
>>> current nodes. Checksums are located in
>>> (/var/lib/glusterd/vols/public/cksum).
>>>
>>> Regards
>>> Rafi KC
>>>
>>> On 09/11/2015 03:24 PM, Davy Croonen wrote:
>>>> Hi all
>>>>
>>>> We have a production cluster with 2 nodes (gfs01a and gfs01b)
in a
>>>> distributed replicate setup with glusterfs 3.6.4. We want to
expand
>>>> the volume with 2 extra nodes (gfs02a and gfs02b) because we
are
>>>> running out of diskspace. Therefor we deployed 2 extra nodes
with
>>>> glusterfs 3.6.4.
>>>>
>>>> Now, while probing the 2 new nodes from a node in the existing
>>>> cluster we got the following error:
>>>>
>>>> root at gfs01a-dcg:~# gluster peer probe gfs02a-dcg.intnet.be
>>>> <http://gfs02a-dcg.intnet.be/>
>>>> peer probe: success.
>>>> root at gfs01a-dcg:~# gluster peer status
>>>> Number of Peers: 2
>>>>
>>>> Hostname: gfs01b-dcg.intnet.be
<http://gfs01b-dcg.intnet.be/>
>>>> Uuid: cfc83cf2-b719-40c7-afea-b23accc714c3
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: gfs02a-dcg.intnet.be
<http://gfs02a-dcg.intnet.be/>
>>>> Uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f
>>>> *State: Peer Rejected (Connected)*
>>>>
>>>> In the log file
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log the
>>>> following entries are written:
>>>>
>>>> [2015-09-11 09:37:49.405906] I
>>>> [glusterd-handler.c:1031:__glusterd_handle_cli_probe]
0-glusterd:
>>>> Received CLI probe req gfs02a-dcg.intnet.be
>>>> <http://gfs02a-dcg.intnet.be/> 24007
>>>> [2015-09-11 09:37:49.428630] I
>>>> [glusterd-handler.c:3198:glusterd_probe_begin] 0-glusterd:
Unable to
>>>> find peerinfo for host: gfs02a-dcg.intnet.be
>>>> <http://gfs02a-dcg.intnet.be/> (24007)
>>>> [2015-09-11 09:37:49.438636] I
>>>> [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting
>>>> frame-timeout to 600
>>>> [2015-09-11 09:37:49.440513] I
>>>> [glusterd-handler.c:3131:glusterd_friend_add] 0-management:
connect
>>>> returned 0
>>>> [2015-09-11 09:37:49.474316] I
>>>> [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management:
Received
>>>> probe resp from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f,
host:
>>>> gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be/>
>>>> [2015-09-11 09:37:49.481801] I
>>>> [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd:
Received
>>>> resp to probe req
>>>> [2015-09-11 09:37:51.650265] I
>>>> [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd:
>>>> Received ACC from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f,
host:
>>>> gfs02a-dcg.intnet.be <http://gfs02a-dcg.intnet.be/>,
port: 0
>>>> [2015-09-11 09:37:51.665861] I
>>>> [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack]
>>>> 0-management: using the op-version 30603
>>>> [2015-09-11 09:37:51.690170] I
>>>> [glusterd-handler.c:2543:__glusterd_handle_probe_query]
0-glusterd:
>>>> Received probe from uuid: 29592d5b-242b-43b5-afc5-5f9a1496d59f
>>>> [2015-09-11 09:37:51.692652] I
>>>> [glusterd-handler.c:2595:__glusterd_handle_probe_query]
0-glusterd:
>>>> Responded to gfs02a-dcg.intnet.be
<http://gfs02a-dcg.intnet.be/>,
>>>> op_ret: 0, op_errno: 0, ret: 0
>>>> [2015-09-11 09:37:51.706203] I
>>>> [glusterd-handler.c:2232:__glusterd_handle_incoming_friend_req]
>>>> 0-glusterd: Received probe from uuid:
>>>> 29592d5b-242b-43b5-afc5-5f9a1496d59f
>>>> *[2015-09-11 09:37:51.708909] E [MSGID: 106010]
>>>> [glusterd-utils.c:3297:glusterd_compare_friend_volume]
0-management:
>>>> Version of Cksums public differ. local cksum = 1932535021,
remote
>>>> cksum = 2474653383 on peer gfs02a-dcg.intnet.be
>>>> <http://gfs02a-dcg.intnet.be/>*
>>>> [2015-09-11 09:37:51.709026] I
>>>> [glusterd-handler.c:3367:glusterd_xfer_friend_add_resp]
0-glusterd:
>>>> Responded to gfs02a-dcg.intnet.be
<http://gfs02a-dcg.intnet.be/> (0),
>>>> ret: 0
>>>> [2015-09-11 09:37:55.537231] I
>>>> [glusterd-handler.c:1241:__glusterd_handle_cli_list_friends]
>>>> 0-glusterd: Received cli list req
>>>>
>>>> The exact same error appears while probing the second node
(gfs02b).
>>>>
>>>> Anyone any idea how to solve this?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Kind regards
>>>> Davy
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gfs01a-dcg.info
Type: application/octet-stream
Size: 457 bytes
Desc: gfs01a-dcg.info
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150911/c39c83d4/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gfs02a-dcg.info
Type: application/octet-stream
Size: 559 bytes
Desc: gfs02a-dcg.info
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150911/c39c83d4/attachment-0001.obj>