[Gluster-users] Fixing a rejected peer

Jamie Lawrence
Tue, 06 Mar 2018 00:30:10 -0000

Hello,

So I'm seeing a rejected peer with 3.12.6. This is with a replica 3 volume.

It actually began as the same problem with a different peer. I noticed with
(call it) gluster-2, when I couldn't make a new volume.  I compared
/var/lib/glusterd between them, and found that somehow the options in one of the
vols differed. (I suspect this was due to attempting to create the volume via
the Ovirt GUI; suffice to say I'm not using it for things like this in the
future.) So I stopped the daemons and corrected that (gluster-2 had a tiering
entry the others didn't).

Started things back up and now gluster-3 is being rejected by the other two. The
error is below.

I'm tempted to repeat - down things, copy the checksum the "good"
ones agree on, start things; but given that this has turned into a
balloon-squeezing exercise, I want to make sure I'm not doing this the wrong
way.

What is the currently accepted best method for fixing this?

And given that this happened on a nearly brand-new deployment, it worries me a
bit that this happened while nothing hinky was going on - I installed Gluster
manually, but the rest of the systems management has been via Ovirt. Has anyone
else seen issues with this?

Thanks,

-j

- - snip - -

[2018-03-06 00:14:06.141281] I [MSGID: 106490]
[glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received
probe from uuid: 77cdfbba-348c-43fe-ab3d-00621904ea9c
[2018-03-06 00:14:06.145540] I [MSGID: 106493]
[glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to
sc5-gluster-1, op_ret: 0, op_errno: 0, ret: 0
[2018-03-06 00:14:06.145697] I [MSGID: 106490]
[glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received
probe from uuid: c1877e0d-ccb2-401e-83a6-e4a680af683a
[2018-03-06 00:14:06.145831] I [MSGID: 106493]
[glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to
sc5-gluster-10g-2, op_ret: 0, op_errno: 0, ret: 0
[2018-03-06 00:14:06.149357] I [MSGID: 106490]
[glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: c1877e0d-ccb2-401e-83a6-e4a680af683a
[2018-03-06 00:14:06.149631] E [MSGID: 106010]
[glusterd-utils.c:3374:glusterd_compare_friend_volume] 0-management: Version of
Cksums sc5-ovirt_engine differ. local cksum = 53769889, remote cksum =
2068896937 on peer sc5-gluster-2
[2018-03-06 00:14:06.149774] I [MSGID: 106493]
[glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
sc5-gluster-2 (0), ret: 0, op_ret: -1
[2018-03-06 00:14:06.151393] I [MSGID: 106490]
[glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 77cdfbba-348c-43fe-ab3d-00621904ea9c
[2018-03-06 00:14:06.152127] E [MSGID: 106010]
[glusterd-utils.c:3374:glusterd_compare_friend_volume] 0-management: Version of
Cksums sc5-ovirt_engine differ. local cksum = 53769889, remote cksum =
2068896937 on peer sc5-gluster-10g-1
[2018-03-06 00:14:06.152314] I [MSGID: 106493]
[glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
sc5-gluster-10g-1 (0), ret: 0, op_ret: -1
[2018-03-06 00:14:06.164819] I [MSGID: 106143]
[glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick
/gluster-bricks/sc5_ovirt_engine/sc5_ovirt_engine on port 49152
[2018-03-06 00:14:06.443882] I [MSGID: 106487]
[glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req