Gluster users - Jan 2018 - Sent and Received peer request (Connected)

This morning I did a rolling update from the latest 3.7.x to 3.12.4,
with no client activity.  "Rolling" as in, shut down the Gluster
services on the first server, update, reboot, wait until up and running,
proceed to the next server.  I anticipated that a 3.12 server might not
properly talk to a 3.7 server but since I had no client activity I was
not overly concerned.

All three servers are now updated to 3.12.4.  The weirdness is that
although the third server reports a proper peer status of "Peer in
Cluster (Connected)" to both server 1 and 2, the first and second
servers report to each other "Sent and Received peer request
(Connected)" but show the proper "Peer in Cluster (Connected)" to
the
third server.


[root at glusterfs1]# gluster peer status
Number of Peers: 2

Hostname: glusterfs2
Uuid: 0f4867d9-7be3-4dbc-83f6-ddcda58df607
State: Sent and Received peer request (Connected)

Hostname: glusterfs3
Uuid: 354fdb76-1205-4a5c-b335-66f2ee3e665f
State: Peer in Cluster (Connected)



[root at glusterfs2]# gluster peer status
Number of Peers: 2

Hostname: glusterfs3
Uuid: 354fdb76-1205-4a5c-b335-66f2ee3e665f
State: Peer in Cluster (Connected)

Hostname: glusterfs1
Uuid: 339533fd-5820-4077-a2a0-d39d21379960
State: Sent and Received peer request (Connected)



[root at glusterfs3]# gluster peer status
Number of Peers: 2

Hostname: glusterfs2
Uuid: 0f4867d9-7be3-4dbc-83f6-ddcda58df607
State: Peer in Cluster (Connected)

Hostname: glusterfs1
Uuid: 339533fd-5820-4077-a2a0-d39d21379960
State: Peer in Cluster (Connected)


In my notes I have a procedure for Rejected peers, which I tried:

   Stop glusterd (systemctl stop glusterd)
   In /var/lib/glusterd, delete everything except glusterd.info (the
UUID file)
   Start glusterd (systemctl start glusterd)
   Probe one of the good peers (gluster peer probe HOSTNAME), then probe
the second
   Restart glusterd, check 'gluster peer status'

This hasn't changed anything.  I've tried reboots of all servers, etc.


Any thoughts on how to correct this?

Thanks,

-Dj

On Fri, 12 Jan 2018 at 01:34, Dj Merrill <gluster at deej.net> wrote:
> This morning I did a rolling update from the latest 3.7.x to 3.12.4,
> with no client activity.  "Rolling" as in, shut down the Gluster
> services on the first server, update, reboot, wait until up and running,
> proceed to the next server.  I anticipated that a 3.12 server might not
> properly talk to a 3.7 server but since I had no client activity I was
> not overly concerned.
>
> All three servers are now updated to 3.12.4.  The weirdness is that
> although the third server reports a proper peer status of "Peer in
> Cluster (Connected)" to both server 1 and 2, the first and second
> servers report to each other "Sent and Received peer request
> (Connected)" but show the proper "Peer in Cluster
(Connected)" to the
> third server.
>
>
> [root at glusterfs1]# gluster peer status
> Number of Peers: 2
>
> Hostname: glusterfs2
> Uuid: 0f4867d9-7be3-4dbc-83f6-ddcda58df607
> State: Sent and Received peer request (Connected)
>
> Hostname: glusterfs3
> Uuid: 354fdb76-1205-4a5c-b335-66f2ee3e665f
> State: Peer in Cluster (Connected)
>
>
>
> [root at glusterfs2]# gluster peer status
> Number of Peers: 2
>
> Hostname: glusterfs3
> Uuid: 354fdb76-1205-4a5c-b335-66f2ee3e665f
> State: Peer in Cluster (Connected)
>
> Hostname: glusterfs1
> Uuid: 339533fd-5820-4077-a2a0-d39d21379960
> State: Sent and Received peer request (Connected)
>
>
>
> [root at glusterfs3]# gluster peer status
> Number of Peers: 2
>
> Hostname: glusterfs2
> Uuid: 0f4867d9-7be3-4dbc-83f6-ddcda58df607
> State: Peer in Cluster (Connected)
>
> Hostname: glusterfs1
> Uuid: 339533fd-5820-4077-a2a0-d39d21379960
> State: Peer in Cluster (Connected)
>
>
> In my notes I have a procedure for Rejected peers, which I tried:
>
>    Stop glusterd (systemctl stop glusterd)
>    In /var/lib/glusterd, delete everything except glusterd.info (the
> UUID file)
>    Start glusterd (systemctl start glusterd)
>    Probe one of the good peers (gluster peer probe HOSTNAME), then probe
> the second
>    Restart glusterd, check 'gluster peer status'
>
> This hasn't changed anything.  I've tried reboots of all servers,
etc.

What you?d need to do is to set ?state=3? for the peer which is not in
connected state in /var/lib/glusterd/peers/<uuid file> and then restart
the
glusterd service.

>
>
> Any thoughts on how to correct this?
>
> Thanks,
>
> -Dj
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-- 
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180115/9a274c49/attachment.html>

On 1/15/2018 8:00 AM, Atin Mukherjee wrote:
> What you?d need to do is to set ?state=3? for the peer which is not in 
> connected state in /var/lib/glusterd/peers/<uuid file> and then
restart
> the glusterd service.

Thank you Atin, that worked perfectly!

On glusterfs2, I edited the uuid file for glusterfs1 and changed the 
state from 5 to 3, then restarted glusterd on glusterfs2.

On glusterfs1, I edited the uuid file for glusterfs2 and changed the 
state from 5 to 3, then restarted glusterd on glusterfs1.

Now all three systems are reporting the proper peer status, and the 
volume status is also now reporting properly, even after a reboot of the 
servers.

Thank you!

-Dj