[Gluster-users] and again peer probe node1 hangs

free.aaa
2015-04-23T14:35:38Z

Hi everybody!

I have "gluster peer probe gfs1" command hung with the result of
"Probe
Sent to Peer (connected)"
> gfs3#gluster peer status
> Number of Peers: 3
>
> Hostname: gfs6
> Uuid: 6bd6ee25-e257-4703-b500-330741b90471
> State: Peer in Cluster (Connected)
>
> Hostname: gfs4
> Uuid: bb1bed20-25bf-43b0-8faa-49f1b5b9ae59
> State: Peer in Cluster (Connected)
>
> Hostname: gfs1
> Uuid: bb67c1da-2698-4c35-b29d-f80f8eb814a6
> State: Probe Sent to Peer (Connected)

I double checked the dns resolution. Forward and reverse resolution
works fine.
I tried to debug on gfs1 and gfs3 at the moment of probe command and it
seems to me that gfs1 does not sync all the gluster configuration from
gfs3. Especially peers folder contains only 1 peer with the following
content:
> gfs1#cat /var/lib/glusterd/peers/192.168.9.53
> uuid=00000000-0000-0000-0000-000000000000
> state=8
> hostname1=192.168.9.53
so there are no uuids both in the content of the file and in filename
itself. Besides there no info about other peers like gfs4, gfs6 in the
folder.
Last lines of the debug log on gfs1 looks like this:
> [2015-04-23 10:12:28.334239] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.334413] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.334557] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.334697] D [common-utils.c:2946:gf_is_local_addr]
> 0-management: gfs4 is not local
> [2015-04-23 10:12:28.334707] D
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
> returning -1
> [2015-04-23 10:12:28.334714] D
> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
> Returning -1
> [2015-04-23 10:12:28.336881] D
> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
> Unable to find friend: gfs6
> [2015-04-23 10:12:28.337490] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.337697] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.337841] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.337981] D [common-utils.c:2946:gf_is_local_addr]
> 0-management: gfs6 is not local
> [2015-04-23 10:12:28.337991] D
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
> returning -1
> [2015-04-23 10:12:28.337998] D
> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
> Returning -1
> [2015-04-23 10:12:28.338858] D
> [glusterd-utils.c:5523:glusterd_friend_find_by_hostname] 0-management:
> Friend gfs3 found.. state: 8
> [2015-04-23 10:12:28.338873] D
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
> returning 0
> [2015-04-23 10:12:28.340989] D
> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
> Unable to find friend: gfs6
> [2015-04-23 10:12:28.341545] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.341697] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.341840] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.341980] D [common-utils.c:2946:gf_is_local_addr]
> 0-management: gfs6 is not local
> [2015-04-23 10:12:28.341991] D
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
> returning -1
> [2015-04-23 10:12:28.341997] D
> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
> [2015-04-23 10:12:28.342003] D
> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
> Returning -1
> [2015-04-23 10:12:28.344065] D
> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
> Unable to find friend: gfs4
> [2015-04-23 10:12:28.344620] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.344772] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.344914] D [common-utils.c:2930:gf_is_local_addr]
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.345054] D [common-utils.c:2946:gf_is_local_addr]
> 0-management: gfs4 is not local
> [2015-04-23 10:12:28.345064] D
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
> returning -1
> [2015-04-23 10:12:28.345071] D
> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
> Returning -1
> [2015-04-23 10:12:28.345543] D [run.c:190:runner_log] 0-: Starting the
> nfs/glustershd services: /usr/sbin/glusterfs -s localhost --volfile-id
> gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l
> /var/log/glusterfs/quotad.log -S
> /var/run/3e619fbfe69c96b1dbc7486a7d38a7be.socket --xlator-option
> *replicate*.data-self-heal=off --xlator-option
> *replicate*.metadata-self-heal=off --xlator-option
> *replicate*.entry-self-heal=off
> ^C[2015-04-23 10:20:50.237403] W [glusterfsd.c:1095:cleanup_and_exit]
> (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_end_reuse+0x26)
> [0x7fa81cd3dfb6]
> (-->/lib/x86_64-linux-gnu/libpthread.so.0(waitpid+0x5b)
> [0x7fa81c499c8b] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x321e0)
> [0x7fa81bd3a1e0]))) 0-: received signum (2), shutting down
> [2015-04-23 10:20:50.237436] D
> [glusterfsd-mgmt.c:2025:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt:
> portmapper signout arguments not given
So I think server hangs when trying to start volume with references to
gfs4 and gfs6 which it does not know about, but why gfs1 peers
configuration is empty?

Any help?

Thanks in advance!
Best regards,
Alex.