thr3ads.net - Gluster users - [Gluster-users] gluster peer probe failing [Jun 2017]

If this information is useful, please help other people find it:
Share via:

Guy Cukierman

2017-Jun-14 15:02 UTC

[Gluster-users] gluster peer probe failing

Hi,
I have a gluster (version 3.10.2) server running on a 3 node (centos7) cluster.
Firewalld and SELinux are disabled, and I see I can telnet from each node to the
other on port 24007.

When I try to create the first peering by running on node1 the command:
gluster peer probe <node2 ip address>

I get the error:
"Connection failed. Please check if gluster daemon is operational."

And Glusterd.log shows:

[2017-06-14 14:46:09.927510] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req 192.168.1.17 24007
[2017-06-14 14:46:09.928560] I [MSGID: 106129]
[glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find
peerinfo for host: 192.168.1.17 (24007)
[2017-06-14 14:46:09.930783] W [MSGID: 106062]
[glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout
[2017-06-14 14:46:09.930837] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2017-06-14 14:46:09
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.10.3
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f69625da4d0]
/lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f69625e3dd4]
/lib64/libc.so.6(+0x35250)[0x7f6960cbe250]
/lib64/libglusterfs.so.0(gf_ports_reserved+0x15c)[0x7f69625e444c]
/lib64/libglusterfs.so.0(gf_process_reserved_ports+0xbe)[0x7f69625e470e]
/usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(+0xd158)[0x7f6954568158]
/usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(client_bind+0x93)[0x7f69545685a3]
/usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(+0xa875)[0x7f6954565875]
/lib64/libgfrpc.so.0(rpc_clnt_reconnect+0xc9)[0x7f69623a3f89]
/lib64/libgfrpc.so.0(rpc_clnt_start+0x39)[0x7f69623a4049]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x24218)[0x7f69570f9218]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x24843)[0x7f69570f9843]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x24ae0)[0x7f69570f9ae0]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x27890)[0x7f69570fc890]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x27e20)[0x7f69570fce20]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x20f5e)[0x7f69570f5f5e]
/lib64/libglusterfs.so.0(synctask_wrap+0x10)[0x7f6962611750]
/lib64/libc.so.6(+0x46cf0)[0x7f6960ccfcf0]

And a file is create under /var/lib/glusterd/peers/<node2 ip address>
which contains:
uuid=00000000-0000-0000-0000-000000000000
state=0
hostname1=192.168.1.17

and the glusterd daemon exits and I cannot restart it until I delete this file
from the peers folder.

Any idea what is wrong?
thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170614/e67ce88e/attachment.html>

Atin Mukherjee

2017-Jun-15 07:56 UTC

head link

[Gluster-users] gluster peer probe failing

https://review.gluster.org/#/c/17494/ will it and the next update of 3.10
should have this fix.

If sysctl net.ipv4.ip_local_reserved_ports has any value > short int
range then this would be a problem with the current version.
Would you be able to reset the reserved ports temporarily to get this going?



On Wed, Jun 14, 2017 at 8:32 PM, Guy Cukierman <guyc at elminda.com>
wrote:
> Hi,
>
> I have a gluster (version 3.10.2) server running on a 3 node (centos7)
> cluster.
>
> Firewalld and SELinux are disabled, and I see I can telnet from each node
> to the other on port 24007.
>
>
>
> When I try to create the first peering by running on node1 the command:
>
> gluster peer probe <node2 ip address>
>
>
>
> I get the error:
>
> ?Connection failed. Please check if gluster daemon is operational.?
>
>
>
> And Glusterd.log shows:
>
>
>
> [2017-06-14 14:46:09.927510] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe]
> 0-glusterd: Received CLI probe req 192.168.1.17 24007
>
> [2017-06-14 14:46:09.928560] I [MSGID: 106129]
[glusterd-handler.c:3690:glusterd_probe_begin]
> 0-glusterd: Unable to find peerinfo for host: 192.168.1.17 (24007)
>
> [2017-06-14 14:46:09.930783] W [MSGID: 106062] [glusterd-handler.c:3466:
> glusterd_transport_inet_options_build] 0-glusterd: Failed to get
> tcp-user-timeout
>
> [2017-06-14 14:46:09.930837] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
>
> pending frames:
>
> frame : type(0) op(0)
>
> patchset: git://git.gluster.org/glusterfs.git
>
> signal received: 11
>
> time of crash:
>
> 2017-06-14 14:46:09
>
> configuration details:
>
> argp 1
>
> backtrace 1
>
> dlfcn 1
>
> libpthread 1
>
> llistxattr 1
>
> setfsid 1
>
> spinlock 1
>
> epoll.h 1
>
> xattr.h 1
>
> st_atim.tv_nsec 1
>
> package-string: glusterfs 3.10.3
>
> /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f69625da4d0]
>
> /lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f69625e3dd4]
>
> /lib64/libc.so.6(+0x35250)[0x7f6960cbe250]
>
> /lib64/libglusterfs.so.0(gf_ports_reserved+0x15c)[0x7f69625e444c]
>
> /lib64/libglusterfs.so.0(gf_process_reserved_ports+0xbe)[0x7f69625e470e]
>
> /usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(+
> 0xd158)[0x7f6954568158]
>
> /usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(client_bind+0x93)[
> 0x7f69545685a3]
>
> /usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(+
> 0xa875)[0x7f6954565875]
>
> /lib64/libgfrpc.so.0(rpc_clnt_reconnect+0xc9)[0x7f69623a3f89]
>
> /lib64/libgfrpc.so.0(rpc_clnt_start+0x39)[0x7f69623a4049]
>
> /usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+
> 0x24218)[0x7f69570f9218]
>
> /usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+
> 0x24843)[0x7f69570f9843]
>
> /usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+
> 0x24ae0)[0x7f69570f9ae0]
>
> /usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+
> 0x27890)[0x7f69570fc890]
>
> /usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+
> 0x27e20)[0x7f69570fce20]
>
> /usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+
> 0x20f5e)[0x7f69570f5f5e]
>
> /lib64/libglusterfs.so.0(synctask_wrap+0x10)[0x7f6962611750]
>
> /lib64/libc.so.6(+0x46cf0)[0x7f6960ccfcf0]
>
>
>
> And a file is create under /var/lib/glusterd/peers/<node2 ip address>
> which contains:
>
> uuid=00000000-0000-0000-0000-000000000000
>
> state=0
>
> hostname1=192.168.1.17
>
>
>
> and the glusterd daemon exits and I cannot restart it until I delete this
> file from the peers folder.
>
>
>
> Any idea what is wrong?
>
> thanks!
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170615/98bc63ce/attachment.html>

Guy Cukierman

2017-Jun-15 09:58 UTC

head link

[Gluster-users] gluster peer probe failing

Thanks, but my current settings are:
net.ipv4.ip_local_reserved_ports = 30000-32767
net.ipv4.ip_local_port_range = 32768    60999
meaning the reserved ports are already in the short int range, so maybe I
misunderstood something? or is it a different issue?

From: Atin Mukherjee [mailto:amukherj at redhat.com]
Sent: Thursday, June 15, 2017 10:56 AM
To: Guy Cukierman <guyc at elminda.com>
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] gluster peer probe failing

https://review.gluster.org/#/c/17494/ will it and the next update of 3.10 should
have this fix.


If sysctl net.ipv4.ip_local_reserved_ports has any value > short int range
then this would be a problem with the current version.
Would you be able to reset the reserved ports temporarily to get this going?


On Wed, Jun 14, 2017 at 8:32 PM, Guy Cukierman <guyc at
elminda.com<mailto:guyc at elminda.com>> wrote:
Hi,
I have a gluster (version 3.10.2) server running on a 3 node (centos7) cluster.
Firewalld and SELinux are disabled, and I see I can telnet from each node to the
other on port 24007.

When I try to create the first peering by running on node1 the command:
gluster peer probe <node2 ip address>

I get the error:
?Connection failed. Please check if gluster daemon is operational.?

And Glusterd.log shows:

[2017-06-14 14:46:09.927510] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req 192.168.1.17 24007
[2017-06-14 14:46:09.928560] I [MSGID: 106129]
[glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find
peerinfo for host: 192.168.1.17 (24007)
[2017-06-14 14:46:09.930783] W [MSGID: 106062]
[glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout
[2017-06-14 14:46:09.930837] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
pending frames:
frame : type(0) op(0)
patchset:
git://git.gluster.org/glusterfs.git<http://git.gluster.org/glusterfs.git>
signal received: 11
time of crash:
2017-06-14 14:46:09
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.10.3
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f69625da4d0]
/lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f69625e3dd4]
/lib64/libc.so.6(+0x35250)[0x7f6960cbe250]
/lib64/libglusterfs.so.0(gf_ports_reserved+0x15c)[0x7f69625e444c]
/lib64/libglusterfs.so.0(gf_process_reserved_ports+0xbe)[0x7f69625e470e]
/usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(+0xd158)[0x7f6954568158]
/usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(client_bind+0x93)[0x7f69545685a3]
/usr/lib64/glusterfs/3.10.3/rpc-transport/socket.so(+0xa875)[0x7f6954565875]
/lib64/libgfrpc.so.0(rpc_clnt_reconnect+0xc9)[0x7f69623a3f89]
/lib64/libgfrpc.so.0(rpc_clnt_start+0x39)[0x7f69623a4049]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x24218)[0x7f69570f9218]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x24843)[0x7f69570f9843]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x24ae0)[0x7f69570f9ae0]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x27890)[0x7f69570fc890]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x27e20)[0x7f69570fce20]
/usr/lib64/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x20f5e)[0x7f69570f5f5e]
/lib64/libglusterfs.so.0(synctask_wrap+0x10)[0x7f6962611750]
/lib64/libc.so.6(+0x46cf0)[0x7f6960ccfcf0]

And a file is create under /var/lib/glusterd/peers/<node2 ip address>
which contains:
uuid=00000000-0000-0000-0000-000000000000
state=0
hostname1=192.168.1.17

and the glusterd daemon exits and I cannot restart it until I delete this file
from the peers folder.

Any idea what is wrong?
thanks!

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170615/8c516c42/attachment.html>

Seemingly Similar Threads

Search for more seemingly similar threads

Gluster users - Jun 2017 - gluster peer probe failing

[Gluster-users] gluster peer probe failing

[Gluster-users] gluster peer probe failing

[Gluster-users] gluster peer probe failing

Seemingly Similar Threads