thr3ads.net - Gluster users - [Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message [Feb 2018]

If this information is useful, please help other people find it:
Share via:

Ercan Aydoğan

2018-Feb-06 14:01 UTC

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

Hello,

i installed glusterfs 3.11.3  version 3 nodes ubuntu 16.04 machine. All machines
have same /etc/hosts.

node1 hostname
pri.ostechnix.lan

node2 hostname
sec.ostechnix.lan

node2 hostname
third.ostechnix.lan 


51.15.77.14     pri.ostechnix.lan     pri
51.15.90.60      sec.ostechnix.lan     sec
163.172.151.120  third.ostechnix.lan   third

volume create command is 

root at pri:/var/log/glusterfs# gluster volume create myvol1 replica 2 transport
tcp pri.ostechnix.lan:/gluster/brick1/mpoint1
sec.ostechnix.lan:/gluster/brick1/mpoint1 force
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid
this. See: 
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.
Do you still want to continue?
 (y/n) y
volume create: myvol1: failed: Host pri.ostechnix.lan is not in 'Peer in
Cluster' state

node 1 glusterd.log is here

root at pri:/var/log/glusterfs# cat glusterd.log 
[2018-02-06 13:28:37.638373] W [glusterfsd.c:1331:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f0232faa6ba]
-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55e17938a8c5]
-->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55e17938a6e4] ) 0-: received
signum (15), shutting down
[2018-02-06 13:29:41.260479] I [MSGID: 100030] [glusterfsd.c:2476:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.3 (args:
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2018-02-06 13:29:41.284367] I [MSGID: 106478] [glusterd.c:1422:init]
0-management: Maximum allowed open file descriptors set to 65536
[2018-02-06 13:29:41.284462] I [MSGID: 106479] [glusterd.c:1469:init]
0-management: Using /var/lib/glusterd as working directory
[2018-02-06 13:29:41.300804] W [MSGID: 103071]
[rdma.c:4591:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel
creation failed [No such device]
[2018-02-06 13:29:41.300969] W [MSGID: 103055] [rdma.c:4898:init]
0-rdma.management: Failed to initialize IB Device
[2018-02-06 13:29:41.301098] W [rpc-transport.c:350:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2018-02-06 13:29:41.301190] W [rpcsvc.c:1660:rpcsvc_create_listener]
0-rpc-service: cannot create listener, initing the transport failed
[2018-02-06 13:29:41.301214] E [MSGID: 106243] [glusterd.c:1693:init]
0-management: creation of 1 listeners failed, continuing with succeeded
transport
[2018-02-06 13:29:44.621889] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-06 13:29:44.621967] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-06 13:29:44.621971] I [MSGID: 106514]
[glusterd-store.c:2215:glusterd_restore_op_version] 0-management: Detected new
install. Setting op-version to maximum : 31100
[2018-02-06 13:29:44.625749] I [MSGID: 106194]
[glusterd-store.c:3772:glusterd_store_retrieve_missed_snaps_list] 0-management:
No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.socket.listen-backlog 128
  8:     option event-threads 1
  9:     option ping-timeout 0
 10:     option transport.socket.read-fail-log off
 11:     option transport.socket.keepalive-interval 2
 12:     option transport.socket.keepalive-time 10
 13:     option transport-type rdma
 14:     option working-directory /var/lib/glusterd
 15: end-volume
 16:  
+------------------------------------------------------------------------------+
[2018-02-06 13:29:44.628451] I [MSGID: 101190]
[event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 1
[2018-02-06 13:46:38.530154] I [MSGID: 106487]
[glusterd-handler.c:1484:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2018-02-06 13:47:05.745357] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req sec.ostechnix.lan 24007
[2018-02-06 13:47:05.746465] I [MSGID: 106129]
[glusterd-handler.c:3623:glusterd_probe_begin] 0-glusterd: Unable to find
peerinfo for host: sec.ostechnix.lan (24007)
[2018-02-06 13:47:05.751131] W [MSGID: 106062]
[glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout
[2018-02-06 13:47:05.751179] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2018-02-06 13:47:05.751345] W [MSGID: 101002] [options.c:954:xl_opt_validate]
0-management: option 'address-family' is deprecated, preferred is
'transport.address-family', continuing with correction
[2018-02-06 13:47:05.751902] I [MSGID: 106498]
[glusterd-handler.c:3549:glusterd_friend_add] 0-management: connect returned 0
[2018-02-06 13:47:05.769054] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-06 13:47:05.769160] I [MSGID: 106477]
[glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID:
476b754c-24cd-4816-a630-99c1b696a9e6
[2018-02-06 13:47:05.806715] I [MSGID: 106511]
[glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp
from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host: sec.ostechnix.lan
[2018-02-06 13:47:05.806764] I [MSGID: 106511]
[glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe
req
[2018-02-06 13:47:05.816670] I [MSGID: 106493]
[glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from
uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host: sec.ostechnix.lan, port: 0
[2018-02-06 13:47:05.831231] I [MSGID: 106163]
[glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 31100
[2018-02-06 13:47:05.845025] I [MSGID: 106490]
[glusterd-handler.c:2890:__glusterd_handle_probe_query] 0-glusterd: Received
probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
[2018-02-06 13:47:05.845156] I [MSGID: 106493]
[glusterd-handler.c:2953:__glusterd_handle_probe_query] 0-glusterd: Responded to
sec.ostechnix.lan, op_ret: 0, op_errno: 0, ret: 0
[2018-02-06 13:47:05.852449] I [MSGID: 106490]
[glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
[2018-02-06 13:47:05.855861] I [MSGID: 106493]
[glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
sec.ostechnix.lan (0), ret: 0, op_ret: 0
[2018-02-06 13:47:05.863598] I [MSGID: 106492]
[glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received
friend update from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
[2018-02-06 13:47:05.863674] I [MSGID: 106502]
[glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received
my uuid as Friend
[2018-02-06 13:47:05.866932] I [MSGID: 106493]
[glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC
from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
[2018-02-06 13:48:28.152542] I [MSGID: 106487]
[glusterd-handler.c:1484:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2018-02-06 13:48:57.283814] E [MSGID: 106429]
[glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management: Failed
to convert hostname pri.ostechnix.lan to uuid
[2018-02-06 13:48:57.283871] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of operation
'Volume Create' failed on localhost : Host pri.ostechnix.lan is not in
'Peer in Cluster' state
[2018-02-06 13:49:33.997818] E [MSGID: 106429]
[glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management: Failed
to convert hostname pri.ostechnix.lan to uuid
[2018-02-06 13:49:33.997826] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of operation
'Volume Create' failed on localhost : Host pri.ostechnix.lan is not in
'Peer in Cluster' state

i need advice how can fix this issue.

Thanks.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180206/f679bb06/attachment.html>

Serkan Çoban

2018-Feb-06 14:05 UTC

head link

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

Did you do gluster peer probe? Check out the documentation:
http://docs.gluster.org/en/latest/Administrator%20Guide/Storage%20Pools/

On Tue, Feb 6, 2018 at 5:01 PM, Ercan Aydo?an <ercan.aydogan at gmail.com>
wrote:> Hello,
>
> i installed glusterfs 3.11.3  version 3 nodes ubuntu 16.04 machine. All
> machines have same /etc/hosts.
>
> node1 hostname
> pri.ostechnix.lan
>
> node2 hostname
> sec.ostechnix.lan
>
> node2 hostname
> third.ostechnix.lan
>
>
> 51.15.77.14     pri.ostechnix.lan     pri
> 51.15.90.60      sec.ostechnix.lan     sec
> 163.172.151.120  third.ostechnix.lan   third
>
> volume create command is
>
> root at pri:/var/log/glusterfs# gluster volume create myvol1 replica 2
> transport tcp pri.ostechnix.lan:/gluster/brick1/mpoint1
> sec.ostechnix.lan:/gluster/brick1/mpoint1 force
> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
> avoid this. See:
>
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.
> Do you still want to continue?
>  (y/n) y
> volume create: myvol1: failed: Host pri.ostechnix.lan is not in 'Peer
in
> Cluster' state
>
> node 1 glusterd.log is here
>
> root at pri:/var/log/glusterfs# cat glusterd.log
> [2018-02-06 13:28:37.638373] W [glusterfsd.c:1331:cleanup_and_exit]
> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f0232faa6ba]
> -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55e17938a8c5]
> -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55e17938a6e4] ) 0-:
received
> signum (15), shutting down
> [2018-02-06 13:29:41.260479] I [MSGID: 100030] [glusterfsd.c:2476:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.3
> (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
> [2018-02-06 13:29:41.284367] I [MSGID: 106478] [glusterd.c:1422:init]
> 0-management: Maximum allowed open file descriptors set to 65536
> [2018-02-06 13:29:41.284462] I [MSGID: 106479] [glusterd.c:1469:init]
> 0-management: Using /var/lib/glusterd as working directory
> [2018-02-06 13:29:41.300804] W [MSGID: 103071]
> [rdma.c:4591:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
> channel creation failed [No such device]
> [2018-02-06 13:29:41.300969] W [MSGID: 103055] [rdma.c:4898:init]
> 0-rdma.management: Failed to initialize IB Device
> [2018-02-06 13:29:41.301098] W [rpc-transport.c:350:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2018-02-06 13:29:41.301190] W [rpcsvc.c:1660:rpcsvc_create_listener]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2018-02-06 13:29:41.301214] E [MSGID: 106243] [glusterd.c:1693:init]
> 0-management: creation of 1 listeners failed, continuing with succeeded
> transport
> [2018-02-06 13:29:44.621889] E [MSGID: 101032]
> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
> /var/lib/glusterd/glusterd.info. [No such file or directory]
> [2018-02-06 13:29:44.621967] E [MSGID: 101032]
> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
> /var/lib/glusterd/glusterd.info. [No such file or directory]
> [2018-02-06 13:29:44.621971] I [MSGID: 106514]
> [glusterd-store.c:2215:glusterd_restore_op_version] 0-management: Detected
> new install. Setting op-version to maximum : 31100
> [2018-02-06 13:29:44.625749] I [MSGID: 106194]
> [glusterd-store.c:3772:glusterd_store_retrieve_missed_snaps_list]
> 0-management: No missed snaps list.
> Final graph:
>
+------------------------------------------------------------------------------+
>   1: volume management
>   2:     type mgmt/glusterd
>   3:     option rpc-auth.auth-glusterfs on
>   4:     option rpc-auth.auth-unix on
>   5:     option rpc-auth.auth-null on
>   6:     option rpc-auth-allow-insecure on
>   7:     option transport.socket.listen-backlog 128
>   8:     option event-threads 1
>   9:     option ping-timeout 0
>  10:     option transport.socket.read-fail-log off
>  11:     option transport.socket.keepalive-interval 2
>  12:     option transport.socket.keepalive-time 10
>  13:     option transport-type rdma
>  14:     option working-directory /var/lib/glusterd
>  15: end-volume
>  16:
>
+------------------------------------------------------------------------------+
> [2018-02-06 13:29:44.628451] I [MSGID: 101190]
> [event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread
with
> index 1
> [2018-02-06 13:46:38.530154] I [MSGID: 106487]
> [glusterd-handler.c:1484:__glusterd_handle_cli_list_friends] 0-glusterd:
> Received cli list req
> [2018-02-06 13:47:05.745357] I [MSGID: 106487]
> [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received
> CLI probe req sec.ostechnix.lan 24007
> [2018-02-06 13:47:05.746465] I [MSGID: 106129]
> [glusterd-handler.c:3623:glusterd_probe_begin] 0-glusterd: Unable to find
> peerinfo for host: sec.ostechnix.lan (24007)
> [2018-02-06 13:47:05.751131] W [MSGID: 106062]
> [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd:
> Failed to get tcp-user-timeout
> [2018-02-06 13:47:05.751179] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2018-02-06 13:47:05.751345] W [MSGID: 101002]
> [options.c:954:xl_opt_validate] 0-management: option
'address-family' is
> deprecated, preferred is 'transport.address-family', continuing
with
> correction
> [2018-02-06 13:47:05.751902] I [MSGID: 106498]
> [glusterd-handler.c:3549:glusterd_friend_add] 0-management: connect
returned
> 0
> [2018-02-06 13:47:05.769054] E [MSGID: 101032]
> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
> /var/lib/glusterd/glusterd.info. [No such file or directory]
> [2018-02-06 13:47:05.769160] I [MSGID: 106477]
> [glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID:
> 476b754c-24cd-4816-a630-99c1b696a9e6
> [2018-02-06 13:47:05.806715] I [MSGID: 106511]
> [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe
> resp from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host:
> sec.ostechnix.lan
> [2018-02-06 13:47:05.806764] I [MSGID: 106511]
> [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to
> probe req
> [2018-02-06 13:47:05.816670] I [MSGID: 106493]
> [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
> from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host: sec.ostechnix.lan,
> port: 0
> [2018-02-06 13:47:05.831231] I [MSGID: 106163]
> [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack]
0-management:
> using the op-version 31100
> [2018-02-06 13:47:05.845025] I [MSGID: 106490]
> [glusterd-handler.c:2890:__glusterd_handle_probe_query] 0-glusterd:
Received
> probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.845156] I [MSGID: 106493]
> [glusterd-handler.c:2953:__glusterd_handle_probe_query] 0-glusterd:
> Responded to sec.ostechnix.lan, op_ret: 0, op_errno: 0, ret: 0
> [2018-02-06 13:47:05.852449] I [MSGID: 106490]
> [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd:
> Received probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.855861] I [MSGID: 106493]
> [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to sec.ostechnix.lan (0), ret: 0, op_ret: 0
> [2018-02-06 13:47:05.863598] I [MSGID: 106492]
> [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd:
> Received friend update from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.863674] I [MSGID: 106502]
> [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management:
> Received my uuid as Friend
> [2018-02-06 13:47:05.866932] I [MSGID: 106493]
> [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management:
Received
> ACC from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:48:28.152542] I [MSGID: 106487]
> [glusterd-handler.c:1484:__glusterd_handle_cli_list_friends] 0-glusterd:
> Received cli list req
> [2018-02-06 13:48:57.283814] E [MSGID: 106429]
> [glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management:
> Failed to convert hostname pri.ostechnix.lan to uuid
> [2018-02-06 13:48:57.283871] E [MSGID: 106301]
> [glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of
> operation 'Volume Create' failed on localhost : Host
pri.ostechnix.lan is
> not in 'Peer in Cluster' state
> [2018-02-06 13:49:33.997818] E [MSGID: 106429]
> [glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management:
> Failed to convert hostname pri.ostechnix.lan to uuid
> [2018-02-06 13:49:33.997826] E [MSGID: 106301]
> [glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of
> operation 'Volume Create' failed on localhost : Host
pri.ostechnix.lan is
> not in 'Peer in Cluster' state
>
> i need advice how can fix this issue.
>
> Thanks.
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

Ercan Aydoğan

2018-Feb-06 14:36 UTC

head link

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

Yes,
on node2 

root at sec:~# gluster peer probe pri.ostechnix.lan
peer probe: success. Host pri.ostechnix.lan port 24007 already in peer list
root at sec:~# gluster peer status
Number of Peers: 1

Hostname: pri.ostechnix.lan
Uuid: 476b754c-24cd-4816-a630-99c1b696a9e6
State: Peer in Cluster (Connected)
root at sec:~# 


 
> On 6 Feb 2018, at 17:05, Serkan ?oban <cobanserkan at gmail.com>
wrote:
> 
> Did you do gluster peer probe? Check out the documentation:
> http://docs.gluster.org/en/latest/Administrator%20Guide/Storage%20Pools/
> 
> On Tue, Feb 6, 2018 at 5:01 PM, Ercan Aydo?an <ercan.aydogan at
gmail.com> wrote:
>> Hello,
>> 
>> i installed glusterfs 3.11.3  version 3 nodes ubuntu 16.04 machine. All
>> machines have same /etc/hosts.
>> 
>> node1 hostname
>> pri.ostechnix.lan
>> 
>> node2 hostname
>> sec.ostechnix.lan
>> 
>> node2 hostname
>> third.ostechnix.lan
>> 
>> 
>> 51.15.77.14     pri.ostechnix.lan     pri
>> 51.15.90.60      sec.ostechnix.lan     sec
>> 163.172.151.120  third.ostechnix.lan   third
>> 
>> volume create command is
>> 
>> root at pri:/var/log/glusterfs# gluster volume create myvol1 replica 2
>> transport tcp pri.ostechnix.lan:/gluster/brick1/mpoint1
>> sec.ostechnix.lan:/gluster/brick1/mpoint1 force
>> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
>> avoid this. See:
>>
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.
>> Do you still want to continue?
>> (y/n) y
>> volume create: myvol1: failed: Host pri.ostechnix.lan is not in
'Peer in
>> Cluster' state
>> 
>> node 1 glusterd.log is here
>> 
>> root at pri:/var/log/glusterfs# cat glusterd.log
>> [2018-02-06 13:28:37.638373] W [glusterfsd.c:1331:cleanup_and_exit]
>> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f0232faa6ba]
>> -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55e17938a8c5]
>> -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55e17938a6e4] ) 0-:
received
>> signum (15), shutting down
>> [2018-02-06 13:29:41.260479] I [MSGID: 100030] [glusterfsd.c:2476:main]
>> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.3
>> (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
>> [2018-02-06 13:29:41.284367] I [MSGID: 106478] [glusterd.c:1422:init]
>> 0-management: Maximum allowed open file descriptors set to 65536
>> [2018-02-06 13:29:41.284462] I [MSGID: 106479] [glusterd.c:1469:init]
>> 0-management: Using /var/lib/glusterd as working directory
>> [2018-02-06 13:29:41.300804] W [MSGID: 103071]
>> [rdma.c:4591:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
>> channel creation failed [No such device]
>> [2018-02-06 13:29:41.300969] W [MSGID: 103055] [rdma.c:4898:init]
>> 0-rdma.management: Failed to initialize IB Device
>> [2018-02-06 13:29:41.301098] W [rpc-transport.c:350:rpc_transport_load]
>> 0-rpc-transport: 'rdma' initialization failed
>> [2018-02-06 13:29:41.301190] W [rpcsvc.c:1660:rpcsvc_create_listener]
>> 0-rpc-service: cannot create listener, initing the transport failed
>> [2018-02-06 13:29:41.301214] E [MSGID: 106243] [glusterd.c:1693:init]
>> 0-management: creation of 1 listeners failed, continuing with succeeded
>> transport
>> [2018-02-06 13:29:44.621889] E [MSGID: 101032]
>> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
>> /var/lib/glusterd/glusterd.info. [No such file or directory]
>> [2018-02-06 13:29:44.621967] E [MSGID: 101032]
>> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
>> /var/lib/glusterd/glusterd.info. [No such file or directory]
>> [2018-02-06 13:29:44.621971] I [MSGID: 106514]
>> [glusterd-store.c:2215:glusterd_restore_op_version] 0-management:
Detected
>> new install. Setting op-version to maximum : 31100
>> [2018-02-06 13:29:44.625749] I [MSGID: 106194]
>> [glusterd-store.c:3772:glusterd_store_retrieve_missed_snaps_list]
>> 0-management: No missed snaps list.
>> Final graph:
>>
+------------------------------------------------------------------------------+
>>  1: volume management
>>  2:     type mgmt/glusterd
>>  3:     option rpc-auth.auth-glusterfs on
>>  4:     option rpc-auth.auth-unix on
>>  5:     option rpc-auth.auth-null on
>>  6:     option rpc-auth-allow-insecure on
>>  7:     option transport.socket.listen-backlog 128
>>  8:     option event-threads 1
>>  9:     option ping-timeout 0
>> 10:     option transport.socket.read-fail-log off
>> 11:     option transport.socket.keepalive-interval 2
>> 12:     option transport.socket.keepalive-time 10
>> 13:     option transport-type rdma
>> 14:     option working-directory /var/lib/glusterd
>> 15: end-volume
>> 16:
>>
+------------------------------------------------------------------------------+
>> [2018-02-06 13:29:44.628451] I [MSGID: 101190]
>> [event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread
with
>> index 1
>> [2018-02-06 13:46:38.530154] I [MSGID: 106487]
>> [glusterd-handler.c:1484:__glusterd_handle_cli_list_friends]
0-glusterd:
>> Received cli list req
>> [2018-02-06 13:47:05.745357] I [MSGID: 106487]
>> [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd:
Received
>> CLI probe req sec.ostechnix.lan 24007
>> [2018-02-06 13:47:05.746465] I [MSGID: 106129]
>> [glusterd-handler.c:3623:glusterd_probe_begin] 0-glusterd: Unable to
find
>> peerinfo for host: sec.ostechnix.lan (24007)
>> [2018-02-06 13:47:05.751131] W [MSGID: 106062]
>> [glusterd-handler.c:3399:glusterd_transport_inet_options_build]
0-glusterd:
>> Failed to get tcp-user-timeout
>> [2018-02-06 13:47:05.751179] I
[rpc-clnt.c:1059:rpc_clnt_connection_init]
>> 0-management: setting frame-timeout to 600
>> [2018-02-06 13:47:05.751345] W [MSGID: 101002]
>> [options.c:954:xl_opt_validate] 0-management: option
'address-family' is
>> deprecated, preferred is 'transport.address-family', continuing
with
>> correction
>> [2018-02-06 13:47:05.751902] I [MSGID: 106498]
>> [glusterd-handler.c:3549:glusterd_friend_add] 0-management: connect
returned
>> 0
>> [2018-02-06 13:47:05.769054] E [MSGID: 101032]
>> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
>> /var/lib/glusterd/glusterd.info. [No such file or directory]
>> [2018-02-06 13:47:05.769160] I [MSGID: 106477]
>> [glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated
UUID:
>> 476b754c-24cd-4816-a630-99c1b696a9e6
>> [2018-02-06 13:47:05.806715] I [MSGID: 106511]
>> [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received
probe
>> resp from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host:
>> sec.ostechnix.lan
>> [2018-02-06 13:47:05.806764] I [MSGID: 106511]
>> [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp
to
>> probe req
>> [2018-02-06 13:47:05.816670] I [MSGID: 106493]
>> [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received
ACC
>> from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host:
sec.ostechnix.lan,
>> port: 0
>> [2018-02-06 13:47:05.831231] I [MSGID: 106163]
>> [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack]
0-management:
>> using the op-version 31100
>> [2018-02-06 13:47:05.845025] I [MSGID: 106490]
>> [glusterd-handler.c:2890:__glusterd_handle_probe_query] 0-glusterd:
Received
>> probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
>> [2018-02-06 13:47:05.845156] I [MSGID: 106493]
>> [glusterd-handler.c:2953:__glusterd_handle_probe_query] 0-glusterd:
>> Responded to sec.ostechnix.lan, op_ret: 0, op_errno: 0, ret: 0
>> [2018-02-06 13:47:05.852449] I [MSGID: 106490]
>> [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req]
0-glusterd:
>> Received probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
>> [2018-02-06 13:47:05.855861] I [MSGID: 106493]
>> [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd:
>> Responded to sec.ostechnix.lan (0), ret: 0, op_ret: 0
>> [2018-02-06 13:47:05.863598] I [MSGID: 106492]
>> [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd:
>> Received friend update from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
>> [2018-02-06 13:47:05.863674] I [MSGID: 106502]
>> [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management:
>> Received my uuid as Friend
>> [2018-02-06 13:47:05.866932] I [MSGID: 106493]
>> [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management:
Received
>> ACC from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
>> [2018-02-06 13:48:28.152542] I [MSGID: 106487]
>> [glusterd-handler.c:1484:__glusterd_handle_cli_list_friends]
0-glusterd:
>> Received cli list req
>> [2018-02-06 13:48:57.283814] E [MSGID: 106429]
>> [glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management:
>> Failed to convert hostname pri.ostechnix.lan to uuid
>> [2018-02-06 13:48:57.283871] E [MSGID: 106301]
>> [glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of
>> operation 'Volume Create' failed on localhost : Host
pri.ostechnix.lan is
>> not in 'Peer in Cluster' state
>> [2018-02-06 13:49:33.997818] E [MSGID: 106429]
>> [glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management:
>> Failed to convert hostname pri.ostechnix.lan to uuid
>> [2018-02-06 13:49:33.997826] E [MSGID: 106301]
>> [glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of
>> operation 'Volume Create' failed on localhost : Host
pri.ostechnix.lan is
>> not in 'Peer in Cluster' state
>> 
>> i need advice how can fix this issue.
>> 
>> Thanks.
>> 
>> 
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180206/7906705d/attachment.html>

Atin Mukherjee

2018-Feb-06 16:01 UTC

head link

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

I'm guessing there's something wrong w.r.t address resolution on node
1.>From the logs it's quite clear to me that node 1 is unable to resolve
theaddress configured in /etc/hosts where as the other nodes do. Could you
paste the gluster peer status output from all the nodes?

Also can you please check if you're able to ping
"pri.ostechnix.lan" from
node1 only? Does volume create go through if you use the IP instead of the
hostname?


On Tue, Feb 6, 2018 at 7:31 PM, Ercan Aydo?an <ercan.aydogan at gmail.com>
wrote:
> Hello,
>
> i installed glusterfs 3.11.3  version 3 nodes ubuntu 16.04 machine. All
> machines have same /etc/hosts.
>
> node1 hostname
> pri.ostechnix.lan
>
> node2 hostname
> sec.ostechnix.lan
>
> node2 hostname
> third.ostechnix.lan
>
>
> 51.15.77.14     pri.ostechnix.lan     pri
> 51.15.90.60      sec.ostechnix.lan     sec
> 163.172.151.120  third.ostechnix.lan   third
>
> volume create command is
>
> root at pri:/var/log/glusterfs# gluster volume create myvol1 replica 2
> transport tcp pri.ostechnix.lan:/gluster/brick1/mpoint1
> sec.ostechnix.lan:/gluster/brick1/mpoint1 force
> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
> avoid this. See:  https://gluster.readthedocs.io/en/latest/Administrator%
> 20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.
> Do you still want to continue?
>  (y/n) y
> volume create: myvol1: failed: Host pri.ostechnix.lan is not in 'Peer
in
> Cluster' state
>
> node 1 glusterd.log is here
>
> root at pri:/var/log/glusterfs# cat glusterd.log
> [2018-02-06 13:28:37.638373] W [glusterfsd.c:1331:cleanup_and_exit]
> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f0232faa6ba]
> -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55e17938a8c5]
> -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55e17938a6e4] ) 0-:
> received signum (15), shutting down
> [2018-02-06 13:29:41.260479] I [MSGID: 100030] [glusterfsd.c:2476:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.3
> (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
> [2018-02-06 13:29:41.284367] I [MSGID: 106478] [glusterd.c:1422:init]
> 0-management: Maximum allowed open file descriptors set to 65536
> [2018-02-06 13:29:41.284462] I [MSGID: 106479] [glusterd.c:1469:init]
> 0-management: Using /var/lib/glusterd as working directory
> [2018-02-06 13:29:41.300804] W [MSGID: 103071]
[rdma.c:4591:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such
device]
> [2018-02-06 13:29:41.300969] W [MSGID: 103055] [rdma.c:4898:init]
> 0-rdma.management: Failed to initialize IB Device
> [2018-02-06 13:29:41.301098] W [rpc-transport.c:350:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2018-02-06 13:29:41.301190] W [rpcsvc.c:1660:rpcsvc_create_listener]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2018-02-06 13:29:41.301214] E [MSGID: 106243] [glusterd.c:1693:init]
> 0-management: creation of 1 listeners failed, continuing with succeeded
> transport
> [2018-02-06 13:29:44.621889] E [MSGID: 101032]
> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
> /var/lib/glusterd/glusterd.info. [No such file or directory]
> [2018-02-06 13:29:44.621967] E [MSGID: 101032]
> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
> /var/lib/glusterd/glusterd.info. [No such file or directory]
> [2018-02-06 13:29:44.621971] I [MSGID: 106514]
[glusterd-store.c:2215:glusterd_restore_op_version]
> 0-management: Detected new install. Setting op-version to maximum : 31100
> [2018-02-06 13:29:44.625749] I [MSGID: 106194] [glusterd-store.c:3772:
> glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps
> list.
> Final graph:
> +-----------------------------------------------------------
> -------------------+
>   1: volume management
>   2:     type mgmt/glusterd
>   3:     option rpc-auth.auth-glusterfs on
>   4:     option rpc-auth.auth-unix on
>   5:     option rpc-auth.auth-null on
>   6:     option rpc-auth-allow-insecure on
>   7:     option transport.socket.listen-backlog 128
>   8:     option event-threads 1
>   9:     option ping-timeout 0
>  10:     option transport.socket.read-fail-log off
>  11:     option transport.socket.keepalive-interval 2
>  12:     option transport.socket.keepalive-time 10
>  13:     option transport-type rdma
>  14:     option working-directory /var/lib/glusterd
>  15: end-volume
>  16:
> +-----------------------------------------------------------
> -------------------+
> [2018-02-06 13:29:44.628451] I [MSGID: 101190]
[event-epoll.c:602:event_dispatch_epoll_worker]
> 0-epoll: Started thread with index 1
> [2018-02-06 13:46:38.530154] I [MSGID: 106487] [glusterd-handler.c:1484:__
> glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
> [2018-02-06 13:47:05.745357] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe]
> 0-glusterd: Received CLI probe req sec.ostechnix.lan 24007
> [2018-02-06 13:47:05.746465] I [MSGID: 106129]
[glusterd-handler.c:3623:glusterd_probe_begin]
> 0-glusterd: Unable to find peerinfo for host: sec.ostechnix.lan (24007)
> [2018-02-06 13:47:05.751131] W [MSGID: 106062] [glusterd-handler.c:3399:
> glusterd_transport_inet_options_build] 0-glusterd: Failed to get
> tcp-user-timeout
> [2018-02-06 13:47:05.751179] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2018-02-06 13:47:05.751345] W [MSGID: 101002]
[options.c:954:xl_opt_validate]
> 0-management: option 'address-family' is deprecated, preferred is
> 'transport.address-family', continuing with correction
> [2018-02-06 13:47:05.751902] I [MSGID: 106498]
[glusterd-handler.c:3549:glusterd_friend_add]
> 0-management: connect returned 0
> [2018-02-06 13:47:05.769054] E [MSGID: 101032]
> [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
> /var/lib/glusterd/glusterd.info. [No such file or directory]
> [2018-02-06 13:47:05.769160] I [MSGID: 106477]
> [glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated
> UUID: 476b754c-24cd-4816-a630-99c1b696a9e6
> [2018-02-06 13:47:05.806715] I [MSGID: 106511]
[glusterd-rpc-ops.c:261:__glusterd_probe_cbk]
> 0-management: Received probe resp from uuid:
1c041dbb-bad3-4158-97b7-fe47cddadada,
> host: sec.ostechnix.lan
> [2018-02-06 13:47:05.806764] I [MSGID: 106511]
[glusterd-rpc-ops.c:421:__glusterd_probe_cbk]
> 0-glusterd: Received resp to probe req
> [2018-02-06 13:47:05.816670] I [MSGID: 106493]
[glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk]
> 0-glusterd: Received ACC from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada,
> host: sec.ostechnix.lan, port: 0
> [2018-02-06 13:47:05.831231] I [MSGID: 106163]
> [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack]
> 0-management: using the op-version 31100
> [2018-02-06 13:47:05.845025] I [MSGID: 106490]
[glusterd-handler.c:2890:__glusterd_handle_probe_query]
> 0-glusterd: Received probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.845156] I [MSGID: 106493]
[glusterd-handler.c:2953:__glusterd_handle_probe_query]
> 0-glusterd: Responded to sec.ostechnix.lan, op_ret: 0, op_errno: 0, ret: 0
> [2018-02-06 13:47:05.852449] I [MSGID: 106490] [glusterd-handler.c:2539:__
> glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from
> uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.855861] I [MSGID: 106493]
[glusterd-handler.c:3799:glusterd_xfer_friend_add_resp]
> 0-glusterd: Responded to sec.ostechnix.lan (0), ret: 0, op_ret: 0
> [2018-02-06 13:47:05.863598] I [MSGID: 106492]
[glusterd-handler.c:2717:__glusterd_handle_friend_update]
> 0-glusterd: Received friend update from uuid: 1c041dbb-bad3-4158-97b7-
> fe47cddadada
> [2018-02-06 13:47:05.863674] I [MSGID: 106502]
[glusterd-handler.c:2762:__glusterd_handle_friend_update]
> 0-management: Received my uuid as Friend
> [2018-02-06 13:47:05.866932] I [MSGID: 106493]
[glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk]
> 0-management: Received ACC from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:48:28.152542] I [MSGID: 106487] [glusterd-handler.c:1484:__
> glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
> [2018-02-06 13:48:57.283814] E [MSGID: 106429] [glusterd-utils.c:1156:
> glusterd_brickinfo_new_from_brick] 0-management: Failed to convert
> hostname pri.ostechnix.lan to uuid
> [2018-02-06 13:48:57.283871] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase]
> 0-management: Staging of operation 'Volume Create' failed on
localhost :
> Host pri.ostechnix.lan is not in 'Peer in Cluster' state
> [2018-02-06 13:49:33.997818] E [MSGID: 106429] [glusterd-utils.c:1156:
> glusterd_brickinfo_new_from_brick] 0-management: Failed to convert
> hostname pri.ostechnix.lan to uuid
> [2018-02-06 13:49:33.997826] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase]
> 0-management: Staging of operation 'Volume Create' failed on
localhost :
> Host pri.ostechnix.lan is not in 'Peer in Cluster' state
>
> i need advice how can fix this issue.
>
> Thanks.
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180206/716d6111/attachment.html>

Ercan Aydoğan

2018-Feb-06 16:37 UTC

head link

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

I changed /etc/hosts

127.0.0.1        pri.ostechnix.lan     pri
51.15.90.60      sec.ostechnix.lan     sec
163.172.151.120  third.ostechnix.lan   third

on every node matching hostname to 127.0.0.1

then 

root at pri:~# apt-get purge glusterfs-server
root at pri:~# rm -rf /var/lib/glusterd/
root at pri:~# rm -rf /var/log/glusterfs/
root at pri:~# apt-get install glusterfs-server
root at pri:~# apt-mark hold glusterfs*
root at pri:~# reboot now
root at pri:~# gluster peer probe sec.ostechnix.lan
peer probe: success.
root at pri:~# gluster peer probe third.ostechnix.lan
peer probe: success.
root at pri:/var/log/glusterfs# gluster volume create myvol1 replica 3 transport
tcp pri.ostechnix.lan:/gluster/brick1/mpoint1
sec.ostechnix.lan:/gluster/brick1/mpoint1
third.ostechnix.lan:/gluster/brick1/mpoint1 force
volume create: myvol1: success: please start the volume to access data 


and volume created
root at pri:~# gluster volume list
myvol1

Thank you @Serkan and @Atin




> On 6 Feb 2018, at 19:01, Atin Mukherjee <amukherj at redhat.com>
wrote:
> 
> I'm guessing there's something wrong w.r.t address resolution on
node 1. From the logs it's quite clear to me that node 1 is unable to
resolve the address configured in /etc/hosts where as the other nodes do. Could
you paste the gluster peer status output from all the nodes?
> 
> Also can you please check if you're able to ping
"pri.ostechnix.lan" from node1 only? Does volume create go through if
you use the IP instead of the hostname?
> 
> 
> On Tue, Feb 6, 2018 at 7:31 PM, Ercan Aydo?an <ercan.aydogan at
gmail.com <mailto:ercan.aydogan at gmail.com>> wrote:
> Hello,
> 
> i installed glusterfs 3.11.3  version 3 nodes ubuntu 16.04 machine. All
machines have same /etc/hosts.
> 
> node1 hostname
> pri.ostechnix.lan
> 
> node2 hostname
> sec.ostechnix.lan
> 
> node2 hostname
> third.ostechnix.lan 
> 
> 
> 51.15.77.14     pri.ostechnix.lan     pri
> 51.15.90.60      sec.ostechnix.lan     sec
> 163.172.151.120  third.ostechnix.lan   third
> 
> volume create command is 
> 
> root at pri:/var/log/glusterfs# gluster volume create myvol1 replica 2
transport tcp pri.ostechnix.lan:/gluster/brick1/mpoint1
sec.ostechnix.lan:/gluster/brick1/mpoint1 force
> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
avoid this. See: 
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/
<https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/>.
> Do you still want to continue?
>  (y/n) y
> volume create: myvol1: failed: Host pri.ostechnix.lan is not in 'Peer
in Cluster' state
> 
> node 1 glusterd.log is here
> 
> root at pri:/var/log/glusterfs# cat glusterd.log 
> [2018-02-06 13:28:37.638373] W [glusterfsd.c:1331:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f0232faa6ba]
-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55e17938a8c5]
-->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55e17938a6e4] ) 0-: received
signum (15), shutting down
> [2018-02-06 13:29:41.260479] I [MSGID: 100030] [glusterfsd.c:2476:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.3 (args:
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
> [2018-02-06 13:29:41.284367] I [MSGID: 106478] [glusterd.c:1422:init]
0-management: Maximum allowed open file descriptors set to 65536
> [2018-02-06 13:29:41.284462] I [MSGID: 106479] [glusterd.c:1469:init]
0-management: Using /var/lib/glusterd as working directory
> [2018-02-06 13:29:41.300804] W [MSGID: 103071]
[rdma.c:4591:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel
creation failed [No such device]
> [2018-02-06 13:29:41.300969] W [MSGID: 103055] [rdma.c:4898:init]
0-rdma.management: Failed to initialize IB Device
> [2018-02-06 13:29:41.301098] W [rpc-transport.c:350:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
> [2018-02-06 13:29:41.301190] W [rpcsvc.c:1660:rpcsvc_create_listener]
0-rpc-service: cannot create listener, initing the transport failed
> [2018-02-06 13:29:41.301214] E [MSGID: 106243] [glusterd.c:1693:init]
0-management: creation of 1 listeners failed, continuing with succeeded
transport
> [2018-02-06 13:29:44.621889] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info <http://glusterd.info/>. [No such file or
directory]
> [2018-02-06 13:29:44.621967] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info <http://glusterd.info/>. [No such file or
directory]
> [2018-02-06 13:29:44.621971] I [MSGID: 106514]
[glusterd-store.c:2215:glusterd_restore_op_version] 0-management: Detected new
install. Setting op-version to maximum : 31100
> [2018-02-06 13:29:44.625749] I [MSGID: 106194]
[glusterd-store.c:3772:glusterd_store_retrieve_missed_snaps_list] 0-management:
No missed snaps list.
> Final graph:
>
+------------------------------------------------------------------------------+
>   1: volume management
>   2:     type mgmt/glusterd
>   3:     option rpc-auth.auth-glusterfs on
>   4:     option rpc-auth.auth-unix on
>   5:     option rpc-auth.auth-null on
>   6:     option rpc-auth-allow-insecure on
>   7:     option transport.socket.listen-backlog 128
>   8:     option event-threads 1
>   9:     option ping-timeout 0
>  10:     option transport.socket.read-fail-log off
>  11:     option transport.socket.keepalive-interval 2
>  12:     option transport.socket.keepalive-time 10
>  13:     option transport-type rdma
>  14:     option working-directory /var/lib/glusterd
>  15: end-volume
>  16:  
>
+------------------------------------------------------------------------------+
> [2018-02-06 13:29:44.628451] I [MSGID: 101190]
[event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 1
> [2018-02-06 13:46:38.530154] I [MSGID: 106487]
[glusterd-handler.c:1484:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
> [2018-02-06 13:47:05.745357] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req sec.ostechnix.lan 24007
> [2018-02-06 13:47:05.746465] I [MSGID: 106129]
[glusterd-handler.c:3623:glusterd_probe_begin] 0-glusterd: Unable to find
peerinfo for host: sec.ostechnix.lan (24007)
> [2018-02-06 13:47:05.751131] W [MSGID: 106062]
[glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout
> [2018-02-06 13:47:05.751179] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
> [2018-02-06 13:47:05.751345] W [MSGID: 101002]
[options.c:954:xl_opt_validate] 0-management: option 'address-family' is
deprecated, preferred is 'transport.address-family', continuing with
correction
> [2018-02-06 13:47:05.751902] I [MSGID: 106498]
[glusterd-handler.c:3549:glusterd_friend_add] 0-management: connect returned 0
> [2018-02-06 13:47:05.769054] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info <http://glusterd.info/>. [No such file or
directory]
> [2018-02-06 13:47:05.769160] I [MSGID: 106477]
[glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID:
476b754c-24cd-4816-a630-99c1b696a9e6
> [2018-02-06 13:47:05.806715] I [MSGID: 106511]
[glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp
from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host: sec.ostechnix.lan
> [2018-02-06 13:47:05.806764] I [MSGID: 106511]
[glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe
req
> [2018-02-06 13:47:05.816670] I [MSGID: 106493]
[glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from
uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host: sec.ostechnix.lan, port: 0
> [2018-02-06 13:47:05.831231] I [MSGID: 106163]
[glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 31100
> [2018-02-06 13:47:05.845025] I [MSGID: 106490]
[glusterd-handler.c:2890:__glusterd_handle_probe_query] 0-glusterd: Received
probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.845156] I [MSGID: 106493]
[glusterd-handler.c:2953:__glusterd_handle_probe_query] 0-glusterd: Responded to
sec.ostechnix.lan, op_ret: 0, op_errno: 0, ret: 0
> [2018-02-06 13:47:05.852449] I [MSGID: 106490]
[glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.855861] I [MSGID: 106493]
[glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
sec.ostechnix.lan (0), ret: 0, op_ret: 0
> [2018-02-06 13:47:05.863598] I [MSGID: 106492]
[glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received
friend update from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.863674] I [MSGID: 106502]
[glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received
my uuid as Friend
> [2018-02-06 13:47:05.866932] I [MSGID: 106493]
[glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC
from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:48:28.152542] I [MSGID: 106487]
[glusterd-handler.c:1484:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
> [2018-02-06 13:48:57.283814] E [MSGID: 106429]
[glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management: Failed
to convert hostname pri.ostechnix.lan to uuid
> [2018-02-06 13:48:57.283871] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of operation
'Volume Create' failed on localhost : Host pri.ostechnix.lan is not in
'Peer in Cluster' state
> [2018-02-06 13:49:33.997818] E [MSGID: 106429]
[glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management: Failed
to convert hostname pri.ostechnix.lan to uuid
> [2018-02-06 13:49:33.997826] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of operation
'Volume Create' failed on localhost : Host pri.ostechnix.lan is not in
'Peer in Cluster' state
> 
> i need advice how can fix this issue.
> 
> Thanks.
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180206/0c62c04b/attachment.html>

Ercan Aydoğan

2018-Feb-06 21:19 UTC

head link

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

With hostname it's ok. But if i peer probe with public ip address error
message come back again.

I removed previous installation then after fresh installation

on node 1

root at pri:/var/log/glusterfs# gluster peer probe 51.15.90.60
peer probe: success. 
root at pri:/var/log/glusterfs# gluster peer probe 163.172.151.120
peer probe: success.

on node 2

root at sec:~# gluster peer probe 51.15.77.14
peer probe: success. Host 51.15.77.14 port 24007 already in peer list


on node 1


root at pri:/var/log/glusterfs# gluster volume create myvol1 replica 3 transport
tcp 51.15.77.14:/gluster/brick1/mpoint1 51.15.90.60:/gluster/brick1/mpoint1 
163.172.151.120:/gluster/brick1/mpoint1 force
volume create: myvol1: failed: Host 51.15.77.14 is not in 'Peer in
Cluster' state

i think /etc/hosts and hostname issue but i?m not sure how can i fix this issue.

i commented previous etc host entries for ip resolve.

127.0.0.1       localhost
#127.0.0.1       pri.ostechnix.lan     pri
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters

#51.15.90.10      sec.ostechnix.lan     sec
#163.172.151.170  third.ostechnix.lan   third
                                              

on node 1 

root at pri:/var/log/glusterfs# gluster peer status
Number of Peers: 2

Hostname: 51.15.90.60
Uuid: 4d0958b1-0897-4d90-9817-f035c8ac5c27
State: Peer in Cluster (Connected)

Hostname: 163.172.151.120
Uuid: fda09db2-d549-4431-91c7-58336ba8b27d
State: Peer in Cluster (Connected)



root at pri:/var/log/glusterfs# cat glusterd.log 

[2018-02-06 21:03:20.744742] I [MSGID: 100030] [glusterfsd.c:2476:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.3 (args:
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2018-02-06 21:03:20.753570] I [MSGID: 106478] [glusterd.c:1422:init]
0-management: Maximum allowed open file descriptors set to 65536
[2018-02-06 21:03:20.753615] I [MSGID: 106479] [glusterd.c:1469:init]
0-management: Using /var/lib/glusterd as working directory
[2018-02-06 21:03:20.760257] W [MSGID: 103071]
[rdma.c:4591:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel
creation failed [No such device]
[2018-02-06 21:03:20.760307] W [MSGID: 103055] [rdma.c:4898:init]
0-rdma.management: Failed to initialize IB Device
[2018-02-06 21:03:20.760331] W [rpc-transport.c:350:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2018-02-06 21:03:20.760407] W [rpcsvc.c:1660:rpcsvc_create_listener]
0-rpc-service: cannot create listener, initing the transport failed
[2018-02-06 21:03:20.760430] E [MSGID: 106243] [glusterd.c:1693:init]
0-management: creation of 1 listeners failed, continuing with succeeded
transport
[2018-02-06 21:03:23.983517] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-06 21:03:23.983595] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-06 21:03:23.983599] I [MSGID: 106514]
[glusterd-store.c:2215:glusterd_restore_op_version] 0-management: Detected new
install. Setting op-version to maximum : 31100
[2018-02-06 21:03:23.983654] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/options. [No such file or directory]
[2018-02-06 21:03:23.987234] I [MSGID: 106194]
[glusterd-store.c:3772:glusterd_store_retrieve_missed_snaps_list] 0-management:
No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.socket.listen-backlog 128
  8:     option event-threads 1
  9:     option ping-timeout 0
 10:     option transport.socket.read-fail-log off
 11:     option transport.socket.keepalive-interval 2
 12:     option transport.socket.keepalive-time 10
 13:     option transport-type rdma
 14:     option working-directory /var/lib/glusterd
 15: end-volume
 16:  
+------------------------------------------------------------------------------+
[2018-02-06 21:03:23.987671] I [MSGID: 101190]
[event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 1
[2018-02-06 21:04:25.506078] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req 51.15.90.60 24007
[2018-02-06 21:04:25.506808] I [MSGID: 106129]
[glusterd-handler.c:3623:glusterd_probe_begin] 0-glusterd: Unable to find
peerinfo for host: 51.15.90.60 (24007)
[2018-02-06 21:04:25.510993] W [MSGID: 106062]
[glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout
[2018-02-06 21:04:25.511039] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2018-02-06 21:04:25.511210] W [MSGID: 101002] [options.c:954:xl_opt_validate]
0-management: option 'address-family' is deprecated, preferred is
'transport.address-family', continuing with correction
[2018-02-06 21:04:25.511930] I [MSGID: 106498]
[glusterd-handler.c:3549:glusterd_friend_add] 0-management: connect returned 0
[2018-02-06 21:04:25.524806] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-06 21:04:25.524931] I [MSGID: 106477]
[glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID:
9a482152-68f6-4909-94ef-73ef6f2aaeeb
[2018-02-06 21:04:25.607450] I [MSGID: 106511]
[glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp
from uuid: d66dc41b-794f-4cc6-a733-f8aecc47c7cc, host: 51.15.90.60
[2018-02-06 21:04:25.607504] I [MSGID: 106511]
[glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe
req
[2018-02-06 21:04:25.659954] I [MSGID: 106493]
[glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from
uuid: d66dc41b-794f-4cc6-a733-f8aecc47c7cc, host: 51.15.90.60, port: 0
[2018-02-06 21:04:25.672115] I [MSGID: 106163]
[glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 31100
[2018-02-06 21:04:25.683412] I [MSGID: 106490]
[glusterd-handler.c:2890:__glusterd_handle_probe_query] 0-glusterd: Received
probe from uuid: d66dc41b-794f-4cc6-a733-f8aecc47c7cc
[2018-02-06 21:04:25.718307] I [MSGID: 106493]
[glusterd-handler.c:2953:__glusterd_handle_probe_query] 0-glusterd: Responded to
51.15.90.60, op_ret: 0, op_errno: 0, ret: 0
[2018-02-06 21:04:25.722659] I [MSGID: 106490]
[glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: d66dc41b-794f-4cc6-a733-f8aecc47c7cc
[2018-02-06 21:04:25.726852] I [MSGID: 106493]
[glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
51.15.90.60 (0), ret: 0, op_ret: 0
[2018-02-06 21:04:25.734638] I [MSGID: 106492]
[glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received
friend update from uuid: d66dc41b-794f-4cc6-a733-f8aecc47c7cc
[2018-02-06 21:04:25.734713] I [MSGID: 106502]
[glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received
my uuid as Friend
[2018-02-06 21:04:25.734812] I [MSGID: 106493]
[glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC
from uuid: d66dc41b-794f-4cc6-a733-f8aecc47c7cc
[2018-02-06 21:04:43.243191] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req 163.172.151.120 24007
[2018-02-06 21:04:43.244017] I [MSGID: 106129]
[glusterd-handler.c:3623:glusterd_probe_begin] 0-glusterd: Unable to find
peerinfo for host: 163.172.151.120 (24007)
[2018-02-06 21:04:43.248000] W [MSGID: 106062]
[glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout
[2018-02-06 21:04:43.248043] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2018-02-06 21:04:43.248208] W [MSGID: 101002] [options.c:954:xl_opt_validate]
0-management: option 'address-family' is deprecated, preferred is
'transport.address-family', continuing with correction
[2018-02-06 21:04:43.248625] I [MSGID: 106498]
[glusterd-handler.c:3549:glusterd_friend_add] 0-management: connect returned 0
[2018-02-06 21:04:43.614068] I [MSGID: 106511]
[glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp
from uuid: a934d4ad-b785-423d-a818-1a73b1340fc1, host: 163.172.151.120
[2018-02-06 21:04:43.614119] I [MSGID: 106511]
[glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe
req
[2018-02-06 21:04:43.669768] I [MSGID: 106493]
[glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from
uuid: a934d4ad-b785-423d-a818-1a73b1340fc1, host: 163.172.151.120, port: 0
[2018-02-06 21:04:43.708761] I [MSGID: 106163]
[glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 31100
[2018-02-06 21:04:43.738003] I [MSGID: 106490]
[glusterd-handler.c:2890:__glusterd_handle_probe_query] 0-glusterd: Received
probe from uuid: a934d4ad-b785-423d-a818-1a73b1340fc1
[2018-02-06 21:04:43.780644] I [MSGID: 106493]
[glusterd-handler.c:2953:__glusterd_handle_probe_query] 0-glusterd: Responded to
163.172.151.120, op_ret: 0, op_errno: 0, ret: 0
[2018-02-06 21:04:43.801411] I [MSGID: 106490]
[glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: a934d4ad-b785-423d-a818-1a73b1340fc1
[2018-02-06 21:04:43.804636] I [MSGID: 106493]
[glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
163.172.151.120 (0), ret: 0, op_ret: 0
[2018-02-06 21:04:43.816462] I [MSGID: 106493]
[glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC
from uuid: d66dc41b-794f-4cc6-a733-f8aecc47c7cc
[2018-02-06 21:04:43.820849] I [MSGID: 106492]
[glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received
friend update from uuid: a934d4ad-b785-423d-a818-1a73b1340fc1
[2018-02-06 21:04:43.820891] I [MSGID: 106502]
[glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received
my uuid as Friend
[2018-02-06 21:04:43.829278] I [MSGID: 106493]
[glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC
from uuid: a934d4ad-b785-423d-a818-1a73b1340fc1
[2018-02-06 21:05:45.857042] E [MSGID: 106429]
[glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management: Failed
to convert hostname 51.15.77.14 to uuid
[2018-02-06 21:05:45.857095] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of operation
'Volume Create' failed on localhost : Host 51.15.77.14 is not in
'Peer in Cluster' state





> On 6 Feb 2018, at 19:01, Atin Mukherjee <amukherj at redhat.com>
wrote:
> 
> I'm guessing there's something wrong w.r.t address resolution on
node 1. From the logs it's quite clear to me that node 1 is unable to
resolve the address configured in /etc/hosts where as the other nodes do. Could
you paste the gluster peer status output from all the nodes?
> 
> Also can you please check if you're able to ping
"pri.ostechnix.lan" from node1 only? Does volume create go through if
you use the IP instead of the hostname?
> 
> 
> On Tue, Feb 6, 2018 at 7:31 PM, Ercan Aydo?an <ercan.aydogan at
gmail.com <mailto:ercan.aydogan at gmail.com>> wrote:
> Hello,
> 
> i installed glusterfs 3.11.3  version 3 nodes ubuntu 16.04 machine. All
machines have same /etc/hosts.
> 
> node1 hostname
> pri.ostechnix.lan
> 
> node2 hostname
> sec.ostechnix.lan
> 
> node2 hostname
> third.ostechnix.lan 
> 
> 
> 51.15.77.14     pri.ostechnix.lan     pri
> 51.15.90.60      sec.ostechnix.lan     sec
> 163.172.151.120  third.ostechnix.lan   third
> 
> volume create command is 
> 
> root at pri:/var/log/glusterfs# gluster volume create myvol1 replica 2
transport tcp pri.ostechnix.lan:/gluster/brick1/mpoint1
sec.ostechnix.lan:/gluster/brick1/mpoint1 force
> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
avoid this. See: 
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/
<https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/>.
> Do you still want to continue?
>  (y/n) y
> volume create: myvol1: failed: Host pri.ostechnix.lan is not in 'Peer
in Cluster' state
> 
> node 1 glusterd.log is here
> 
> root at pri:/var/log/glusterfs# cat glusterd.log 
> [2018-02-06 13:28:37.638373] W [glusterfsd.c:1331:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f0232faa6ba]
-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55e17938a8c5]
-->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55e17938a6e4] ) 0-: received
signum (15), shutting down
> [2018-02-06 13:29:41.260479] I [MSGID: 100030] [glusterfsd.c:2476:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.3 (args:
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
> [2018-02-06 13:29:41.284367] I [MSGID: 106478] [glusterd.c:1422:init]
0-management: Maximum allowed open file descriptors set to 65536
> [2018-02-06 13:29:41.284462] I [MSGID: 106479] [glusterd.c:1469:init]
0-management: Using /var/lib/glusterd as working directory
> [2018-02-06 13:29:41.300804] W [MSGID: 103071]
[rdma.c:4591:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel
creation failed [No such device]
> [2018-02-06 13:29:41.300969] W [MSGID: 103055] [rdma.c:4898:init]
0-rdma.management: Failed to initialize IB Device
> [2018-02-06 13:29:41.301098] W [rpc-transport.c:350:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
> [2018-02-06 13:29:41.301190] W [rpcsvc.c:1660:rpcsvc_create_listener]
0-rpc-service: cannot create listener, initing the transport failed
> [2018-02-06 13:29:41.301214] E [MSGID: 106243] [glusterd.c:1693:init]
0-management: creation of 1 listeners failed, continuing with succeeded
transport
> [2018-02-06 13:29:44.621889] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info <http://glusterd.info/>. [No such file or
directory]
> [2018-02-06 13:29:44.621967] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info <http://glusterd.info/>. [No such file or
directory]
> [2018-02-06 13:29:44.621971] I [MSGID: 106514]
[glusterd-store.c:2215:glusterd_restore_op_version] 0-management: Detected new
install. Setting op-version to maximum : 31100
> [2018-02-06 13:29:44.625749] I [MSGID: 106194]
[glusterd-store.c:3772:glusterd_store_retrieve_missed_snaps_list] 0-management:
No missed snaps list.
> Final graph:
>
+------------------------------------------------------------------------------+
>   1: volume management
>   2:     type mgmt/glusterd
>   3:     option rpc-auth.auth-glusterfs on
>   4:     option rpc-auth.auth-unix on
>   5:     option rpc-auth.auth-null on
>   6:     option rpc-auth-allow-insecure on
>   7:     option transport.socket.listen-backlog 128
>   8:     option event-threads 1
>   9:     option ping-timeout 0
>  10:     option transport.socket.read-fail-log off
>  11:     option transport.socket.keepalive-interval 2
>  12:     option transport.socket.keepalive-time 10
>  13:     option transport-type rdma
>  14:     option working-directory /var/lib/glusterd
>  15: end-volume
>  16:  
>
+------------------------------------------------------------------------------+
> [2018-02-06 13:29:44.628451] I [MSGID: 101190]
[event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 1
> [2018-02-06 13:46:38.530154] I [MSGID: 106487]
[glusterd-handler.c:1484:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
> [2018-02-06 13:47:05.745357] I [MSGID: 106487]
[glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req sec.ostechnix.lan 24007
> [2018-02-06 13:47:05.746465] I [MSGID: 106129]
[glusterd-handler.c:3623:glusterd_probe_begin] 0-glusterd: Unable to find
peerinfo for host: sec.ostechnix.lan (24007)
> [2018-02-06 13:47:05.751131] W [MSGID: 106062]
[glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout
> [2018-02-06 13:47:05.751179] I [rpc-clnt.c:1059:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
> [2018-02-06 13:47:05.751345] W [MSGID: 101002]
[options.c:954:xl_opt_validate] 0-management: option 'address-family' is
deprecated, preferred is 'transport.address-family', continuing with
correction
> [2018-02-06 13:47:05.751902] I [MSGID: 106498]
[glusterd-handler.c:3549:glusterd_friend_add] 0-management: connect returned 0
> [2018-02-06 13:47:05.769054] E [MSGID: 101032]
[store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/glusterd.info <http://glusterd.info/>. [No such file or
directory]
> [2018-02-06 13:47:05.769160] I [MSGID: 106477]
[glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID:
476b754c-24cd-4816-a630-99c1b696a9e6
> [2018-02-06 13:47:05.806715] I [MSGID: 106511]
[glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp
from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host: sec.ostechnix.lan
> [2018-02-06 13:47:05.806764] I [MSGID: 106511]
[glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe
req
> [2018-02-06 13:47:05.816670] I [MSGID: 106493]
[glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from
uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada, host: sec.ostechnix.lan, port: 0
> [2018-02-06 13:47:05.831231] I [MSGID: 106163]
[glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 31100
> [2018-02-06 13:47:05.845025] I [MSGID: 106490]
[glusterd-handler.c:2890:__glusterd_handle_probe_query] 0-glusterd: Received
probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.845156] I [MSGID: 106493]
[glusterd-handler.c:2953:__glusterd_handle_probe_query] 0-glusterd: Responded to
sec.ostechnix.lan, op_ret: 0, op_errno: 0, ret: 0
> [2018-02-06 13:47:05.852449] I [MSGID: 106490]
[glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.855861] I [MSGID: 106493]
[glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
sec.ostechnix.lan (0), ret: 0, op_ret: 0
> [2018-02-06 13:47:05.863598] I [MSGID: 106492]
[glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received
friend update from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:47:05.863674] I [MSGID: 106502]
[glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received
my uuid as Friend
> [2018-02-06 13:47:05.866932] I [MSGID: 106493]
[glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC
from uuid: 1c041dbb-bad3-4158-97b7-fe47cddadada
> [2018-02-06 13:48:28.152542] I [MSGID: 106487]
[glusterd-handler.c:1484:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
> [2018-02-06 13:48:57.283814] E [MSGID: 106429]
[glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management: Failed
to convert hostname pri.ostechnix.lan to uuid
> [2018-02-06 13:48:57.283871] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of operation
'Volume Create' failed on localhost : Host pri.ostechnix.lan is not in
'Peer in Cluster' state
> [2018-02-06 13:49:33.997818] E [MSGID: 106429]
[glusterd-utils.c:1156:glusterd_brickinfo_new_from_brick] 0-management: Failed
to convert hostname pri.ostechnix.lan to uuid
> [2018-02-06 13:49:33.997826] E [MSGID: 106301]
[glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of operation
'Volume Create' failed on localhost : Host pri.ostechnix.lan is not in
'Peer in Cluster' state
> 
> i need advice how can fix this issue.
> 
> Thanks.
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180207/731675cc/attachment.html>

Maybe Matching Threads

Search for more apparently analagous threads

Gluster users - Feb 2018 - strange hostname issue on volume create command with famous Peer in Cluster state error message

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

[Gluster-users] strange hostname issue on volume create command with famous Peer in Cluster state error message

Maybe Matching Threads