huangql
2015-Jun-05 02:57 UTC
[Gluster-users] 回复： 答复: 答复: 答复: Gluster peer rejected and failed to start
Maybe you can check selinux on the server side. Make sure selinux is disabled. ???? ???:vyyy???yuyangyang at Ctrip.com ???:Atin Mukherjeeamukherj at redhat.com; Gluster-users at gluster.orgGluster-users@gluster.org ????:2015?6?4?(??)?21:24 ??:[Gluster-users] ??: ??: ??: Gluster peer rejected and failed to start Can anyone help me? New nodes still can't join the cluster. The new nodes can peer each other, just not exist cluster nodes. I can not find any network problem, there were no firewall, port 24007 is connected, there were all at the same subnet... Thanks [root at SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log [2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) [2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory [2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans port/rdma.so: cannot open shared object file: No such file or directory [2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport- type 'rdma' is not valid or not found on this machine [2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra nsport failed [2015-06-05 01:59:02.343346] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-06-05 01:59:02.351861] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-06-05 01:59:02.363138] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap s list. Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option ping-timeout 30 8: option transport.socket.read-fail-log off 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-time 10 11: option transport-type rdma 12: option working-directory /var/lib/glusterd 13: end-volume 14: +------------------------------------------------------------------------------+ [2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 1b98af34-08f9-492f-b54e-7608 2588d63b [2015-06-05 02:00:43.025462] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:00:44.025069] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:00:44.599862] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:00:46.029823] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:00:46.701845] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:01:43.119612] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:01:44.165527] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:01:44.851903] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:01:46.190401] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:01:46.722394] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:01:58.719968] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:32.584999] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust erfs05.sh2.ctripcorp.com 24007 [2015-06-05 02:02:32.586281] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g lusterfs05.sh2.ctripcorp.com (24007) [2015-06-05 02:02:32.602192] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-05 02:02:32.606053] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0 [2015-06-05 02:02:32.620958] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the 'versions' from peer (10.8.230.209:24007) [2015-06-05 02:02:32.621020] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer 000000 00-0000-0000-0000-000000000000, in Establishing Connection state, has disconnected from glusterd. [2015-06-05 02:02:43.264141] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:02:44.312918] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:02:44.781434] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:46.176888] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:46.799916] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:03:42.720994] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:03:43.771987] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:03:44.659775] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:03:45.956807] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:03:46.688787] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:04:42.891131] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:04:43.957423] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:04:44.995638] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:04:46.062171] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:04:46.957327] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:05:42.966651] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:05:44.008498] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:05:45.049936] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:05:46.087007] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:05:47.029325] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:06:42.874089] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:06:43.910998] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:06:44.932802] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:06:46.053731] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:06:47.055859] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:07:43.099647] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:07:44.146454] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:07:45.218982] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:07:46.246303] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:07:47.106954] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:08:43.226090] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:08:44.292041] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:08:45.351830] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:08:45.688349] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:08:46.651768] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:09:42.748628] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed Best Regards -----????----- ???: vyyy??? ????: Thursday, June 04, 2015 11:00 AM ???: 'Atin Mukherjee'; Gluster-users at gluster.org ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start ?? The problem 2 has resolved. From glusterd log file, I find Cksums error as following: [2015-06-04 01:25:03.137798] E [MSGID: 106010] [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of Cksums JQStore2 differ. local cksum = 2617028277, remote cksum = 2210669223 on peer glusterfs09.sh2.ctripcorp.com The solution for this problem was to do the following, on the BAD node: /etc/init.d/glusterd stop rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/ /etc/init.d/glusterd start The problem 1 is still under inverstigate: New nodes can not join the cluster, log file shown: failed to get the 'versions' from peer I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are " operating-version=2" ?? ??[2015-06-04 02:13:49.199416] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the ??'versions' from peer (10.8.230.211:24007) ??[2015-06-04 02:14:07.112577] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust ??erfs05.sh2.ctripcorp.com 24007 ??[2015-06-04 02:14:07.113778] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g ??lusterfs05.sh2.ctripcorp.com (24007) ??[2015-06-04 02:14:07.131023] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 ??[2015-06-04 02:14:07.134829] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0 ??[ Best Regards ??? Site Operations Center Ext.15554 Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----????----- ???: vyyy??? ????: Wednesday, June 03, 2015 5:37 PM ???: 'Atin Mukherjee'; Gluster-users at gluster.org ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Hi Atin and everybody, I have forced peer detach the news nodes ,The peer status seems even strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help me? Any advice is appreciated. Glusterfs05 ?? ??[root at sh02svr5956 ~]# gluster peer status ??Number of Peers: 5 ?? ??Hostname: glusterfs06.sh2.ctripcorp.com ??Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs07.sh2.ctripcorp.com ??Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs08.sh2.ctripcorp.com ??Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs10.sh2.ctripcorp.com ??Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c ??State: Peer in Cluster (Connected) ?? ??Hostname: glusterfs09.sh2.ctripcorp.com ??Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d ??State: Peer Rejected (Connected) ??[root at sh02svr5956 ~]# ?? ?? ??Glusterfs06 ?? ??[root at SH02SVR5955 ~]# gluster peer status ??Number of Peers: 5 ?? ??Hostname: glusterfs10.sh2.ctripcorp.com ??Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs08.sh2.ctripcorp.com ??Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 ??State: Peer in Cluster (Connected) ?? ??Hostname: glusterfs07.sh2.ctripcorp.com ??Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 ??State: Peer in Cluster (Connected) ?? ??Hostname: 10.8.230.209 ??Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs09.sh2.ctripcorp.com ??Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d ??State: Peer in Cluster (Connected) ??[root at SH02SVR5955 ~]# ?? Glusterfs07 [root at SH02SVR5954 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) [root at SH02SVR5954 ~]# Glusterfs08 [root at SH02SVR5953 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) [root at SH02SVR5953 ~]# [root at SH02SVR5952 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) [root at SH02SVR5952 ~]# Glusterfs10 [root at SH02SVR5951 ~]# gluster peer status Number of Peers: 6 Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer Rejected (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer Rejected (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer Rejected (Connected) [root at SH02SVR5951 ~]# Best Regards ??? Site Operations Center Ext.15554 Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----????----- ???: vyyy??? ????: Wednesday, June 03, 2015 2:55 PM ???: 'Atin Mukherjee' ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Can I delete the files in /var/lib/glusterd/peers which state is not =3 on all nodes, then restart glusterd? [root at sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15 uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15 state=6 hostname1=glusterfs12.sh2.ctripcorp.com Best Regards Yang -----????----- ???: vyyy??? ????: Wednesday, June 03, 2015 2:39 PM ???: 'Atin Mukherjee' ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Since it's intranet, You can only control my desktop, and then access these node. Should I apply a webex room? Best Regards ??? Site Operations Center Ext.15554 Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----????----- ???: Atin Mukherjee [mailto:amukherj at redhat.com] ????: Wednesday, June 03, 2015 2:36 PM ???: vyyy??? ??: Re: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Is it possible to share your ip/user/passwd for few of these nodes? On 06/03/2015 11:50 AM, vyyy??? wrote: Can you remote control my desktop (like webex)? The nodes host bricks. And on the new nodes, I did "clean /var/lib/glusterd and restart" several times, but have no effect. Is there any other work around? Best Regards Yang -----????----- ???: Atin Mukherjee [mailto:amukherj at redhat.com] ????: Wednesday, June 03, 2015 12:33 PM ???: vyyy???; Gluster-users at gluster.org ??: Re: ??: [Gluster-users] Gluster peer rejected and failed to start I can't comment on what went wrong till I can have visibility on the entire cluster. Does the node which fails to start host any brick? If not then work around could be to clean /var/lib/glusterd and restart glusterd, that should sync up data and bring this node in consistency. HTH, Atin On 06/03/2015 08:34 AM, vyyy??? wrote: I found that /var/lib/glusterd/peers can't include node's own peer, I delete this file. Glusterd started finally. But the peer status still not normal. Seems that glusterfs10 leave the cluster and join the new nodes alone. Peer status on glusterfs10 [root at SH02SVR5951 ~]# gluster peer status Number of Peers: 7 Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer Rejected (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Accepted peer request (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer Rejected (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer Rejected (Connected) Hostname: glusterfs12.sh2.ctripcorp.com Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 State: Peer in Cluster (Connected) Peer status on glusterfs10 [root at SVR6993HW2285 ~]# gluster peer status Number of Peers: 6 Hostname: glusterfs05.sh2.ctripcorp.com Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer Rejected (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer Rejected (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer Rejected (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer in Cluster (Connected) Peer status on glusterfs05, this is the node that I tried to peer probe new nodes (glusterfs11 to glusterfs14) [root at sh02svr5956 ~]# gluster peer status Number of Peers: 9 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) Hostname: glusterfs12.sh2.ctripcorp.com Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 State: Peer in Cluster (Connected) Hostname: glusterfs13.sh2.ctripcorp.com Uuid: 43e96d31-6c6d-4096-832a-440e544554ca State: Peer Rejected (Connected) Hostname: glusterfs14.sh2.ctripcorp.com Uuid: 45c11af9-7850-4598-ad62-4bb343b62524 State: Peer Rejected (Connected) [root at sh02svr5956 ~]# Best Regards Yang -----????----- ???: vyyy??? ????: Wednesday, June 03, 2015 10:04 AM ???: 'Atin Mukherjee'; 'Gluster-users at gluster.org' ??: [Gluster-users] Gluster peer rejected and failed to start I am still messed, can you help me figure out problem 2 (A node of cluster failed to start glusterd) priority The glusterd log on the node gluster10 shown ' resolve brick failed in restore' I checked /var/lib/glusterd/vols, seems ok Glusterd log file Error message: [2015-06-03 01:31:32.564010] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again Best Regards Yang -----????----- ???: vyyy??? ????: Tuesday, June 02, 2015 6:27 PM ???: 'Atin Mukherjee'; Gluster-users at gluster.org ??: ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Gluster11 to gluster14 are new nodes I tried to join exit cluster but failed ?? ?? ??On gluster12: [root at SVR6994HW2285 ~]# gluster peer status Number of Peers: 1 Hostname: glusterfs05.sh2.ctripcorp.com Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Accepted peer request (Connected) [root at SVR6994HW2285 ~]# On gluster05: (peer status seems strange on all those nodes): [root at sh02svr5956 ~]# gluster peer status Number of Peers: 9 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer in Cluster (Disconnected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) Hostname: glusterfs12.sh2.ctripcorp.com Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 State: Peer in Cluster (Connected) Hostname: glusterfs13.sh2.ctripcorp.com Uuid: 43e96d31-6c6d-4096-832a-440e544554ca State: Peer Rejected (Connected) Hostname: glusterfs14.sh2.ctripcorp.com Uuid: 45c11af9-7850-4598-ad62-4bb343b62524 State: Peer Rejected (Connected) [root at sh02svr5956 ~]# The glusterd log file [root at SVR6994HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log [2015-06-02 10:23:48.368697] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) [2015-06-02 10:23:48.372587] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-06-02 10:23:48.372625] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory [2015-06-02 10:23:48.374890] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans port/rdma.so: cannot open shared object file: No such file or directory [2015-06-02 10:23:48.374908] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport- type 'rdma' is not valid or not found on this machine [2015-06-02 10:23:48.374920] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra nsport failed [2015-06-02 10:23:48.376539] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-06-02 10:23:48.377464] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-06-02 10:23:48.378122] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 10:23:48.378191] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 10:23:48.383072] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap s list. Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option ping-timeout 30 8: option transport.socket.read-fail-log off 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-time 10 11: option transport-type rdma 12: option working-directory /var/lib/glusterd 13: end-volume 14: +------------------------------------------------------------------------------+ [2015-06-02 10:23:48.413755] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 0475e11a-3c43-4b76-99dd-1d2e 814c5f15 [2015-06-02 10:23:48.414345] I [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 04f 22ee8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com [2015-06-02 10:23:48.414883] I [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2015-06-02 10:23:50.088645] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 04f22ee 8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com, port: 0 [2015-06-02 10:23:50.090245] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04 f22ee8-8e00-4c32-a924-b40a0e413aa6 [2015-06-02 10:23:50.107013] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1016 [2015-06-02 10:23:50.107805] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1018 [2015-06-02 10:23:50.108358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1020 [2015-06-02 10:23:50.139963] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1008 [2015-06-02 10:23:53.286070] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1006 [2015-06-02 10:23:53.893225] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1019 [2015-06-02 10:23:53.928314] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1017 [2015-06-02 10:23:54.013229] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1014 [2015-06-02 10:23:54.273022] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-02 10:23:55.531073] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-02 10:23:56.293171] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1004 [2015-06-02 10:23:56.900676] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1018 [2015-06-02 10:23:56.935113] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-02 10:23:56.936892] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1016 [2015-06-02 10:23:57.020995] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1012 [2015-06-02 10:23:57.659165] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-02 10:23:58.332111] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-02 10:23:58.929018] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-02 10:23:59.100566] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-vers ion 2 [2015-06-02 10:23:59.103665] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe fro m uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 [2015-06-02 10:23:59.110710] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick resides [2015-06-02 10:23:59.110735] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick for JQStore2 [2015-06-02 10:23:59.131973] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 resides [2015-06-02 10:23:59.131994] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick3 for Webresource [2015-06-02 10:23:59.152949] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 resides [2015-06-02 10:23:59.152969] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick2 for ccim [2015-06-02 10:23:59.172948] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 resides [2015-06-02 10:23:59.172968] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct ripcorp.com:/export/sdb/brick4 for cloudimage [2015-06-02 10:23:59.189033] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 resides [2015-06-02 10:23:59.189053] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct ripcorp.com:/export/sdb/brick1 for filews [2015-06-02 10:23:59.203985] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 resides [2015-06-02 10:23:59.204005] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick1 for filex [2015-06-02 10:23:59.212727] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 resides [2015-06-02 10:23:59.212747] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick4 for pubmail [2015-06-02 10:23:59.237408] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 10:23:59.243681] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 10:23:59.243947] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glusterfs05.s h2.ctripcorp.com (0), ret: 0 [2015-06-02 10:23:59.245638] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/5dd5697669073fb549309773b211e62a.s ocket failed (Invalid argument) [2015-06-02 10:23:59.245679] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd. [2015-06-02 10:23:59.245703] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/1578ba566e5a93f452fe8ad2e91b8682.s ocket failed (Invalid argument) [2015-06-02 10:23:59.245719] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glusters hd has disconnected from glusterd. [2015-06-02 10:23:59.245748] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04 f22ee8-8e00-4c32-a924-b40a0e413aa6 [2015-06-02 10:23:59.300666] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1002 [2015-06-02 10:23:59.569403] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.570725] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.571481] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.572218] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.572858] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.573422] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.574062] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.574655] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.908305] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1017 [2015-06-02 10:23:59.944630] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1014 [2015-06-02 10:24:00.028556] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1011 [2015-06-02 10:24:02.309238] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1000 [2015-06-02 10:24:02.915960] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1016 [2015-06-02 10:24:02.952056] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1013 [2015-06-02 10:24:03.040358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1010 [2015-06-02 10:24:05.317047] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:996 [2015-06-02 10:24:05.923400] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1015 [2015-06-02 10:24:05.959144] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1012 [2015-06-02 10:24:06.044976] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1009 [2015-06-02 10:24:08.324821] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:994 [2015-06-02 10:24:08.932287] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1014 [2015-06-02 10:24:08.967433] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1011 [2015-06-02 10:24:09.052451] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1006 [2015-06-02 10:24:11.332558] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:993 [2015-06-02 10:24:11.939689] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1013 [2015-06-02 10:24:11.975001] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1010 [2015-06-02 10:24:12.060625] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1001 [2015-06-02 10:24:14.339808] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:992 [2015-06-02 10:24:14.947317] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1012 [2015-06-02 10:24:14.983732] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1007 [2015-06-02 10:24:15.068444] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1000 [root at SVR6994HW2285 glusterfs]# Best Regards Yuyang Yang -----????----- ???: Atin Mukherjee [mailto:amukherj at redhat.com] ????: Tuesday, June 02, 2015 5:49 PM ???: vyyy???; Gluster-users at gluster.org ??: Re: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start On 06/02/2015 12:23 PM, vyyy??? wrote: Attached glusterd log files for 10 11 [2015-06-02 06:33:42.668268] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management handshake request from unknown peer 10.8.230.212:1002 From the above log it looks like node 11 tried to handshake with 12, but how come 12 is part of the cluster? Could you run gluster peer status in 12 and share glusterd log? I am still under the same opinion which I had earlier - you are trying to add node which is already a member of another cluster. Best Regards Yuyang Yang -----????----- ???: Atin Mukherjee [mailto:amukherj at redhat.com] ????: Tuesday, June 02, 2015 2:42 PM ???: vyyy???; Gluster-users at gluster.org ??: Re: ??: ??: [Gluster-users] Gluster peer rejected and failed to start On 06/02/2015 12:04 PM, vyyy??? wrote: Glusterfs05~glusterfs10 are clustered for 2 years, recently upgrade to 3.6.3 Glusterfs11~glusterfs14 are new nodes need to join the cluster On glusterfs09: [root at SH02SVR5952 ~]# gluster peer status Number of Peers: 6 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer in Cluster (Disconnected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) [root at SH02SVR5952 ~]# Can you attach glusterd log files for 10 11? [root at SH02SVR5952 ~]# gluster volume status Status of volume: JQStore2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2782 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2744 Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 5307 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 3986 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 Task Status of Volume JQStore2 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: Webresource Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2787 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2753 Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 5313 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 3992 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 Task Status of Volume Webresource ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: ccim Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2793 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2745 Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 5320 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 3999 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 Task Status of Volume ccim ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cloudimage Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 5325 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 4005 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 Task Status of Volume cloudimage ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: filews Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 5330 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 4010 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 Task Status of Volume filews ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: filex Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2794 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2755 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 Task Status of Volume filex ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: pubmail Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2792 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2754 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 Task Status of Volume pubmail ------------------------------------------------------------------------------ There are no active volume tasks [root at SH02SVR5952 ~]# Best Regards ??? Site Operations Center Ext.15554 Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----????----- ???: Atin Mukherjee [mailto:amukherj at redhat.com] ????: Tuesday, June 02, 2015 2:20 PM ???: vyyy???; Gluster-users at gluster.org ??: Re: ??: [Gluster-users] Gluster peer rejected and failed to start On 06/02/2015 11:33 AM, vyyy??? wrote: Actually I have 2 problems 1?New nodes can't add to the clusters I cleaned /var/lib/glusterd, now status is State: Accepted peer request (Connected) 2?One of clusterd nodes shown 'Peer rejected' and glusterd failed to start. The log is attached in pre-mail This is a product cluster, this problem is more egent From the log I can clearly see the problematic node is glusterfs09.sh2.ctripcorp.com. Your existing cluster configuration has bricks hosted in glusterfs09.sh2.ctripcorp.com however the same is not part of the cluster. Could you paste the output of gluster peer status and gluster volume status ? Best Regards Yuyang Yang -----????----- ???: Atin Mukherjee [mailto:amukherj at redhat.com] ????: Tuesday, June 02, 2015 12:52 PM ???: vyyy???; Gluster-users at gluster.org ??: Re: [Gluster-users] Gluster peer rejected and failed to start On 06/02/2015 10:00 AM, vyyy??? wrote: Hi We have a gluster (Version 3.6.3) cluster with 6 nodes, I tried to add 4 more nodes, but ?Peer Rejected?, then I tried to resolve it by dump /var/lib/glusterd and probe again, not success, this is a question, But strange thing is: A node already in cluster also shown ?Peer Reject? I tried to restart glusterd, It failed I found that /var/lib/glusterd/peers is empty, I copied the files from other nodes, still can?t start glusterd It seems like you are trying to peer probe nodes which are either either part of some other clusters (uncleaned nodes). Could you check whether the nodes which you are adding have empty /var/lib/glusterd? If not clean them and retry. ~Atin etc-glusterfs-glusterd.vol.log shown that cluster member as ?unknown peer ? [2015-06-02 01:52:14.650635] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 04f22ee8-8e00-4c32-a924-b40a0e413aa6 [2015-06-02 01:52:14.650786] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 674a78b5-0590-48d4-8752-d4608832ed1d [2015-06-02 01:52:14.657881] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 83e1a9db-3134-45e4-acd2-387b12b5b207 [2015-06-02 01:52:17.747865] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 doesn't belong to the cluster. Ignoring request. [2015-06-02 01:52:17.747908] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2015-06-02 01:52:40.338885] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request. [2015-06-02 01:52:40.338929] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2015-06-02 01:52:41.310451] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request. [2015-06-02 01:52:41.310486] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully Debug info is as following, /usr/sbin/glusterd [root at SH02SVR5951 peers]# /usr/sbin/glusterd --debug [2015-06-02 04:09:24.626690] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.3 (args: /usr/sbin/glusterd --debug) [2015-06-02 04:09:24.626739] D [logging.c:1763:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [2015-06-02 04:09:24.627052] D [MSGID: 0] [glusterfsd.c:613:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2015-06-02 04:09:24.629683] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-06-02 04:09:24.629706] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory [2015-06-02 04:09:24.629764] D [glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128 [2015-06-02 04:09:24.629895] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited. [2015-06-02 04:09:24.629904] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2015-06-02 04:09:24.629930] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.631989] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.632005] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.632013] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread [2015-06-02 04:09:24.632024] D [name.c:550:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet [2015-06-02 04:09:24.632072] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so [2015-06-02 04:09:24.632102] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2015-06-02 04:09:24.632112] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2015-06-02 04:09:24.632122] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-06-02 04:09:24.632132] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2015-06-02 04:09:24.632140] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2015-06-02 04:09:24.632148] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2015-06-02 04:09:24.632155] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2015-06-02 04:09:24.632163] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2015-06-02 04:09:24.632170] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2015-06-02 04:09:24.632177] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2015-06-02 04:09:24.632212] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited. [2015-06-02 04:09:24.632220] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2015-06-02 04:09:24.632236] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.632302] D [socket.c:3692:socket_init] 0-socket.management: disabling nodelay [2015-06-02 04:09:24.632314] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.632321] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.632329] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread [2015-06-02 04:09:24.632402] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2015-06-02 04:09:24.632414] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2015-06-02 04:09:24.632443] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.632455] D [glusterd.c:1484:init] 0-management: cannot get run-with-valgrind value [2015-06-02 04:09:24.633086] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-06-02 04:09:24.633518] D [glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning -1 [2015-06-02 04:09:24.634031] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.634057] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.634115] D [store.c:304:gf_store_retrieve_value] 0-: key operating-version found [2015-06-02 04:09:24.634141] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-06-02 04:09:24.634196] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.634436] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.634446] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.634469] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.634488] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634496] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.634510] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634517] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 [2015-06-02 04:09:24.634528] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634534] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.634544] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634551] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.634562] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634569] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.634580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.634597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634603] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 24 [2015-06-02 04:09:24.634613] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634620] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.634631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 7bf7d81a-f835-4c3b-9c58-4d4ad7ec6e07 [2015-06-02 04:09:24.634652] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634659] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c7374917-2344-4c42-aeae-1d0bdb7ece0a [2015-06-02 04:09:24.634678] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634689] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 703c3b9d-4343-4546-9ca3-350aecdd52fe [2015-06-02 04:09:24.634700] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634707] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.634719] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634726] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.634736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634742] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on [2015-06-02 04:09:24.634773] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on [2015-06-02 04:09:24.634785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634791] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on [2015-06-02 04:09:24.634803] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on [2015-06-02 04:09:24.634814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634821] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.* [2015-06-02 04:09:24.634838] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.* [2015-06-02 04:09:24.634849] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634856] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634868] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634874] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634891] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634908] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634925] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634935] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634942] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.634984] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.635002] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.635019] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635039] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635048] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635056] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635074] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635083] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635092] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635101] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635110] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635118] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635127] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635136] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635144] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635153] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635161] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635368] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.635380] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.635396] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.635416] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635426] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635436] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635447] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635456] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.635488] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.635501] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635748] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.635766] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.635786] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.635807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635828] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635838] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635847] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635856] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.635909] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.635922] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636158] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.636167] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.636180] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.636202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636222] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636250] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.636274] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.636286] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636471] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.636479] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.636493] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.636514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636524] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636533] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636542] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.636586] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.636598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636856] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.636889] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.636922] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.636943] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636964] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636975] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636994] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.637020] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.637036] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.637307] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.637325] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.637346] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.637367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.637378] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.637388] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.819347] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.819374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.819385] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.819413] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.819707] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.820187] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.820222] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.820239] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.820532] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.820585] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.820959] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.821064] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.821090] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.837290] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.837540] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.837551] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.837568] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.837589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.837600] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.837862] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.837880] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.837896] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.837918] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.837929] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.837939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.837964] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.838216] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.838257] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.838273] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.838295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838303] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.838314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838320] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 [2015-06-02 04:09:24.838330] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838336] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.838346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838352] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.838361] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838367] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.838377] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838383] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.838393] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838399] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 6 [2015-06-02 04:09:24.838408] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838414] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.838423] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838430] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = cdc279f4-6cd0-4087-8d2a-440bccb51e33 [2015-06-02 04:09:24.838443] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838449] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 0b6d3496-0865-45e4-a45b-c221b22d3d92 [2015-06-02 04:09:24.838459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838465] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = ecedfb4d-3308-4f6c-b188-4d5b868958e1 [2015-06-02 04:09:24.838475] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.838493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838499] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.838508] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838515] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.68.* [2015-06-02 04:09:24.838533] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.68.* [2015-06-02 04:09:24.838545] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838552] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838563] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838570] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838614] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838621] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838637] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838649] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.838697] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.838719] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.838738] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838747] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838756] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838764] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838781] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838797] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838806] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838815] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838840] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838857] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839051] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.839063] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.839084] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.839102] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839113] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839123] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839142] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839151] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.839177] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.839189] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839382] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.839390] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.839403] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.839421] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839431] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839468] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.839491] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.839502] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839700] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.839712] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.839732] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.839750] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.839822] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.839834] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840057] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.840075] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.840096] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.840117] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840128] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840137] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840156] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840166] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.840192] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.840206] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840402] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.840410] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.840424] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.840440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840469] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840478] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.840513] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.840525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840799] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.840818] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.840839] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.840859] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840871] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840880] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840900] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.840931] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.841118] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.841359] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.841396] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.841413] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.841557] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.841611] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.841799] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.841875] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.841901] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.842350] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.842469] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.842477] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.842489] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.842514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.842636] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.842645] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.842656] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.842696] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842708] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842718] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.842738] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.842848] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.842859] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.842872] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.842891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842898] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.842908] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842914] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4 [2015-06-02 04:09:24.842924] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842930] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.842939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842945] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.842954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842960] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.842969] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842975] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.842984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842990] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3 [2015-06-02 04:09:24.842999] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843005] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.843015] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843021] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = a3369f63-9fce-4a56-a921-f7206cb8d3ae [2015-06-02 04:09:24.843033] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843039] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c6012954-928a-43c1-b7ea-336d6ee0bea9 [2015-06-02 04:09:24.843050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843056] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = bfbd7007-0e0c-45f9-8d59-05e1aee66728 [2015-06-02 04:09:24.843066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843072] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.843082] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843089] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.843098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843104] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.83.* [2015-06-02 04:09:24.843119] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.83.* [2015-06-02 04:09:24.843130] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843136] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick4 [2015-06-02 04:09:24.843147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843154] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick4 [2015-06-02 04:09:24.843164] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843171] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick4 [2015-06-02 04:09:24.843181] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843188] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick4 [2015-06-02 04:09:24.843198] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.843221] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.843238] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.843255] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843264] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843273] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843281] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843290] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843298] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843306] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843323] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843332] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843340] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843357] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843365] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843476] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.843484] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.843499] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.843519] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843720] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843739] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843751] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843761] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843771] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.843802] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.843817] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843984] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.843996] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.844013] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.844030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844041] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844060] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844069] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844078] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.844104] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.844116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844222] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.844230] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.844243] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.844266] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844276] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844285] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844294] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844303] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844312] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.844334] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.844346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844443] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.844451] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.844468] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.844487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844497] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844516] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844534] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.844550] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.844753] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.845000] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.845033] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.845049] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.845199] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.845244] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.845396] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.845448] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.845471] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.846053] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.846195] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.846206] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.846220] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.846241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846252] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.846362] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.846371] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.846382] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.846404] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846424] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.846442] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.846543] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.846552] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.846562] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.846585] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846592] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.846603] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846609] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 [2015-06-02 04:09:24.846618] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846624] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.846633] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846639] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.846648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.846663] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846690] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.846702] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846708] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3 [2015-06-02 04:09:24.846717] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846723] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.846733] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846739] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 8869b21b-1bf1-4eba-a52a-1158abde7d23 [2015-06-02 04:09:24.846752] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846758] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 27298b95-919d-43f7-9507-8cbfbe5292e3 [2015-06-02 04:09:24.846769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846775] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 47f52166-020b-4bef-ada1-7e65fcd2834f [2015-06-02 04:09:24.846785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846792] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.846802] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846808] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.846818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846824] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on [2015-06-02 04:09:24.846837] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on [2015-06-02 04:09:24.846848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846854] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on [2015-06-02 04:09:24.846864] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on [2015-06-02 04:09:24.846875] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846881] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846892] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846899] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846916] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846927] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846934] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846944] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846952] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846969] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846981] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.847003] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.847027] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.847045] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847055] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847064] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847072] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847081] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847089] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847106] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847115] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847124] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847150] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847159] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847167] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847176] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847286] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.847294] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.847308] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.847328] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847339] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847358] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847376] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.847402] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.847414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847516] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.847523] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.847537] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.847559] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847569] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847579] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847588] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847607] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.847628] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.847640] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847747] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.847756] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.847771] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.847794] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847804] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847841] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.847862] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.847873] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847969] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.847977] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.847990] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.848012] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848022] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848031] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848040] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848049] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848058] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.848080] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.848091] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848187] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.848195] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.848208] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.848230] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848240] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848249] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848259] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848267] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848277] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.848299] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.848310] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848411] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.848419] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.848432] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.848455] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848474] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848484] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848503] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.848519] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.848727] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.848975] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.849002] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.849019] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.849167] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.849212] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.849367] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.849419] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.849442] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.849880] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.850003] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.850012] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.850024] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.850043] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.850157] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.850166] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.850177] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.850192] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850211] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.850228] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.850402] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.850414] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.850426] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.850444] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850451] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.850461] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850467] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4 [2015-06-02 04:09:24.850476] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.850491] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850497] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.850506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850512] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.850522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850528] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.850537] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850543] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = disperse_count value = 0 [2015-06-02 04:09:24.850552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850558] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = redundancy_count value = 0 [2015-06-02 04:09:24.850567] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850573] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 11 [2015-06-02 04:09:24.850582] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850588] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.850598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = ea46609c-be9a-4283-8357-8fdc2e5a190e [2015-06-02 04:09:24.850616] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850622] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 21b247ce-58ce-4087-8c32-3d578b4cc22d [2015-06-02 04:09:24.850632] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 0724171a-c7ce-4528-b54b-2d983f969537 [2015-06-02 04:09:24.850648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.850664] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850692] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.850703] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850709] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.92.*,10.8.230.*,192.168.98.* [2015-06-02 04:09:24.850724] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.92.*,10.8.230.*,192.168.98.* [2015-06-02 04:09:24.850736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850743] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick1 [2015-06-02 04:09:24.850754] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850760] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick1 [2015-06-02 04:09:24.850772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850778] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick1 [2015-06-02 04:09:24.850789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850795] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick1 [2015-06-02 04:09:24.850807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.850830] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.850850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.850867] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850877] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850894] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850911] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850928] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850936] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850945] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850953] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850971] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850980] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850988] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850996] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851005] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851164] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.851173] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.851187] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.851203] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851223] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851242] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851251] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851261] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.851283] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.851295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851457] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.851465] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.851478] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.851493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851504] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851513] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851531] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851540] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851549] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.851571] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.851583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851732] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.851741] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.851755] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.851770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851816] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851826] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.851850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.851863] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851959] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.851974] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.852005] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.852026] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852037] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852047] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852061] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852070] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852080] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852090] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.852111] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.852298] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.852524] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.852556] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.852576] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.852723] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.852776] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.852930] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.852982] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.853006] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.853412] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.853524] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.853539] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.853553] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.853573] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.853699] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.853718] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.853735] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.853757] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853768] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853778] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853787] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853798] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.853820] D [glusterd-store.c:3094:glusterd_store_retrieve_volumes] 0-: Returning with 0 [2015-06-02 04:09:24.853962] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.853972] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.853983] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854000] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854011] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854023] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854032] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854043] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854070] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.854162] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.854171] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.854181] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854197] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854205] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854216] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854225] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854232] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854255] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.854345] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.854354] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.854364] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854380] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854390] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854402] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854411] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854417] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854427] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854439] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.854527] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.854536] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.854546] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854569] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854596] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854605] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854617] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.854704] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.854713] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.854725] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854740] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854749] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854776] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854803] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.855064] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.855072] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.855083] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.855098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.855106] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.855116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.855125] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.855132] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.855141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.855154] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.855186] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.855210] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.855218] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.855231] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.855324] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.855333] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.855340] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.859610] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.211 (port-24007) for hostname: glusterfs07.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.859628] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.859707] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.859733] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.859772] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.859781] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.859794] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.859905] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.859916] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.859923] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.863505] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.231.160 (port-24007) for hostname: glusterfs11.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.863521] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.863562] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.863585] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.863607] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.863615] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.863625] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.863754] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.863765] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.863772] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.867255] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.210 (port-24007) for hostname: glusterfs06.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.867271] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.867314] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.867337] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.867358] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.867366] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.867377] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.867463] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.867476] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.867483] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.870920] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.214 (port-24007) for hostname: glusterfs10.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.870935] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.870997] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.871021] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.871042] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.871050] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.871061] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.871129] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.871138] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.871144] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.874102] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.209 (port-24007) for hostname: 10.8.230.209 and port: 24007 [2015-06-02 04:09:24.874116] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.874159] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.874181] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.874202] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.874210] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.874219] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.874292] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.874301] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.874326] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.877759] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.212 (port-24007) for hostname: glusterfs08.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.877774] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.877818] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.877831] D [glusterd-store.c:4106:glusterd_store_retrieve_peers] 0-: Returning with 0 [2015-06-02 04:09:24.877863] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2015-06-02 04:09:24.877873] D [glusterd-store.c:3612:glusterd_store_retrieve_snaps] 0-: Returning with 0 [2015-06-02 04:09:24.881125] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-06-02 04:09:24.881140] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 [2015-06-02 04:09:24.881148] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs06.sh2.ctripcorp.com found.. state: 3 [2015-06-02 04:09:24.881156] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-06-02 04:09:24.881162] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 [2015-06-02 04:09:24.881169] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs07.sh2.ctripcorp.com found.. state: 3 [2015-06-02 04:09:24.881176] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-06-02 04:09:24.881182] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 [2015-06-02 04:09:24.881188] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs08.sh2.ctripcorp.com found.. state: 3 [2015-06-02 04:09:24.881195] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-06-02 04:09:24.881201] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 [2015-06-02 04:09:24.890739] D [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: glusterfs09.sh2.ctripcorp.com [2015-06-02 04:09:24.891607] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 [2015-06-02 04:09:24.891750] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 [2015-06-02 04:09:24.891820] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 [2015-06-02 04:09:24.891881] D [common-utils.c:2913:gf_is_local_addr] 0-management: glusterfs09.sh2.ctripcorp.com is not local [2015-06-02 04:09:24.891890] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning -1 [2015-06-02 04:09:24.891898] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning -1 [2015-06-02 04:09:24.891904] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2015-06-02 04:09:24.891911] D [glusterd-store.c:4321:glusterd_restore] 0-: Returning -1 [2015-06-02 04:09:24.891923] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2015-06-02 04:09:24.891933] E [graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed [2015-06-02 04:09:24.891941] E [graph.c:525:glusterfs_graph_activate] 0-graph: init failed [2015-06-02 04:09:24.892164] D [logging.c:1740:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 2 extra log messages [2015-06-02 04:09:24.892178] D [logging.c:1743:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 2 extra log messages [2015-06-02 04:09:24.892325] W [glusterfsd.c:1194:cleanup_and_exit] (-- 0-: received signum (0), shutting down [2015-06-02 04:09:24.892345] D [glusterfsd-mgmt.c:2244:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given [root at SH02SVR5951 peers]# Best Regards Yuyang Yang _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -- ~Atin _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150604/6d8de576/attachment.html>
vyyy杨雨阳
2015-Jun-05 03:04 UTC
[Gluster-users] 答复: 回复： 答复: 答复: 答复: Gluster peer rejected and failed to start
Selinux is disabled on all server
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=disabled
Best Regards
???
Site Operations Center
Ext.15554 Seat.1#3F049
We are the F.I.R.S.T.
Fast, Innovation, Reliability , Secure, Team
???: huangql [mailto:huangql at ihep.ac.cn]
????: Friday, June 05, 2015 10:57 AM
???: Atin Mukherjee; Gluster-users at gluster.org; vyyy???
??: ???[Gluster-users] ??: ??: ??: Gluster peer rejected and failed to start
Maybe you can check selinux on the server side. Make sure selinux is disabled.
????
???: vyyy???<yuyangyang at Ctrip.com<mailto:yuyangyang at
Ctrip.com>>
???: Atin Mukherjee<amukherj at redhat.com<mailto:amukherj at
redhat.com>>; Gluster-users at gluster.org<Gluster-users at
gluster.org<mailto:Gluster-users at gluster.org%3cGluster-users at
gluster.org>>
????: 2015?6?4?(??)?21:24
??: [Gluster-users] ??: ??: ??: Gluster peer rejected and failed to start
Can anyone help me?
New nodes still can't join the cluster.
The new nodes can peer each other, just not exist cluster nodes.
I can not find any network problem, there were no firewall, port 24007 is
connected, there were all at the same subnet...
Thanks
[root at SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log
[2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glust
erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management: Maximum
allowed open file descriptors set to 65536
[2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management: Using
/var/lib/glusterd as working directory
[2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load]
0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans
port/rdma.so: cannot open shared object file: No such file or directory
[2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load]
0-rpc-transport: volume 'rdma.management': transport-
type 'rdma' is not valid or not found on this machine
[2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the tra
nsport failed
[2015-06-05 01:59:02.343346] I [glusterd.c:413:glusterd_check_gsync_present]
0-glusterd: geo-replication module not installed
in the system
[2015-06-05 01:59:02.351861] I
[glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 2
[2015-06-05 01:59:02.363138] I
[glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management:
No missed snap
s list.
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option transport.socket.listen-backlog 128
7: option ping-timeout 30
8: option transport.socket.read-fail-log off
9: option transport.socket.keepalive-interval 2
10: option transport.socket.keepalive-time 10
11: option transport-type rdma
12: option working-directory /var/lib/glusterd
13: end-volume
14:
+------------------------------------------------------------------------------+
[2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init] 0-management:
retrieved UUID: 1b98af34-08f9-492f-b54e-7608
2588d63b
[2015-06-05 02:00:43.025462] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:00:44.025069] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:00:44.599862] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:00:46.029823] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:00:46.701845] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:01:43.119612] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:01:44.165527] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:01:44.851903] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:01:46.190401] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:01:46.722394] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:01:58.719968] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:02:32.584999] I
[glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req glust
erfs05.sh2.ctripcorp.com 24007
[2015-06-05 02:02:32.586281] I [glusterd-handler.c:3165:glusterd_probe_begin]
0-glusterd: Unable to find peerinfo for host: g
lusterfs05.sh2.ctripcorp.com (24007)
[2015-06-05 02:02:32.602192] I [rpc-clnt.c:969:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-06-05 02:02:32.606053] I [glusterd-handler.c:3098:glusterd_friend_add]
0-management: connect returned 0
[2015-06-05 02:02:32.620958] E
[glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management:
failed to get the
'versions' from peer (10.8.230.209:24007)
[2015-06-05 02:02:32.621020] I [MSGID: 106004]
[glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer 000000
00-0000-0000-0000-000000000000, in Establishing Connection state, has
disconnected from glusterd.
[2015-06-05 02:02:43.264141] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:02:44.312918] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:02:44.781434] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:02:46.176888] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:02:46.799916] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:03:42.720994] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:03:43.771987] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:03:44.659775] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:03:45.956807] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:03:46.688787] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:04:42.891131] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:04:43.957423] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:04:44.995638] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:04:46.062171] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:04:46.957327] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:05:42.966651] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:05:44.008498] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:05:45.049936] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:05:46.087007] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:05:47.029325] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:06:42.874089] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:06:43.910998] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:06:44.932802] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:06:46.053731] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:06:47.055859] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:07:43.099647] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:07:44.146454] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:07:45.218982] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:07:46.246303] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:07:47.106954] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:08:43.226090] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:08:44.292041] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
[2015-06-05 02:08:45.351830] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:08:45.688349] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2015-06-05 02:08:46.651768] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
[2015-06-05 02:09:42.748628] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx]
0-management: op_ctx modification failed
Best Regards
-----????-----
???: vyyy???
????: Thursday, June 04, 2015 11:00 AM
???: 'Atin Mukherjee'; Gluster-users at
gluster.org<mailto:Gluster-users at gluster.org>
??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start
??
The problem 2 has resolved.
From glusterd log file, I find Cksums error as following:
[2015-06-04 01:25:03.137798] E [MSGID: 106010]
[glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of
Cksums JQStore2 differ. local cksum = 2617028277, remote cksum = 2210669223 on
peer glusterfs09.sh2.ctripcorp.com
The solution for this problem was to do the following, on the BAD node:
/etc/init.d/glusterd stop
rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/
/var/lib/glusterd/vols/
/etc/init.d/glusterd start
The problem 1 is still under inverstigate:
New nodes can not join the cluster, log file shown: failed to get the
'versions' from peer
I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are "
operating-version=2"
??
??[2015-06-04 02:13:49.199416] E
[glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management:
failed to get the
??'versions' from peer (10.8.230.211:24007)
??[2015-06-04 02:14:07.112577] I
[glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI
probe req glust
??erfs05.sh2.ctripcorp.com 24007
??[2015-06-04 02:14:07.113778] I [glusterd-handler.c:3165:glusterd_probe_begin]
0-glusterd: Unable to find peerinfo for host: g
??lusterfs05.sh2.ctripcorp.com (24007)
??[2015-06-04 02:14:07.131023] I [rpc-clnt.c:969:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
??[2015-06-04 02:14:07.134829] I [glusterd-handler.c:3098:glusterd_friend_add]
0-management: connect returned 0
??[
Best Regards
???
Site Operations Center
Ext.15554 Seat.1#3F049
We are the F.I.R.S.T.
Fast, Innovation, Reliability , Secure, Team
-----????-----
???: vyyy???
????: Wednesday, June 03, 2015 5:37 PM
???: 'Atin Mukherjee'; Gluster-users at
gluster.org<mailto:Gluster-users at gluster.org>
??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start
Hi Atin and everybody,
I have forced peer detach the news nodes ,The peer status seems even
strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help me? Any
advice is appreciated.
Glusterfs05
??
??[root at sh02svr5956 ~]# gluster peer status
??Number of Peers: 5
??
??Hostname: glusterfs06.sh2.ctripcorp.com
??Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
??State: Peer Rejected (Connected)
??
??Hostname: glusterfs07.sh2.ctripcorp.com
??Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
??State: Peer Rejected (Connected)
??
??Hostname: glusterfs08.sh2.ctripcorp.com
??Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
??State: Peer Rejected (Connected)
??
??Hostname: glusterfs10.sh2.ctripcorp.com
??Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
??State: Peer in Cluster (Connected)
??
??Hostname: glusterfs09.sh2.ctripcorp.com
??Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
??State: Peer Rejected (Connected)
??[root at sh02svr5956 ~]#
??
??
??Glusterfs06
??
??[root at SH02SVR5955 ~]# gluster peer status
??Number of Peers: 5
??
??Hostname: glusterfs10.sh2.ctripcorp.com
??Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
??State: Peer Rejected (Connected)
??
??Hostname: glusterfs08.sh2.ctripcorp.com
??Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
??State: Peer in Cluster (Connected)
??
??Hostname: glusterfs07.sh2.ctripcorp.com
??Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
??State: Peer in Cluster (Connected)
??
??Hostname: 10.8.230.209
??Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
??State: Peer Rejected (Connected)
??
??Hostname: glusterfs09.sh2.ctripcorp.com
??Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
??State: Peer in Cluster (Connected)
??[root at SH02SVR5955 ~]#
??
Glusterfs07
[root at SH02SVR5954 ~]# gluster peer status
Number of Peers: 5
Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)
Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer in Cluster (Connected)
Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer in Cluster (Connected)
[root at SH02SVR5954 ~]#
Glusterfs08
[root at SH02SVR5953 ~]# gluster peer status
Number of Peers: 5
Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer in Cluster (Connected)
Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)
Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer in Cluster (Connected)
[root at SH02SVR5953 ~]#
[root at SH02SVR5952 ~]# gluster peer status
Number of Peers: 5
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)
Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer in Cluster (Connected)
Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer in Cluster (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)
Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)
[root at SH02SVR5952 ~]#
Glusterfs10
[root at SH02SVR5951 ~]# gluster peer status
Number of Peers: 6
Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer Rejected (Connected)
Hostname: glusterfs11.sh2.ctripcorp.com
Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
State: Peer Rejected (Connected)
Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer Rejected (Connected)
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer Rejected (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer in Cluster (Connected)
Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer Rejected (Connected)
[root at SH02SVR5951 ~]#
Best Regards
???
Site Operations Center
Ext.15554 Seat.1#3F049
We are the F.I.R.S.T.
Fast, Innovation, Reliability , Secure, Team
-----????-----
???: vyyy???
????: Wednesday, June 03, 2015 2:55 PM
???: 'Atin Mukherjee'
??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start
Can I delete the files in /var/lib/glusterd/peers which state is not =3 on all
nodes, then restart glusterd?
[root at sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15
uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15
state=6
hostname1=glusterfs12.sh2.ctripcorp.com
Best Regards
Yang
-----????-----
???: vyyy???
????: Wednesday, June 03, 2015 2:39 PM
???: 'Atin Mukherjee'
??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start
Since it's intranet, You can only control my desktop, and then access these
node.
Should I apply a webex room?
Best Regards
???
Site Operations Center
Ext.15554 Seat.1#3F049
We are the F.I.R.S.T.
Fast, Innovation, Reliability , Secure, Team
-----????-----
???: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj at
redhat.com>]
????: Wednesday, June 03, 2015 2:36 PM
???: vyyy???
??: Re: ??: ??: [Gluster-users] Gluster peer rejected and failed to start
Is it possible to share your ip/user/passwd for few of these nodes?
On 06/03/2015 11:50 AM, vyyy??? wrote:
> Can you remote control my desktop (like webex)?
> The nodes host bricks. And on the new nodes, I did "clean
/var/lib/glusterd and restart" several times, but have no effect.
> Is there any other work around?
>
>
> Best Regards
> Yang
>
> -----????-----
> ???: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj at
redhat.com>]
> ????: Wednesday, June 03, 2015 12:33 PM
> ???: vyyy???; Gluster-users at gluster.org<mailto:Gluster-users at
gluster.org>
> ??: Re: ??: [Gluster-users] Gluster peer rejected and failed to start
>
> I can't comment on what went wrong till I can have visibility on the
> entire cluster. Does the node which fails to start host any brick? If
> not then work around could be to clean /var/lib/glusterd and restart
> glusterd, that should sync up data and bring this node in consistency.
>
> HTH,
> Atin
>
> On 06/03/2015 08:34 AM, vyyy??? wrote:
>> I found that /var/lib/glusterd/peers can't include node's own
peer, I delete this file. Glusterd started finally.
>> But the peer status still not normal.
>> Seems that glusterfs10 leave the cluster and join the new nodes alone.
>>
>> Peer status on glusterfs10
>> [root at SH02SVR5951 ~]# gluster peer status
>> Number of Peers: 7
>>
>> Hostname: glusterfs07.sh2.ctripcorp.com
>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs11.sh2.ctripcorp.com
>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>> State: Accepted peer request (Connected)
>>
>> Hostname: glusterfs09.sh2.ctripcorp.com
>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs06.sh2.ctripcorp.com
>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>> State: Peer Rejected (Connected)
>>
>> Hostname: 10.8.230.209
>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs08.sh2.ctripcorp.com
>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs12.sh2.ctripcorp.com
>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>> State: Peer in Cluster (Connected)
>>
>>
>>
>> Peer status on glusterfs10
>>
>> [root at SVR6993HW2285 ~]# gluster peer status
>> Number of Peers: 6
>>
>> Hostname: glusterfs05.sh2.ctripcorp.com
>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs09.sh2.ctripcorp.com
>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs07.sh2.ctripcorp.com
>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs08.sh2.ctripcorp.com
>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs06.sh2.ctripcorp.com
>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs10.sh2.ctripcorp.com
>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>> State: Peer in Cluster (Connected)
>>
>>
>> Peer status on glusterfs05, this is the node that I tried to peer probe
new nodes (glusterfs11 to glusterfs14)
>>
>> [root at sh02svr5956 ~]# gluster peer status
>> Number of Peers: 9
>>
>> Hostname: glusterfs06.sh2.ctripcorp.com
>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs07.sh2.ctripcorp.com
>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs08.sh2.ctripcorp.com
>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs10.sh2.ctripcorp.com
>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs09.sh2.ctripcorp.com
>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs11.sh2.ctripcorp.com
>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs12.sh2.ctripcorp.com
>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs13.sh2.ctripcorp.com
>> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs14.sh2.ctripcorp.com
>> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524
>> State: Peer Rejected (Connected)
>> [root at sh02svr5956 ~]#
>>
>>
>>
>>
>> Best Regards
>> Yang
>>
>>
>> -----????-----
>> ???: vyyy???
>> ????: Wednesday, June 03, 2015 10:04 AM
>> ???: 'Atin Mukherjee'; 'Gluster-users at
gluster.org<mailto:Gluster-users at gluster.org>'
>> ??: [Gluster-users] Gluster peer rejected and failed to start
>>
>> I am still messed, can you help me figure out problem 2 (A node of
cluster failed to start glusterd) priority
>>
>> The glusterd log on the node gluster10 shown ' resolve brick failed
in restore'
>> I checked /var/lib/glusterd/vols, seems ok
>>
>> Glusterd log file Error message:
>>
>> [2015-06-03 01:31:32.564010] E
[glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick
failed in restore
>> [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] 0-management:
Initialization of volume 'management' failed, review
>> your volfile again
>>
>>
>>
>>
>> Best Regards
>> Yang
>>
>> -----????-----
>> ???: vyyy???
>> ????: Tuesday, June 02, 2015 6:27 PM
>> ???: 'Atin Mukherjee'; Gluster-users at
gluster.org<mailto:Gluster-users at gluster.org>
>> ??: ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to
start
>>
>> Gluster11 to gluster14 are new nodes I tried to join exit cluster but
failed
>> ??
>> ??
>> ??On gluster12:
>> [root at SVR6994HW2285 ~]# gluster peer status
>> Number of Peers: 1
>>
>> Hostname: glusterfs05.sh2.ctripcorp.com
>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>> State: Accepted peer request (Connected)
>> [root at SVR6994HW2285 ~]#
>>
>>
>>
>> On gluster05: (peer status seems strange on all those nodes):
>>
>> [root at sh02svr5956 ~]# gluster peer status
>> Number of Peers: 9
>>
>> Hostname: glusterfs06.sh2.ctripcorp.com
>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs07.sh2.ctripcorp.com
>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs08.sh2.ctripcorp.com
>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs10.sh2.ctripcorp.com
>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>> State: Peer in Cluster (Disconnected)
>>
>> Hostname: glusterfs09.sh2.ctripcorp.com
>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs11.sh2.ctripcorp.com
>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs12.sh2.ctripcorp.com
>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs13.sh2.ctripcorp.com
>> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs14.sh2.ctripcorp.com
>> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524
>> State: Peer Rejected (Connected)
>> [root at sh02svr5956 ~]#
>>
>>
>>
>> The glusterd log file
>>
>> [root at SVR6994HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log
>> [2015-06-02 10:23:48.368697] I [MSGID: 100030] [glusterfsd.c:2018:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glust
>> erd version 3.6.3 (args: /usr/sbin/glusterd
--pid-file=/var/run/glusterd.pid)
>> [2015-06-02 10:23:48.372587] I [glusterd.c:1214:init] 0-management:
Maximum allowed open file descriptors set to 65536
>> [2015-06-02 10:23:48.372625] I [glusterd.c:1259:init] 0-management:
Using /var/lib/glusterd as working directory
>> [2015-06-02 10:23:48.374890] E [rpc-transport.c:266:rpc_transport_load]
0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans
>> port/rdma.so: cannot open shared object file: No such file or directory
>> [2015-06-02 10:23:48.374908] W [rpc-transport.c:270:rpc_transport_load]
0-rpc-transport: volume 'rdma.management': transport-
>> type 'rdma' is not valid or not found on this machine
>> [2015-06-02 10:23:48.374920] W [rpcsvc.c:1524:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the tra
>> nsport failed
>> [2015-06-02 10:23:48.376539] I
[glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module
not installed
>> in the system
>> [2015-06-02 10:23:48.377464] I
[glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 2
>> [2015-06-02 10:23:48.378122] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
>> [2015-06-02 10:23:48.378191] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>> [2015-06-02 10:23:48.383072] I
[glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management:
No missed snap
>> s list.
>> Final graph:
>>
+------------------------------------------------------------------------------+
>> 1: volume management
>> 2: type mgmt/glusterd
>> 3: option rpc-auth.auth-glusterfs on
>> 4: option rpc-auth.auth-unix on
>> 5: option rpc-auth.auth-null on
>> 6: option transport.socket.listen-backlog 128
>> 7: option ping-timeout 30
>> 8: option transport.socket.read-fail-log off
>> 9: option transport.socket.keepalive-interval 2
>> 10: option transport.socket.keepalive-time 10
>> 11: option transport-type rdma
>> 12: option working-directory /var/lib/glusterd
>> 13: end-volume
>> 14:
>>
+------------------------------------------------------------------------------+
>> [2015-06-02 10:23:48.413755] I [glusterd.c:146:glusterd_uuid_init]
0-management: retrieved UUID: 0475e11a-3c43-4b76-99dd-1d2e
>> 814c5f15
>> [2015-06-02 10:23:48.414345] I
[glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp
from uuid: 04f
>> 22ee8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com
>> [2015-06-02 10:23:48.414883] I
[glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe
req
>> [2015-06-02 10:23:50.088645] I
[glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from
uuid: 04f22ee
>> 8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com,
port: 0
>> [2015-06-02 10:23:50.090245] I
[glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC
from uuid: 04
>> f22ee8-8e00-4c32-a924-b40a0e413aa6
>> [2015-06-02 10:23:50.107013] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1016
>> [2015-06-02 10:23:50.107805] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1018
>> [2015-06-02 10:23:50.108358] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1020
>> [2015-06-02 10:23:50.139963] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:1008
>> [2015-06-02 10:23:53.286070] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:1006
>> [2015-06-02 10:23:53.893225] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1019
>> [2015-06-02 10:23:53.928314] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1017
>> [2015-06-02 10:23:54.013229] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1014
>> [2015-06-02 10:23:54.273022] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
>> [2015-06-02 10:23:55.531073] W
[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
modification failed
>> [2015-06-02 10:23:56.293171] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:1004
>> [2015-06-02 10:23:56.900676] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1018
>> [2015-06-02 10:23:56.935113] W
[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
modification failed
>> [2015-06-02 10:23:56.936892] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1016
>> [2015-06-02 10:23:57.020995] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1012
>> [2015-06-02 10:23:57.659165] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
>> [2015-06-02 10:23:58.332111] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
>> [2015-06-02 10:23:58.929018] I
[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
>> [2015-06-02 10:23:59.100566] I
[glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-vers
>> ion 2
>> [2015-06-02 10:23:59.103665] I
[glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe fro
>> m uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>> [2015-06-02 10:23:59.110710] E
[glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer
on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick resides
>> [2015-06-02 10:23:59.110735] E
[glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start
glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick for JQStore2
>> [2015-06-02 10:23:59.131973] E
[glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer
on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 resides
>> [2015-06-02 10:23:59.131994] E
[glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start
glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick3 for Webresource
>> [2015-06-02 10:23:59.152949] E
[glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer
on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 resides
>> [2015-06-02 10:23:59.152969] E
[glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start
glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick2 for ccim
>> [2015-06-02 10:23:59.172948] E
[glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer
on which brick
>> glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 resides
>> [2015-06-02 10:23:59.172968] E
[glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start
glusterfs07.sh2.ct
>> ripcorp.com:/export/sdb/brick4 for cloudimage
>> [2015-06-02 10:23:59.189033] E
[glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer
on which brick
>> glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 resides
>> [2015-06-02 10:23:59.189053] E
[glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start
glusterfs07.sh2.ct
>> ripcorp.com:/export/sdb/brick1 for filews
>> [2015-06-02 10:23:59.203985] E
[glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer
on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 resides
>> [2015-06-02 10:23:59.204005] E
[glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start
glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick1 for filex
>> [2015-06-02 10:23:59.212727] E
[glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer
on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 resides
>> [2015-06-02 10:23:59.212747] E
[glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start
glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick4 for pubmail
>> [2015-06-02 10:23:59.237408] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>> [2015-06-02 10:23:59.243681] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>> [2015-06-02 10:23:59.243947] I
[glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to
glusterfs05.s
>> h2.ctripcorp.com (0), ret: 0
>> [2015-06-02 10:23:59.245638] W [socket.c:620:__socket_rwv]
0-management: readv on /var/run/5dd5697669073fb549309773b211e62a.s
>> ocket failed (Invalid argument)
>> [2015-06-02 10:23:59.245679] I [MSGID: 106006]
[glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has
>> disconnected from glusterd.
>> [2015-06-02 10:23:59.245703] W [socket.c:620:__socket_rwv]
0-management: readv on /var/run/1578ba566e5a93f452fe8ad2e91b8682.s
>> ocket failed (Invalid argument)
>> [2015-06-02 10:23:59.245719] I [MSGID: 106006]
[glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glusters
>> hd has disconnected from glusterd.
>> [2015-06-02 10:23:59.245748] I
[glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC
from uuid: 04
>> f22ee8-8e00-4c32-a924-b40a0e413aa6
>> [2015-06-02 10:23:59.300666] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:1002
>> [2015-06-02 10:23:59.569403] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
>> [2015-06-02 10:23:59.570725] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
>> [2015-06-02 10:23:59.571481] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
>> [2015-06-02 10:23:59.572218] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
>> [2015-06-02 10:23:59.572858] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
>> [2015-06-02 10:23:59.573422] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
>> [2015-06-02 10:23:59.574062] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
>> [2015-06-02 10:23:59.574655] I
[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
>> [2015-06-02 10:23:59.908305] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1017
>> [2015-06-02 10:23:59.944630] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1014
>> [2015-06-02 10:24:00.028556] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1011
>> [2015-06-02 10:24:02.309238] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:1000
>> [2015-06-02 10:24:02.915960] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1016
>> [2015-06-02 10:24:02.952056] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1013
>> [2015-06-02 10:24:03.040358] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1010
>> [2015-06-02 10:24:05.317047] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:996
>> [2015-06-02 10:24:05.923400] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1015
>> [2015-06-02 10:24:05.959144] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1012
>> [2015-06-02 10:24:06.044976] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1009
>> [2015-06-02 10:24:08.324821] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:994
>> [2015-06-02 10:24:08.932287] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1014
>> [2015-06-02 10:24:08.967433] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1011
>> [2015-06-02 10:24:09.052451] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1006
>> [2015-06-02 10:24:11.332558] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:993
>> [2015-06-02 10:24:11.939689] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1013
>> [2015-06-02 10:24:11.975001] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1010
>> [2015-06-02 10:24:12.060625] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1001
>> [2015-06-02 10:24:14.339808] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.213:992
>> [2015-06-02 10:24:14.947317] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.210:1012
>> [2015-06-02 10:24:14.983732] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.211:1007
>> [2015-06-02 10:24:15.068444] E
[glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting
management hands
>> hake request from unknown peer 10.8.230.212:1000
>> [root at SVR6994HW2285 glusterfs]#
>>
>>
>>
>>
>> Best Regards
>> Yuyang Yang
>>
>>
>> -----????-----
>> ???: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj
at redhat.com>]
>> ????: Tuesday, June 02, 2015 5:49 PM
>> ???: vyyy???; Gluster-users at gluster.org<mailto:Gluster-users at
gluster.org>
>> ??: Re: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to
start
>>
>>
>>
>> On 06/02/2015 12:23 PM, vyyy??? wrote:
>>> Attached glusterd log files for 10 & 11
>> [2015-06-02 06:33:42.668268] E
>> [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management:
>> Rejecting management handshake request from unknown peer
10.8.230.212:1002
>>
>> From the above log it looks like node 11 tried to handshake with 12,
but
>> how come 12 is part of the cluster? Could you run gluster peer status
in
>> 12 and share glusterd log? I am still under the same opinion which I
had
>> earlier - you are trying to add node which is already a member of
>> another cluster.
>>>
>>> Best Regards
>>> Yuyang Yang
>>>
>>> -----????-----
>>> ???: Atin Mukherjee [mailto:amukherj at
redhat.com<mailto:amukherj at redhat.com>]
>>> ????: Tuesday, June 02, 2015 2:42 PM
>>> ???: vyyy???; Gluster-users at gluster.org<mailto:Gluster-users
at gluster.org>
>>> ??: Re: ??: ??: [Gluster-users] Gluster peer rejected and failed to
start
>>>
>>>
>>>
>>> On 06/02/2015 12:04 PM, vyyy??? wrote:
>>>> Glusterfs05~glusterfs10 are clustered for 2 years, recently
upgrade to 3.6.3
>>>> Glusterfs11~glusterfs14 are new nodes need to join the cluster
>>>>
>>>> On glusterfs09:
>>>>
>>>> [root at SH02SVR5952 ~]# gluster peer status
>>>> Number of Peers: 6
>>>>
>>>> Hostname: glusterfs06.sh2.ctripcorp.com
>>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: glusterfs07.sh2.ctripcorp.com
>>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: glusterfs08.sh2.ctripcorp.com
>>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: 10.8.230.209
>>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: glusterfs10.sh2.ctripcorp.com
>>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>>>> State: Peer in Cluster (Disconnected)
>>>>
>>>> Hostname: glusterfs11.sh2.ctripcorp.com
>>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>>>> State: Peer Rejected (Connected)
>>>> [root at SH02SVR5952 ~]#
>>> Can you attach glusterd log files for 10 & 11?
>>>>
>>>> [root at SH02SVR5952 ~]# gluster volume status
>>>> Status of volume: JQStore2
>>>> Gluster process Port
Online Pid
>>>>
------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick 49152
Y 2782
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick 49152
Y 2744
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick 49152
Y 5307
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick 49152
Y 3986
>>>> NFS Server on localhost 2049
Y 51697
>>>> Self-heal Daemon on localhost N/A
Y 51710
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049
Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A
Y 110905
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049
Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A
Y 22192
>>>> NFS Server on 10.8.230.209 2049
Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y
4104
>>>>
>>>> Task Status of Volume JQStore2
>>>>
------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: Webresource
>>>> Gluster process Port
Online Pid
>>>>
------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick3 49155
Y 2787
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 49155
Y 2753
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick3 49155
Y 5313
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick3 49155
Y 3992
>>>> NFS Server on localhost 2049
Y 51697
>>>> Self-heal Daemon on localhost N/A
Y 51710
>>>> NFS Server on 10.8.230.209 2049
Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y
4104
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049
Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A
Y 22192
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049
Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A
Y 110905
>>>>
>>>> Task Status of Volume Webresource
>>>>
------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: ccim
>>>> Gluster process Port
Online Pid
>>>>
------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick2 49154
Y 2793
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 49154
Y 2745
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick2 49154
Y 5320
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick2 49154
Y 3999
>>>> NFS Server on localhost 2049
Y 51697
>>>> Self-heal Daemon on localhost N/A
Y 51710
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049
Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A
Y 22192
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049
Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A
Y 110905
>>>> NFS Server on 10.8.230.209 2049
Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y
4104
>>>>
>>>> Task Status of Volume ccim
>>>>
------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: cloudimage
>>>> Gluster process Port
Online Pid
>>>>
------------------------------------------------------------------------------
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 49156
Y 5325
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick4 49156
Y 4005
>>>> NFS Server on localhost 2049
Y 51697
>>>> Self-heal Daemon on localhost N/A
Y 51710
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049
Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A
Y 22192
>>>> NFS Server on 10.8.230.209 2049
Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y
4104
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049
Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A
Y 110905
>>>>
>>>> Task Status of Volume cloudimage
>>>>
------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: filews
>>>> Gluster process Port
Online Pid
>>>>
------------------------------------------------------------------------------
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 49153
Y 5330
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick1 49153
Y 4010
>>>> NFS Server on localhost 2049
Y 51697
>>>> Self-heal Daemon on localhost N/A
Y 51710
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049
Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A
Y 22192
>>>> NFS Server on 10.8.230.209 2049
Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y
4104
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049
Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A
Y 110905
>>>>
>>>> Task Status of Volume filews
>>>>
------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: filex
>>>> Gluster process Port
Online Pid
>>>>
------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick1 49153
Y 2794
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 49153
Y 2755
>>>> NFS Server on localhost 2049
Y 51697
>>>> Self-heal Daemon on localhost N/A
Y 51710
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049
Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A
Y 110905
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049
Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A
Y 22192
>>>> NFS Server on 10.8.230.209 2049
Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y
4104
>>>>
>>>> Task Status of Volume filex
>>>>
------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: pubmail
>>>> Gluster process Port
Online Pid
>>>>
------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick4 49156
Y 2792
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 49156
Y 2754
>>>> NFS Server on localhost 2049
Y 51697
>>>> Self-heal Daemon on localhost N/A
Y 51710
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049
Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A
Y 110905
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049
Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A
Y 22192
>>>> NFS Server on 10.8.230.209 2049
Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y
4104
>>>>
>>>> Task Status of Volume pubmail
>>>>
------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> [root at SH02SVR5952 ~]#
>>>>
>>>>
>>>>
>>>> Best Regards
>>>> ???
>>>> Site Operations Center
>>>> Ext.15554 Seat.1#3F049
>>>> We are the F.I.R.S.T.
>>>> Fast, Innovation, Reliability , Secure, Team
>>>>
>>>>
>>>>
>>>> -----????-----
>>>> ???: Atin Mukherjee [mailto:amukherj at
redhat.com<mailto:amukherj at redhat.com>]
>>>> ????: Tuesday, June 02, 2015 2:20 PM
>>>> ???: vyyy???; Gluster-users at
gluster.org<mailto:Gluster-users at gluster.org>
>>>> ??: Re: ??: [Gluster-users] Gluster peer rejected and failed to
start
>>>>
>>>>
>>>>
>>>> On 06/02/2015 11:33 AM, vyyy??? wrote:
>>>>> Actually I have 2 problems
>>>>> 1?New nodes can't add to the clusters
>>>>> I cleaned /var/lib/glusterd, now status is
>>>>> State: Accepted peer request (Connected)
>>>>>
>>>>> 2?One of clusterd nodes shown 'Peer rejected' and
glusterd failed to start.
>>>>> The log is attached in pre-mail
>>>>> This is a product cluster, this problem is more egent
>>>> From the log I can clearly see the problematic node is
>>>> glusterfs09.sh2.ctripcorp.com. Your existing cluster
configuration has
>>>> bricks hosted in glusterfs09.sh2.ctripcorp.com however the same
is not
>>>> part of the cluster. Could you paste the output of gluster peer
status
>>>> and gluster volume status ?
>>>>>
>>>>>
>>>>>
>>>>> Best Regards
>>>>> Yuyang Yang
>>>>>
>>>>> -----????-----
>>>>> ???: Atin Mukherjee [mailto:amukherj at
redhat.com<mailto:amukherj at redhat.com>]
>>>>> ????: Tuesday, June 02, 2015 12:52 PM
>>>>> ???: vyyy???; Gluster-users at
gluster.org<mailto:Gluster-users at gluster.org>
>>>>> ??: Re: [Gluster-users] Gluster peer rejected and failed to
start
>>>>>
>>>>>
>>>>>
>>>>> On 06/02/2015 10:00 AM, vyyy??? wrote:
>>>>>> Hi
>>>>>>
>>>>>> We have a gluster (Version 3.6.3) cluster with 6 nodes,
I tried to add 4 more nodes, but ?Peer Rejected?, then I tried to resolve it by
dump /var/lib/glusterd and probe again, not success, this is a question, But
strange thing is:
>>>>>>
>>>>>> A node already in cluster also shown ?Peer Reject?
>>>>>>
>>>>>> I tried to restart glusterd, It failed
>>>>>>
>>>>>> I found that /var/lib/glusterd/peers is empty, I copied
the files from other nodes, still can?t start glusterd
>>>>> It seems like you are trying to peer probe nodes which are
either either
>>>>> part of some other clusters (uncleaned nodes). Could you
check whether
>>>>> the nodes which you are adding have empty
/var/lib/glusterd? If not
>>>>> clean them and retry.
>>>>>
>>>>> ~Atin
>>>>>>
>>>>>>
>>>>>>
>>>>>> etc-glusterfs-glusterd.vol.log shown that cluster
member as ?unknown peer ?
>>>>>>
>>>>>> [2015-06-02 01:52:14.650635] C
[glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend
update request from unknown peer 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>>>>>> [2015-06-02 01:52:14.650786] C
[glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend
update request from unknown peer 674a78b5-0590-48d4-8752-d4608832ed1d
>>>>>> [2015-06-02 01:52:14.657881] C
[glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend
update request from unknown peer 83e1a9db-3134-45e4-acd2-387b12b5b207
>>>>>> [2015-06-02 01:52:17.747865] W
[glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management:
04f22ee8-8e00-4c32-a924-b40a0e413aa6 doesn't belong to the cluster. Ignoring
request.
>>>>>> [2015-06-02 01:52:17.747908] E
[rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to
complete successfully
>>>>>> [2015-06-02 01:52:40.338885] W
[glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management:
674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring
request.
>>>>>> [2015-06-02 01:52:40.338929] E
[rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to
complete successfully
>>>>>> [2015-06-02 01:52:41.310451] W
[glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management:
674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring
request.
>>>>>> [2015-06-02 01:52:41.310486] E
[rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to
complete successfully
>>>>>>
>>>>>>
>>>>>>
>>>>>> Debug info is as following,
>>>>>>
>>>>>>
>>>>>>
>>>>>> /usr/sbin/glusterd
>>>>>> [root at SH02SVR5951 peers]# /usr/sbin/glusterd --debug
>>>>>> [2015-06-02 04:09:24.626690] I [MSGID: 100030]
[glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running
/usr/sbin/glusterd version 3.6.3 (args: /usr/sbin/glusterd --debug)
>>>>>> [2015-06-02 04:09:24.626739] D
[logging.c:1763:__gf_log_inject_timer_event] 0-logging-infra: Starting timer
now. Timeout = 120, current buf size = 5
>>>>>> [2015-06-02 04:09:24.627052] D [MSGID: 0]
[glusterfsd.c:613:get_volfp] 0-glusterfsd: loading volume file
/etc/glusterfs/glusterd.vol
>>>>>> [2015-06-02 04:09:24.629683] I [glusterd.c:1214:init]
0-management: Maximum allowed open file descriptors set to 65536
>>>>>> [2015-06-02 04:09:24.629706] I [glusterd.c:1259:init]
0-management: Using /var/lib/glusterd as working directory
>>>>>> [2015-06-02 04:09:24.629764] D
[glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
>>>>>> [2015-06-02 04:09:24.629895] D
[rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited.
>>>>>> [2015-06-02 04:09:24.629904] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
GF-DUMP, Num: 123451501, Ver: 1, Port: 0
>>>>>> [2015-06-02 04:09:24.629930] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.631989] D
[socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is
NOT enabled
>>>>>> [2015-06-02 04:09:24.632005] D
[socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT
enabled
>>>>>> [2015-06-02 04:09:24.632013] D
[socket.c:3827:socket_init] 0-socket.management: using system polling thread
>>>>>> [2015-06-02 04:09:24.632024] D
[name.c:550:server_fill_address_family] 0-socket.management: option
address-family not specified, defaulting to inet
>>>>>> [2015-06-02 04:09:24.632072] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so
>>>>>> [2015-06-02 04:09:24.632102] E
[rpc-transport.c:266:rpc_transport_load] 0-rpc-transport:
/usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so: cannot open shared object
file: No such file or directory
>>>>>> [2015-06-02 04:09:24.632112] W
[rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume
'rdma.management': transport-type 'rdma' is not valid or not
found on this machine
>>>>>> [2015-06-02 04:09:24.632122] W
[rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener,
initing the transport failed
>>>>>> [2015-06-02 04:09:24.632132] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632140] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632148] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632155] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0
>>>>>> [2015-06-02 04:09:24.632163] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
Gluster Portmap, Num: 34123456, Ver: 1, Port: 0
>>>>>> [2015-06-02 04:09:24.632170] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
Gluster Handshake, Num: 14398633, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632177] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0
>>>>>> [2015-06-02 04:09:24.632212] D
[rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited.
>>>>>> [2015-06-02 04:09:24.632220] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
GF-DUMP, Num: 123451501, Ver: 1, Port: 0
>>>>>> [2015-06-02 04:09:24.632236] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.632302] D
[socket.c:3692:socket_init] 0-socket.management: disabling nodelay
>>>>>> [2015-06-02 04:09:24.632314] D
[socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is
NOT enabled
>>>>>> [2015-06-02 04:09:24.632321] D
[socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT
enabled
>>>>>> [2015-06-02 04:09:24.632329] D
[socket.c:3827:socket_init] 0-socket.management: using system polling thread
>>>>>> [2015-06-02 04:09:24.632402] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632414] D
[rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered:
Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632443] D
[glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.632455] D [glusterd.c:1484:init]
0-management: cannot get run-with-valgrind value
>>>>>> [2015-06-02 04:09:24.633086] I
[glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module
not installed in the system
>>>>>> [2015-06-02 04:09:24.633518] D
[glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning -1
>>>>>> [2015-06-02 04:09:24.634031] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.634057] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.634115] D
[store.c:304:gf_store_retrieve_value] 0-: key operating-version found
>>>>>> [2015-06-02 04:09:24.634141] I
[glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 2
>>>>>> [2015-06-02 04:09:24.634196] D
[glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.634436] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.634446] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.634469] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634488] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634496] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.634510] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634517] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>>>>>> [2015-06-02 04:09:24.634528] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634534] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.634544] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634551] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value
= 2
>>>>>> [2015-06-02 04:09:24.634562] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634569] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count
value = 1
>>>>>> [2015-06-02 04:09:24.634580] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634587] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count
value = 2
>>>>>> [2015-06-02 04:09:24.634597] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634603] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value =
24
>>>>>> [2015-06-02 04:09:24.634613] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634620] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type
value = 0
>>>>>> [2015-06-02 04:09:24.634631] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634638] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value
= 7bf7d81a-f835-4c3b-9c58-4d4ad7ec6e07
>>>>>> [2015-06-02 04:09:24.634652] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634659] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value =
c7374917-2344-4c42-aeae-1d0bdb7ece0a
>>>>>> [2015-06-02 04:09:24.634678] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634689] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value =
703c3b9d-4343-4546-9ca3-350aecdd52fe
>>>>>> [2015-06-02 04:09:24.634700] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634707] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value
= 2
>>>>>> [2015-06-02 04:09:24.634719] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634726] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.634736] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634742] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
diagnostics.latency-measurement value = on
>>>>>> [2015-06-02 04:09:24.634773] D
[glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as
Volume-set:key=diagnostics.latency-measurement,value:on
>>>>>> [2015-06-02 04:09:24.634785] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634791] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
diagnostics.count-fop-hits value = on
>>>>>> [2015-06-02 04:09:24.634803] D
[glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as
Volume-set:key=diagnostics.count-fop-hits,value:on
>>>>>> [2015-06-02 04:09:24.634814] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634821] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
nfs.rpc-auth-allow value = 10.8.*
>>>>>> [2015-06-02 04:09:24.634838] D
[glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as
Volume-set:key=nfs.rpc-auth-allow,value:10.8.*
>>>>>> [2015-06-02 04:09:24.634849] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634856] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value =
glusterfs05.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634868] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634874] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value =
glusterfs06.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634885] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634891] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value =
glusterfs07.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634902] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634908] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value =
glusterfs08.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634919] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634925] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value =
glusterfs09.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634935] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634942] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value =
glusterfs10.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634954] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.634984] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635002] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.635019] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635030] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635039] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635048] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635056] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635066] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635074] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635083] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635092] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635101] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635110] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635118] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635127] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635136] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635144] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635153] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635161] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635368] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.635380] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.635396] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635416] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635426] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635436] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635447] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635456] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635465] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.635488] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.635501] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635748] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.635766] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.635786] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635807] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635818] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635828] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635838] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635847] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635856] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.635909] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.635922] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636158] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636167] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636180] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636202] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636213] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636222] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636232] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636241] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636250] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.636274] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.636286] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636471] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636479] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636493] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636514] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636524] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636533] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636542] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636552] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636561] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.636586] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.636598] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636856] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636889] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636922] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636943] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636954] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636964] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636975] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636984] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636994] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.637020] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.637036] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.637307] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.637325] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.637346] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.637367] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.637378] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.637388] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.819347] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.819374] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.819385] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.819413] D
[glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.819707] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.820187] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.820222] D
[store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.820239] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.820532] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.820585] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.820959] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.821064] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.821090] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.837290] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837540] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837551] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837568] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837589] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837600] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.837862] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837880] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837896] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837918] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837929] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837939] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.837964] D
[glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.838216] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.838257] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.838273] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838295] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838303] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.838314] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838320] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>>>>>> [2015-06-02 04:09:24.838330] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838336] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.838346] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838352] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value
= 2
>>>>>> [2015-06-02 04:09:24.838361] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838367] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count
value = 1
>>>>>> [2015-06-02 04:09:24.838377] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838383] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count
value = 2
>>>>>> [2015-06-02 04:09:24.838393] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838399] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value =
6
>>>>>> [2015-06-02 04:09:24.838408] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838414] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type
value = 0
>>>>>> [2015-06-02 04:09:24.838423] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838430] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value
= cdc279f4-6cd0-4087-8d2a-440bccb51e33
>>>>>> [2015-06-02 04:09:24.838443] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838449] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value =
0b6d3496-0865-45e4-a45b-c221b22d3d92
>>>>>> [2015-06-02 04:09:24.838459] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838465] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value =
ecedfb4d-3308-4f6c-b188-4d5b868958e1
>>>>>> [2015-06-02 04:09:24.838475] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838482] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value
= 2
>>>>>> [2015-06-02 04:09:24.838493] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838499] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.838508] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838515] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
nfs.rpc-auth-allow value = 10.8.68.*
>>>>>> [2015-06-02 04:09:24.838533] D
[glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as
Volume-set:key=nfs.rpc-auth-allow,value:10.8.68.*
>>>>>> [2015-06-02 04:09:24.838545] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838552] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value =
glusterfs05.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838563] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838570] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value =
glusterfs06.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838580] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838587] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value =
glusterfs07.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838597] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838604] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value =
glusterfs08.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838614] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838621] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value =
glusterfs09.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838631] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838637] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value =
glusterfs10.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838649] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.838697] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838719] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.838738] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838747] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838756] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838764] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838772] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838781] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838789] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838797] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838806] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838815] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838823] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838832] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838840] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838848] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838857] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839051] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839063] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839084] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839102] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839113] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839123] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839133] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839142] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839151] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.839177] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.839189] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839382] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839390] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839403] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839421] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839431] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839440] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839450] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839459] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839468] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.839491] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.839502] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839700] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839712] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839732] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839750] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839760] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839770] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839780] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839789] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839799] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.839822] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.839834] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840057] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840075] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840096] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840117] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840128] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840137] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840147] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840156] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840166] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.840192] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.840206] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840402] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840410] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840424] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840440] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840450] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840459] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840469] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840478] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840487] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.840513] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.840525] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840799] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840818] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840839] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840859] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840871] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840880] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840891] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840900] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840910] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.840931] D
[glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.841118] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.841359] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.841396] D
[store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.841413] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.841557] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.841611] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.841799] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.841875] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.841901] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.842350] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842469] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842477] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842489] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842514] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842525] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.842636] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842645] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842656] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842696] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842708] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842718] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.842738] D
[glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.842848] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842859] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842872] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842891] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842898] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.842908] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842914] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4
>>>>>> [2015-06-02 04:09:24.842924] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842930] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.842939] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842945] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value
= 2
>>>>>> [2015-06-02 04:09:24.842954] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842960] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count
value = 1
>>>>>> [2015-06-02 04:09:24.842969] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842975] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count
value = 2
>>>>>> [2015-06-02 04:09:24.842984] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842990] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value =
3
>>>>>> [2015-06-02 04:09:24.842999] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843005] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type
value = 0
>>>>>> [2015-06-02 04:09:24.843015] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843021] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value
= a3369f63-9fce-4a56-a921-f7206cb8d3ae
>>>>>> [2015-06-02 04:09:24.843033] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843039] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value =
c6012954-928a-43c1-b7ea-336d6ee0bea9
>>>>>> [2015-06-02 04:09:24.843050] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843056] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value =
bfbd7007-0e0c-45f9-8d59-05e1aee66728
>>>>>> [2015-06-02 04:09:24.843066] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843072] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value
= 2
>>>>>> [2015-06-02 04:09:24.843082] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843089] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.843098] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843104] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
nfs.rpc-auth-allow value = 10.8.83.*
>>>>>> [2015-06-02 04:09:24.843119] D
[glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as
Volume-set:key=nfs.rpc-auth-allow,value:10.8.83.*
>>>>>> [2015-06-02 04:09:24.843130] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843136] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value =
glusterfs07.sh2.ctripcorp.com:-export-sdb-brick4
>>>>>> [2015-06-02 04:09:24.843147] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843154] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value =
glusterfs08.sh2.ctripcorp.com:-export-sdb-brick4
>>>>>> [2015-06-02 04:09:24.843164] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843171] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value =
glusterfs09.sh2.ctripcorp.com:-export-sdb-brick4
>>>>>> [2015-06-02 04:09:24.843181] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843188] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value =
glusterfs10.sh2.ctripcorp.com:-export-sdb-brick4
>>>>>> [2015-06-02 04:09:24.843198] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.843221] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843238] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.843255] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843264] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843273] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843281] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843290] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843298] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843306] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843314] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843323] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843332] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843340] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843348] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843357] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843365] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843374] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843476] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.843484] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.843499] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843519] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843720] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843739] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843751] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843761] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843771] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.843802] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.843817] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843984] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.843996] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844013] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844030] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844041] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844050] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844060] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844069] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844078] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.844104] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.844116] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844222] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844230] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844243] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844266] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844276] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844285] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844294] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844303] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844312] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.844334] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.844346] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844443] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844451] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844468] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844487] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844497] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844506] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844516] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844525] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844534] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.844550] D
[glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844753] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.845000] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.845033] D
[store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.845049] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.845199] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.845244] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.845396] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.845448] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.845471] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.846053] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846195] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846206] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846220] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846241] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846252] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.846362] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846371] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846382] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846404] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846414] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846424] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.846442] D
[glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.846543] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846552] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846562] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846585] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846592] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.846603] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846609] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>>>>>> [2015-06-02 04:09:24.846618] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846624] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.846633] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846639] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value
= 2
>>>>>> [2015-06-02 04:09:24.846648] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846654] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count
value = 1
>>>>>> [2015-06-02 04:09:24.846663] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846690] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count
value = 2
>>>>>> [2015-06-02 04:09:24.846702] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846708] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value =
3
>>>>>> [2015-06-02 04:09:24.846717] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846723] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type
value = 0
>>>>>> [2015-06-02 04:09:24.846733] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846739] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value
= 8869b21b-1bf1-4eba-a52a-1158abde7d23
>>>>>> [2015-06-02 04:09:24.846752] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846758] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value =
27298b95-919d-43f7-9507-8cbfbe5292e3
>>>>>> [2015-06-02 04:09:24.846769] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846775] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value =
47f52166-020b-4bef-ada1-7e65fcd2834f
>>>>>> [2015-06-02 04:09:24.846785] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846792] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value
= 2
>>>>>> [2015-06-02 04:09:24.846802] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846808] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.846818] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846824] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
diagnostics.count-fop-hits value = on
>>>>>> [2015-06-02 04:09:24.846837] D
[glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as
Volume-set:key=diagnostics.count-fop-hits,value:on
>>>>>> [2015-06-02 04:09:24.846848] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846854] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
diagnostics.latency-measurement value = on
>>>>>> [2015-06-02 04:09:24.846864] D
[glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as
Volume-set:key=diagnostics.latency-measurement,value:on
>>>>>> [2015-06-02 04:09:24.846875] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846881] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value =
glusterfs05.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846892] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846899] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value =
glusterfs06.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846910] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846916] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value =
glusterfs07.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846927] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846934] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value =
glusterfs08.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846944] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846952] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value =
glusterfs09.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846962] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846969] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value =
glusterfs10.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846981] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.847003] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847027] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.847045] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847055] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847064] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847072] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847081] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847089] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847098] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847106] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847115] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847124] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847133] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847141] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847150] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847159] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847167] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847176] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847286] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847294] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847308] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847328] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847339] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847348] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847358] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847367] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847376] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.847402] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.847414] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847516] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847523] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847537] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847559] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847569] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847579] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847588] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847597] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847607] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.847628] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.847640] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847747] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847756] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847771] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847794] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847804] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847814] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847823] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847832] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847841] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.847862] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.847873] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847969] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847977] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847990] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848012] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848022] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848031] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848040] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848049] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848058] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.848080] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.848091] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848187] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.848195] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.848208] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848230] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848240] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848249] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848259] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848267] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848277] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.848299] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.848310] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848411] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.848419] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.848432] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848455] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848465] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848474] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848484] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848493] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848503] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.848519] D
[glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848727] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.848975] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.849002] D
[store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.849019] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.849167] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.849212] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.849367] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.849419] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.849442] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.849880] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850003] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850012] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850024] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850043] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850054] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.850157] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850166] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850177] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850192] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850202] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850211] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.850228] D
[glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.850402] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850414] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850426] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850444] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850451] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.850461] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850467] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4
>>>>>> [2015-06-02 04:09:24.850476] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850482] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.850491] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850497] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value
= 2
>>>>>> [2015-06-02 04:09:24.850506] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850512] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count
value = 1
>>>>>> [2015-06-02 04:09:24.850522] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850528] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count
value = 2
>>>>>> [2015-06-02 04:09:24.850537] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850543] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = disperse_count
value = 0
>>>>>> [2015-06-02 04:09:24.850552] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850558] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = redundancy_count
value = 0
>>>>>> [2015-06-02 04:09:24.850567] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850573] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value =
11
>>>>>> [2015-06-02 04:09:24.850582] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850588] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type
value = 0
>>>>>> [2015-06-02 04:09:24.850598] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850604] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value
= ea46609c-be9a-4283-8357-8fdc2e5a190e
>>>>>> [2015-06-02 04:09:24.850616] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850622] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value =
21b247ce-58ce-4087-8c32-3d578b4cc22d
>>>>>> [2015-06-02 04:09:24.850632] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850638] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value =
0724171a-c7ce-4528-b54b-2d983f969537
>>>>>> [2015-06-02 04:09:24.850648] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850654] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value
= 2
>>>>>> [2015-06-02 04:09:24.850664] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850692] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.850703] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850709] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key =
nfs.rpc-auth-allow value = 10.8.92.*,10.8.230.*,192.168.98.*
>>>>>> [2015-06-02 04:09:24.850724] D
[glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as
Volume-set:key=nfs.rpc-auth-allow,value:10.8.92.*,10.8.230.*,192.168.98.*
>>>>>> [2015-06-02 04:09:24.850736] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850743] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value =
glusterfs07.sh2.ctripcorp.com:-export-sdb-brick1
>>>>>> [2015-06-02 04:09:24.850754] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850760] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value =
glusterfs08.sh2.ctripcorp.com:-export-sdb-brick1
>>>>>> [2015-06-02 04:09:24.850772] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850778] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value =
glusterfs09.sh2.ctripcorp.com:-export-sdb-brick1
>>>>>> [2015-06-02 04:09:24.850789] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850795] D
[glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value =
glusterfs10.sh2.ctripcorp.com:-export-sdb-brick1
>>>>>> [2015-06-02 04:09:24.850807] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.850830] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850850] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.850867] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850877] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850885] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850894] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850902] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850911] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850919] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850928] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850936] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850945] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850953] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850962] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850971] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850980] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850988] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850996] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851005] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851164] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851173] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851187] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851203] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851213] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851223] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851232] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851242] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851251] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851261] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.851283] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.851295] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851457] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851465] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851478] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851493] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851504] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851513] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851522] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851531] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851540] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851549] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.851571] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.851583] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851732] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851741] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851755] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851770] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851780] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851789] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851799] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851807] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851816] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851826] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.851850] D
[glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.851863] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851959] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851974] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852005] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852026] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852037] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852047] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852061] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852070] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852080] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852090] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.852111] D
[glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852298] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.852524] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852556] D
[store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.852576] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852723] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852776] D
[glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning
with 0
>>>>>> [2015-06-02 04:09:24.852930] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852982] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.853006] D
[store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.853412] D
[store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853524] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853539] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853553] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853573] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853583] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.853699] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853718] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853735] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853757] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853768] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853778] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853787] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853798] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.853820] D
[glusterd-store.c:3094:glusterd_store_retrieve_volumes] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853962] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853972] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853983] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854000] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854011] D
[glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854023] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854032] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854043] D
[glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854054] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854070] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
>>>>>> [2015-06-02 04:09:24.854162] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854171] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854181] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854197] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854205] D
[glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854216] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854225] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854232] D
[glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854241] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854255] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
>>>>>> [2015-06-02 04:09:24.854345] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854354] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854364] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854380] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854390] D
[glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854402] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854411] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854417] D
[glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854427] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854439] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
>>>>>> [2015-06-02 04:09:24.854527] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854536] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854546] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854561] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854569] D
[glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854580] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854589] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854596] D
[glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854605] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854617] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
>>>>>> [2015-06-02 04:09:24.854704] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854713] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854725] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854740] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854749] D
[glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854760] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854769] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854776] D
[glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854789] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854803] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
>>>>>> [2015-06-02 04:09:24.855064] D
[store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.855072] D
[store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.855083] D
[store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.855098] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.855106] D
[glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.855116] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.855125] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.855132] D
[glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.855141] D
[store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.855154] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
>>>>>> [2015-06-02 04:09:24.855186] D
[glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd:
Returning 0
>>>>>> [2015-06-02 04:09:24.855210] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>>>>>> [2015-06-02 04:09:24.855218] D
[rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to
30
>>>>>> [2015-06-02 04:09:24.855231] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.855324] D
[socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT
enabled
>>>>>> [2015-06-02 04:09:24.855333] D
[socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT
enabled
>>>>>> [2015-06-02 04:09:24.855340] D
[socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.859610] D
[common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.211
(port-24007) for hostname: glusterfs07.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.859628] D
[socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper
connection
>>>>>> [2015-06-02 04:09:24.859707] D
[glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.859733] D
[glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd:
Returning 0
>>>>>> [2015-06-02 04:09:24.859772] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>>>>>> [2015-06-02 04:09:24.859781] D
[rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to
30
>>>>>> [2015-06-02 04:09:24.859794] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.859905] D
[socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT
enabled
>>>>>> [2015-06-02 04:09:24.859916] D
[socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT
enabled
>>>>>> [2015-06-02 04:09:24.859923] D
[socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.863505] D
[common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.231.160
(port-24007) for hostname: glusterfs11.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.863521] D
[socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper
connection
>>>>>> [2015-06-02 04:09:24.863562] D
[glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.863585] D
[glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd:
Returning 0
>>>>>> [2015-06-02 04:09:24.863607] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>>>>>> [2015-06-02 04:09:24.863615] D
[rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to
30
>>>>>> [2015-06-02 04:09:24.863625] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.863754] D
[socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT
enabled
>>>>>> [2015-06-02 04:09:24.863765] D
[socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT
enabled
>>>>>> [2015-06-02 04:09:24.863772] D
[socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.867255] D
[common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.210
(port-24007) for hostname: glusterfs06.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.867271] D
[socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper
connection
>>>>>> [2015-06-02 04:09:24.867314] D
[glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.867337] D
[glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd:
Returning 0
>>>>>> [2015-06-02 04:09:24.867358] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>>>>>> [2015-06-02 04:09:24.867366] D
[rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to
30
>>>>>> [2015-06-02 04:09:24.867377] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.867463] D
[socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT
enabled
>>>>>> [2015-06-02 04:09:24.867476] D
[socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT
enabled
>>>>>> [2015-06-02 04:09:24.867483] D
[socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.870920] D
[common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.214
(port-24007) for hostname: glusterfs10.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.870935] D
[socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper
connection
>>>>>> [2015-06-02 04:09:24.870997] D
[glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.871021] D
[glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd:
Returning 0
>>>>>> [2015-06-02 04:09:24.871042] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>>>>>> [2015-06-02 04:09:24.871050] D
[rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to
30
>>>>>> [2015-06-02 04:09:24.871061] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.871129] D
[socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT
enabled
>>>>>> [2015-06-02 04:09:24.871138] D
[socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT
enabled
>>>>>> [2015-06-02 04:09:24.871144] D
[socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.874102] D
[common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.209
(port-24007) for hostname: 10.8.230.209 and port: 24007
>>>>>> [2015-06-02 04:09:24.874116] D
[socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper
connection
>>>>>> [2015-06-02 04:09:24.874159] D
[glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.874181] D
[glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd:
Returning 0
>>>>>> [2015-06-02 04:09:24.874202] I
[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to
600
>>>>>> [2015-06-02 04:09:24.874210] D
[rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to
30
>>>>>> [2015-06-02 04:09:24.874219] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.874292] D
[socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT
enabled
>>>>>> [2015-06-02 04:09:24.874301] D
[socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT
enabled
>>>>>> [2015-06-02 04:09:24.874326] D
[socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.877759] D
[common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.212
(port-24007) for hostname: glusterfs08.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.877774] D
[socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper
connection
>>>>>> [2015-06-02 04:09:24.877818] D
[glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.877831] D
[glusterd-store.c:4106:glusterd_store_retrieve_peers] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.877863] I
[glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management:
No missed snaps list.
>>>>>> [2015-06-02 04:09:24.877873] D
[glusterd-store.c:3612:glusterd_store_retrieve_snaps] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.881125] D
[glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.881140] D
[glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.881148] D
[glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend
glusterfs06.sh2.ctripcorp.com found.. state: 3
>>>>>> [2015-06-02 04:09:24.881156] D
[glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.881162] D
[glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.881169] D
[glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend
glusterfs07.sh2.ctripcorp.com found.. state: 3
>>>>>> [2015-06-02 04:09:24.881176] D
[glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.881182] D
[glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.881188] D
[glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend
glusterfs08.sh2.ctripcorp.com found.. state: 3
>>>>>> [2015-06-02 04:09:24.881195] D
[glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.881201] D
[glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.890739] D
[glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] 0-management:
Unable to find friend: glusterfs09.sh2.ctripcorp.com
>>>>>> [2015-06-02 04:09:24.891607] D
[common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>>>>>> [2015-06-02 04:09:24.891750] D
[common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>>>>>> [2015-06-02 04:09:24.891820] D
[common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>>>>>> [2015-06-02 04:09:24.891881] D
[common-utils.c:2913:gf_is_local_addr] 0-management:
glusterfs09.sh2.ctripcorp.com is not local
>>>>>> [2015-06-02 04:09:24.891890] D
[glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning -1
>>>>>> [2015-06-02 04:09:24.891898] D
[glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning -1
>>>>>> [2015-06-02 04:09:24.891904] E
[glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick
failed in restore
>>>>>> [2015-06-02 04:09:24.891911] D
[glusterd-store.c:4321:glusterd_restore] 0-: Returning -1
>>>>>> [2015-06-02 04:09:24.891923] E
[xlator.c:425:xlator_init] 0-management: Initialization of volume
'management' failed, review your volfile again
>>>>>> [2015-06-02 04:09:24.891933] E
[graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed
>>>>>> [2015-06-02 04:09:24.891941] E
[graph.c:525:glusterfs_graph_activate] 0-graph: init failed
>>>>>> [2015-06-02 04:09:24.892164] D
[logging.c:1740:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size
reduced. About to flush 2 extra log messages
>>>>>> [2015-06-02 04:09:24.892178] D
[logging.c:1743:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 2 extra
log messages
>>>>>> [2015-06-02 04:09:24.892325] W
[glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received signum (0), shutting
down
>>>>>> [2015-06-02 04:09:24.892345] D
[glusterfsd-mgmt.c:2244:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper
signout arguments not given
>>>>>> [root at SH02SVR5951 peers]#
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best Regards
>>>>>> Yuyang Yang
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org<mailto:Gluster-users at
gluster.org>
>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
~Atin
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150605/f10c78c5/attachment-0001.html>