vyyy杨雨阳
2015-Jun-05 02:24 UTC
[Gluster-users] 答复: 答复: 答复: Gluster peer rejected and failed to start
Can anyone help me? New nodes still can't join the cluster. The new nodes can peer each other, just not exist cluster nodes. I can not find any network problem, there were no firewall, port 24007 is connected, there were all at the same subnet... Thanks [root at SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log [2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) [2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory [2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans port/rdma.so: cannot open shared object file: No such file or directory [2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport- type 'rdma' is not valid or not found on this machine [2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra nsport failed [2015-06-05 01:59:02.343346] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-06-05 01:59:02.351861] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-06-05 01:59:02.363138] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap s list. Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option ping-timeout 30 8: option transport.socket.read-fail-log off 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-time 10 11: option transport-type rdma 12: option working-directory /var/lib/glusterd 13: end-volume 14: +------------------------------------------------------------------------------+ [2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 1b98af34-08f9-492f-b54e-7608 2588d63b [2015-06-05 02:00:43.025462] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:00:44.025069] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:00:44.599862] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:00:46.029823] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:00:46.701845] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:01:43.119612] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:01:44.165527] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:01:44.851903] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:01:46.190401] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:01:46.722394] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:01:58.719968] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:32.584999] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust erfs05.sh2.ctripcorp.com 24007 [2015-06-05 02:02:32.586281] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g lusterfs05.sh2.ctripcorp.com (24007) [2015-06-05 02:02:32.602192] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-05 02:02:32.606053] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0 [2015-06-05 02:02:32.620958] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the 'versions' from peer (10.8.230.209:24007) [2015-06-05 02:02:32.621020] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer 000000 00-0000-0000-0000-000000000000, in Establishing Connection state, has disconnected from glusterd. [2015-06-05 02:02:43.264141] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:02:44.312918] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:02:44.781434] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:46.176888] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:46.799916] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:03:42.720994] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:03:43.771987] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:03:44.659775] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:03:45.956807] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:03:46.688787] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:04:42.891131] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:04:43.957423] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:04:44.995638] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:04:46.062171] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:04:46.957327] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:05:42.966651] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:05:44.008498] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:05:45.049936] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:05:46.087007] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:05:47.029325] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:06:42.874089] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:06:43.910998] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:06:44.932802] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:06:46.053731] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:06:47.055859] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:07:43.099647] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:07:44.146454] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:07:45.218982] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:07:46.246303] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:07:47.106954] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:08:43.226090] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:08:44.292041] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:08:45.351830] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:08:45.688349] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:08:46.651768] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:09:42.748628] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed Best Regards -----????----- ???: vyyy??? ????: Thursday, June 04, 2015 11:00 AM ???: 'Atin Mukherjee'; Gluster-users at gluster.org ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start ?? The problem 2 has resolved. From glusterd log file, I find Cksums error as following: [2015-06-04 01:25:03.137798] E [MSGID: 106010] [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of Cksums JQStore2 differ. local cksum = 2617028277, remote cksum = 2210669223 on peer glusterfs09.sh2.ctripcorp.com The solution for this problem was to do the following, on the BAD node: /etc/init.d/glusterd stop rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/ /etc/init.d/glusterd start The problem 1 is still under inverstigate: New nodes can not join the cluster, log file shown: failed to get the 'versions' from peer I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are " operating-version=2" ?? ??[2015-06-04 02:13:49.199416] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the ??'versions' from peer (10.8.230.211:24007) ??[2015-06-04 02:14:07.112577] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust ??erfs05.sh2.ctripcorp.com 24007 ??[2015-06-04 02:14:07.113778] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g ??lusterfs05.sh2.ctripcorp.com (24007) ??[2015-06-04 02:14:07.131023] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 ??[2015-06-04 02:14:07.134829] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0 ??[ Best Regards ??? Site Operations Center Ext.15554? Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----????----- ???: vyyy??? ????: Wednesday, June 03, 2015 5:37 PM ???: 'Atin Mukherjee'; Gluster-users at gluster.org ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Hi Atin and everybody, I have forced peer detach the news nodes ,The peer status seems even strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help me? Any advice is appreciated. Glusterfs05 ?? ??[root at sh02svr5956 ~]# gluster peer status ??Number of Peers: 5 ?? ??Hostname: glusterfs06.sh2.ctripcorp.com ??Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs07.sh2.ctripcorp.com ??Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs08.sh2.ctripcorp.com ??Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs10.sh2.ctripcorp.com ??Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c ??State: Peer in Cluster (Connected) ?? ??Hostname: glusterfs09.sh2.ctripcorp.com ??Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d ??State: Peer Rejected (Connected) ??[root at sh02svr5956 ~]# ?? ?? ??Glusterfs06 ?? ??[root at SH02SVR5955 ~]# gluster peer status ??Number of Peers: 5 ?? ??Hostname: glusterfs10.sh2.ctripcorp.com ??Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs08.sh2.ctripcorp.com ??Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 ??State: Peer in Cluster (Connected) ?? ??Hostname: glusterfs07.sh2.ctripcorp.com ??Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 ??State: Peer in Cluster (Connected) ?? ??Hostname: 10.8.230.209 ??Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 ??State: Peer Rejected (Connected) ?? ??Hostname: glusterfs09.sh2.ctripcorp.com ??Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d ??State: Peer in Cluster (Connected) ??[root at SH02SVR5955 ~]# ?? Glusterfs07 [root at SH02SVR5954 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) [root at SH02SVR5954 ~]# Glusterfs08 [root at SH02SVR5953 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) [root at SH02SVR5953 ~]# [root at SH02SVR5952 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) [root at SH02SVR5952 ~]# Glusterfs10 [root at SH02SVR5951 ~]# gluster peer status Number of Peers: 6 Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer Rejected (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer Rejected (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer Rejected (Connected) [root at SH02SVR5951 ~]# Best Regards ??? Site Operations Center Ext.15554? Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----????----- ???: vyyy??? ????: Wednesday, June 03, 2015 2:55 PM ???: 'Atin Mukherjee' ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Can I delete the files in /var/lib/glusterd/peers which state is not =3 on all nodes, then restart glusterd? [root at sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15 uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15 state=6 hostname1=glusterfs12.sh2.ctripcorp.com Best Regards Yang -----????----- ???: vyyy??? ????: Wednesday, June 03, 2015 2:39 PM ???: 'Atin Mukherjee' ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Since it's intranet, You can only control my desktop, and then access these node. Should I apply a webex room? Best Regards ??? Site Operations Center Ext.15554? Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----????----- ???: Atin Mukherjee [mailto:amukherj at redhat.com] ????: Wednesday, June 03, 2015 2:36 PM ???: vyyy??? ??: Re: ??: ??: [Gluster-users] Gluster peer rejected and failed to start Is it possible to share your ip/user/passwd for few of these nodes? On 06/03/2015 11:50 AM, vyyy??? wrote:> Can you remote control my desktop (like webex)? > The nodes host bricks. And on the new nodes, I did "clean /var/lib/glusterd and restart" several times, but have no effect. > Is there any other work around? > > > Best Regards > Yang > > -----????----- > ???: Atin Mukherjee [mailto:amukherj at redhat.com] > ????: Wednesday, June 03, 2015 12:33 PM > ???: vyyy???; Gluster-users at gluster.org > ??: Re: ??: [Gluster-users] Gluster peer rejected and failed to start > > I can't comment on what went wrong till I can have visibility on the > entire cluster. Does the node which fails to start host any brick? If > not then work around could be to clean /var/lib/glusterd and restart > glusterd, that should sync up data and bring this node in consistency. > > HTH, > Atin > > On 06/03/2015 08:34 AM, vyyy??? wrote: >> I found that /var/lib/glusterd/peers can't include node's own peer, I delete this file. Glusterd started finally. >> But the peer status still not normal. >> Seems that glusterfs10 leave the cluster and join the new nodes alone. >> >> Peer status on glusterfs10 >> [root at SH02SVR5951 ~]# gluster peer status >> Number of Peers: 7 >> >> Hostname: glusterfs07.sh2.ctripcorp.com >> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs11.sh2.ctripcorp.com >> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 >> State: Accepted peer request (Connected) >> >> Hostname: glusterfs09.sh2.ctripcorp.com >> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs06.sh2.ctripcorp.com >> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >> State: Peer Rejected (Connected) >> >> Hostname: 10.8.230.209 >> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs08.sh2.ctripcorp.com >> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs12.sh2.ctripcorp.com >> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 >> State: Peer in Cluster (Connected) >> >> >> >> Peer status on glusterfs10 >> >> [root at SVR6993HW2285 ~]# gluster peer status >> Number of Peers: 6 >> >> Hostname: glusterfs05.sh2.ctripcorp.com >> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs09.sh2.ctripcorp.com >> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs07.sh2.ctripcorp.com >> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs08.sh2.ctripcorp.com >> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs06.sh2.ctripcorp.com >> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs10.sh2.ctripcorp.com >> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >> State: Peer in Cluster (Connected) >> >> >> Peer status on glusterfs05, this is the node that I tried to peer probe new nodes (glusterfs11 to glusterfs14) >> >> [root at sh02svr5956 ~]# gluster peer status >> Number of Peers: 9 >> >> Hostname: glusterfs06.sh2.ctripcorp.com >> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs07.sh2.ctripcorp.com >> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs08.sh2.ctripcorp.com >> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs10.sh2.ctripcorp.com >> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs09.sh2.ctripcorp.com >> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs11.sh2.ctripcorp.com >> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs12.sh2.ctripcorp.com >> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs13.sh2.ctripcorp.com >> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs14.sh2.ctripcorp.com >> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524 >> State: Peer Rejected (Connected) >> [root at sh02svr5956 ~]# >> >> >> >> >> Best Regards >> Yang >> >> >> -----????----- >> ???: vyyy??? >> ????: Wednesday, June 03, 2015 10:04 AM >> ???: 'Atin Mukherjee'; 'Gluster-users at gluster.org' >> ??: [Gluster-users] Gluster peer rejected and failed to start >> >> I am still messed, can you help me figure out problem 2 (A node of cluster failed to start glusterd) priority >> >> The glusterd log on the node gluster10 shown ' resolve brick failed in restore' >> I checked /var/lib/glusterd/vols, seems ok >> >> Glusterd log file Error message: >> >> [2015-06-03 01:31:32.564010] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore >> [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review >> your volfile again >> >> >> >> >> Best Regards >> Yang >> >> -----????----- >> ???: vyyy??? >> ????: Tuesday, June 02, 2015 6:27 PM >> ???: 'Atin Mukherjee'; Gluster-users at gluster.org >> ??: ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start >> >> Gluster11 to gluster14 are new nodes I tried to join exit cluster but failed >> ?? >> ?? >> ??On gluster12: >> [root at SVR6994HW2285 ~]# gluster peer status >> Number of Peers: 1 >> >> Hostname: glusterfs05.sh2.ctripcorp.com >> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >> State: Accepted peer request (Connected) >> [root at SVR6994HW2285 ~]# >> >> >> >> On gluster05: (peer status seems strange on all those nodes): >> >> [root at sh02svr5956 ~]# gluster peer status >> Number of Peers: 9 >> >> Hostname: glusterfs06.sh2.ctripcorp.com >> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs07.sh2.ctripcorp.com >> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs08.sh2.ctripcorp.com >> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs10.sh2.ctripcorp.com >> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >> State: Peer in Cluster (Disconnected) >> >> Hostname: glusterfs09.sh2.ctripcorp.com >> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs11.sh2.ctripcorp.com >> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs12.sh2.ctripcorp.com >> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 >> State: Peer in Cluster (Connected) >> >> Hostname: glusterfs13.sh2.ctripcorp.com >> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca >> State: Peer Rejected (Connected) >> >> Hostname: glusterfs14.sh2.ctripcorp.com >> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524 >> State: Peer Rejected (Connected) >> [root at sh02svr5956 ~]# >> >> >> >> The glusterd log file >> >> [root at SVR6994HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log >> [2015-06-02 10:23:48.368697] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust >> erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) >> [2015-06-02 10:23:48.372587] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 >> [2015-06-02 10:23:48.372625] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory >> [2015-06-02 10:23:48.374890] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans >> port/rdma.so: cannot open shared object file: No such file or directory >> [2015-06-02 10:23:48.374908] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport- >> type 'rdma' is not valid or not found on this machine >> [2015-06-02 10:23:48.374920] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra >> nsport failed >> [2015-06-02 10:23:48.376539] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed >> in the system >> [2015-06-02 10:23:48.377464] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 >> [2015-06-02 10:23:48.378122] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >> [2015-06-02 10:23:48.378191] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >> [2015-06-02 10:23:48.383072] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap >> s list. >> Final graph: >> +------------------------------------------------------------------------------+ >> 1: volume management >> 2: type mgmt/glusterd >> 3: option rpc-auth.auth-glusterfs on >> 4: option rpc-auth.auth-unix on >> 5: option rpc-auth.auth-null on >> 6: option transport.socket.listen-backlog 128 >> 7: option ping-timeout 30 >> 8: option transport.socket.read-fail-log off >> 9: option transport.socket.keepalive-interval 2 >> 10: option transport.socket.keepalive-time 10 >> 11: option transport-type rdma >> 12: option working-directory /var/lib/glusterd >> 13: end-volume >> 14: >> +------------------------------------------------------------------------------+ >> [2015-06-02 10:23:48.413755] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 0475e11a-3c43-4b76-99dd-1d2e >> 814c5f15 >> [2015-06-02 10:23:48.414345] I [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 04f >> 22ee8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com >> [2015-06-02 10:23:48.414883] I [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req >> [2015-06-02 10:23:50.088645] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 04f22ee >> 8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com, port: 0 >> [2015-06-02 10:23:50.090245] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04 >> f22ee8-8e00-4c32-a924-b40a0e413aa6 >> [2015-06-02 10:23:50.107013] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1016 >> [2015-06-02 10:23:50.107805] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1018 >> [2015-06-02 10:23:50.108358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1020 >> [2015-06-02 10:23:50.139963] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:1008 >> [2015-06-02 10:23:53.286070] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:1006 >> [2015-06-02 10:23:53.893225] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1019 >> [2015-06-02 10:23:53.928314] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1017 >> [2015-06-02 10:23:54.013229] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1014 >> [2015-06-02 10:23:54.273022] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req >> [2015-06-02 10:23:55.531073] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed >> [2015-06-02 10:23:56.293171] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:1004 >> [2015-06-02 10:23:56.900676] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1018 >> [2015-06-02 10:23:56.935113] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed >> [2015-06-02 10:23:56.936892] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1016 >> [2015-06-02 10:23:57.020995] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1012 >> [2015-06-02 10:23:57.659165] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req >> [2015-06-02 10:23:58.332111] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req >> [2015-06-02 10:23:58.929018] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req >> [2015-06-02 10:23:59.100566] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-vers >> ion 2 >> [2015-06-02 10:23:59.103665] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe fro >> m uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >> [2015-06-02 10:23:59.110710] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick >> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick resides >> [2015-06-02 10:23:59.110735] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct >> ripcorp.com:/export/sdb/brick for JQStore2 >> [2015-06-02 10:23:59.131973] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick >> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 resides >> [2015-06-02 10:23:59.131994] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct >> ripcorp.com:/export/sdb/brick3 for Webresource >> [2015-06-02 10:23:59.152949] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick >> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 resides >> [2015-06-02 10:23:59.152969] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct >> ripcorp.com:/export/sdb/brick2 for ccim >> [2015-06-02 10:23:59.172948] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick >> glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 resides >> [2015-06-02 10:23:59.172968] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct >> ripcorp.com:/export/sdb/brick4 for cloudimage >> [2015-06-02 10:23:59.189033] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick >> glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 resides >> [2015-06-02 10:23:59.189053] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct >> ripcorp.com:/export/sdb/brick1 for filews >> [2015-06-02 10:23:59.203985] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick >> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 resides >> [2015-06-02 10:23:59.204005] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct >> ripcorp.com:/export/sdb/brick1 for filex >> [2015-06-02 10:23:59.212727] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick >> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 resides >> [2015-06-02 10:23:59.212747] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct >> ripcorp.com:/export/sdb/brick4 for pubmail >> [2015-06-02 10:23:59.237408] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >> [2015-06-02 10:23:59.243681] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >> [2015-06-02 10:23:59.243947] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glusterfs05.s >> h2.ctripcorp.com (0), ret: 0 >> [2015-06-02 10:23:59.245638] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/5dd5697669073fb549309773b211e62a.s >> ocket failed (Invalid argument) >> [2015-06-02 10:23:59.245679] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has >> disconnected from glusterd. >> [2015-06-02 10:23:59.245703] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/1578ba566e5a93f452fe8ad2e91b8682.s >> ocket failed (Invalid argument) >> [2015-06-02 10:23:59.245719] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glusters >> hd has disconnected from glusterd. >> [2015-06-02 10:23:59.245748] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04 >> f22ee8-8e00-4c32-a924-b40a0e413aa6 >> [2015-06-02 10:23:59.300666] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:1002 >> [2015-06-02 10:23:59.569403] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req >> [2015-06-02 10:23:59.570725] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req >> [2015-06-02 10:23:59.571481] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req >> [2015-06-02 10:23:59.572218] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req >> [2015-06-02 10:23:59.572858] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req >> [2015-06-02 10:23:59.573422] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req >> [2015-06-02 10:23:59.574062] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req >> [2015-06-02 10:23:59.574655] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req >> [2015-06-02 10:23:59.908305] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1017 >> [2015-06-02 10:23:59.944630] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1014 >> [2015-06-02 10:24:00.028556] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1011 >> [2015-06-02 10:24:02.309238] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:1000 >> [2015-06-02 10:24:02.915960] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1016 >> [2015-06-02 10:24:02.952056] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1013 >> [2015-06-02 10:24:03.040358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1010 >> [2015-06-02 10:24:05.317047] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:996 >> [2015-06-02 10:24:05.923400] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1015 >> [2015-06-02 10:24:05.959144] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1012 >> [2015-06-02 10:24:06.044976] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1009 >> [2015-06-02 10:24:08.324821] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:994 >> [2015-06-02 10:24:08.932287] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1014 >> [2015-06-02 10:24:08.967433] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1011 >> [2015-06-02 10:24:09.052451] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1006 >> [2015-06-02 10:24:11.332558] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:993 >> [2015-06-02 10:24:11.939689] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1013 >> [2015-06-02 10:24:11.975001] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1010 >> [2015-06-02 10:24:12.060625] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1001 >> [2015-06-02 10:24:14.339808] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.213:992 >> [2015-06-02 10:24:14.947317] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.210:1012 >> [2015-06-02 10:24:14.983732] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.211:1007 >> [2015-06-02 10:24:15.068444] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands >> hake request from unknown peer 10.8.230.212:1000 >> [root at SVR6994HW2285 glusterfs]# >> >> >> >> >> Best Regards >> Yuyang Yang >> >> >> -----????----- >> ???: Atin Mukherjee [mailto:amukherj at redhat.com] >> ????: Tuesday, June 02, 2015 5:49 PM >> ???: vyyy???; Gluster-users at gluster.org >> ??: Re: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to start >> >> >> >> On 06/02/2015 12:23 PM, vyyy??? wrote: >>> Attached glusterd log files for 10 & 11 >> [2015-06-02 06:33:42.668268] E >> [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: >> Rejecting management handshake request from unknown peer 10.8.230.212:1002 >> >> From the above log it looks like node 11 tried to handshake with 12, but >> how come 12 is part of the cluster? Could you run gluster peer status in >> 12 and share glusterd log? I am still under the same opinion which I had >> earlier - you are trying to add node which is already a member of >> another cluster. >>> >>> Best Regards >>> Yuyang Yang >>> >>> -----????----- >>> ???: Atin Mukherjee [mailto:amukherj at redhat.com] >>> ????: Tuesday, June 02, 2015 2:42 PM >>> ???: vyyy???; Gluster-users at gluster.org >>> ??: Re: ??: ??: [Gluster-users] Gluster peer rejected and failed to start >>> >>> >>> >>> On 06/02/2015 12:04 PM, vyyy??? wrote: >>>> Glusterfs05~glusterfs10 are clustered for 2 years, recently upgrade to 3.6.3 >>>> Glusterfs11~glusterfs14 are new nodes need to join the cluster >>>> >>>> On glusterfs09: >>>> >>>> [root at SH02SVR5952 ~]# gluster peer status >>>> Number of Peers: 6 >>>> >>>> Hostname: glusterfs06.sh2.ctripcorp.com >>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >>>> State: Peer in Cluster (Connected) >>>> >>>> Hostname: glusterfs07.sh2.ctripcorp.com >>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >>>> State: Peer in Cluster (Connected) >>>> >>>> Hostname: glusterfs08.sh2.ctripcorp.com >>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >>>> State: Peer in Cluster (Connected) >>>> >>>> Hostname: 10.8.230.209 >>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >>>> State: Peer in Cluster (Connected) >>>> >>>> Hostname: glusterfs10.sh2.ctripcorp.com >>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >>>> State: Peer in Cluster (Disconnected) >>>> >>>> Hostname: glusterfs11.sh2.ctripcorp.com >>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 >>>> State: Peer Rejected (Connected) >>>> [root at SH02SVR5952 ~]# >>> Can you attach glusterd log files for 10 & 11? >>>> >>>> [root at SH02SVR5952 ~]# gluster volume status >>>> Status of volume: JQStore2 >>>> Gluster process Port Online Pid >>>> ------------------------------------------------------------------------------ >>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2782 >>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2744 >>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 5307 >>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 3986 >>>> NFS Server on localhost 2049 Y 51697 >>>> Self-heal Daemon on localhost N/A Y 51710 >>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 >>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 >>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 >>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 >>>> NFS Server on 10.8.230.209 2049 Y 4091 >>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104 >>>> >>>> Task Status of Volume JQStore2 >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> Status of volume: Webresource >>>> Gluster process Port Online Pid >>>> ------------------------------------------------------------------------------ >>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2787 >>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2753 >>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 5313 >>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 3992 >>>> NFS Server on localhost 2049 Y 51697 >>>> Self-heal Daemon on localhost N/A Y 51710 >>>> NFS Server on 10.8.230.209 2049 Y 4091 >>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104 >>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 >>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 >>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 >>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 >>>> >>>> Task Status of Volume Webresource >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> Status of volume: ccim >>>> Gluster process Port Online Pid >>>> ------------------------------------------------------------------------------ >>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2793 >>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2745 >>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 5320 >>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 3999 >>>> NFS Server on localhost 2049 Y 51697 >>>> Self-heal Daemon on localhost N/A Y 51710 >>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 >>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 >>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 >>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 >>>> NFS Server on 10.8.230.209 2049 Y 4091 >>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104 >>>> >>>> Task Status of Volume ccim >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> Status of volume: cloudimage >>>> Gluster process Port Online Pid >>>> ------------------------------------------------------------------------------ >>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 5325 >>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 4005 >>>> NFS Server on localhost 2049 Y 51697 >>>> Self-heal Daemon on localhost N/A Y 51710 >>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 >>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 >>>> NFS Server on 10.8.230.209 2049 Y 4091 >>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104 >>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 >>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 >>>> >>>> Task Status of Volume cloudimage >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> Status of volume: filews >>>> Gluster process Port Online Pid >>>> ------------------------------------------------------------------------------ >>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 5330 >>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 4010 >>>> NFS Server on localhost 2049 Y 51697 >>>> Self-heal Daemon on localhost N/A Y 51710 >>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 >>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 >>>> NFS Server on 10.8.230.209 2049 Y 4091 >>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104 >>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 >>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 >>>> >>>> Task Status of Volume filews >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> Status of volume: filex >>>> Gluster process Port Online Pid >>>> ------------------------------------------------------------------------------ >>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2794 >>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2755 >>>> NFS Server on localhost 2049 Y 51697 >>>> Self-heal Daemon on localhost N/A Y 51710 >>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 >>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 >>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 >>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 >>>> NFS Server on 10.8.230.209 2049 Y 4091 >>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104 >>>> >>>> Task Status of Volume filex >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> Status of volume: pubmail >>>> Gluster process Port Online Pid >>>> ------------------------------------------------------------------------------ >>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2792 >>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2754 >>>> NFS Server on localhost 2049 Y 51697 >>>> Self-heal Daemon on localhost N/A Y 51710 >>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 >>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 >>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 >>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 >>>> NFS Server on 10.8.230.209 2049 Y 4091 >>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104 >>>> >>>> Task Status of Volume pubmail >>>> ------------------------------------------------------------------------------ >>>> There are no active volume tasks >>>> >>>> [root at SH02SVR5952 ~]# >>>> >>>> >>>> >>>> Best Regards >>>> ??? >>>> Site Operations Center >>>> Ext.15554 Seat.1#3F049 >>>> We are the F.I.R.S.T. >>>> Fast, Innovation, Reliability , Secure, Team >>>> >>>> >>>> >>>> -----????----- >>>> ???: Atin Mukherjee [mailto:amukherj at redhat.com] >>>> ????: Tuesday, June 02, 2015 2:20 PM >>>> ???: vyyy???; Gluster-users at gluster.org >>>> ??: Re: ??: [Gluster-users] Gluster peer rejected and failed to start >>>> >>>> >>>> >>>> On 06/02/2015 11:33 AM, vyyy??? wrote: >>>>> Actually I have 2 problems >>>>> 1?New nodes can't add to the clusters >>>>> I cleaned /var/lib/glusterd, now status is >>>>> State: Accepted peer request (Connected) >>>>> >>>>> 2?One of clusterd nodes shown 'Peer rejected' and glusterd failed to start. >>>>> The log is attached in pre-mail >>>>> This is a product cluster, this problem is more egent >>>> From the log I can clearly see the problematic node is >>>> glusterfs09.sh2.ctripcorp.com. Your existing cluster configuration has >>>> bricks hosted in glusterfs09.sh2.ctripcorp.com however the same is not >>>> part of the cluster. Could you paste the output of gluster peer status >>>> and gluster volume status ? >>>>> >>>>> >>>>> >>>>> Best Regards >>>>> Yuyang Yang >>>>> >>>>> -----????----- >>>>> ???: Atin Mukherjee [mailto:amukherj at redhat.com] >>>>> ????: Tuesday, June 02, 2015 12:52 PM >>>>> ???: vyyy???; Gluster-users at gluster.org >>>>> ??: Re: [Gluster-users] Gluster peer rejected and failed to start >>>>> >>>>> >>>>> >>>>> On 06/02/2015 10:00 AM, vyyy??? wrote: >>>>>> Hi >>>>>> >>>>>> We have a gluster (Version 3.6.3) cluster with 6 nodes, I tried to add 4 more nodes, but ?Peer Rejected?, then I tried to resolve it by dump /var/lib/glusterd and probe again, not success, this is a question, But strange thing is: >>>>>> >>>>>> A node already in cluster also shown ?Peer Reject? >>>>>> >>>>>> I tried to restart glusterd, It failed >>>>>> >>>>>> I found that /var/lib/glusterd/peers is empty, I copied the files from other nodes, still can?t start glusterd >>>>> It seems like you are trying to peer probe nodes which are either either >>>>> part of some other clusters (uncleaned nodes). Could you check whether >>>>> the nodes which you are adding have empty /var/lib/glusterd? If not >>>>> clean them and retry. >>>>> >>>>> ~Atin >>>>>> >>>>>> >>>>>> >>>>>> etc-glusterfs-glusterd.vol.log shown that cluster member as ?unknown peer ? >>>>>> >>>>>> [2015-06-02 01:52:14.650635] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >>>>>> [2015-06-02 01:52:14.650786] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 674a78b5-0590-48d4-8752-d4608832ed1d >>>>>> [2015-06-02 01:52:14.657881] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 83e1a9db-3134-45e4-acd2-387b12b5b207 >>>>>> [2015-06-02 01:52:17.747865] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 doesn't belong to the cluster. Ignoring request. >>>>>> [2015-06-02 01:52:17.747908] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully >>>>>> [2015-06-02 01:52:40.338885] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request. >>>>>> [2015-06-02 01:52:40.338929] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully >>>>>> [2015-06-02 01:52:41.310451] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request. >>>>>> [2015-06-02 01:52:41.310486] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully >>>>>> >>>>>> >>>>>> >>>>>> Debug info is as following, >>>>>> >>>>>> >>>>>> >>>>>> /usr/sbin/glusterd >>>>>> [root at SH02SVR5951 peers]# /usr/sbin/glusterd --debug >>>>>> [2015-06-02 04:09:24.626690] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.3 (args: /usr/sbin/glusterd --debug) >>>>>> [2015-06-02 04:09:24.626739] D [logging.c:1763:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 >>>>>> [2015-06-02 04:09:24.627052] D [MSGID: 0] [glusterfsd.c:613:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol >>>>>> [2015-06-02 04:09:24.629683] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 >>>>>> [2015-06-02 04:09:24.629706] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory >>>>>> [2015-06-02 04:09:24.629764] D [glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128 >>>>>> [2015-06-02 04:09:24.629895] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited. >>>>>> [2015-06-02 04:09:24.629904] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 >>>>>> [2015-06-02 04:09:24.629930] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so >>>>>> [2015-06-02 04:09:24.631989] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled >>>>>> [2015-06-02 04:09:24.632005] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled >>>>>> [2015-06-02 04:09:24.632013] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread >>>>>> [2015-06-02 04:09:24.632024] D [name.c:550:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet >>>>>> [2015-06-02 04:09:24.632072] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so >>>>>> [2015-06-02 04:09:24.632102] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so: cannot open shared object file: No such file or directory >>>>>> [2015-06-02 04:09:24.632112] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine >>>>>> [2015-06-02 04:09:24.632122] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed >>>>>> [2015-06-02 04:09:24.632132] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 >>>>>> [2015-06-02 04:09:24.632140] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 >>>>>> [2015-06-02 04:09:24.632148] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 >>>>>> [2015-06-02 04:09:24.632155] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 >>>>>> [2015-06-02 04:09:24.632163] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 >>>>>> [2015-06-02 04:09:24.632170] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 >>>>>> [2015-06-02 04:09:24.632177] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 >>>>>> [2015-06-02 04:09:24.632212] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited. >>>>>> [2015-06-02 04:09:24.632220] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 >>>>>> [2015-06-02 04:09:24.632236] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so >>>>>> [2015-06-02 04:09:24.632302] D [socket.c:3692:socket_init] 0-socket.management: disabling nodelay >>>>>> [2015-06-02 04:09:24.632314] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled >>>>>> [2015-06-02 04:09:24.632321] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled >>>>>> [2015-06-02 04:09:24.632329] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread >>>>>> [2015-06-02 04:09:24.632402] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 >>>>>> [2015-06-02 04:09:24.632414] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 >>>>>> [2015-06-02 04:09:24.632443] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 >>>>>> [2015-06-02 04:09:24.632455] D [glusterd.c:1484:init] 0-management: cannot get run-with-valgrind value >>>>>> [2015-06-02 04:09:24.633086] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system >>>>>> [2015-06-02 04:09:24.633518] D [glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning -1 >>>>>> [2015-06-02 04:09:24.634031] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.634057] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.634115] D [store.c:304:gf_store_retrieve_value] 0-: key operating-version found >>>>>> [2015-06-02 04:09:24.634141] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 >>>>>> [2015-06-02 04:09:24.634196] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.634436] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.634446] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.634469] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634488] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634496] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 >>>>>> [2015-06-02 04:09:24.634510] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634517] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 >>>>>> [2015-06-02 04:09:24.634528] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634534] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 >>>>>> [2015-06-02 04:09:24.634544] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634551] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 >>>>>> [2015-06-02 04:09:24.634562] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634569] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 >>>>>> [2015-06-02 04:09:24.634580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 >>>>>> [2015-06-02 04:09:24.634597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634603] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 24 >>>>>> [2015-06-02 04:09:24.634613] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634620] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 >>>>>> [2015-06-02 04:09:24.634631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 7bf7d81a-f835-4c3b-9c58-4d4ad7ec6e07 >>>>>> [2015-06-02 04:09:24.634652] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634659] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c7374917-2344-4c42-aeae-1d0bdb7ece0a >>>>>> [2015-06-02 04:09:24.634678] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634689] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 703c3b9d-4343-4546-9ca3-350aecdd52fe >>>>>> [2015-06-02 04:09:24.634700] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634707] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 >>>>>> [2015-06-02 04:09:24.634719] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634726] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 >>>>>> [2015-06-02 04:09:24.634736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634742] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on >>>>>> [2015-06-02 04:09:24.634773] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on >>>>>> [2015-06-02 04:09:24.634785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634791] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on >>>>>> [2015-06-02 04:09:24.634803] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on >>>>>> [2015-06-02 04:09:24.634814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634821] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.* >>>>>> [2015-06-02 04:09:24.634838] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.* >>>>>> [2015-06-02 04:09:24.634849] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634856] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick3 >>>>>> [2015-06-02 04:09:24.634868] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634874] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick3 >>>>>> [2015-06-02 04:09:24.634885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634891] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick3 >>>>>> [2015-06-02 04:09:24.634902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634908] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick3 >>>>>> [2015-06-02 04:09:24.634919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634925] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick3 >>>>>> [2015-06-02 04:09:24.634935] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.634942] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick3 >>>>>> [2015-06-02 04:09:24.634954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.634984] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635002] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.635019] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635039] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635048] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635056] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635074] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635083] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635092] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635101] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635110] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635118] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635127] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635136] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635144] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635153] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635161] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635368] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.635380] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.635396] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635416] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635426] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635436] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635447] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635456] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.635488] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.635501] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635748] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.635766] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.635786] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635828] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635838] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635847] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.635856] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.635909] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.635922] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636158] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.636167] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.636180] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636222] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636250] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.636274] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.636286] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636471] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.636479] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.636493] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636524] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636533] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636542] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.636586] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.636598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636856] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.636889] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.636922] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636943] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636964] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636975] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.636994] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.637020] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.637036] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.637307] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.637325] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.637346] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.637367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.637378] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.637388] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.819347] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.819374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.819385] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.819413] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.819707] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.820187] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.820222] D [store.c:304:gf_store_retrieve_value] 0-: key version found >>>>>> [2015-06-02 04:09:24.820239] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.820532] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.820585] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.820959] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.821064] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.821090] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.837290] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.837540] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.837551] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.837568] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.837589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.837600] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.837862] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.837880] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.837896] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.837918] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.837929] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.837939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.837964] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.838216] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.838257] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.838273] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838303] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 >>>>>> [2015-06-02 04:09:24.838314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838320] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 >>>>>> [2015-06-02 04:09:24.838330] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838336] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 >>>>>> [2015-06-02 04:09:24.838346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838352] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 >>>>>> [2015-06-02 04:09:24.838361] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838367] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 >>>>>> [2015-06-02 04:09:24.838377] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838383] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 >>>>>> [2015-06-02 04:09:24.838393] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838399] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 6 >>>>>> [2015-06-02 04:09:24.838408] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838414] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 >>>>>> [2015-06-02 04:09:24.838423] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838430] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = cdc279f4-6cd0-4087-8d2a-440bccb51e33 >>>>>> [2015-06-02 04:09:24.838443] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838449] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 0b6d3496-0865-45e4-a45b-c221b22d3d92 >>>>>> [2015-06-02 04:09:24.838459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838465] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = ecedfb4d-3308-4f6c-b188-4d5b868958e1 >>>>>> [2015-06-02 04:09:24.838475] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 >>>>>> [2015-06-02 04:09:24.838493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838499] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 >>>>>> [2015-06-02 04:09:24.838508] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838515] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.68.* >>>>>> [2015-06-02 04:09:24.838533] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.68.* >>>>>> [2015-06-02 04:09:24.838545] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838552] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick2 >>>>>> [2015-06-02 04:09:24.838563] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838570] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick2 >>>>>> [2015-06-02 04:09:24.838580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick2 >>>>>> [2015-06-02 04:09:24.838597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick2 >>>>>> [2015-06-02 04:09:24.838614] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838621] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick2 >>>>>> [2015-06-02 04:09:24.838631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838637] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick2 >>>>>> [2015-06-02 04:09:24.838649] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.838697] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838719] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.838738] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838747] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838756] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838764] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838781] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838797] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838806] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838815] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838840] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.838857] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839051] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.839063] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.839084] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839102] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839113] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839123] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839142] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839151] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.839177] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.839189] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839382] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.839390] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.839403] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839421] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839431] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839468] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.839491] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.839502] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839700] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.839712] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.839732] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839750] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.839799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.839822] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.839834] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840057] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.840075] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.840096] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840117] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840128] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840137] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840156] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840166] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.840192] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.840206] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840402] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.840410] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.840424] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840469] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840478] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.840513] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.840525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840799] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.840818] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.840839] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840859] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840871] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840880] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840900] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.840910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.840931] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.841118] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.841359] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.841396] D [store.c:304:gf_store_retrieve_value] 0-: key version found >>>>>> [2015-06-02 04:09:24.841413] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.841557] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.841611] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.841799] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.841875] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.841901] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.842350] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.842469] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.842477] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.842489] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.842636] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.842645] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.842656] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842696] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842708] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842718] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.842738] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.842848] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.842859] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.842872] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842898] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 >>>>>> [2015-06-02 04:09:24.842908] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842914] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4 >>>>>> [2015-06-02 04:09:24.842924] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842930] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 >>>>>> [2015-06-02 04:09:24.842939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842945] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 >>>>>> [2015-06-02 04:09:24.842954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842960] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 >>>>>> [2015-06-02 04:09:24.842969] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842975] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 >>>>>> [2015-06-02 04:09:24.842984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.842990] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3 >>>>>> [2015-06-02 04:09:24.842999] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843005] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 >>>>>> [2015-06-02 04:09:24.843015] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843021] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = a3369f63-9fce-4a56-a921-f7206cb8d3ae >>>>>> [2015-06-02 04:09:24.843033] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843039] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c6012954-928a-43c1-b7ea-336d6ee0bea9 >>>>>> [2015-06-02 04:09:24.843050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843056] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = bfbd7007-0e0c-45f9-8d59-05e1aee66728 >>>>>> [2015-06-02 04:09:24.843066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843072] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 >>>>>> [2015-06-02 04:09:24.843082] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843089] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 >>>>>> [2015-06-02 04:09:24.843098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843104] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.83.* >>>>>> [2015-06-02 04:09:24.843119] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.83.* >>>>>> [2015-06-02 04:09:24.843130] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843136] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick4 >>>>>> [2015-06-02 04:09:24.843147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843154] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick4 >>>>>> [2015-06-02 04:09:24.843164] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843171] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick4 >>>>>> [2015-06-02 04:09:24.843181] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843188] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick4 >>>>>> [2015-06-02 04:09:24.843198] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.843221] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843238] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.843255] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843264] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843273] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843281] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843290] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843298] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843306] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843323] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843332] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843340] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843357] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843365] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843476] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.843484] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.843499] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843519] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843720] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843739] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843751] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843761] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843771] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.843802] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.843817] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.843984] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.843996] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.844013] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844041] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844060] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844069] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844078] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.844104] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.844116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844222] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.844230] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.844243] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844266] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844276] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844285] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844294] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844303] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844312] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.844334] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.844346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844443] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.844451] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.844468] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844497] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844516] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844534] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.844550] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.844753] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.845000] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.845033] D [store.c:304:gf_store_retrieve_value] 0-: key version found >>>>>> [2015-06-02 04:09:24.845049] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.845199] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.845244] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.845396] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.845448] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.845471] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.846053] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.846195] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.846206] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.846220] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846252] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.846362] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.846371] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.846382] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846404] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846424] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.846442] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.846543] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.846552] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.846562] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846585] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846592] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 >>>>>> [2015-06-02 04:09:24.846603] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846609] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 >>>>>> [2015-06-02 04:09:24.846618] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846624] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 >>>>>> [2015-06-02 04:09:24.846633] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846639] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 >>>>>> [2015-06-02 04:09:24.846648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 >>>>>> [2015-06-02 04:09:24.846663] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846690] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 >>>>>> [2015-06-02 04:09:24.846702] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846708] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3 >>>>>> [2015-06-02 04:09:24.846717] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846723] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 >>>>>> [2015-06-02 04:09:24.846733] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846739] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 8869b21b-1bf1-4eba-a52a-1158abde7d23 >>>>>> [2015-06-02 04:09:24.846752] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846758] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 27298b95-919d-43f7-9507-8cbfbe5292e3 >>>>>> [2015-06-02 04:09:24.846769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846775] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 47f52166-020b-4bef-ada1-7e65fcd2834f >>>>>> [2015-06-02 04:09:24.846785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846792] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 >>>>>> [2015-06-02 04:09:24.846802] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846808] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 >>>>>> [2015-06-02 04:09:24.846818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846824] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on >>>>>> [2015-06-02 04:09:24.846837] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on >>>>>> [2015-06-02 04:09:24.846848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846854] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on >>>>>> [2015-06-02 04:09:24.846864] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on >>>>>> [2015-06-02 04:09:24.846875] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846881] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick >>>>>> [2015-06-02 04:09:24.846892] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846899] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick >>>>>> [2015-06-02 04:09:24.846910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846916] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick >>>>>> [2015-06-02 04:09:24.846927] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846934] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick >>>>>> [2015-06-02 04:09:24.846944] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846952] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick >>>>>> [2015-06-02 04:09:24.846962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.846969] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick >>>>>> [2015-06-02 04:09:24.846981] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.847003] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847027] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.847045] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847055] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847064] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847072] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847081] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847089] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847106] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847115] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847124] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847150] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847159] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847167] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847176] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847286] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.847294] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.847308] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847328] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847339] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847358] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847376] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.847402] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.847414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847516] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.847523] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.847537] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847559] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847569] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847579] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847588] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847607] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.847628] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.847640] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847747] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.847756] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.847771] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847794] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847804] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847841] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.847862] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.847873] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.847969] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.847977] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.847990] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848012] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848022] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848031] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848040] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848049] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848058] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.848080] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.848091] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848187] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.848195] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.848208] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848230] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848240] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848249] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848259] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848267] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848277] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.848299] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.848310] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848411] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.848419] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.848432] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848455] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848474] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848484] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848503] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.848519] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.848727] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.848975] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.849002] D [store.c:304:gf_store_retrieve_value] 0-: key version found >>>>>> [2015-06-02 04:09:24.849019] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.849167] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.849212] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.849367] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.849419] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.849442] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.849880] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.850003] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.850012] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.850024] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850043] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.850157] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.850166] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.850177] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850192] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850211] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.850228] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.850402] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.850414] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.850426] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850444] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850451] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 >>>>>> [2015-06-02 04:09:24.850461] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850467] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4 >>>>>> [2015-06-02 04:09:24.850476] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 >>>>>> [2015-06-02 04:09:24.850491] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850497] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 >>>>>> [2015-06-02 04:09:24.850506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850512] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 >>>>>> [2015-06-02 04:09:24.850522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850528] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 >>>>>> [2015-06-02 04:09:24.850537] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850543] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = disperse_count value = 0 >>>>>> [2015-06-02 04:09:24.850552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850558] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = redundancy_count value = 0 >>>>>> [2015-06-02 04:09:24.850567] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850573] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 11 >>>>>> [2015-06-02 04:09:24.850582] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850588] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 >>>>>> [2015-06-02 04:09:24.850598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = ea46609c-be9a-4283-8357-8fdc2e5a190e >>>>>> [2015-06-02 04:09:24.850616] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850622] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 21b247ce-58ce-4087-8c32-3d578b4cc22d >>>>>> [2015-06-02 04:09:24.850632] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 0724171a-c7ce-4528-b54b-2d983f969537 >>>>>> [2015-06-02 04:09:24.850648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 >>>>>> [2015-06-02 04:09:24.850664] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850692] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 >>>>>> [2015-06-02 04:09:24.850703] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850709] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.92.*,10.8.230.*,192.168.98.* >>>>>> [2015-06-02 04:09:24.850724] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.92.*,10.8.230.*,192.168.98.* >>>>>> [2015-06-02 04:09:24.850736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850743] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick1 >>>>>> [2015-06-02 04:09:24.850754] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850760] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick1 >>>>>> [2015-06-02 04:09:24.850772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850778] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick1 >>>>>> [2015-06-02 04:09:24.850789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850795] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick1 >>>>>> [2015-06-02 04:09:24.850807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.850830] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.850867] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850877] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850894] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850911] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850928] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850936] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850945] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850953] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850971] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850980] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850988] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.850996] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851005] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851164] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.851173] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.851187] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851203] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851223] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851242] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851251] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851261] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.851283] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.851295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851457] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.851465] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.851478] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851504] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851513] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851531] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851540] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851549] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.851571] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.851583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851732] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.851741] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.851755] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851816] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851826] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.851850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.851863] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.851959] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.851974] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.852005] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.852026] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.852037] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.852047] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.852061] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.852070] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.852080] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.852090] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.852111] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.852298] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.852524] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.852556] D [store.c:304:gf_store_retrieve_value] 0-: key version found >>>>>> [2015-06-02 04:09:24.852576] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.852723] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.852776] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 >>>>>> [2015-06-02 04:09:24.852930] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.852982] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.853006] D [store.c:372:gf_store_save_value] 0-: returning: 0 >>>>>> [2015-06-02 04:09:24.853412] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.853524] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.853539] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.853553] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.853573] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.853583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.853699] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.853718] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.853735] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.853757] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.853768] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.853778] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.853787] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.853798] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.853820] D [glusterd-store.c:3094:glusterd_store_retrieve_volumes] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.853962] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.853972] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.853983] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854000] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854011] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 >>>>>> [2015-06-02 04:09:24.854023] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854032] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854043] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.854054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.854070] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >>>>>> [2015-06-02 04:09:24.854162] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.854171] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.854181] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854197] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854205] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 >>>>>> [2015-06-02 04:09:24.854216] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854225] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854232] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.854241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.854255] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >>>>>> [2015-06-02 04:09:24.854345] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.854354] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.854364] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854380] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854390] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 >>>>>> [2015-06-02 04:09:24.854402] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854411] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854417] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.854427] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.854439] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >>>>>> [2015-06-02 04:09:24.854527] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.854536] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.854546] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854569] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 >>>>>> [2015-06-02 04:09:24.854580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854596] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.854605] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.854617] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >>>>>> [2015-06-02 04:09:24.854704] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.854713] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.854725] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854740] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854749] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 >>>>>> [2015-06-02 04:09:24.854760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.854776] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.854789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.854803] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >>>>>> [2015-06-02 04:09:24.855064] D [store.c:418:gf_store_handle_new] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.855072] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 >>>>>> [2015-06-02 04:09:24.855083] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.855098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.855106] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 >>>>>> [2015-06-02 04:09:24.855116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.855125] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.855132] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.855141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 >>>>>> [2015-06-02 04:09:24.855154] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >>>>>> [2015-06-02 04:09:24.855186] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.855210] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >>>>>> [2015-06-02 04:09:24.855218] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 >>>>>> [2015-06-02 04:09:24.855231] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so >>>>>> [2015-06-02 04:09:24.855324] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled >>>>>> [2015-06-02 04:09:24.855333] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled >>>>>> [2015-06-02 04:09:24.855340] D [socket.c:3827:socket_init] 0-management: using system polling thread >>>>>> [2015-06-02 04:09:24.859610] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.211 (port-24007) for hostname: glusterfs07.sh2.ctripcorp.com and port: 24007 >>>>>> [2015-06-02 04:09:24.859628] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection >>>>>> [2015-06-02 04:09:24.859707] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.859733] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.859772] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >>>>>> [2015-06-02 04:09:24.859781] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 >>>>>> [2015-06-02 04:09:24.859794] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so >>>>>> [2015-06-02 04:09:24.859905] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled >>>>>> [2015-06-02 04:09:24.859916] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled >>>>>> [2015-06-02 04:09:24.859923] D [socket.c:3827:socket_init] 0-management: using system polling thread >>>>>> [2015-06-02 04:09:24.863505] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.231.160 (port-24007) for hostname: glusterfs11.sh2.ctripcorp.com and port: 24007 >>>>>> [2015-06-02 04:09:24.863521] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection >>>>>> [2015-06-02 04:09:24.863562] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.863585] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.863607] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >>>>>> [2015-06-02 04:09:24.863615] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 >>>>>> [2015-06-02 04:09:24.863625] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so >>>>>> [2015-06-02 04:09:24.863754] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled >>>>>> [2015-06-02 04:09:24.863765] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled >>>>>> [2015-06-02 04:09:24.863772] D [socket.c:3827:socket_init] 0-management: using system polling thread >>>>>> [2015-06-02 04:09:24.867255] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.210 (port-24007) for hostname: glusterfs06.sh2.ctripcorp.com and port: 24007 >>>>>> [2015-06-02 04:09:24.867271] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection >>>>>> [2015-06-02 04:09:24.867314] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.867337] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.867358] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >>>>>> [2015-06-02 04:09:24.867366] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 >>>>>> [2015-06-02 04:09:24.867377] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so >>>>>> [2015-06-02 04:09:24.867463] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled >>>>>> [2015-06-02 04:09:24.867476] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled >>>>>> [2015-06-02 04:09:24.867483] D [socket.c:3827:socket_init] 0-management: using system polling thread >>>>>> [2015-06-02 04:09:24.870920] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.214 (port-24007) for hostname: glusterfs10.sh2.ctripcorp.com and port: 24007 >>>>>> [2015-06-02 04:09:24.870935] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection >>>>>> [2015-06-02 04:09:24.870997] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.871021] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.871042] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >>>>>> [2015-06-02 04:09:24.871050] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 >>>>>> [2015-06-02 04:09:24.871061] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so >>>>>> [2015-06-02 04:09:24.871129] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled >>>>>> [2015-06-02 04:09:24.871138] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled >>>>>> [2015-06-02 04:09:24.871144] D [socket.c:3827:socket_init] 0-management: using system polling thread >>>>>> [2015-06-02 04:09:24.874102] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.209 (port-24007) for hostname: 10.8.230.209 and port: 24007 >>>>>> [2015-06-02 04:09:24.874116] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection >>>>>> [2015-06-02 04:09:24.874159] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.874181] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 >>>>>> [2015-06-02 04:09:24.874202] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >>>>>> [2015-06-02 04:09:24.874210] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 >>>>>> [2015-06-02 04:09:24.874219] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so >>>>>> [2015-06-02 04:09:24.874292] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled >>>>>> [2015-06-02 04:09:24.874301] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled >>>>>> [2015-06-02 04:09:24.874326] D [socket.c:3827:socket_init] 0-management: using system polling thread >>>>>> [2015-06-02 04:09:24.877759] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.212 (port-24007) for hostname: glusterfs08.sh2.ctripcorp.com and port: 24007 >>>>>> [2015-06-02 04:09:24.877774] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection >>>>>> [2015-06-02 04:09:24.877818] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.877831] D [glusterd-store.c:4106:glusterd_store_retrieve_peers] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.877863] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. >>>>>> [2015-06-02 04:09:24.877873] D [glusterd-store.c:3612:glusterd_store_retrieve_snaps] 0-: Returning with 0 >>>>>> [2015-06-02 04:09:24.881125] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.881140] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.881148] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs06.sh2.ctripcorp.com found.. state: 3 >>>>>> [2015-06-02 04:09:24.881156] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.881162] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.881169] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs07.sh2.ctripcorp.com found.. state: 3 >>>>>> [2015-06-02 04:09:24.881176] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.881182] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.881188] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs08.sh2.ctripcorp.com found.. state: 3 >>>>>> [2015-06-02 04:09:24.881195] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 >>>>>> [2015-06-02 04:09:24.881201] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 >>>>>> [2015-06-02 04:09:24.890739] D [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: glusterfs09.sh2.ctripcorp.com >>>>>> [2015-06-02 04:09:24.891607] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 >>>>>> [2015-06-02 04:09:24.891750] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 >>>>>> [2015-06-02 04:09:24.891820] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 >>>>>> [2015-06-02 04:09:24.891881] D [common-utils.c:2913:gf_is_local_addr] 0-management: glusterfs09.sh2.ctripcorp.com is not local >>>>>> [2015-06-02 04:09:24.891890] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning -1 >>>>>> [2015-06-02 04:09:24.891898] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning -1 >>>>>> [2015-06-02 04:09:24.891904] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore >>>>>> [2015-06-02 04:09:24.891911] D [glusterd-store.c:4321:glusterd_restore] 0-: Returning -1 >>>>>> [2015-06-02 04:09:24.891923] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again >>>>>> [2015-06-02 04:09:24.891933] E [graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed >>>>>> [2015-06-02 04:09:24.891941] E [graph.c:525:glusterfs_graph_activate] 0-graph: init failed >>>>>> [2015-06-02 04:09:24.892164] D [logging.c:1740:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 2 extra log messages >>>>>> [2015-06-02 04:09:24.892178] D [logging.c:1743:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 2 extra log messages >>>>>> [2015-06-02 04:09:24.892325] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received signum (0), shutting down >>>>>> [2015-06-02 04:09:24.892345] D [glusterfsd-mgmt.c:2244:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given >>>>>> [root at SH02SVR5951 peers]# >>>>>> >>>>>> >>>>>> >>>>>> Best Regards >>>>>> Yuyang Yang >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>> >>>> >>> >> >-- ~Atin
Joe Julian
2015-Jun-05 02:44 UTC
[Gluster-users] 答复: 答复: 答复: Gluster peer rejected and failed to start
You cannot join a trusted pool, you must be invited. In other words, a server inside the trusted pool must probe the new server, and that new server must not be part of another pool. On June 4, 2015 7:24:28 PM PDT, "vyyy???" <yuyangyang at Ctrip.com> wrote:>Can anyone help me? > >New nodes still can't join the cluster. >The new nodes can peer each other, just not exist cluster nodes. >I can not find any network problem, there were no firewall, port 24007 >is connected, there were all at the same subnet... > >Thanks > > > >[root at SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log >[2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main] >0-/usr/sbin/glusterd: Started running /usr/sbin/glust >erd version 3.6.3 (args: /usr/sbin/glusterd >--pid-file=/var/run/glusterd.pid) >[2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management: >Maximum allowed open file descriptors set to 65536 >[2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management: >Using /var/lib/glusterd as working directory >[2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load] >0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans >port/rdma.so: cannot open shared object file: No such file or directory >[2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load] >0-rpc-transport: volume 'rdma.management': transport- >type 'rdma' is not valid or not found on this machine >[2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create] >0-rpc-service: cannot create listener, initing the tra >nsport failed >[2015-06-05 01:59:02.343346] I >[glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: >geo-replication module not installed > in the system >[2015-06-05 01:59:02.351861] I >[glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: >retrieved op-version: 2 >[2015-06-05 01:59:02.363138] I >[glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] >0-management: No missed snap >s list. >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option ping-timeout 30 > 8: option transport.socket.read-fail-log off > 9: option transport.socket.keepalive-interval 2 > 10: option transport.socket.keepalive-time 10 > 11: option transport-type rdma > 12: option working-directory /var/lib/glusterd > 13: end-volume > 14: >+------------------------------------------------------------------------------+ >[2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init] >0-management: retrieved UUID: 1b98af34-08f9-492f-b54e-7608 >2588d63b >[2015-06-05 02:00:43.025462] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:00:44.025069] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:00:44.599862] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:00:46.029823] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:00:46.701845] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:01:43.119612] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:01:44.165527] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:01:44.851903] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:01:46.190401] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:01:46.722394] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:01:58.719968] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:02:32.584999] I >[glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: >Received CLI probe req glust >erfs05.sh2.ctripcorp.com 24007 >[2015-06-05 02:02:32.586281] I >[glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to >find peerinfo for host: g >lusterfs05.sh2.ctripcorp.com (24007) >[2015-06-05 02:02:32.602192] I >[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting >frame-timeout to 600 >[2015-06-05 02:02:32.606053] I >[glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect >returned 0 >[2015-06-05 02:02:32.620958] E >[glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] >0-management: failed to get the >'versions' from peer (10.8.230.209:24007) >[2015-06-05 02:02:32.621020] I [MSGID: 106004] >[glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer >000000 >00-0000-0000-0000-000000000000, in Establishing Connection state, has >disconnected from glusterd. >[2015-06-05 02:02:43.264141] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:02:44.312918] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:02:44.781434] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:02:46.176888] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:02:46.799916] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:03:42.720994] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:03:43.771987] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:03:44.659775] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:03:45.956807] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:03:46.688787] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:04:42.891131] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:04:43.957423] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:04:44.995638] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:04:46.062171] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:04:46.957327] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:05:42.966651] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:05:44.008498] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:05:45.049936] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:05:46.087007] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:05:47.029325] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:06:42.874089] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:06:43.910998] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:06:44.932802] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:06:46.053731] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:06:47.055859] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:07:43.099647] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:07:44.146454] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:07:45.218982] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:07:46.246303] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:07:47.106954] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:08:43.226090] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:08:44.292041] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed >[2015-06-05 02:08:45.351830] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:08:45.688349] I >[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] >0-glusterd: Received cli list req >[2015-06-05 02:08:46.651768] I >[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: >Received get vol req >[2015-06-05 02:09:42.748628] W >[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx >modification failed > > > >Best Regards > > >-----????----- >???: vyyy??? >????: Thursday, June 04, 2015 11:00 AM >???: 'Atin Mukherjee'; Gluster-users at gluster.org >??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to >start > >?? >The problem 2 has resolved. > >From glusterd log file, I find Cksums error as following: >[2015-06-04 01:25:03.137798] E [MSGID: 106010] >[glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: >Version of Cksums JQStore2 differ. local cksum = 2617028277, remote >cksum = 2210669223 on peer glusterfs09.sh2.ctripcorp.com > >The solution for this problem was to do the following, on the BAD node: >/etc/init.d/glusterd stop >rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/ >/var/lib/glusterd/vols/ >/etc/init.d/glusterd start > > > >The problem 1 is still under inverstigate: > >New nodes can not join the cluster, log file shown: failed to get the >'versions' from peer >I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are >" operating-version=2" > >?? >??[2015-06-04 02:13:49.199416] E >[glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] >0-management: failed to get the >??'versions' from peer (10.8.230.211:24007) >??[2015-06-04 02:14:07.112577] I >[glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: >Received CLI probe req glust >??erfs05.sh2.ctripcorp.com 24007 >??[2015-06-04 02:14:07.113778] I >[glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to >find peerinfo for host: g >??lusterfs05.sh2.ctripcorp.com (24007) >??[2015-06-04 02:14:07.131023] I >[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting >frame-timeout to 600 >??[2015-06-04 02:14:07.134829] I >[glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect >returned 0 >??[ > > > > >Best Regards >??? >Site Operations Center >Ext.15554? Seat.1#3F049 >We are the F.I.R.S.T. >Fast, Innovation, Reliability , Secure, Team > > >-----????----- >???: vyyy??? >????: Wednesday, June 03, 2015 5:37 PM >???: 'Atin Mukherjee'; Gluster-users at gluster.org >??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to >start > >Hi Atin and everybody, > > I have forced peer detach the news nodes ,The peer status seems even >strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help >me? Any advice is appreciated. > >Glusterfs05 >?? >??[root at sh02svr5956 ~]# gluster peer status >??Number of Peers: 5 >?? >??Hostname: glusterfs06.sh2.ctripcorp.com >??Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >??State: Peer Rejected (Connected) >?? >??Hostname: glusterfs07.sh2.ctripcorp.com >??Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >??State: Peer Rejected (Connected) >?? >??Hostname: glusterfs08.sh2.ctripcorp.com >??Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >??State: Peer Rejected (Connected) >?? >??Hostname: glusterfs10.sh2.ctripcorp.com >??Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >??State: Peer in Cluster (Connected) >?? >??Hostname: glusterfs09.sh2.ctripcorp.com >??Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >??State: Peer Rejected (Connected) >??[root at sh02svr5956 ~]# >?? >?? >??Glusterfs06 >?? >??[root at SH02SVR5955 ~]# gluster peer status >??Number of Peers: 5 >?? >??Hostname: glusterfs10.sh2.ctripcorp.com >??Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >??State: Peer Rejected (Connected) >?? >??Hostname: glusterfs08.sh2.ctripcorp.com >??Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >??State: Peer in Cluster (Connected) >?? >??Hostname: glusterfs07.sh2.ctripcorp.com >??Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >??State: Peer in Cluster (Connected) >?? >??Hostname: 10.8.230.209 >??Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >??State: Peer Rejected (Connected) >?? >??Hostname: glusterfs09.sh2.ctripcorp.com >??Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >??State: Peer in Cluster (Connected) >??[root at SH02SVR5955 ~]# >?? > >Glusterfs07 > >[root at SH02SVR5954 ~]# gluster peer status >Number of Peers: 5 > >Hostname: glusterfs10.sh2.ctripcorp.com >Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >State: Peer Rejected (Connected) > >Hostname: 10.8.230.209 >Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >State: Peer Rejected (Connected) > >Hostname: glusterfs06.sh2.ctripcorp.com >Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >State: Peer in Cluster (Connected) > >Hostname: glusterfs08.sh2.ctripcorp.com >Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >State: Peer in Cluster (Connected) > >Hostname: glusterfs09.sh2.ctripcorp.com >Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >State: Peer in Cluster (Connected) >[root at SH02SVR5954 ~]# > > > >Glusterfs08 > >[root at SH02SVR5953 ~]# gluster peer status >Number of Peers: 5 > >Hostname: glusterfs07.sh2.ctripcorp.com >Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >State: Peer in Cluster (Connected) > >Hostname: glusterfs10.sh2.ctripcorp.com >Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >State: Peer Rejected (Connected) > >Hostname: glusterfs06.sh2.ctripcorp.com >Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >State: Peer in Cluster (Connected) > >Hostname: 10.8.230.209 >Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >State: Peer Rejected (Connected) > >Hostname: glusterfs09.sh2.ctripcorp.com >Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >State: Peer in Cluster (Connected) >[root at SH02SVR5953 ~]# > > >[root at SH02SVR5952 ~]# gluster peer status >Number of Peers: 5 > >Hostname: glusterfs06.sh2.ctripcorp.com >Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >State: Peer in Cluster (Connected) > >Hostname: glusterfs07.sh2.ctripcorp.com >Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >State: Peer in Cluster (Connected) > >Hostname: glusterfs08.sh2.ctripcorp.com >Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >State: Peer in Cluster (Connected) > >Hostname: 10.8.230.209 >Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >State: Peer Rejected (Connected) > >Hostname: glusterfs10.sh2.ctripcorp.com >Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >State: Peer Rejected (Connected) >[root at SH02SVR5952 ~]# > > >Glusterfs10 > >[root at SH02SVR5951 ~]# gluster peer status >Number of Peers: 6 > >Hostname: glusterfs07.sh2.ctripcorp.com >Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >State: Peer Rejected (Connected) > >Hostname: glusterfs11.sh2.ctripcorp.com >Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 >State: Peer Rejected (Connected) > >Hostname: glusterfs09.sh2.ctripcorp.com >Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >State: Peer Rejected (Connected) > >Hostname: glusterfs06.sh2.ctripcorp.com >Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >State: Peer Rejected (Connected) > >Hostname: 10.8.230.209 >Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >State: Peer in Cluster (Connected) > >Hostname: glusterfs08.sh2.ctripcorp.com >Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >State: Peer Rejected (Connected) >[root at SH02SVR5951 ~]# > > > > > > > >Best Regards >??? >Site Operations Center >Ext.15554? Seat.1#3F049 >We are the F.I.R.S.T. >Fast, Innovation, Reliability , Secure, Team > > > >-----????----- >???: vyyy??? >????: Wednesday, June 03, 2015 2:55 PM >???: 'Atin Mukherjee' >??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to >start > >Can I delete the files in /var/lib/glusterd/peers which state is not =3 >on all nodes, then restart glusterd? > > >[root at sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15 >uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15 >state=6 >hostname1=glusterfs12.sh2.ctripcorp.com > > > > >Best Regards >Yang > > >-----????----- >???: vyyy??? >????: Wednesday, June 03, 2015 2:39 PM >???: 'Atin Mukherjee' >??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed to >start > >Since it's intranet, You can only control my desktop, and then access >these node. >Should I apply a webex room? > > >Best Regards >??? >Site Operations Center >Ext.15554? Seat.1#3F049 >We are the F.I.R.S.T. >Fast, Innovation, Reliability , Secure, Team > > > >-----????----- >???: Atin Mukherjee [mailto:amukherj at redhat.com] >????: Wednesday, June 03, 2015 2:36 PM >???: vyyy??? >??: Re: ??: ??: [Gluster-users] Gluster peer rejected and failed to >start > > >Is it possible to share your ip/user/passwd for few of these nodes? > >On 06/03/2015 11:50 AM, vyyy??? wrote: >> Can you remote control my desktop (like webex)? >> The nodes host bricks. And on the new nodes, I did "clean >/var/lib/glusterd and restart" several times, but have no effect. >> Is there any other work around? >> >> >> Best Regards >> Yang >> >> -----????----- >> ???: Atin Mukherjee [mailto:amukherj at redhat.com] >> ????: Wednesday, June 03, 2015 12:33 PM >> ???: vyyy???; Gluster-users at gluster.org >> ??: Re: ??: [Gluster-users] Gluster peer rejected and failed to start >> >> I can't comment on what went wrong till I can have visibility on the >> entire cluster. Does the node which fails to start host any brick? If >> not then work around could be to clean /var/lib/glusterd and restart >> glusterd, that should sync up data and bring this node in >consistency. >> >> HTH, >> Atin >> >> On 06/03/2015 08:34 AM, vyyy??? wrote: >>> I found that /var/lib/glusterd/peers can't include node's own peer, >I delete this file. Glusterd started finally. >>> But the peer status still not normal. >>> Seems that glusterfs10 leave the cluster and join the new nodes >alone. >>> >>> Peer status on glusterfs10 >>> [root at SH02SVR5951 ~]# gluster peer status >>> Number of Peers: 7 >>> >>> Hostname: glusterfs07.sh2.ctripcorp.com >>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs11.sh2.ctripcorp.com >>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 >>> State: Accepted peer request (Connected) >>> >>> Hostname: glusterfs09.sh2.ctripcorp.com >>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs06.sh2.ctripcorp.com >>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >>> State: Peer Rejected (Connected) >>> >>> Hostname: 10.8.230.209 >>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs08.sh2.ctripcorp.com >>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs12.sh2.ctripcorp.com >>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 >>> State: Peer in Cluster (Connected) >>> >>> >>> >>> Peer status on glusterfs10 >>> >>> [root at SVR6993HW2285 ~]# gluster peer status >>> Number of Peers: 6 >>> >>> Hostname: glusterfs05.sh2.ctripcorp.com >>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs09.sh2.ctripcorp.com >>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs07.sh2.ctripcorp.com >>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs08.sh2.ctripcorp.com >>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs06.sh2.ctripcorp.com >>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs10.sh2.ctripcorp.com >>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >>> State: Peer in Cluster (Connected) >>> >>> >>> Peer status on glusterfs05, this is the node that I tried to peer >probe new nodes (glusterfs11 to glusterfs14) >>> >>> [root at sh02svr5956 ~]# gluster peer status >>> Number of Peers: 9 >>> >>> Hostname: glusterfs06.sh2.ctripcorp.com >>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f >>> State: Peer in Cluster (Connected) >>> >>> Hostname: glusterfs07.sh2.ctripcorp.com >>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 >>> State: Peer in Cluster (Connected) >>> >>> Hostname: glusterfs08.sh2.ctripcorp.com >>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 >>> State: Peer in Cluster (Connected) >>> >>> Hostname: glusterfs10.sh2.ctripcorp.com >>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs09.sh2.ctripcorp.com >>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d >>> State: Peer in Cluster (Connected) >>> >>> Hostname: glusterfs11.sh2.ctripcorp.com >>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs12.sh2.ctripcorp.com >>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 >>> State: Peer in Cluster (Connected) >>> >>> Hostname: glusterfs13.sh2.ctripcorp.com >>> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca >>> State: Peer Rejected (Connected) >>> >>> Hostname: glusterfs14.sh2.ctripcorp.com >>> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524 >>> State: Peer Rejected (Connected) >>> [root at sh02svr5956 ~]# >>> >>> >>> >>> >>> Best Regards >>> Yang >>> >>> >>> -----????----- >>> ???: vyyy??? >>> ????: Wednesday, June 03, 2015 10:04 AM >>> ???: 'Atin Mukherjee'; 'Gluster-users at gluster.org' >>> ??: [Gluster-users] Gluster peer rejected and failed to start >>> >>> I am still messed, can you help me figure out problem 2 (A node of >cluster failed to start glusterd) priority >>> >>> The glusterd log on the node gluster10 shown ' resolve brick failed >in restore' >>> I checked /var/lib/glusterd/vols, seems ok >>> >>> Glusterd log file Error message: >>> >>> [2015-06-03 01:31:32.564010] E >[glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve >brick failed in restore >>> [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] >0-management: Initialization of volume 'management' failed, review >>> your volfile again >>> >>> >>> >>> >>> Best Regards >>> Yang >>> >>> -----????----- >>> ???: vyyy??? >>> ????: Tuesday, June 02, 2015 6:27 PM >>> ???: 'Atin Mukherjee'; Gluster-users at gluster.org >>> ??: ??: ??: ??: ??: [Gluster-users] Gluster peer rejected and failed >to start >>> >>> Gluster11 to gluster14 are new nodes I tried to join exit cluster >but failed >>> ?? >>> ?? >>> ??On gluster12: >>> [root at SVR6994HW2285 ~]# gluster peer status >>> Number of Peers: 1 >>> >>> Hostname: glusterfs05.sh2.ctripcorp.com >>> Uuid: 04f22ee8-8e0-- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150604/eb213d49/attachment.html>