Changliang Chen
2011-Dec-14 09:36 UTC
[Gluster-users] glusterfs crash when the one of replicate node restart
Hi,we have use glusterfs for two years. After upgraded to 3.2.5,we discover that when one of replicate node reboot and startup the glusterd daemon,the gluster will crash cause by the other replicate node cpu usage reach 100%. Our gluster info: Type: Distributed-Replicate Status: Started Number of Bricks: 5 x 2 = 10 Transport-type: tcp Options Reconfigured: performance.cache-size: 3GB performance.cache-max-file-size: 512KB network.frame-timeout: 30 network.ping-timeout: 25 cluster.min-free-disk: 10% Our device? Dell R710 600Gsas *6 3*8Gmem The error info: [2011-12-14 13:24:10.483812] E [rdma.c:4813:init] 0-rdma.management: Failed to initialize IB Device [2011-12-14 13:24:10.483828] E [rpc-transport.c:742:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2011-12-14 13:24:10.483841] W [rpcsvc.c:1288:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2011-12-14 13:24:11.967621] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2011-12-14 13:24:11.967665] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2011-12-14 13:24:11.967681] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2011-12-14 13:24:11.967695] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2011-12-14 13:24:11.967709] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4 [2011-12-14 13:24:11.967723] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5 [2011-12-14 13:24:11.967736] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6 [2011-12-14 13:24:11.967750] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7 [2011-12-14 13:24:11.967764] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-8 [2011-12-14 13:24:11.967777] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-9 [2011-12-14 13:24:12.465565] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.17:1013) [2011-12-14 13:24:12.465623] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.8:1013) [2011-12-14 13:24:12.465656] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.10:1013) [2011-12-14 13:24:12.465686] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.11:1013) [2011-12-14 13:24:12.465716] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.125:1013) [2011-12-14 13:24:12.633288] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.65:1006) [2011-12-14 13:24:13.138150] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.1:1013) [2011-12-14 13:24:13.284665] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.3:1013) [2011-12-14 13:24:15.790805] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.8:1013) [2011-12-14 13:24:16.113430] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.125:1013) [2011-12-14 13:24:16.259040] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.10:1013) [2011-12-14 13:24:16.392058] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.17:1013) [2011-12-14 13:24:16.429444] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.11:1013) [2011-12-14 13:26:05.787680] W [glusterfsd.c:727:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37c8ed3c2d] (-->/lib64/libpthread.so.0 [0x37c96064a7] (-->/opt/glusterfs/3.2.5/sbin/glusterd(glusterfs_sigwaiter+0x17c) [0x40477c]))) 0-: received signum (15), shutting down -- Regards, Cocl -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111214/4efe1c65/attachment.html> -------------- next part -------------- [2011-12-14 13:24:10.381798] I [glusterfsd.c:1493:main] 0-/opt/glusterfs/3.2.5/sbin/glusterd: Started running /opt/glusterfs/3.2.5/sbin/glusterd version 3.2.5 [2011-12-14 13:24:10.456111] I [glusterd.c:550:init] 0-management: Using /etc/glusterd as working directory [2011-12-14 13:24:10.483696] C [rdma.c:3934:rdma_init] 0-rpc-transport/rdma: Failed to get IB devices [2011-12-14 13:24:10.483812] E [rdma.c:4813:init] 0-rdma.management: Failed to initialize IB Device [2011-12-14 13:24:10.483828] E [rpc-transport.c:742:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2011-12-14 13:24:10.483841] W [rpcsvc.c:1288:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2011-12-14 13:24:10.492780] I [glusterd.c:88:glusterd_uuid_init] 0-glusterd: retrieved UUID: 18ce0543-24f2-4e8f-a13d-3d5a4a7dff1b [2011-12-14 13:24:11.967621] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2011-12-14 13:24:11.967665] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2011-12-14 13:24:11.967681] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2011-12-14 13:24:11.967695] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2011-12-14 13:24:11.967709] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4 [2011-12-14 13:24:11.967723] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5 [2011-12-14 13:24:11.967736] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6 [2011-12-14 13:24:11.967750] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7 [2011-12-14 13:24:11.967764] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-8 [2011-12-14 13:24:11.967777] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-9 [2011-12-14 13:24:12.318700] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.325662] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.333999] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.342100] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.348505] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.355102] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.361994] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.368569] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.377832] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-12-14 13:24:12.383343] I [glusterd-utils.c:1097:glusterd_volume_start_glusterfs] 0-: About to start glusterfs for brick x.x.x.x:/data0 Given volfile: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option working-directory /etc/glusterd 4: option transport-type socket,rdma 5: option transport.socket.keepalive-time 10 6: option transport.socket.keepalive-interval 2 7: end-volume 8: +------------------------------------------------------------------------------+ [2011-12-14 13:24:12.462383] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.462428] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.462457] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.462505] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.462549] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.462608] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.462637] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.462663] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.462740] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) [2011-12-14 13:24:12.465565] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.17:1013) [2011-12-14 13:24:12.465623] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.8:1013) [2011-12-14 13:24:12.465656] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.x.x.x) [2011-12-14 13:24:12.465686] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.11:1013) [2011-12-14 13:24:12.465716] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.125:1013) [2011-12-14 13:24:12.469566] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd [2011-12-14 13:24:12.469691] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:12.469850] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:12.469873] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:12.469889] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:12.469904] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:12.469919] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:12.469934] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:12.469950] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:12.469965] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:12.469980] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:12.470508] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.470714] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.474314] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22 [2011-12-14 13:24:12.474424] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:12.474575] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:12.474608] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:12.474625] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:12.474640] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:12.474655] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:12.474670] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:12.474685] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:12.474701] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:12.474716] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:12.475100] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.475305] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.477877] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 493352a0-912c-4298-b987-a0f039559f45 [2011-12-14 13:24:12.477977] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:12.478101] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:12.478122] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:12.478138] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:12.478153] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:12.478167] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:12.478183] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:12.478198] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:12.478213] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:12.478228] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:12.478609] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.479016] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.480496] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad [2011-12-14 13:24:12.480622] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:12.480746] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:12.480767] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:12.480783] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:12.480806] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:12.480822] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:12.480837] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:12.480852] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:12.480868] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:12.480883] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:12.481550] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.481709] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.484275] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.484546] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.484774] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.484981] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.485041] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.485235] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.485453] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.485681] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.492618] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.492965] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.493040] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.493184] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.493382] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.493430] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.493751] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.493782] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.503875] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.504289] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.504573] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.504718] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.518166] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.518803] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.519241] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.519625] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.522356] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.522430] I [glusterd-pmap.c:237:pmap_registry_bind] 0-pmap: adding brick /data0 on port 24009 [2011-12-14 13:24:12.522699] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.522849] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.523066] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.625851] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672 [2011-12-14 13:24:12.625974] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:12.626124] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:12.626146] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:12.626163] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:12.626178] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:12.626193] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:12.626208] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:12.626223] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:12.626239] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:12.626254] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:12.626763] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.626805] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.626829] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.626860] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.626884] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.626905] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.626926] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.626947] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.633288] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.65:1006) [2011-12-14 13:24:12.638589] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.731673] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3 [2011-12-14 13:24:12.731784] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:12.731930] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:12.731962] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:12.731979] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:12.731994] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:12.732009] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:12.732024] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:12.732039] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:12.732055] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:12.732070] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:12.732607] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.732659] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.732687] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.732716] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.732742] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.732767] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.732792] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.732981] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.758831] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.929723] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884 [2011-12-14 13:24:12.929832] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:12.929970] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:12.929992] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:12.930009] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:12.930024] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:12.930039] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:12.930054] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:12.930069] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:12.930084] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:12.930099] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:12.930570] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.930644] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.930676] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.930701] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.930726] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.930750] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.930777] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.930804] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:12.979192] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.65011] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36 [2011-12-14 13:24:13.65118] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:13.65247] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:13.65269] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:13.65285] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:13.65300] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:13.65314] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:13.65329] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:13.65344] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:13.65360] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:13.65375] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:13.65931] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.65973] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.65999] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.66024] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.66048] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.66072] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.66097] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.66120] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.82588] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.138150] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.1:1013) [2011-12-14 13:24:13.275230] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c [2011-12-14 13:24:13.275358] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0 [2011-12-14 13:24:13.275504] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64 [2011-12-14 13:24:13.275558] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23 [2011-12-14 13:24:13.275578] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24 [2011-12-14 13:24:13.275593] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67 [2011-12-14 13:24:13.275608] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68 [2011-12-14 13:24:13.275623] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28 [2011-12-14 13:24:13.275638] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27 [2011-12-14 13:24:13.275654] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25 [2011-12-14 13:24:13.275669] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26 [2011-12-14 13:24:13.276064] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.276105] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.276130] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.276155] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.276188] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.276262] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.276305] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.276330] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:13.284665] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.3:1013) [2011-12-14 13:24:13.300639] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: [2011-12-14 13:24:15.790805] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.8:1013) [2011-12-14 13:24:16.113430] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.125:1013) [2011-12-14 13:24:16.259040] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.x.x.x) [2011-12-14 13:24:16.392058] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.17:1013) [2011-12-14 13:24:16.429444] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1 0.1.1.11:1013) [2011-12-14 13:26:05.787680] W [glusterfsd.c:727:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37c8ed3c2d] (-->/lib64/libpthread.so.0 [0x37c96064a7] (-->/opt/glusterfs/3 .2.5/sbin/glusterd(glusterfs_sigwaiter+0x17c) [0x40477c]))) 0-: received signum (15), shutting down
Pranith Kumar K
2011-Dec-14 10:49 UTC
[Gluster-users] glusterfs crash when the one of replicate node restart
On 12/14/2011 03:06 PM, Changliang Chen wrote:> Hi,we have use glusterfs for two years. After upgraded to 3.2.5,we > discover that when one of replicate node reboot and startup the > glusterd daemon,the gluster will crash cause by the other > > replicate node cpu usage reach 100%. > > Our gluster info: > > Type: Distributed-Replicate > Status: Started > Number of Bricks: 5 x 2 = 10 > Transport-type: tcp > Options Reconfigured: > performance.cache-size: 3GB > performance.cache-max-file-size: 512KB > network.frame-timeout: 30 > network.ping-timeout: 25 > cluster.min-free-disk: 10% > > Our device: > > Dell R710 > 600Gsas *6 > 3*8Gmem > > The error info: > > [2011-12-14 13:24:10.483812] E [rdma.c:4813:init] 0-rdma.management: > Failed to initialize IB Device > [2011-12-14 13:24:10.483828] E > [rpc-transport.c:742:rpc_transport_load] 0-rpc-transport: 'rdma' > initialization failed > [2011-12-14 13:24:10.483841] W [rpcsvc.c:1288:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2011-12-14 13:24:11.967621] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-0 > [2011-12-14 13:24:11.967665] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-1 > [2011-12-14 13:24:11.967681] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-2 > [2011-12-14 13:24:11.967695] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-3 > [2011-12-14 13:24:11.967709] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-4 > [2011-12-14 13:24:11.967723] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-5 > [2011-12-14 13:24:11.967736] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-6 > [2011-12-14 13:24:11.967750] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-7 > [2011-12-14 13:24:11.967764] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-8 > [2011-12-14 13:24:11.967777] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-9 > [2011-12-14 13:24:12.465565] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.17:1013 <http://10.1.1.17:1013>) > [2011-12-14 13:24:12.465623] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.8:1013 <http://10.1.1.8:1013>) > [2011-12-14 13:24:12.465656] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.10:1013 <http://10.1.1.10:1013>) > [2011-12-14 13:24:12.465686] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.11:1013 <http://10.1.1.11:1013>) > [2011-12-14 13:24:12.465716] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.125:1013 <http://10.1.1.125:1013>) > [2011-12-14 13:24:12.633288] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.65:1006 <http://10.1.1.65:1006>) > [2011-12-14 13:24:13.138150] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.1:1013 <http://10.1.1.1:1013>) > [2011-12-14 13:24:13.284665] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.3:1013 <http://10.1.1.3:1013>) > [2011-12-14 13:24:15.790805] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.8:1013 <http://10.1.1.8:1013>) > [2011-12-14 13:24:16.113430] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.125:1013 <http://10.1.1.125:1013>) > [2011-12-14 13:24:16.259040] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.10:1013 <http://10.1.1.10:1013>) > [2011-12-14 13:24:16.392058] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.17:1013 <http://10.1.1.17:1013>) > [2011-12-14 13:24:16.429444] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.11:1013 <http://10.1.1.11:1013>) > [2011-12-14 13:26:05.787680] W [glusterfsd.c:727:cleanup_and_exit] > (-->/lib64/libc.so.6(clone+0x6d) [0x37c8ed3c2d] > (-->/lib64/libpthread.so.0 [0x37c96064a7] > (-->/opt/glusterfs/3.2.5/sbin/glusterd(glusterfs_sigwaiter+0x17c) > [0x40477c]))) 0-: received signum (15), shutting down > > -- > > Regards, > > Cocl > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-usershi Changliang, Could you specify which process crashed. Is it glusterd or glusterfs? Could you provide the stack trace that is present in it's respective logfile. I dont see any stack trace in the logs you have provided. Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111214/58d1ae25/attachment.html>