Kaamesh Kamalaaharan
2015-Feb-26 00:58 UTC
[Gluster-users] 3 node 2 brick server keeps going offline. Runs excruciatingly slow when both bricks online
Hi guys, My glusterfs setup is across 3 servers (gfs1 ,gfs2, hpc1) with 2 bricks (on gfs1 and gfs2). The reason i have 3 servers is that i wanted the gluster to stay online when one server goes down as i frequently lose my gfs2 server. I have no idea what causes this and im at a dead end. I reinstalled glusterfs-server on my gfs2 server and i found that my performance dropped to the point my R/W speed is around 1.5 kB/S on my 10G LAN . Pressing tab when i ls into a folder takes around 20-30 seconds to autocomplete, sometimes longer. Previously i had blazing fast speeds. When i take gluster2 off the network, the speed is back up. Both my servers are now on the fritz and the bricks keep going offline and i have no idea what to do. Can anyone help shed some light on this? I have no idea what i need to provide so ill provide the logs and config. ::EDIT:: As i was typing, gluster went down again and when i brought it back up, the speed went back to normal. This didnt happen the last 3 or four times i restarted the servers.. I would still like to know what is going on... :::::::: gluster volume info Volume Name: gfsvolume Type: Replicate Volume ID: a29bd2fb-b1ef-4481-be10-c2f4faf4059b Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gfs1:/export/sda/brick Brick2: gfs2:/export/sda/brick Options Reconfigured: performance.quick-read: off network.ping-timeout: 30 network.frame-timeout: 90 performance.cache-max-file-size: 2MB cluster.server-quorum-type: none nfs.addr-namelookup: off nfs.trusted-write: off performance.write-behind-window-size: 4MB cluster.data-self-heal-algorithm: diff performance.cache-refresh-timeout: 60 performance.cache-size: 6442450944 cluster.quorum-type: fixed auth.allow: 172.* cluster.quorum-count: 1 cluster.server-quorum-ratio: 50% /var/log//glusterfs/ [2015-02-26 00:13:36.207408] I [glusterd-volume-ops.c:481:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gfsvolume [2015-02-26 00:13:36.211810] E [glusterd-syncop.c:101:gd_collate_errors] 0-: Commit failed on gfs2. Please check log file for details. [2015-02-26 00:14:17.939391] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f13408f8e6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f1340fa6b50] (-->/us r/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f1342832d55]))) 0-: received signum (15), shutting down [2015-02-26 00:14:20.127606] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid) [2015-02-26 00:14:20.136366] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory [2015-02-26 00:14:20.137768] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-26 00:14:20.137797] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread [2015-02-26 00:14:20.139452] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) [2015-02-26 00:14:20.139483] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device [2015-02-26 00:14:20.139497] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-02-26 00:14:20.139568] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-02-26 00:14:20.139719] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-26 00:14:20.139741] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread [2015-02-26 00:14:23.351872] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-02-26 00:14:23.355502] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-02-26 00:14:23.355552] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2015-02-26 00:14:23.584714] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2015-02-26 00:14:23.588106] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2015-02-26 00:14:23.588223] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:23.588328] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:23.588347] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:23.589290] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:23.589378] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:23.589397] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:23.589885] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option transport.socket.read-fail-log off 8: option transport.socket.keepalive-interval 2 9: option transport.socket.keepalive-time 10 10: option transport-type rdma 11: option working-directory /var/lib/glusterd 12: end-volume 13: +------------------------------------------------------------------------------+ [2015-02-26 00:14:23.780644] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0 [2015-02-26 00:14:23.798102] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:23.798242] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:23.798276] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:24.800202] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: Permission denied [2015-02-26 00:14:24.800858] I [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully [2015-02-26 00:14:24.801261] I [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully [2015-02-26 00:14:24.801649] I [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully [2015-02-26 00:14:24.802028] I [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully [2015-02-26 00:14:24.802410] I [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully [2015-02-26 00:14:24.802793] I [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully [2015-02-26 00:14:24.806276] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:24.806387] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:24.806406] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:25.807341] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No such file or directory [2015-02-26 00:14:25.810987] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:25.811100] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled[2015-02-26 00:14:25.811119] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:25.811329] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c [2015-02-26 00:14:25.811370] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 [2015-02-26 00:14:25.811391] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2015-02-26 00:14:25.811407] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1 [2015-02-26 00:14:25.811510] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2 [2015-02-26 00:14:25.982101] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now [2015-02-26 00:14:25.982172] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now [2015-02-26 00:14:25.982265] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0 [2015-02-26 00:14:26.003332] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /export/sda/brick on port 49153 [2015-02-26 00:14:26.159051] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae [2015-02-26 00:14:26.159112] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 [2015-02-26 00:14:26.159130] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2015-02-26 00:14:26.178634] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c [2015-02-26 00:14:26.178794] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to gfs2 (0), ret: 0 [2015-02-26 00:14:26.194289] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 [2015-02-26 00:14:26.194348] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 [2015-02-26 00:14:26.208631] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c [2015-02-26 00:14:26.373031] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae [2015-02-26 00:14:26.373101] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2 [2015-02-26 00:14:26.385510] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae [2015-02-26 00:14:26.385645] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to hpc1 (0), ret: 0 [2015-02-26 00:14:26.399963] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 [2015-02-26 00:14:26.400010] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 [2015-02-26 00:14:26.414448] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c [2015-02-26 00:14:26.668753] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae [2015-02-26 00:14:32.167298] W [socket.c:522:__socket_rwv] 0-management: readv on 172.20.20.22:24007 failed (No data available) [2015-02-26 00:14:33.590941] E [socket.c:2161:socket_connect_finish] 0-management: connection to 172.20.20.22:24007 failed (Connection refused) [2015-02-26 00:14:39.060572] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f727130ce6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f72719bab50] (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f7273246d55]))) 0-: received signum (15), shutting down [2015-02-26 00:14:42.990079] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid) [2015-02-26 00:14:42.994671] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory [2015-02-26 00:14:42.996068] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-26 00:14:42.996098] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread [2015-02-26 00:14:42.996892] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) [2015-02-26 00:14:42.996919] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device [2015-02-26 00:14:42.996932] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-02-26 00:14:42.997004] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-02-26 00:14:42.997138] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-26 00:14:42.997156] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread [2015-02-26 00:14:46.319737] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-02-26 00:14:46.323519] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-02-26 00:14:46.323567] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2015-02-26 00:14:46.561399] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2015-02-26 00:14:46.564795] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2015-02-26 00:14:46.564919] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:46.565035] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:46.565054] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:46.566216] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:46.566329] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:46.566348] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:46.566832] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option transport.socket.read-fail-log off 8: option transport.socket.keepalive-interval 2 9: option transport.socket.keepalive-time 10 10: option transport-type rdma 11: option working-directory /var/lib/glusterd 12: end-volume 13: +------------------------------------------------------------------------------+ [2015-02-26 00:14:46.570634] E [socket.c:2161:socket_connect_finish] 0-management: connection to 172.20.20.22:24007 failed (Connection refused) [2015-02-26 00:14:46.571244] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2 [2015-02-26 00:14:47.001039] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae [2015-02-26 00:14:47.001183] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to hpc1 (0), ret: 0 [2015-02-26 00:14:47.015772] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 [2015-02-26 00:14:47.015820] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 [2015-02-26 00:14:47.030090] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:47.030178] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:47.030196] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:48.031850] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: Permission denied [2015-02-26 00:14:48.032392] I [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully [2015-02-26 00:14:48.032794] I [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully [2015-02-26 00:14:48.033182] I [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully [2015-02-26 00:14:48.033571] I [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully [2015-02-26 00:14:48.033964] I [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully [2015-02-26 00:14:48.034356] I [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully [2015-02-26 00:14:48.037872] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:48.037982] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:48.038001] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:49.039002] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No such file or directory [2015-02-26 00:14:49.042564] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-26 00:14:49.042670] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2015-02-26 00:14:49.042690] I [socket.c:3576:socket_init] 0-management: using system polling thread [2015-02-26 00:14:49.042958] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae [2015-02-26 00:14:49.042998] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 [2015-02-26 00:14:49.043013] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2015-02-26 00:14:49.043319] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0 [2015-02-26 00:14:49.204554] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now [2015-02-26 00:14:49.204657] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now [2015-02-26 00:14:49.204717] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae [2015-02-26 00:14:49.814028] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /export/sda/brick on port 49153 [2015-02-26 00:14:53.678347] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2 [2015-02-26 00:14:53.848276] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c [2015-02-26 00:14:53.848407] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to gfs2 (0), ret: 0 [2015-02-26 00:14:53.862892] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 [2015-02-26 00:14:53.862952] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 [2015-02-26 00:14:53.877685] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae [2015-02-26 00:14:55.217486] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0 [2015-02-26 00:14:55.234370] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c [2015-02-26 00:14:55.234433] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 [2015-02-26 00:14:55.234451] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2015-02-26 00:14:55.234468] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1 [2015-02-26 00:15:14.637613] E [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: Volume gfsvolume already started [2015-02-26 00:15:14.637661] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Start' failed on localhost : Volume gfsvolume already started [2015-02-26 00:24:41.265078] W [socket.c:522:__socket_rwv] 0-management: readv on 172.20.20.22:24007 failed (No data available) [2015-02-26 00:24:43.631739] E [socket.c:2161:socket_connect_finish] 0-management: connection to 172.20.20.22:24007 failed (Connection refused) [2015-02-26 00:24:56.174790] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/91bfef953906a83fc14aa435a886ac4d.socket failed (No data available) [2015-02-26 00:24:56.175299] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1:/export/sda/brick [2015-02-26 00:25:24.076560] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-02-26 00:25:24.078109] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gfsvolume [2015-02-26 00:25:31.041343] E [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: Volume gfsvolume already started [2015-02-26 00:25:31.041404] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Start' failed on localhost : Volume gfsvolume already started [2015-02-26 00:25:43.389260] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8f8cfa7e6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f8f8d655b50] (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f8f8eee1d55]))) 0-: received signum (15), shutting down [2015-02-26 00:25:45.437325] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid) [2015-02-26 00:25:45.487307] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory [2015-02-26 00:25:45.488734] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-26 00:25:45.488765] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread [2015-02-26 00:25:45.489557] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) [2015-02-26 00:25:45.489584] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device [2015-02-26 00:25:45.489597] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-02-26 00:25:45.489668] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-02-26 00:25:45.489799] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-26 00:25:45.489818] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread [2015-02-26 00:25:48.706235] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 Please Do let me know if there is anything else i can provide to help. Apologies if this is a simple fix but i have never found an answer after a week of searching. Thank You Kindly, Kaamesh -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150226/61854e00/attachment.html>
Kaamesh Kamalaaharan
2015-Feb-26 01:07 UTC
[Gluster-users] 3 node 2 brick server keeps going offline. Runs excruciatingly slow when both bricks online
Update The speed is back down, but when the gfs is mounted via nfs instead of through the gluster client, i get good speeds Thank You Kindly, Kaamesh Bioinformatician Novocraft Technologies Sdn Bhd C-23A-05, 3 Two Square, Section 19, 46300 Petaling Jaya Selangor Darul Ehsan Malaysia Mobile: +60176562635 Ph: +60379600541 Fax: +60379600540 On Thu, Feb 26, 2015 at 8:58 AM, Kaamesh Kamalaaharan <kaamesh at novocraft.com> wrote:> Hi guys, > > My glusterfs setup is across 3 servers (gfs1 ,gfs2, hpc1) with 2 bricks > (on gfs1 and gfs2). > > The reason i have 3 servers is that i wanted the gluster to stay online > when one server goes down as i frequently lose my gfs2 server. I have no > idea what causes this and im at a dead end. > > I reinstalled glusterfs-server on my gfs2 server and i found that my > performance dropped to the point my R/W speed is around 1.5 kB/S on my 10G > LAN . Pressing tab when i ls into a folder takes around 20-30 seconds to > autocomplete, sometimes longer. Previously i had blazing fast speeds. When > i take gluster2 off the network, the speed is back up. > > Both my servers are now on the fritz and the bricks keep going offline and > i have no idea what to do. Can anyone help shed some light on this? I have > no idea what i need to provide so ill provide the logs and config. > > > ::EDIT:: > As i was typing, gluster went down again and when i brought it back up, > the speed went back to normal. This didnt happen the last 3 or four times i > restarted the servers.. I would still like to know what is going on... > :::::::: > > > > gluster volume info > > Volume Name: gfsvolume > Type: Replicate > Volume ID: a29bd2fb-b1ef-4481-be10-c2f4faf4059b > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: gfs1:/export/sda/brick > Brick2: gfs2:/export/sda/brick > Options Reconfigured: > performance.quick-read: off > network.ping-timeout: 30 > network.frame-timeout: 90 > performance.cache-max-file-size: 2MB > cluster.server-quorum-type: none > nfs.addr-namelookup: off > nfs.trusted-write: off > performance.write-behind-window-size: 4MB > cluster.data-self-heal-algorithm: diff > performance.cache-refresh-timeout: 60 > performance.cache-size: 6442450944 > cluster.quorum-type: fixed > auth.allow: 172.* > cluster.quorum-count: 1 > cluster.server-quorum-ratio: 50% > > /var/log//glusterfs/ > > > > [2015-02-26 00:13:36.207408] I > [glusterd-volume-ops.c:481:__glusterd_handle_cli_heal_volume] 0-management: > Received heal vol req for volume gfsvolume > > [2015-02-26 00:13:36.211810] E [glusterd-syncop.c:101:gd_collate_errors] > 0-: Commit failed on gfs2. Please check log file for details. > [2015-02-26 00:14:17.939391] W [glusterfsd.c:1095:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f13408f8e6d] > (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f1340fa6b50] (-->/us > r/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f1342832d55]))) 0-: received > signum (15), shutting down > [2015-02-26 00:14:20.127606] I [glusterfsd.c:1959:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 > (/usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-02-26 00:14:20.136366] I [glusterd.c:1148:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-02-26 00:14:20.137768] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:14:20.137797] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:14:20.139452] W [rdma.c:4194:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-26 00:14:20.139483] E [rdma.c:4482:init] 0-rdma.management: > Failed to initialize IB Device > [2015-02-26 00:14:20.139497] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-02-26 00:14:20.139568] W [rpcsvc.c:1483:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-26 00:14:20.139719] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:14:20.139741] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:14:23.351872] I > [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 2 > [2015-02-26 00:14:23.355502] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-0 > [2015-02-26 00:14:23.355552] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-1 > [2015-02-26 00:14:23.584714] I > [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect > returned 0 > [2015-02-26 00:14:23.588106] I > [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect > returned 0 > [2015-02-26 00:14:23.588223] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:23.588328] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:23.588347] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:23.589290] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:23.589378] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:23.589397] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:23.589885] I [glusterd.c:138:glusterd_uuid_init] > 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070 > Final graph: > > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option transport.socket.read-fail-log off > 8: option transport.socket.keepalive-interval 2 > 9: option transport.socket.keepalive-time 10 > 10: option transport-type rdma > 11: option working-directory /var/lib/glusterd > 12: end-volume > 13: > > +------------------------------------------------------------------------------+ > [2015-02-26 00:14:23.780644] I > [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC > from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0 > [2015-02-26 00:14:23.798102] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:23.798242] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:23.798276] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:24.800202] E > [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: > Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: > Permission denied > [2015-02-26 00:14:24.800858] I > [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV3 successfully > [2015-02-26 00:14:24.801261] I > [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV1 successfully > [2015-02-26 00:14:24.801649] I > [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered > NFSV3 successfully > [2015-02-26 00:14:24.802028] I > [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM > v4 successfully > [2015-02-26 00:14:24.802410] I > [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM > v1 successfully > [2015-02-26 00:14:24.802793] I > [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL > v3 successfully > [2015-02-26 00:14:24.806276] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:24.806387] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:24.806406] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:25.807341] E > [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: > Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No > such file or directory > [2015-02-26 00:14:25.810987] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:25.811100] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled[2015-02-26 00:14:25.811119] I > [socket.c:3576:socket_init] 0-management: using system polling thread > [2015-02-26 00:14:25.811329] I > [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: > Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:25.811370] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 > [2015-02-26 00:14:25.811391] I > [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my > uuid as Friend > [2015-02-26 00:14:25.811407] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1 > [2015-02-26 00:14:25.811510] I > [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: > using the op-version 2 > [2015-02-26 00:14:25.982101] I [socket.c:2238:socket_event_handler] > 0-transport: disconnecting now > [2015-02-26 00:14:25.982172] I [socket.c:2238:socket_event_handler] > 0-transport: disconnecting now > [2015-02-26 00:14:25.982265] I > [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC > from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0 > [2015-02-26 00:14:26.003332] I [glusterd-pmap.c:227:pmap_registry_bind] > 0-pmap: adding brick /export/sda/brick on port 49153 > [2015-02-26 00:14:26.159051] I > [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: > Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:26.159112] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 > [2015-02-26 00:14:26.159130] I > [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my > uuid as Friend > [2015-02-26 00:14:26.178634] I > [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: > Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:26.178794] I > [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to gfs2 (0), ret: 0 > [2015-02-26 00:14:26.194289] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 > [2015-02-26 00:14:26.194348] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 > [2015-02-26 00:14:26.208631] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:26.373031] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:26.373101] I > [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: > using the op-version 2 > [2015-02-26 00:14:26.385510] I > [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: > Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:26.385645] I > [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to hpc1 (0), ret: 0 > [2015-02-26 00:14:26.399963] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 > [2015-02-26 00:14:26.400010] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 > [2015-02-26 00:14:26.414448] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:26.668753] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:32.167298] W [socket.c:522:__socket_rwv] 0-management: > readv on 172.20.20.22:24007 failed (No data available) > [2015-02-26 00:14:33.590941] E [socket.c:2161:socket_connect_finish] > 0-management: connection to 172.20.20.22:24007 failed (Connection refused) > [2015-02-26 00:14:39.060572] W [glusterfsd.c:1095:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f727130ce6d] > (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f72719bab50] > (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f7273246d55]))) 0-: > received signum (15), shutting down > [2015-02-26 00:14:42.990079] I [glusterfsd.c:1959:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 > (/usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-02-26 00:14:42.994671] I [glusterd.c:1148:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-02-26 00:14:42.996068] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:14:42.996098] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:14:42.996892] W [rdma.c:4194:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-26 00:14:42.996919] E [rdma.c:4482:init] 0-rdma.management: > Failed to initialize IB Device > [2015-02-26 00:14:42.996932] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-02-26 00:14:42.997004] W [rpcsvc.c:1483:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-26 00:14:42.997138] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:14:42.997156] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:14:46.319737] I > [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 2 > [2015-02-26 00:14:46.323519] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-0 > [2015-02-26 00:14:46.323567] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-1 > [2015-02-26 00:14:46.561399] I > [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect > returned 0 > [2015-02-26 00:14:46.564795] I > [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect > returned 0 > [2015-02-26 00:14:46.564919] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:46.565035] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:46.565054] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:46.566216] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:46.566329] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:46.566348] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:46.566832] I [glusterd.c:138:glusterd_uuid_init] > 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070 > Final graph: > > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option transport.socket.read-fail-log off > 8: option transport.socket.keepalive-interval 2 > 9: option transport.socket.keepalive-time 10 > 10: option transport-type rdma > 11: option working-directory /var/lib/glusterd > 12: end-volume > 13: > > +------------------------------------------------------------------------------+ > [2015-02-26 00:14:46.570634] E [socket.c:2161:socket_connect_finish] > 0-management: connection to 172.20.20.22:24007 failed (Connection refused) > [2015-02-26 00:14:46.571244] I > [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: > using the op-version 2 > [2015-02-26 00:14:47.001039] I > [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: > Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:47.001183] I > [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to hpc1 (0), ret: 0 > [2015-02-26 00:14:47.015772] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 > [2015-02-26 00:14:47.015820] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 > [2015-02-26 00:14:47.030090] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:47.030178] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:47.030196] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:48.031850] E > [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: > Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: > Permission denied > [2015-02-26 00:14:48.032392] I > [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV3 successfully > [2015-02-26 00:14:48.032794] I > [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV1 successfully > [2015-02-26 00:14:48.033182] I > [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered > NFSV3 successfully > [2015-02-26 00:14:48.033571] I > [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM > v4 successfully > [2015-02-26 00:14:48.033964] I > [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM > v1 successfully > [2015-02-26 00:14:48.034356] I > [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL > v3 successfully > [2015-02-26 00:14:48.037872] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:48.037982] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:48.038001] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:49.039002] E > [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: > Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No > such file or directory > [2015-02-26 00:14:49.042564] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-26 00:14:49.042670] I [socket.c:3561:socket_init] 0-management: > SSL support is NOT enabled > [2015-02-26 00:14:49.042690] I [socket.c:3576:socket_init] 0-management: > using system polling thread > [2015-02-26 00:14:49.042958] I > [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: > Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:49.042998] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 > [2015-02-26 00:14:49.043013] I > [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my > uuid as Friend > [2015-02-26 00:14:49.043319] I > [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC > from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0 > [2015-02-26 00:14:49.204554] I [socket.c:2238:socket_event_handler] > 0-transport: disconnecting now > [2015-02-26 00:14:49.204657] I [socket.c:2238:socket_event_handler] > 0-transport: disconnecting now > [2015-02-26 00:14:49.204717] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:49.814028] I [glusterd-pmap.c:227:pmap_registry_bind] > 0-pmap: adding brick /export/sda/brick on port 49153 > [2015-02-26 00:14:53.678347] I > [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: > using the op-version 2 > [2015-02-26 00:14:53.848276] I > [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: > Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:53.848407] I > [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: > Responded to gfs2 (0), ret: 0 > [2015-02-26 00:14:53.862892] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1 > [2015-02-26 00:14:53.862952] I > [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: > adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2 > [2015-02-26 00:14:53.877685] I > [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: > Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae > [2015-02-26 00:14:55.217486] I > [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC > from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0 > [2015-02-26 00:14:55.234370] I > [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: > Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c > [2015-02-26 00:14:55.234433] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1 > [2015-02-26 00:14:55.234451] I > [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my > uuid as Friend > [2015-02-26 00:14:55.234468] I > [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received > uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1 > [2015-02-26 00:15:14.637613] E > [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: > Volume gfsvolume already started > [2015-02-26 00:15:14.637661] E [glusterd-syncop.c:912:gd_stage_op_phase] > 0-management: Staging of operation 'Volume Start' failed on localhost : > Volume gfsvolume already started > [2015-02-26 00:24:41.265078] W [socket.c:522:__socket_rwv] 0-management: > readv on 172.20.20.22:24007 failed (No data available) > [2015-02-26 00:24:43.631739] E [socket.c:2161:socket_connect_finish] > 0-management: connection to 172.20.20.22:24007 failed (Connection refused) > [2015-02-26 00:24:56.174790] W [socket.c:522:__socket_rwv] 0-management: > readv on /var/run/91bfef953906a83fc14aa435a886ac4d.socket failed (No data > available) > [2015-02-26 00:24:56.175299] I > [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: > Disconnected from gfs1:/export/sda/brick > [2015-02-26 00:25:24.076560] W > [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx > modification failed > [2015-02-26 00:25:24.078109] I > [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume gfsvolume > [2015-02-26 00:25:31.041343] E > [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: > Volume gfsvolume already started > [2015-02-26 00:25:31.041404] E [glusterd-syncop.c:912:gd_stage_op_phase] > 0-management: Staging of operation 'Volume Start' failed on localhost : > Volume gfsvolume already started > [2015-02-26 00:25:43.389260] W [glusterfsd.c:1095:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8f8cfa7e6d] > (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f8f8d655b50] > (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f8f8eee1d55]))) 0-: > received signum (15), shutting down > [2015-02-26 00:25:45.437325] I [glusterfsd.c:1959:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 > (/usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-02-26 00:25:45.487307] I [glusterd.c:1148:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-02-26 00:25:45.488734] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:25:45.488765] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:25:45.489557] W [rdma.c:4194:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-26 00:25:45.489584] E [rdma.c:4482:init] 0-rdma.management: > Failed to initialize IB Device > [2015-02-26 00:25:45.489597] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-02-26 00:25:45.489668] W [rpcsvc.c:1483:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-26 00:25:45.489799] I [socket.c:3561:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-26 00:25:45.489818] I [socket.c:3576:socket_init] > 0-socket.management: using system polling thread > [2015-02-26 00:25:48.706235] I > [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 2 > > > > > Please Do let me know if there is anything else i can provide to help. > Apologies if this is a simple fix but i have never found an answer after a > week of searching. > > Thank You Kindly, > Kaamesh >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150226/93d1eec7/attachment.html>