Игорь Бирюлин
2015-Aug-21 14:59 UTC
[Gluster-users] Gluster peer status disconnected on node where glusterd has been started first
Hello, I am testing SSL/TLS support in gluster. I have gluster replica set with 2 node: # gluster volume info Volume Name: repofiles Type: Replicate Volume ID: 4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: dist-int-master03:/storage/gluster_brick_repofiles Brick2: dist-int-master04:/storage/gluster_brick_repofiles Options Reconfigured: performance.readdir-ahead: on auth.ssl-allow: dist-master03,dist-master04 client.ssl: on server.ssl: on If umount all shares, stop all gluster process and try to start glusterd process I found strange behavior on node where I has started first glusterd. On gluster 3.7.3 it look like output of gluster peer status is disconnected. But the second node show gluster peer status connected. Restart glusterd service on node where I has start first glusterd have solved the problem. 1. # cat /etc/issue Ubuntu 14.04.2 LTS \n \l # dpkg -l | grep gluster ii glusterfs-client 3.7.3-ubuntu1~trusty1 amd64 clustered file-system (client package) ii glusterfs-common 3.7.3-ubuntu1~trusty1 amd64 GlusterFS common libraries and translator modules ii glusterfs-server 3.7.3-ubuntu1~trusty1 amd64 clustered file-system (server package) In log before restart glusterd: [2015-08-21 13:31:51.474562] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.3 (args: /usr/sbin /glusterd -p /var/run/glusterd.pid) [2015-08-21 13:31:51.477772] I [MSGID: 106478] [glusterd.c:1346:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-08-21 13:31:51.477807] I [MSGID: 106479] [glusterd.c:1395:init] 0-management: Using /var/lib/glusterd as working directory [2015-08-21 13:31:51.480745] I [socket.c:3968:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED [2015-08-21 13:31:51.480775] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED [2015-08-21 13:31:51.480785] I [socket.c:3988:socket_init] 0-socket.management: using private polling thread [2015-08-21 13:31:51.481924] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2015-08-21 13:31:51.481948] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device [2015-08-21 13:31:51.481959] W [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-08-21 13:31:51.482005] W [rpcsvc.c:1597:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-08-21 13:31:51.482021] E [MSGID: 106243] [glusterd.c:1619:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2015-08-21 13:31:51.482131] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED [2015-08-21 13:31:54.273846] I [MSGID: 106513] [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30703 [2015-08-21 13:31:54.506187] I [MSGID: 106498] [glusterd-handler.c:3570:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-08-21 13:31:54.506288] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-08-21 13:31:54.506416] I [socket.c:3968:socket_init] 0-management: SSL support on the I/O path is ENABLED [2015-08-21 13:31:54.506430] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED [2015-08-21 13:31:54.506439] I [socket.c:3988:socket_init] 0-management: using private polling thread [2015-08-21 13:31:54.508248] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument [2015-08-21 13:31:54.508277] E [socket.c:3019:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2015-08-21 13:31:54.510377] E [socket.c:3075:socket_connect] 0-management: connection attempt on failed, (Connection refused) [2015-08-21 13:31:54.510826] I [MSGID: 106004] [glusterd-handler.c:5051:__glusterd_peer_rpc_notify] 0-management: Peer <dist-int-master04.xxx> (<264cf70c-688c-4d67-b835-5ec7b7689894>), in state <Peer in Cluster>, has disconnected from glusterd. [2015-08-21 13:31:54.510856] E [socket.c:406:ssl_setup_connection] 0-management: SSL connect error [2015-08-21 13:31:54.510889] E [socket.c:2442:socket_poller] 0-management: client setup failed [2015-08-21 13:31:54.510885] E [MSGID: 106155] [glusterd-utils.c:199:glusterd_unlock] 0-management: Cluster lock not held! [2015-08-21 13:31:54.510936] I [MSGID: 106004] [glusterd-handler.c:5051:__glusterd_peer_rpc_notify] 0-management: Peer <dist-int-master04.xxx> (<264cf70c-688c-4d67-b835-5ec7b7689894>), in state <Peer in Cluster>, has disconnected from glusterd. [2015-08-21 13:31:54.510960] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=588 max=0 total=0 [2015-08-21 13:31:54.510971] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=124 max=0 total=0 [2015-08-21 13:31:54.516188] I [MSGID: 106544] [glusterd.c:156:glusterd_uuid_init] 0-management: retrieved UUID: c90033d0-1253-40bf-b700-5f65648940dd Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option transport.socket.ssl-enabled on 4: option rpc-auth.auth-glusterfs on 5: option rpc-auth.auth-unix on 6: option rpc-auth.auth-null on 7: option rpc-auth-allow-insecure on 8: option transport.socket.listen-backlog 128 9: option ping-timeout 30 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2015-08-21 13:31:54.517254] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-08-21 13:32:05.981534] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:32:05.987833] I [MSGID: 106163] [glusterd-handshake.c:1198:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30703 [2015-08-21 13:32:05.992637] I [MSGID: 106490] [glusterd-handler.c:2530:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894 [2015-08-21 13:32:13.248817] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-08-21 13:32:13.251969] I [MSGID: 106493] [glusterd-handler.c:3771:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dist-int-master04.xxx (0), ret: 0 [2015-08-21 13:32:13.758331] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:32:13.764531] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop [2015-08-21 13:32:17.734396] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:32:17.740518] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop [2015-08-21 13:32:20.736654] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:32:20.742658] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop [2015-08-21 13:32:23.740409] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:32:23.871236] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop [2015-08-21 13:32:26.743118] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:32:26.749210] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop In log after restart: [2015-08-21 13:35:29.934968] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:35:29.941086] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop [2015-08-21 13:35:30.581835] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-08-21 13:35:32.937617] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:35:32.943770] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop [2015-08-21 13:35:34.699874] W [glusterfsd.c:1219:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fee4bafc182] -->/usr/sbin/glusterd(g lusterfs_sigwaiter+0xd5) [0x7fee4c8396f5] -->/usr/sbin/glusterd(cleanup_and_exit+0x69) [0x7fee4c839589] ) 0-: received signum (15), shutting down [2015-08-21 13:35:34.700241] E [rpcsvc.c:1390:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap [2015-08-21 13:35:34.700260] E [rpcsvc.c:1516:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed [2015-08-21 13:35:34.700269] E [rpcsvc.c:1554:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2015-08-21 13:35:34.700507] E [rpcsvc.c:1390:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap [2015-08-21 13:35:34.700527] E [rpcsvc.c:1516:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed [2015-08-21 13:35:34.700535] E [rpcsvc.c:1554:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: Gluster Handshake, Num: 14398633, Ver: 2 , Port: 0 [2015-08-21 13:35:36.711372] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.3 (args: /usr/sbin /glusterd -p /var/run/glusterd.pid) [2015-08-21 13:35:36.714487] I [MSGID: 106478] [glusterd.c:1346:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-08-21 13:35:36.714522] I [MSGID: 106479] [glusterd.c:1395:init] 0-management: Using /var/lib/glusterd as working directory [2015-08-21 13:35:36.717424] I [socket.c:3968:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED [2015-08-21 13:35:36.717446] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED [2015-08-21 13:35:36.717455] I [socket.c:3988:socket_init] 0-socket.management: using private polling thread [2015-08-21 13:35:36.718591] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2015-08-21 13:35:36.718616] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device [2015-08-21 13:35:36.718627] W [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-08-21 13:35:36.718672] W [rpcsvc.c:1597:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-08-21 13:35:36.718690] E [MSGID: 106243] [glusterd.c:1619:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2015-08-21 13:35:36.718799] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED [2015-08-21 13:35:38.987950] I [MSGID: 106513] [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30703 [2015-08-21 13:35:39.388369] I [MSGID: 106498] [glusterd-handler.c:3570:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-08-21 13:35:39.388463] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-08-21 13:35:39.388588] I [socket.c:3968:socket_init] 0-management: SSL support on the I/O path is ENABLED [2015-08-21 13:35:39.388601] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED [2015-08-21 13:35:39.388610] I [socket.c:3988:socket_init] 0-management: using private polling thread [2015-08-21 13:35:39.390521] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument [2015-08-21 13:35:39.390542] E [socket.c:3019:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2015-08-21 13:35:39.397966] I [MSGID: 106544] [glusterd.c:156:glusterd_uuid_init] 0-management: retrieved UUID: c90033d0-1253-40bf-b700-5f65648940dd Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option transport.socket.ssl-enabled on 4: option rpc-auth.auth-glusterfs on 5: option rpc-auth.auth-unix on 6: option rpc-auth.auth-null on 7: option rpc-auth-allow-insecure on 8: option transport.socket.listen-backlog 128 9: option ping-timeout 30 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2015-08-21 13:35:39.398830] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-08-21 13:35:39.404083] I [socket.c:401:ssl_setup_connection] 0-management: peer CN = dist-master04 [2015-08-21 13:35:42.215075] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-08-21 13:35:45.552171] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04 [2015-08-21 13:35:45.786872] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894, host: dist-int-master04.xxx, port: 0 [2015-08-21 13:35:45.789585] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2015-08-21 13:35:45.789699] I [socket.c:3971:socket_init] 0-nfs: SSL support for glusterd is ENABLED [2015-08-21 13:35:45.790201] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2015-08-21 13:35:45.798535] W [socket.c:3063:socket_connect] 0-nfs: Ignore failed connection attempt on , (No such file or directory) [2015-08-21 13:35:45.798614] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2015-08-21 13:35:45.798698] I [socket.c:3971:socket_init] 0-glustershd: SSL support for glusterd is ENABLED [2015-08-21 13:35:45.799560] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2015-08-21 13:35:45.801467] W [socket.c:3063:socket_connect] 0-glustershd: Ignore failed connection attempt on , (No such file or directory) [2015-08-21 13:35:45.801557] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2015-08-21 13:35:45.801655] I [socket.c:3971:socket_init] 0-quotad: SSL support for glusterd is ENABLED [2015-08-21 13:35:45.802201] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2015-08-21 13:35:45.802267] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2015-08-21 13:35:45.802352] I [socket.c:3971:socket_init] 0-bitd: SSL support for glusterd is ENABLED [2015-08-21 13:35:45.802825] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2015-08-21 13:35:45.802892] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2015-08-21 13:35:45.802976] I [socket.c:3971:socket_init] 0-scrub: SSL support for glusterd is ENABLED [2015-08-21 13:35:45.803458] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2015-08-21 13:35:45.803929] [run.c:190:runner_log] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7ff79c3e7f46] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0xfc)[0x7ff79c439abc] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_volume_start_glusterfs+0x736)[0x7ff7974e3996] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_brick_start+0xfe)[0x7ff7974e8eae] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_restart_bricks+0xed)[0x7ff7974e90ad] ))))) 0-: Starting GlusterFS: /usr/sbin/glusterfsd -s dist-int-master03.xxx --volfile-id repofiles.dist-int-master03.xxx.storage-gluster_brick_repofiles -p /var/lib/glusterd/vols/repofiles/run/dist-int-master03.xxx-storage-gluster_brick_repofiles.pid -S /var/run/gluster/1603c345d3b0bb50423077d097db360b.socket --brick-name /storage/gluster_brick_repofiles -l /var/log/glusterfs/bricks/storage-gluster_brick_repofiles.log --xlator-option *-posix.glusterd-uuid=c90033d0-1253-40bf-b700-5f65648940dd --brick-port 49152 --xlator-option repofiles-server.listen-port=49152 [2015-08-21 13:35:45.806005] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-08-21 13:35:45.806118] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED [2015-08-21 13:35:45.806698] I [MSGID: 106492] [glusterd-handler.c:2706:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894 [2015-08-21 13:35:45.806743] I [MSGID: 106502] [glusterd-handler.c:2751:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-08-21 13:35:45.806851] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-08-21 13:35:45.806966] I [MSGID: 106006] [glusterd-svc-mgmt.c:319:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd. [2015-08-21 13:35:45.807016] W [socket.c:642:__socket_rwv] 0-glustershd: readv on /var/run/gluster/fb1f2970d6c956f3831f6976e5c51a0e.socket failed (Invalid argument) [2015-08-21 13:35:45.807035] I [MSGID: 106006] [glusterd-svc-mgmt.c:319:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. [2015-08-21 13:35:45.812411] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/1603c345d3b0bb50423077d097db360b.socket failed (Invalid argument) [2015-08-21 13:35:45.817497] I [MSGID: 106005] [glusterd-handler.c:4899:__glusterd_brick_rpc_notify] 0-management: Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles has disconnected from glusterd. [2015-08-21 13:35:45.817545] I [MSGID: 106163] [glusterd-handshake.c:1198:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30703 [2015-08-21 13:35:45.825665] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03 [2015-08-21 13:35:45.825763] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03 [2015-08-21 13:35:45.825829] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03 [2015-08-21 13:35:45.890352] I [MSGID: 106490] [glusterd-handler.c:2530:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894 [2015-08-21 13:35:45.900221] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop [2015-08-21 13:35:45.902916] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03 [2015-08-21 13:35:45.942628] I [MSGID: 106493] [glusterd-handler.c:3771:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dist-int-master04.xxx (0), ret: 0 [2015-08-21 13:35:45.948258] I [MSGID: 106143] [glusterd-pmap.c:229:pmap_registry_bind] 0-pmap: adding brick /storage/gluster_brick_repofiles on port 49152 [2015-08-21 13:35:45.949411] I [MSGID: 106493] [glusterd-rpc-ops.c:692:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894 On gluster 3.6.4 and ubuntu 12.04 it look like on node where I has started glusterd first gluster peer status return "peer status: failed Connection failed. Please check if gluster daemon is operational." But glusterd started: # ps aux | grep gluster root 731287 0.0 0.0 45740 10432 ? Ss 13:51 0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid root 731288 0.0 0.0 158760 12268 ? Ssl 13:51 0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid root 731309 0.0 0.0 11100 940 pts/10 S+ 13:51 0:00 grep --color=auto gluster This error disappeared after start glusterd service on second node. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150821/5b04441d/attachment.html>