Leonid Isaev
2017-Aug-06 03:54 UTC
[Gluster-users] [3.11.2] Bricks disconnect from gluster with 0-transport: EPOLLERR
Hi, I have a distributed volume which runs on Fedora 26 systems with glusterfs 3.11.2 from gluster.org repos: ---------- [root at taupo ~]# glusterd --version glusterfs 3.11.2 gluster> volume info gv2 Volume Name: gv2 Type: Distribute Volume ID: 6b468f43-3857-4506-917c-7eaaaef9b6ee Status: Started Snapshot Count: 0 Number of Bricks: 6 Transport-type: tcp Bricks: Brick1: kiwi:/srv/gluster/gv2/brick1/gvol Brick2: kiwi:/srv/gluster/gv2/brick2/gvol Brick3: taupo:/srv/gluster/gv2/brick1/gvol Brick4: fox:/srv/gluster/gv2/brick1/gvol Brick5: fox:/srv/gluster/gv2/brick2/gvol Brick6: logan:/srv/gluster/gv2/brick1/gvol Options Reconfigured: performance.readdir-ahead: on nfs.disable: on gluster> volume status gv2 Status of volume: gv2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick kiwi:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1128 Brick kiwi:/srv/gluster/gv2/brick2/gvol 49153 0 Y 1134 Brick taupo:/srv/gluster/gv2/brick1/gvol N/A N/A N N/A Brick fox:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1169 Brick fox:/srv/gluster/gv2/brick2/gvol 49153 0 Y 1175 Brick logan:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1003 ---------- The machine in question is TAUPO which has one brick that refuses to connect to the cluster. All installations were migrated from glusterfs 3.8.14 on Fedora 24: I simply rsync'ed /var/lib/glusterd to new systems. On all other machines glusterd starts fine and all bricks come up. Hence I suspect a race condition somewhere. The glusterd.log file (attached) shows that the brick connects, and then suddenly disconnects from the cluster: ---------- [2017-08-06 03:12:38.536409] I [glusterd-utils.c:5468:glusterd_brick_start] 0-management: discovered already-running brick /srv/gluster/gv2/brick1/gvol [2017-08-06 03:12:38.536414] I [MSGID: 106143] [glusterd-pmap.c:279:pmap_registry_bind] 0-pmap: adding brick /srv/gluster/gv2/brick1/gvol on port 49153 [2017-08-06 03:12:38.536427] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:38.536500] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-08-06 03:12:38.536556] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-08-06 03:12:38.536616] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.584598] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.599340] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:38.613745] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. ---------- I checked that cluster.brick-multiplex is off. How can I debug this further? Thanks in advance, -- Leonid Isaev -------------- next part -------------- [2017-08-06 03:12:36.559795] I [MSGID: 100030] [glusterfsd.c:2476:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) [2017-08-06 03:12:36.564906] I [MSGID: 106478] [glusterd.c:1422:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-08-06 03:12:36.564936] I [MSGID: 106479] [glusterd.c:1469:init] 0-management: Using /var/lib/glusterd as working directory [2017-08-06 03:12:36.567903] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.11.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2017-08-06 03:12:36.567916] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2017-08-06 03:12:36.567922] W [rpcsvc.c:1660:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-08-06 03:12:36.567929] E [MSGID: 106243] [glusterd.c:1693:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-08-06 03:12:36.568821] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory] [2017-08-06 03:12:36.573114] I [MSGID: 106513] [glusterd-store.c:2193:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30800 [2017-08-06 03:12:36.691076] I [MSGID: 106498] [glusterd-handler.c:3602:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 The message "I [MSGID: 106498] [glusterd-handler.c:3602:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0" repeated 6 times between [2017-08-06 03:12:36.691076] and [2017-08-06 03:12:36.713022] [2017-08-06 03:12:36.713040] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.713057] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.713102] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.715561] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.715573] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.715612] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.716599] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.716610] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.716647] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.717624] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.717634] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.717672] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.718664] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.718673] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.718710] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.719693] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.719703] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.719740] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.720718] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.720728] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.720764] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.736985] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 4fc4d794-e463-4424-8531-13c639fedfe9 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-08-06 03:12:36.737096] I [MSGID: 101190] [event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-08-06 03:12:38.525090] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9, host: 128.138.140.58, port: 0 [2017-08-06 03:12:38.535858] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-08-06 03:12:38.535924] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-08-06 03:12:38.535934] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-08-06 03:12:38.535941] I [MSGID: 106600] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed [2017-08-06 03:12:38.535956] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-08-06 03:12:38.536002] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-08-06 03:12:38.536008] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-08-06 03:12:38.536023] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-08-06 03:12:38.536088] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-08-06 03:12:38.536096] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-08-06 03:12:38.536109] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-08-06 03:12:38.536175] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-08-06 03:12:38.536182] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-08-06 03:12:38.536196] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-08-06 03:12:38.536257] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-08-06 03:12:38.536267] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-08-06 03:12:38.536409] I [glusterd-utils.c:5468:glusterd_brick_start] 0-management: discovered already-running brick /srv/gluster/gv2/brick1/gvol [2017-08-06 03:12:38.536414] I [MSGID: 106143] [glusterd-pmap.c:279:pmap_registry_bind] 0-pmap: adding brick /srv/gluster/gv2/brick1/gvol on port 49153 [2017-08-06 03:12:38.536427] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:38.536500] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-08-06 03:12:38.536556] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-08-06 03:12:38.536616] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.584598] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.599340] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:38.613745] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:12:38.613783] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.614144] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:38.622382] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.636536] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.58 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:38.654277] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225, host: 128.138.140.223, port: 0 [2017-08-06 03:12:38.661217] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.719619] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.719700] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.719719] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.719732] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.719749] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.789917] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:38.800810] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.808300] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.223 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:38.821959] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.821982] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.873314] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a, host: takahe.colorado.edu, port: 0 [2017-08-06 03:12:38.880937] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:38.880974] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.929723] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.975223] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5, host: 128.138.140.48, port: 0 [2017-08-06 03:12:38.982264] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:38.982651] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:38.993222] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:38.993246] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.061315] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:39.067929] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to takahe.colorado.edu (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.089534] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:39.089579] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5, host: 128.138.140.227, port: 0 [2017-08-06 03:12:39.100236] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:39.100258] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.151399] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:39.158128] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.158154] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c, host: 128.138.140.154, port: 0 [2017-08-06 03:12:39.165171] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.185496] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.213247] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:39.223893] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.48 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.242085] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:39.242115] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.290703] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.290740] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.290751] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b, host: pukeko.colorado.edu, port: 0 [2017-08-06 03:12:39.297783] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:39.297806] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.350544] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:39.357462] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:39.357480] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:39.364344] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.405649] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:39.415886] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.426685] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.227 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.443877] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:39.443916] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.455961] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.154 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.473703] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.493665] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.518102] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:39.525093] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.525112] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.576708] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.576744] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.576761] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:39.587450] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to pukeko.colorado.edu (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.609156] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:39.616031] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.658011] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:41.583404] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:41.602493] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:12:44.578101] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:47.582732] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:50.583198] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:53.584918] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:56.583692] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:59.584040] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:02.585104] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:05.583899] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:08.583690] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:11.585503] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:14.585056] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:17.586200] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:20.586345] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:23.586469] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:26.584786] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:29.585524] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:32.587889] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:35.588441] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:38.588356] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:41.589152] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:44.589795] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:47.590841] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:50.590175] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:53.591069] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:56.589404] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:59.591044] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:02.592658] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:05.594516] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:08.593540] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:11.592057] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:14.592725] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:17.593480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:20.593690] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:23.593413] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:26.592297] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:29.596041] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:32.595171] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:35.594801] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 38 times between [2017-08-06 03:12:41.602493] and [2017-08-06 03:14:35.612349] [2017-08-06 03:14:38.595965] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:38.612558] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:14:41.594507] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:44.595193] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:47.596333] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:50.595273] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:53.596662] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:56.596728] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:59.596660] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:02.598157] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:05.599272] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:08.598349] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:11.597804] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:14.598448] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:17.599280] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:20.600455] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:23.599751] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:26.600506] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:29.601173] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:32.601107] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:35.601696] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:38.602591] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:41.601936] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:44.599341] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:47.603964] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:50.604358] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:53.600733] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:56.601888] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:59.605305] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:02.601723] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:05.605308] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:08.605716] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:11.606025] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:12.478137] I [MSGID: 100030] [glusterfsd.c:2476:main] 0-glusterd: Started running glusterd version 3.11.2 (args: glusterd) [2017-08-06 03:16:12.484758] I [MSGID: 106478] [glusterd.c:1422:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-08-06 03:16:12.484797] I [MSGID: 106479] [glusterd.c:1469:init] 0-management: Using /var/lib/glusterd as working directory [2017-08-06 03:16:12.489459] E [socket.c:796:__socket_server_bind] 0-socket.management: binding to failed: Address already in use [2017-08-06 03:16:12.489485] E [socket.c:799:__socket_server_bind] 0-socket.management: Port is already in use [2017-08-06 03:16:12.489503] W [rpcsvc.c:1667:rpcsvc_create_listener] 0-rpc-service: listening on transport failed [2017-08-06 03:16:12.489515] E [MSGID: 106243] [glusterd.c:1686:init] 0-management: creation of listener failed [2017-08-06 03:16:12.489535] E [MSGID: 101019] [xlator.c:503:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2017-08-06 03:16:12.489546] E [MSGID: 101066] [graph.c:325:glusterfs_graph_init] 0-management: initializing translator failed [2017-08-06 03:16:12.489558] E [MSGID: 101176] [graph.c:696:glusterfs_graph_activate] 0-graph: init failed [2017-08-06 03:16:12.490488] W [glusterfsd.c:1331:cleanup_and_exit] (-->glusterd(glusterfs_volumes_init+0xf7) [0x5604900d02c7] -->glusterd(glusterfs_process_volfp+0x183) [0x5604900d0183] -->glusterd(cleanup_and_exit+0x5a) [0x5604900ccb7a] ) 0-: received signum (-1), shutting down [2017-08-06 03:16:14.605509] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:17.604803] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:20.608876] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:23.603836] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:26.607734] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:29.609353] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:32.609900] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:35.608544] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:14:38.612558] and [2017-08-06 03:16:35.625771] [2017-08-06 03:16:38.609485] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:38.625303] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:16:41.610589] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:44.610563] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:47.610414] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:50.611649] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:53.609751] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:56.611203] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:59.613105] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:02.611623] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:05.611041] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:08.611022] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:11.614079] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:14.613345] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:17.616445] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:20.613447] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:23.614405] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:26.615806] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:29.616985] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:32.617296] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:35.617071] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:38.618057] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:41.613480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:44.618631] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:47.618622] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:50.619449] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:53.619548] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:56.619893] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:59.620349] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:02.620491] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:05.622153] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:08.618072] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:11.620160] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:14.620953] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:17.623081] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:20.624232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:23.623405] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:26.621556] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:29.622174] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:32.624101] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:35.624159] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:16:38.625303] and [2017-08-06 03:18:35.643628] [2017-08-06 03:18:38.623154] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:38.638874] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:18:41.625459] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:44.625504] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:47.624487] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:50.625552] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:53.627476] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:56.626804] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:59.625783] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:02.625682] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:05.628513] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:08.629161] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:11.628698] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:14.627382] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:17.628452] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:20.624656] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:23.629982] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:26.630140] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:29.631511] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:32.631489] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:33.030413] I [MSGID: 106488] [glusterd-handler.c:1547:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-08-06 03:19:35.630630] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:38.628652] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:41.633954] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:44.629301] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:47.634167] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:50.632232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:53.630952] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:56.630006] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:56.645139] I [MSGID: 106499] [glusterd-handler.c:4302:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv2 [2017-08-06 03:19:59.635084] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:02.636198] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:05.635633] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:08.636366] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:11.637104] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:14.635935] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:17.638419] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:20.638921] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:23.638394] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:26.637497] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:29.638240] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:32.640077] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:35.639375] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106488] [glusterd-handler.c:1547:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2017-08-06 03:19:33.030413] and [2017-08-06 03:19:33.032367] The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:18:38.638874] and [2017-08-06 03:20:35.657497] [2017-08-06 03:20:38.638998] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:38.655345] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:20:41.639249] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:44.641530] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:47.640078] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:50.639738] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:53.642584] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:56.639418] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:59.641479] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:02.641310] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:05.642192] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:08.643265] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:11.645152] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:14.643147] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:17.642636] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:20.644147] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:23.644211] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:26.645538] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:29.646314] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:32.644232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:35.646143] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:38.646014] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:41.647145] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:44.645480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:47.646920] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:50.645961] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:53.648376] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:56.647314] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:59.647240] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:02.649525] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:05.650677] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:08.649927] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:11.651573] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:14.650324] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:17.651756] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:20.648295] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:23.652112] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:26.651214] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:29.649616] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:32.653143] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:35.651358] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:20:38.655345] and [2017-08-06 03:22:35.667266] [2017-08-06 03:22:38.653869] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:38.670345] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:22:41.654410] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:44.652675] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:47.654514] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:50.654210] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:53.654301] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:56.655478] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:59.654460] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:02.657118] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:05.655457] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:08.656861] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:11.656538] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:14.657619] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:17.659736] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:20.655873] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:23.658645] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:26.657419] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:29.659674] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:32.659161] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:35.660123] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:38.660054] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:41.660019] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:44.659646] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:47.660895] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:50.661367] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:53.663192] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:56.662402] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:59.661518] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:02.663579] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:05.664634] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:08.663792] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:11.664684] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:14.663878] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:17.665416] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:20.664696] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:23.662762] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:26.663935] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 36 times between [2017-08-06 03:22:38.670345] and [2017-08-06 03:24:26.680441] [2017-08-06 03:24:28.820960] W [glusterfsd.c:1331:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x736d) [0x7f8adc8c036d] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xf5) [0x55ceb1ffcd85] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a) [0x55ceb1ffcb7a] ) 0-: received signum (15), shutting down
Ben Werthmann
2017-Sep-13 19:28 UTC
[Gluster-users] [3.11.2] Bricks disconnect from gluster with 0-transport: EPOLLERR
I ran into something like this in 3.10.4 and filed two bugs for it: https://bugzilla.redhat.com/show_bug.cgi?id=1491059 https://bugzilla.redhat.com/show_bug.cgi?id=1491060 Please see the above bugs for full detail. In summary, my issue was related to glusterd's pid handling of pid files when is starts self-heal and bricks. The issues are: a. brick pid file leaves stale pid and brick fails to start when glusterd is started. pid files are stored in `/var/lib/glusterd` which persists across reboots. When glusterd is started (or restarted or host rebooted) and the pid of any process matching the pid in the brick pid file, brick fails to start. b. self-heal-deamon pid file leave stale pid and indiscriminately kills pid when glusterd is started. pid files are stored in `/var/lib/glusterd` which persists across reboots. When glusterd is started (or restarted or host rebooted) the pid of any process matching the pid in the shd pid file is killed. due to the nature of these bugs sometimes bricks/shd will start, sometimes they will not, restart success may be intermittent. This bug is most likely to occur when services were running with a low pid, then the host is rebooted since reboots tend to densely group pids in lower pid numbers. You might also see it if you have high pid churn due to short lived processes. In the case of self-heal daemon, you may also see other processes "randomly" being terminated. resulting in: 1a. pid file /var/lib/glusterd/glustershd/run/glustershd.pid remains after shd is stopped 2a. glusterd kills any process number in the stale shd pid file. 1b. brick pid file(s) remain after brick is stopped 2b. glusterd fails to start brick when the pid in a pid file matches any running process Workaround: in our automation, when we stop all gluster processes (reboot, upgrade, etc.) we ensure all processes are stopped and then cleanup the pids with: 'find /var/lib/glusterd/ -name '*pid' -delete' This is not a complete solution, but works in our most critical times. We may develop something more complete if the bug is not addressed promptly. On Sat, Aug 5, 2017 at 11:54 PM, Leonid Isaev < leonid.isaev at jila.colorado.edu> wrote:> Hi, > > I have a distributed volume which runs on Fedora 26 systems with > glusterfs 3.11.2 from gluster.org repos: > ---------- > [root at taupo ~]# glusterd --version > glusterfs 3.11.2 > > gluster> volume info gv2 > Volume Name: gv2 > Type: Distribute > Volume ID: 6b468f43-3857-4506-917c-7eaaaef9b6ee > Status: Started > Snapshot Count: 0 > Number of Bricks: 6 > Transport-type: tcp > Bricks: > Brick1: kiwi:/srv/gluster/gv2/brick1/gvol > Brick2: kiwi:/srv/gluster/gv2/brick2/gvol > Brick3: taupo:/srv/gluster/gv2/brick1/gvol > Brick4: fox:/srv/gluster/gv2/brick1/gvol > Brick5: fox:/srv/gluster/gv2/brick2/gvol > Brick6: logan:/srv/gluster/gv2/brick1/gvol > Options Reconfigured: > performance.readdir-ahead: on > nfs.disable: on > > gluster> volume status gv2 > Status of volume: gv2 > Gluster process TCP Port RDMA Port Online > Pid > ------------------------------------------------------------ > ------------------ > Brick kiwi:/srv/gluster/gv2/brick1/gvol 49152 0 Y > 1128 > Brick kiwi:/srv/gluster/gv2/brick2/gvol 49153 0 Y > 1134 > Brick taupo:/srv/gluster/gv2/brick1/gvol N/A N/A N > N/A > Brick fox:/srv/gluster/gv2/brick1/gvol 49152 0 Y > 1169 > Brick fox:/srv/gluster/gv2/brick2/gvol 49153 0 Y > 1175 > Brick logan:/srv/gluster/gv2/brick1/gvol 49152 0 Y > 1003 > ---------- > > The machine in question is TAUPO which has one brick that refuses to > connect to > the cluster. All installations were migrated from glusterfs 3.8.14 on > Fedora > 24: I simply rsync'ed /var/lib/glusterd to new systems. On all other > machines > glusterd starts fine and all bricks come up. Hence I suspect a race > condition > somewhere. The glusterd.log file (attached) shows that the brick connects, > and > then suddenly disconnects from the cluster: > ---------- > [2017-08-06 03:12:38.536409] I [glusterd-utils.c:5468:glusterd_brick_start] > 0-management: discovered already-running brick /srv/gluster/gv2/brick1/gvol > [2017-08-06 03:12:38.536414] I [MSGID: 106143] [glusterd-pmap.c:279:pmap_registry_bind] > 0-pmap: adding brick /srv/gluster/gv2/brick1/gvol on port 49153 > [2017-08-06 03:12:38.536427] I [rpc-clnt.c:1059:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2017-08-06 03:12:38.536500] I [rpc-clnt.c:1059:rpc_clnt_connection_init] > 0-snapd: setting frame-timeout to 600 > [2017-08-06 03:12:38.536556] I [rpc-clnt.c:1059:rpc_clnt_connection_init] > 0-snapd: setting frame-timeout to 600 > [2017-08-06 03:12:38.536616] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] > 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff- > b8d85fd51da9 > [2017-08-06 03:12:38.584598] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] > 0-management: Received my uuid as Friend > [2017-08-06 03:12:38.599340] I [socket.c:2474:socket_event_handler] > 0-transport: EPOLLERR - disconnecting now > [2017-08-06 03:12:38.613745] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] > 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected > from glusterd. > ---------- > > I checked that cluster.brick-multiplex is off. How can I debug this > further? > > Thanks in advance, > -- > Leonid Isaev > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170913/541a2891/attachment.html>