Dear Group. Many days ago I had to restart one of the servers and after that, it doesn?t heal anymore. I tried lots of thinks, no answer on Internet and the server stills down by now. Could please lend me a hand. Logs reports [client-handshake.c:1760:client_query_portmap_cbk] 0-data-cotelser-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. But brick processes run okay without any problems. root at ipred01:/var/log/glusterfs# gluster volume status Status of volume: data-cotelser Gluster process Port Online Pid ---------------------------------------------------------------------------- -- Brick cotelser01:/data/sdb1/data-cotelser 49152 Y 2188 Brick ipred01:/data/sdb1/data-cotelser 49154 Y 2138 Brick cotelser01:/data/sdc1/data-cotelser 49153 Y 2201 Brick ipred01:/data/sdc1/data-cotelser 49155 Y 2152 NFS Server on localhost 2049 Y 2162 Self-heal Daemon on localhost N/A Y 2167 NFS Server on cotelser01 2049 Y 9893 Self-heal Daemon on cotelser01 N/A Y 9900 Task Status of Volume data-cotelser ---------------------------------------------------------------------------- -- There are no active volume tasks Status of volume: data-ipred Gluster process Port Online Pid ---------------------------------------------------------------------------- -- Brick cotelser01:/data/sdb1/data-ipred 49154 Y 2202 Brick ipred01:/data/sdb1/data-ipred 49156 Y 2140 Brick cotelser01:/data/sdc1/data-ipred 49155 Y 2212 Brick ipred01:/data/sdc1/data-ipred 49157 Y 2142 NFS Server on localhost 2049 Y 2162 Self-heal Daemon on localhost N/A Y 2167 NFS Server on cotelser01 2049 Y 9893 Self-heal Daemon on cotelser01 N/A Y 9900 Task Status of Volume data-ipred ---------------------------------------------------------------------------- -- There are no active volume tasks root at ipred01:/var/log/glusterfs# LOG glustershd.log (bellow) [2015-08-19 21:19:38.200445] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/17c62c5aeddf9c73dfed4aa2d7407aaa.socket --xlator-option *replicate*.node-uuid=c53ad0c0-3940-443a-9687-597f6a14b783) [2015-08-19 21:19:38.201761] I [socket.c:3578:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled [2015-08-19 21:19:38.201778] I [socket.c:3593:socket_init] 0-socket.glusterfsd: using system polling thread [2015-08-19 21:19:38.201873] I [socket.c:3578:socket_init] 0-glusterfs: SSL support is NOT enabled [2015-08-19 21:19:38.201882] I [socket.c:3593:socket_init] 0-glusterfs: using system polling thread [2015-08-19 21:19:38.679468] I [graph.c:254:gf_add_cmdline_options] 0-data-cotelser-replicate-1: adding option 'node-uuid' for volume 'data-cotelser-replicate-1' with value 'c53ad0c0-3940-443a-9687-597f6a14b783' [2015-08-19 21:19:38.679484] I [graph.c:254:gf_add_cmdline_options] 0-data-cotelser-replicate-0: adding option 'node-uuid' for volume 'data-cotelser-replicate-0' with value 'c53ad0c0-3940-443a-9687-597f6a14b783' [2015-08-19 21:19:38.679490] I [graph.c:254:gf_add_cmdline_options] 0-data-ipred-replicate-1: adding option 'node-uuid' for volume 'data-ipred-replicate-1' with value 'c53ad0c0-3940-443a-9687-597f6a14b783' [2015-08-19 21:19:38.679500] I [graph.c:254:gf_add_cmdline_options] 0-data-ipred-replicate-0: adding option 'node-uuid' for volume 'data-ipred-replicate-0' with value 'c53ad0c0-3940-443a-9687-597f6a14b783' [2015-08-19 21:19:38.682874] I [socket.c:3578:socket_init] 0-data-cotelser-client-3: SSL support is NOT enabled [2015-08-19 21:19:38.682904] I [socket.c:3593:socket_init] 0-data-cotelser-client-3: using system polling thread [2015-08-19 21:19:38.683110] I [socket.c:3578:socket_init] 0-data-cotelser-client-2: SSL support is NOT enabled [2015-08-19 21:19:38.683117] I [socket.c:3593:socket_init] 0-data-cotelser-client-2: using system polling thread [2015-08-19 21:19:38.683314] I [socket.c:3578:socket_init] 0-data-cotelser-client-1: SSL support is NOT enabled [2015-08-19 21:19:38.683321] I [socket.c:3593:socket_init] 0-data-cotelser-client-1: using system polling thread [2015-08-19 21:19:38.683516] I [socket.c:3578:socket_init] 0-data-cotelser-client-0: SSL support is NOT enabled [2015-08-19 21:19:38.683523] I [socket.c:3593:socket_init] 0-data-cotelser-client-0: using system polling thread [2015-08-19 21:19:38.686272] I [socket.c:3578:socket_init] 0-data-ipred-client-3: SSL support is NOT enabled [2015-08-19 21:19:38.686295] I [socket.c:3593:socket_init] 0-data-ipred-client-3: using system polling thread [2015-08-19 21:19:38.686446] I [socket.c:3578:socket_init] 0-data-ipred-client-2: SSL support is NOT enabled [2015-08-19 21:19:38.686451] I [socket.c:3593:socket_init] 0-data-ipred-client-2: using system polling thread [2015-08-19 21:19:38.686599] I [socket.c:3578:socket_init] 0-data-ipred-client-1: SSL support is NOT enabled [2015-08-19 21:19:38.686604] I [socket.c:3593:socket_init] 0-data-ipred-client-1: using system polling thread [2015-08-19 21:19:38.686750] I [socket.c:3578:socket_init] 0-data-ipred-client-0: SSL support is NOT enabled [2015-08-19 21:19:38.686755] I [socket.c:3593:socket_init] 0-data-ipred-client-0: using system polling thread [2015-08-19 21:19:38.686789] I [client.c:2294:notify] 0-data-cotelser-client-0: parent translators are ready, attempting connect on transport [2015-08-19 21:19:38.687080] I [client.c:2294:notify] 0-data-cotelser-client-1: parent translators are ready, attempting connect on transport [2015-08-19 21:19:38.687269] I [client.c:2294:notify] 0-data-cotelser-client-2: parent translators are ready, attempting connect on transport [2015-08-19 21:19:38.687427] I [client.c:2294:notify] 0-data-cotelser-client-3: parent translators are ready, attempting connect on transport [2015-08-19 21:19:38.687591] I [client.c:2294:notify] 0-data-ipred-client-0: parent translators are ready, attempting connect on transport [2015-08-19 21:19:38.687755] I [client.c:2294:notify] 0-data-ipred-client-1: parent translators are ready, attempting connect on transport [2015-08-19 21:19:38.687916] I [client.c:2294:notify] 0-data-ipred-client-2: parent translators are ready, attempting connect on transport [2015-08-19 21:19:38.688066] I [client.c:2294:notify] 0-data-ipred-client-3: parent translators are ready, attempting connect on transport Final graph: +--------------------------------------------------------------------------- ---+ 1: volume data-cotelser-client-0 2: type protocol/client 3: option remote-host cotelser01 4: option remote-subvolume /data/sdb1/data-cotelser 5: option transport-type socket 6: option username 691af945-5b60-4c66-9246-1834143ace9b 7: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c 8: end-volume 9: 10: volume data-cotelser-client-1 11: type protocol/client 12: option remote-host ipred01 13: option remote-subvolume /data/sdb1/data-cotelser 14: option transport-type socket 15: option username 691af945-5b60-4c66-9246-1834143ace9b 16: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c 17: end-volume 18: 19: volume data-cotelser-replicate-0 20: type cluster/replicate 21: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783 22: option background-self-heal-count 0 23: option metadata-self-heal on 24: option data-self-heal on 25: option entry-self-heal on 26: option self-heal-daemon on 27: option iam-self-heal-daemon yes 28: subvolumes data-cotelser-client-0 data-cotelser-client-1 29: end-volume 30: 31: volume data-cotelser-client-2 32: type protocol/client 33: option remote-host cotelser01 34: option remote-subvolume /data/sdc1/data-cotelser 35: option transport-type socket 36: option username 691af945-5b60-4c66-9246-1834143ace9b 37: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c 38: end-volume 39: 40: volume data-cotelser-client-3 41: type protocol/client 42: option remote-host ipred01 43: option remote-subvolume /data/sdc1/data-cotelser 44: option transport-type socket 45: option username 691af945-5b60-4c66-9246-1834143ace9b 46: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c 47: end-volume 48: 49: volume data-cotelser-replicate-1 50: type cluster/replicate 51: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783 52: option background-self-heal-count 0 53: option metadata-self-heal on 54: option data-self-heal on 55: option entry-self-heal on 56: option self-heal-daemon on 57: option iam-self-heal-daemon yes 58: subvolumes data-cotelser-client-2 data-cotelser-client-3 59: end-volume 60: 61: volume data-ipred-client-0 62: type protocol/client 63: option remote-host cotelser01 64: option remote-subvolume /data/sdb1/data-ipred 65: option transport-type socket 66: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1 67: option password c1ab7f12-fe61-43e4-af17-67906d09cea5 68: end-volume 69: 70: volume data-ipred-client-1 71: type protocol/client 72: option remote-host ipred01 73: option remote-subvolume /data/sdb1/data-ipred 74: option transport-type socket 75: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1 76: option password c1ab7f12-fe61-43e4-af17-67906d09cea5 77: end-volume 78: 79: volume data-ipred-replicate-0 80: type cluster/replicate 81: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783 82: option background-self-heal-count 0 83: option metadata-self-heal on 84: option data-self-heal on 85: option entry-self-heal on 86: option self-heal-daemon on 87: option iam-self-heal-daemon yes 88: subvolumes data-ipred-client-0 data-ipred-client-1 89: end-volume 90: 91: volume data-ipred-client-2 92: type protocol/client 93: option remote-host cotelser01 94: option remote-subvolume /data/sdc1/data-ipred 95: option transport-type socket 96: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1 97: option password c1ab7f12-fe61-43e4-af17-67906d09cea5 98: end-volume 99: 100: volume data-ipred-client-3 101: type protocol/client 102: option remote-host ipred01 103: option remote-subvolume /data/sdc1/data-ipred 104: option transport-type socket 105: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1 106: option password c1ab7f12-fe61-43e4-af17-67906d09cea5 107: end-volume 108: 109: volume data-ipred-replicate-1 110: type cluster/replicate 111: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783 112: option background-self-heal-count 0 113: option metadata-self-heal on 114: option data-self-heal on 115: option entry-self-heal on 116: option self-heal-daemon on 117: option iam-self-heal-daemon yes 118: subvolumes data-ipred-client-2 data-ipred-client-3 119: end-volume 120: 121: volume glustershd 122: type debug/io-stats 123: subvolumes data-cotelser-replicate-0 data-cotelser-replicate-1 data-ipred-replicate-0 data-ipred-replicate-1 124: end-volume 125: +--------------------------------------------------------------------------- ---+ [2015-08-19 21:19:38.688612] E [client-handshake.c:1760:client_query_portmap_cbk] 0-data-cotelser-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-08-19 21:19:38.688650] I [client.c:2229:client_rpc_notify] 0-data-cotelser-client-1: disconnected from 172.25.30.2:24007. Client process will keep trying to connect to glusterd until brick's port is available [2015-08-19 21:19:38.688662] E [client-handshake.c:1760:client_query_portmap_cbk] 0-data-ipred-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-08-19 21:19:38.688693] E [client-handshake.c:1760:client_query_portmap_cbk] 0-data-cotelser-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-08-19 21:19:38.688715] I [client.c:2229:client_rpc_notify] 0-data-ipred-client-3: disconnected from 172.25.30.2:24007. Client process will keep trying to connect to glusterd until brick's port is available [2015-08-19 21:19:38.688724] I [client.c:2229:client_rpc_notify] 0-data-cotelser-client-3: disconnected from 172.25.30.2:24007. Client process will keep trying to connect to glusterd until brick's port is available [2015-08-19 21:19:38.688732] E [client-handshake.c:1760:client_query_portmap_cbk] 0-data-ipred-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-08-19 21:19:38.688771] I [client.c:2229:client_rpc_notify] 0-data-ipred-client-1: disconnected from 172.25.30.2:24007. Client process will keep trying to connect to glusterd until brick's port is available [2015-08-19 21:19:38.688860] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-cotelser-client-0: changing port to 49152 (from 0) [2015-08-19 21:19:38.689076] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-cotelser-client-2: changing port to 49153 (from 0) [2015-08-19 21:19:38.689098] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-ipred-client-0: changing port to 49154 (from 0) [2015-08-19 21:19:38.689110] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-ipred-client-2: changing port to 49155 (from 0) [2015-08-19 21:19:38.689762] I [client-handshake.c:1677:select_server_supported_programs] 0-data-cotelser-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:19:38.689824] I [client-handshake.c:1677:select_server_supported_programs] 0-data-ipred-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:19:38.689852] I [client-handshake.c:1677:select_server_supported_programs] 0-data-cotelser-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:19:38.689887] I [client-handshake.c:1677:select_server_supported_programs] 0-data-ipred-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:19:38.690090] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-2: Connected to 172.25.30.1:49153, attached to remote volume '/data/sdc1/data-cotelser'. [2015-08-19 21:19:38.690099] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-2: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:19:38.690121] I [afr-common.c:4131:afr_notify] 0-data-cotelser-replicate-1: Subvolume 'data-cotelser-client-2' came back up; going online. [2015-08-19 21:19:38.690193] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-0: Connected to 172.25.30.1:49152, attached to remote volume '/data/sdb1/data-cotelser'. [2015-08-19 21:19:38.690204] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:19:38.690227] I [afr-common.c:4131:afr_notify] 0-data-cotelser-replicate-0: Subvolume 'data-cotelser-client-0' came back up; going online. [2015-08-19 21:19:38.690283] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-0: Connected to 172.25.30.1:49154, attached to remote volume '/data/sdb1/data-ipred'. [2015-08-19 21:19:38.690296] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:19:38.690346] I [afr-common.c:4131:afr_notify] 0-data-ipred-replicate-0: Subvolume 'data-ipred-client-0' came back up; going online. [2015-08-19 21:19:38.690382] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-0: Server lk version = 1 [2015-08-19 21:19:38.690399] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-2: Connected to 172.25.30.1:49155, attached to remote volume '/data/sdc1/data-ipred'. [2015-08-19 21:19:38.690405] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-2: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:19:38.690433] I [afr-common.c:4131:afr_notify] 0-data-ipred-replicate-1: Subvolume 'data-ipred-client-2' came back up; going online. [2015-08-19 21:19:38.690486] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-2: Server lk version = 1 [2015-08-19 21:19:38.690534] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-0: Server lk version = 1 [2015-08-19 21:19:38.690582] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-2: Server lk version = 1 [2015-08-19 21:19:42.204115] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-cotelser-client-1: changing port to 49154 (from 0) [2015-08-19 21:19:42.204180] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-cotelser-client-3: changing port to 49155 (from 0) [2015-08-19 21:19:42.204637] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-ipred-client-1: changing port to 49156 (from 0) [2015-08-19 21:19:42.204869] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-ipred-client-3: changing port to 49157 (from 0) [2015-08-19 21:19:42.205205] I [client-handshake.c:1677:select_server_supported_programs] 0-data-cotelser-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:19:42.205258] I [client-handshake.c:1677:select_server_supported_programs] 0-data-cotelser-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:19:42.205335] I [client-handshake.c:1677:select_server_supported_programs] 0-data-ipred-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:19:42.205384] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-1: Connected to 172.25.30.2:49154, attached to remote volume '/data/sdb1/data-cotelser'. [2015-08-19 21:19:42.205396] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:19:42.205517] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-3: Connected to 172.25.30.2:49155, attached to remote volume '/data/sdc1/data-cotelser'. [2015-08-19 21:19:42.205539] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-3: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:19:42.205652] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-3: Server lk version = 1 [2015-08-19 21:19:42.205674] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-1: Connected to 172.25.30.2:49156, attached to remote volume '/data/sdb1/data-ipred'. [2015-08-19 21:19:42.205686] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:19:42.205848] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-1: Server lk version = 1 [2015-08-19 21:19:42.206392] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-1: Server lk version = 1 [2015-08-19 21:19:42.206415] I [client-handshake.c:1677:select_server_supported_programs] 0-data-ipred-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:19:42.206605] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-3: Connected to 172.25.30.2:49157, attached to remote volume '/data/sdc1/data-ipred'. [2015-08-19 21:19:42.206622] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-3: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:19:42.206816] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-3: Server lk version = 1 [2015-08-19 21:19:42.207424] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-1: Another crawl is in progress for data-cotelser-client-3 [2015-08-19 21:19:42.208042] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-0: Another crawl is in progress for data-ipred-client-1 [2015-08-19 21:19:42.208066] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-1: Another crawl is in progress for data-ipred-client-3 [2015-08-19 21:19:42.208760] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0: Another crawl is in progress for data-cotelser-client-1 [2015-08-19 21:26:20.555892] W [socket.c:523:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available) [2015-08-19 21:26:21.647156] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f55d518004d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f55d58460a4] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x4073bd]))) 0-: received signum (15), shutting down [2015-08-19 21:27:25.946841] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/17c62c5aeddf9c73dfed4aa2d7407aaa.socket --xlator-option *replicate*.node-uuid=c53ad0c0-3940-443a-9687-597f6a14b783) [2015-08-19 21:27:26.162911] I [socket.c:3578:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled [2015-08-19 21:27:26.162929] I [socket.c:3593:socket_init] 0-socket.glusterfsd: using system polling thread [2015-08-19 21:27:26.163006] I [socket.c:3578:socket_init] 0-glusterfs: SSL support is NOT enabled [2015-08-19 21:27:26.163014] I [socket.c:3593:socket_init] 0-glusterfs: using system polling thread [2015-08-19 21:27:31.860980] I [graph.c:254:gf_add_cmdline_options] 0-data-cotelser-replicate-1: adding option 'node-uuid' for volume 'data-cotelser-replicate-1' with value 'c53ad0c0-3940-443a-9687-597f6a14b783' [2015-08-19 21:27:31.860999] I [graph.c:254:gf_add_cmdline_options] 0-data-cotelser-replicate-0: adding option 'node-uuid' for volume 'data-cotelser-replicate-0' with value 'c53ad0c0-3940-443a-9687-597f6a14b783' [2015-08-19 21:27:31.861007] I [graph.c:254:gf_add_cmdline_options] 0-data-ipred-replicate-1: adding option 'node-uuid' for volume 'data-ipred-replicate-1' with value 'c53ad0c0-3940-443a-9687-597f6a14b783' [2015-08-19 21:27:31.861025] I [graph.c:254:gf_add_cmdline_options] 0-data-ipred-replicate-0: adding option 'node-uuid' for volume 'data-ipred-replicate-0' with value 'c53ad0c0-3940-443a-9687-597f6a14b783' [2015-08-19 21:27:31.864670] I [socket.c:3578:socket_init] 0-data-cotelser-client-3: SSL support is NOT enabled [2015-08-19 21:27:31.864698] I [socket.c:3593:socket_init] 0-data-cotelser-client-3: using system polling thread [2015-08-19 21:27:31.864975] I [socket.c:3578:socket_init] 0-data-cotelser-client-2: SSL support is NOT enabled [2015-08-19 21:27:31.864990] I [socket.c:3593:socket_init] 0-data-cotelser-client-2: using system polling thread [2015-08-19 21:27:31.865260] I [socket.c:3578:socket_init] 0-data-cotelser-client-1: SSL support is NOT enabled [2015-08-19 21:27:31.865273] I [socket.c:3593:socket_init] 0-data-cotelser-client-1: using system polling thread [2015-08-19 21:27:31.865496] I [socket.c:3578:socket_init] 0-data-cotelser-client-0: SSL support is NOT enabled [2015-08-19 21:27:31.865511] I [socket.c:3593:socket_init] 0-data-cotelser-client-0: using system polling thread [2015-08-19 21:27:31.868522] I [socket.c:3578:socket_init] 0-data-ipred-client-3: SSL support is NOT enabled [2015-08-19 21:27:31.868540] I [socket.c:3593:socket_init] 0-data-ipred-client-3: using system polling thread [2015-08-19 21:27:31.868721] I [socket.c:3578:socket_init] 0-data-ipred-client-2: SSL support is NOT enabled [2015-08-19 21:27:31.868731] I [socket.c:3593:socket_init] 0-data-ipred-client-2: using system polling thread [2015-08-19 21:27:31.868909] I [socket.c:3578:socket_init] 0-data-ipred-client-1: SSL support is NOT enabled [2015-08-19 21:27:31.868919] I [socket.c:3593:socket_init] 0-data-ipred-client-1: using system polling thread [2015-08-19 21:27:31.869099] I [socket.c:3578:socket_init] 0-data-ipred-client-0: SSL support is NOT enabled [2015-08-19 21:27:31.869108] I [socket.c:3593:socket_init] 0-data-ipred-client-0: using system polling thread [2015-08-19 21:27:31.869144] I [client.c:2294:notify] 0-data-cotelser-client-0: parent translators are ready, attempting connect on transport [2015-08-19 21:27:31.869468] I [client.c:2294:notify] 0-data-cotelser-client-1: parent translators are ready, attempting connect on transport [2015-08-19 21:27:31.869725] I [client.c:2294:notify] 0-data-cotelser-client-2: parent translators are ready, attempting connect on transport [2015-08-19 21:27:31.869958] I [client.c:2294:notify] 0-data-cotelser-client-3: parent translators are ready, attempting connect on transport [2015-08-19 21:27:31.870195] I [client.c:2294:notify] 0-data-ipred-client-0: parent translators are ready, attempting connect on transport [2015-08-19 21:27:31.870418] I [client.c:2294:notify] 0-data-ipred-client-1: parent translators are ready, attempting connect on transport [2015-08-19 21:27:31.870670] I [client.c:2294:notify] 0-data-ipred-client-2: parent translators are ready, attempting connect on transport [2015-08-19 21:27:31.870905] I [client.c:2294:notify] 0-data-ipred-client-3: parent translators are ready, attempting connect on transport Final graph: +--------------------------------------------------------------------------- ---+ 1: volume data-cotelser-client-0 2: type protocol/client 3: option remote-host cotelser01 4: option remote-subvolume /data/sdb1/data-cotelser 5: option transport-type socket 6: option username 691af945-5b60-4c66-9246-1834143ace9b 7: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c 8: end-volume 9: 10: volume data-cotelser-client-1 11: type protocol/client 12: option remote-host ipred01 13: option remote-subvolume /data/sdb1/data-cotelser 14: option transport-type socket 15: option username 691af945-5b60-4c66-9246-1834143ace9b 16: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c 17: end-volume 18: 19: volume data-cotelser-replicate-0 20: type cluster/replicate 21: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783 22: option background-self-heal-count 0 23: option metadata-self-heal on 24: option data-self-heal on 25: option entry-self-heal on 26: option self-heal-daemon on 27: option iam-self-heal-daemon yes 28: subvolumes data-cotelser-client-0 data-cotelser-client-1 29: end-volume 30: 31: volume data-cotelser-client-2 32: type protocol/client 33: option remote-host cotelser01 34: option remote-subvolume /data/sdc1/data-cotelser 35: option transport-type socket 36: option username 691af945-5b60-4c66-9246-1834143ace9b 37: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c 38: end-volume 39: 40: volume data-cotelser-client-3 41: type protocol/client 42: option remote-host ipred01 43: option remote-subvolume /data/sdc1/data-cotelser 44: option transport-type socket 45: option username 691af945-5b60-4c66-9246-1834143ace9b 46: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c 47: end-volume 48: 49: volume data-cotelser-replicate-1 50: type cluster/replicate 51: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783 52: option background-self-heal-count 0 53: option metadata-self-heal on 54: option data-self-heal on 55: option entry-self-heal on 56: option self-heal-daemon on 57: option iam-self-heal-daemon yes 58: subvolumes data-cotelser-client-2 data-cotelser-client-3 59: end-volume 60: 61: volume data-ipred-client-0 62: type protocol/client 63: option remote-host cotelser01 64: option remote-subvolume /data/sdb1/data-ipred 65: option transport-type socket 66: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1 67: option password c1ab7f12-fe61-43e4-af17-67906d09cea5 68: end-volume 69: 70: volume data-ipred-client-1 71: type protocol/client 72: option remote-host ipred01 73: option remote-subvolume /data/sdb1/data-ipred 74: option transport-type socket 75: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1 76: option password c1ab7f12-fe61-43e4-af17-67906d09cea5 77: end-volume 78: 79: volume data-ipred-replicate-0 80: type cluster/replicate 81: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783 82: option background-self-heal-count 0 83: option metadata-self-heal on 84: option data-self-heal on 85: option entry-self-heal on 86: option self-heal-daemon on 87: option iam-self-heal-daemon yes 88: subvolumes data-ipred-client-0 data-ipred-client-1 89: end-volume 90: 91: volume data-ipred-client-2 92: type protocol/client 93: option remote-host cotelser01 94: option remote-subvolume /data/sdc1/data-ipred 95: option transport-type socket 96: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1 97: option password c1ab7f12-fe61-43e4-af17-67906d09cea5 98: end-volume 99: 100: volume data-ipred-client-3 101: type protocol/client 102: option remote-host ipred01 103: option remote-subvolume /data/sdc1/data-ipred 104: option transport-type socket 105: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1 106: option password c1ab7f12-fe61-43e4-af17-67906d09cea5 107: end-volume 108: 109: volume data-ipred-replicate-1 110: type cluster/replicate 111: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783 112: option background-self-heal-count 0 113: option metadata-self-heal on 114: option data-self-heal on 115: option entry-self-heal on 116: option self-heal-daemon on 117: option iam-self-heal-daemon yes 118: subvolumes data-ipred-client-2 data-ipred-client-3 119: end-volume 120: 121: volume glustershd 122: type debug/io-stats 123: subvolumes data-cotelser-replicate-0 data-cotelser-replicate-1 data-ipred-replicate-0 data-ipred-replicate-1 124: end-volume 125: +--------------------------------------------------------------------------- ---+ [2015-08-19 21:27:31.871484] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-cotelser-client-1: changing port to 49154 (from 0) [2015-08-19 21:27:31.871790] E [client-handshake.c:1760:client_query_portmap_cbk] 0-data-cotelser-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-08-19 21:27:31.871826] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-ipred-client-1: changing port to 49156 (from 0) [2015-08-19 21:27:31.871871] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-cotelser-client-0: changing port to 49152 (from 0) [2015-08-19 21:27:31.871894] I [client.c:2229:client_rpc_notify] 0-data-cotelser-client-3: disconnected from 172.25.30.2:24007. Client process will keep trying to connect to glusterd until brick's port is available [2015-08-19 21:27:31.872242] E [client-handshake.c:1760:client_query_portmap_cbk] 0-data-ipred-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-08-19 21:27:31.872269] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-cotelser-client-2: changing port to 49153 (from 0) [2015-08-19 21:27:31.872293] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-ipred-client-0: changing port to 49154 (from 0) [2015-08-19 21:27:31.872343] I [client.c:2229:client_rpc_notify] 0-data-ipred-client-3: disconnected from 172.25.30.2:24007. Client process will keep trying to connect to glusterd until brick's port is available [2015-08-19 21:27:31.872676] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-ipred-client-2: changing port to 49155 (from 0) [2015-08-19 21:27:31.872883] I [client-handshake.c:1677:select_server_supported_programs] 0-data-cotelser-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:27:31.872948] I [client-handshake.c:1677:select_server_supported_programs] 0-data-ipred-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:27:31.873009] I [client-handshake.c:1677:select_server_supported_programs] 0-data-cotelser-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:27:31.873067] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-1: Connected to 172.25.30.2:49156, attached to remote volume '/data/sdb1/data-ipred'. [2015-08-19 21:27:31.873075] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:27:31.873101] I [afr-common.c:4131:afr_notify] 0-data-ipred-replicate-0: Subvolume 'data-ipred-client-1' came back up; going online. [2015-08-19 21:27:31.873144] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-1: Connected to 172.25.30.2:49154, attached to remote volume '/data/sdb1/data-cotelser'. [2015-08-19 21:27:31.873154] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:27:31.873184] I [afr-common.c:4131:afr_notify] 0-data-cotelser-replicate-0: Subvolume 'data-cotelser-client-1' came back up; going online. [2015-08-19 21:27:31.873223] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-1: Server lk version = 1 [2015-08-19 21:27:31.873240] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-1: Server lk version = 1 [2015-08-19 21:27:31.873252] I [client-handshake.c:1677:select_server_supported_programs] 0-data-cotelser-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:27:31.873287] I [client-handshake.c:1677:select_server_supported_programs] 0-data-ipred-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:27:31.873322] I [client-handshake.c:1677:select_server_supported_programs] 0-data-ipred-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:27:31.873347] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-0: Connected to 172.25.30.1:49152, attached to remote volume '/data/sdb1/data-cotelser'. [2015-08-19 21:27:31.873359] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:27:31.873469] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-2: Connected to 172.25.30.1:49153, attached to remote volume '/data/sdc1/data-cotelser'. [2015-08-19 21:27:31.873480] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-2: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:27:31.873499] I [afr-common.c:4131:afr_notify] 0-data-cotelser-replicate-1: Subvolume 'data-cotelser-client-2' came back up; going online. [2015-08-19 21:27:31.873534] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-2: Connected to 172.25.30.1:49155, attached to remote volume '/data/sdc1/data-ipred'. [2015-08-19 21:27:31.873544] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-2: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:27:31.873565] I [afr-common.c:4131:afr_notify] 0-data-ipred-replicate-1: Subvolume 'data-ipred-client-2' came back up; going online. [2015-08-19 21:27:31.873594] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-0: Connected to 172.25.30.1:49154, attached to remote volume '/data/sdb1/data-ipred'. [2015-08-19 21:27:31.873604] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:27:31.873653] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-0: Server lk version = 1 [2015-08-19 21:27:31.873670] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-2: Server lk version = 1 [2015-08-19 21:27:31.873709] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-2: Server lk version = 1 [2015-08-19 21:27:31.873755] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-0: Server lk version = 1 [2015-08-19 21:27:31.873911] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-0: Another crawl is in progress for data-ipred-client-1 [2015-08-19 21:27:31.874180] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-0: Another crawl is in progress for data-ipred-client-1 [2015-08-19 21:27:31.907888] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0: Another crawl is in progress for data-cotelser-client-1 [2015-08-19 21:27:31.908846] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0: Another crawl is in progress for data-cotelser-client-1 [2015-08-19 21:27:35.164347] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-cotelser-client-3: changing port to 49155 (from 0) [2015-08-19 21:27:35.164812] I [client-handshake.c:1677:select_server_supported_programs] 0-data-cotelser-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:27:35.165001] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-3: Connected to 172.25.30.2:49155, attached to remote volume '/data/sdc1/data-cotelser'. [2015-08-19 21:27:35.165014] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-3: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:27:35.165112] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-data-ipred-client-3: changing port to 49157 (from 0) [2015-08-19 21:27:35.165150] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-3: Server lk version = 1 [2015-08-19 21:27:35.165488] I [client-handshake.c:1677:select_server_supported_programs] 0-data-ipred-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-08-19 21:27:35.165698] I [client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-3: Connected to 172.25.30.2:49157, attached to remote volume '/data/sdc1/data-ipred'. [2015-08-19 21:27:35.165712] I [client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-3: Server and Client lk-version numbers are not same, reopening the fds [2015-08-19 21:27:35.165811] I [client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-3: Server lk version = 1 [2015-08-19 21:27:35.166070] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-1: Another crawl is in progress for data-cotelser-client-3 [2015-08-19 21:27:35.166336] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-1: Another crawl is in progress for data-ipred-client-3 [2015-08-19 21:37:32.234149] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0: Another crawl is in progress for data-cotelser-client-1 [2015-08-19 21:37:35.237022] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-1: Another crawl is in progress for data-ipred-client-3 [2015-08-19 21:47:32.322400] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0: Another crawl is in progress for data-cotelser-client-1 [2015-08-19 21:47:35.323534] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-1: Another crawl is in progress for data-ipred-client-3 root at ipred01:/var/log/glusterfs# Saludos cordiales, Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: Descripci?n: cid:image001.jpg at 01CCA872.FCC7C370 NOC IP?RED ? BUNAR SRL Maip? S/N ? CASA CENTRAL 2126 ? General Lagos - SF Fijo: <tel:%2B54-341-4760459> +54-341-4760459 / +54-3402-490890 E-Mail: <mailto:noc at ipred.com.ar> noc at ipred.com.ar Skype: nocipred -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150819/dea57922/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1669 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150819/dea57922/attachment.jpg>