Hi all! I have setup a replicated/distributed gluster cluster 2 x (2 + 1). Centos 7 and gluster version 3.12.6 on server. All machines have two network interfaces and connected to two different networks, 10.10.0.0/16 (with hostnames in /etc/hosts, gluster version 3.12.6) 192.168.67.0/24 (with ldap, gluster version 3.13.1) Gluster cluster was created on the 10.10.0.0/16 net, gluster peer probe ...and so on. All nodes are available on both networks and have the same names on both networks. Now to my problem, the gluster cluster is mounted on multiple clients on the 192.168.67.0/24 net and a process was running on one of the clients, reading and writing to files. At the same time I mounted the cluster on a client on the 10.10.0.0/16 net and started to create and edit files on the cluster. Around the same time the process on the 192-net stopped without any specific errors. Started other processes on the 192-net and continued to make changes on the 10-net and got the same behavior with stopping processes on the 192-net. Is there any known problems with this type of setup? How do I proceed to figure out a solution as I need access from both networks? Following error shows a couple of times on server (systemd -> glusterd): [2018-04-09 11:46:46.254071] C [mem-pool.c:613:mem_pools_init_early] 0-mem-pool: incorrect order of mem-pool initialization (init_done=3) Client logs: Client on 192-net: [2018-04-09 11:35:31.402979] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 5-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached to remote volume '/urd-gds/gluster'. [2018-04-09 11:35:31.403019] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 5-urd-gds-volume-client-1: Server and Client lk-version numbers are not same, reopening the fds [2018-04-09 11:35:31.403051] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 5-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-vo\ lume'. [2018-04-09 11:35:31.403091] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 5-urd-gds-volume-snapd-client: Server and Client lk-version numbers are not same, reopening the fds [2018-04-09 11:35:31.403271] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 5-urd-gds-volume-client-3: Server lk version = 1 [2018-04-09 11:35:31.403325] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 5-urd-gds-volume-client-4: Server lk version = 1 [2018-04-09 11:35:31.403349] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 5-urd-gds-volume-client-0: Server lk version = 1 [2018-04-09 11:35:31.403367] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 5-urd-gds-volume-client-2: Server lk version = 1 [2018-04-09 11:35:31.403616] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 5-urd-gds-volume-client-1: Server lk version = 1 [2018-04-09 11:35:31.403751] I [MSGID: 114057] [client-handshake.c:1484:select_server_supported_programs] 5-urd-gds-volume-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-04-09 11:35:31.404174] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 5-urd-gds-volume-snapd-client: Server lk version = 1 [2018-04-09 11:35:31.405030] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 5-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'. [2018-04-09 11:35:31.405069] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 5-urd-gds-volume-client-5: Server and Client lk-version numbers are not same, reopening the fds [2018-04-09 11:35:31.405585] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 5-urd-gds-volume-client-5: Server lk version = 1 [2018-04-09 11:42:29.622006] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: switched to graph 5 [2018-04-09 11:42:29.627533] I [MSGID: 109005] [dht-selfheal.c:2458:dht_selfheal_directory] 5-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory / [2018-04-09 11:42:29.627935] I [MSGID: 114021] [client.c:2369:notify] 2-urd-gds-volume-client-0: current graph is no longer active, destroying rpc_client [2018-04-09 11:42:29.628013] I [MSGID: 114021] [client.c:2369:notify] 2-urd-gds-volume-client-1: current graph is no longer active, destroying rpc_client [2018-04-09 11:42:29.628047] I [MSGID: 114021] [client.c:2369:notify] 2-urd-gds-volume-client-2: current graph is no longer active, destroying rpc_client [2018-04-09 11:42:29.628069] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-0: disconnected from urd-gds-volume-client-0. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:42:29.628077] I [MSGID: 114021] [client.c:2369:notify] 2-urd-gds-volume-client-3: current graph is no longer active, destroying rpc_client [2018-04-09 11:42:29.628184] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:42:29.628191] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:42:29.628272] W [MSGID: 108001] [afr-common.c:5370:afr_notify] 2-urd-gds-volume-replicate-0: Client-quorum is not met [2018-04-09 11:42:29.628299] I [MSGID: 114021] [client.c:2369:notify] 2-urd-gds-volume-client-4: current graph is no longer active, destroying rpc_client [2018-04-09 11:42:29.628349] I [MSGID: 114021] [client.c:2369:notify] 2-urd-gds-volume-client-5: current graph is no longer active, destroying rpc_client [2018-04-09 11:42:29.628382] I [MSGID: 114021] [client.c:2369:notify] 2-urd-gds-volume-snapd-client: current graph is no longer active, destroying rpc_client [2018-04-09 11:42:29.632749] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:42:29.632804] E [MSGID: 108006] [afr-common.c:5143:__afr_handle_child_down_event] 2-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2018-04-09 11:42:29.637247] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:42:29.637294] W [MSGID: 108001] [afr-common.c:5370:afr_notify] 2-urd-gds-volume-replicate-1: Client-quorum is not met [2018-04-09 11:42:29.637330] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:42:29.641674] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-snapd-client: disconnected from urd-gds-volume-snapd-client. Client process will keep trying to connect to glust\ erd until brick's port is available [2018-04-09 11:42:29.641701] E [MSGID: 108006] [afr-common.c:5143:__afr_handle_child_down_event] 2-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. Other client on 192-net: [2018-04-09 14:13:57.816783] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-urd-gds-volume-client-1: Server lk version = 1 [2018-04-09 14:13:57.817092] I [MSGID: 114057] [client-handshake.c:1484:select_server_supported_programs] 0-urd-gds-volume-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-04-09 14:13:57.817208] I [rpc-clnt.c:1994:rpc_clnt_reconfig] 0-urd-gds-volume-client-4: changing port to 49152 (from 0) [2018-04-09 14:13:57.817388] W [socket.c:3216:socket_connect] 0-urd-gds-volume-client-2: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" [2018-04-09 14:13:57.817623] I [rpc-clnt.c:1994:rpc_clnt_reconfig] 0-urd-gds-volume-client-5: changing port to 49153 (from 0) [2018-04-09 14:13:57.817658] I [rpc-clnt.c:1994:rpc_clnt_reconfig] 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0) [2018-04-09 14:13:57.822047] W [socket.c:3216:socket_connect] 0-urd-gds-volume-client-4: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" [2018-04-09 14:13:57.823419] W [socket.c:3216:socket_connect] 0-urd-gds-volume-client-5: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" [2018-04-09 14:13:57.823613] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached to remote volume '/urd-gds/gluster'. [2018-04-09 14:13:57.823634] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-urd-gds-volume-client-3: Server and Client lk-version numbers are not same, reopening the fds [2018-04-09 14:13:57.823684] I [MSGID: 108005] [afr-common.c:5066:__afr_handle_child_up_event] 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back up; going online. [2018-04-09 14:13:57.825689] W [socket.c:3216:socket_connect] 0-urd-gds-volume-snapd-client: Error disabling sockopt IPV6_V6ONLY: "Protocol not available" [2018-04-09 14:13:57.825845] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-urd-gds-volume-client-3: Server lk version = 1 [2018-04-09 14:13:57.825873] I [MSGID: 114057] [client-handshake.c:1484:select_server_supported_programs] 0-urd-gds-volume-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-04-09 14:13:57.826270] I [MSGID: 114057] [client-handshake.c:1484:select_server_supported_programs] 0-urd-gds-volume-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-04-09 14:13:57.826414] I [MSGID: 114057] [client-handshake.c:1484:select_server_supported_programs] 0-urd-gds-volume-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-04-09 14:13:57.826562] I [MSGID: 114057] [client-handshake.c:1484:select_server_supported_programs] 0-urd-gds-volume-snapd-client: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-04-09 14:13:57.827226] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached to remote volume '/urd-gds/gluster1'. [2018-04-09 14:13:57.827245] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-urd-gds-volume-client-2: Server and Client lk-version numbers are not same, reopening the fds [2018-04-09 14:13:57.827594] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached to remote volume '/urd-gds/gluster'. [2018-04-09 14:13:57.827630] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-urd-gds-volume-client-4: Server and Client lk-version numbers are not same, reopening the fds [2018-04-09 14:13:57.827750] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached to remote volume '/urd-gds/gluster2'. [2018-04-09 14:13:57.827775] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-urd-gds-volume-client-5: Server and Client lk-version numbers are not same, reopening the fds [2018-04-09 14:13:57.827782] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, attached to remote volume 'snapd-urd-gds-vo\ lume'. [2018-04-09 14:13:57.827802] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-urd-gds-volume-snapd-client: Server and Client lk-version numbers are not same, reopening the fds [2018-04-09 14:13:57.829136] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-urd-gds-volume-client-2: Server lk version = 1 [2018-04-09 14:13:57.829173] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-urd-gds-volume-client-5: Server lk version = 1 [2018-04-09 14:13:57.829180] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-urd-gds-volume-client-4: Server lk version = 1 [2018-04-09 14:13:57.829210] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-urd-gds-volume-snapd-client: Server lk version = 1 [2018-04-09 14:13:57.829295] I [fuse-bridge.c:4205:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.26 [2018-04-09 14:13:57.829320] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: switched to graph 0 [2018-04-09 14:13:57.833539] I [MSGID: 109005] [dht-selfheal.c:2458:dht_selfheal_directory] 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for directory / Client on 10-net: [2018-04-09 11:35:31.113283] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:35:31.113289] W [MSGID: 108001] [afr-common.c:5233:afr_notify] 2-urd-gds-volume-replicate-0: Client-quorum is not met [2018-04-09 11:35:31.113289] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:35:31.113351] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 2-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2018-04-09 11:35:31.113367] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:35:31.113492] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:35:31.113500] W [MSGID: 108001] [afr-common.c:5233:afr_notify] 2-urd-gds-volume-replicate-1: Client-quorum is not met [2018-04-09 11:35:31.113511] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 11:35:31.113554] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 2-urd-gds-volume-snapd-client: disconnected from urd-gds-volume-snapd-client. Client process will keep trying to connect to glust\ erd until brick's port is available [2018-04-09 11:35:31.113567] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 2-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. [2018-04-09 12:05:35.111892] I [fuse-bridge.c:4835:fuse_graph_sync] 0-fuse: switched to graph 5 [2018-04-09 12:05:35.116187] I [MSGID: 114021] [client.c:2369:notify] 0-urd-gds-volume-client-0: current graph is no longer active, destroying rpc_client [2018-04-09 12:05:35.116214] I [MSGID: 114021] [client.c:2369:notify] 0-urd-gds-volume-client-1: current graph is no longer active, destroying rpc_client [2018-04-09 12:05:35.116223] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-urd-gds-volume-client-0: disconnected from urd-gds-volume-client-0. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 12:05:35.116227] I [MSGID: 114021] [client.c:2369:notify] 0-urd-gds-volume-client-2: current graph is no longer active, destroying rpc_client [2018-04-09 12:05:35.116252] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 12:05:35.116257] I [MSGID: 114021] [client.c:2369:notify] 0-urd-gds-volume-client-3: current graph is no longer active, destroying rpc_client [2018-04-09 12:05:35.116258] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 12:05:35.116273] I [MSGID: 114021] [client.c:2369:notify] 0-urd-gds-volume-client-4: current graph is no longer active, destroying rpc_client [2018-04-09 12:05:35.116273] W [MSGID: 108001] [afr-common.c:5233:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum is not met [2018-04-09 12:05:35.116288] I [MSGID: 114021] [client.c:2369:notify] 0-urd-gds-volume-client-5: current graph is no longer active, destroying rpc_client [2018-04-09 12:05:35.116393] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 0-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2018-04-09 12:05:35.116397] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 12:05:35.116574] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 12:05:35.116575] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. Client process will keep trying to connect to glusterd unti\ l brick's port is available [2018-04-09 12:05:35.116592] W [MSGID: 108001] [afr-common.c:5233:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum is not met [2018-04-09 12:05:35.116646] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 0-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. [2018-04-09 12:13:18.767382] I [MSGID: 109066] [dht-rename.c:1741:dht_rename] 5-urd-gds-volume-dht: renaming /interbull/backup/scripts/backup/gsnapshotctl.sh (hash=urd-gds-volume-replicate-0/cache=urd-gds-volum\ e-replicate-0) => /interbull/backup/scripts/backup/gsnapshotctl.sh~ (hash=urd-gds-volume-replicate-1/cache=<nul>) [2018-04-09 13:34:54.031860] I [MSGID: 109066] [dht-rename.c:1741:dht_rename] 5-urd-gds-volume-dht: renaming /interbull/backup/scripts/backup/bkp_gluster_to_ribston.sh (hash=urd-gds-volume-replicate-0/cache=urd\ -gds-volume-replicate-0) => /interbull/backup/scripts/backup/bkp_gluster_to_ribston.sh~ (hash=urd-gds-volume-replicate-1/cache=urd-gds-volume-replicate-0) Many thanks in advance!! Best regards Marcus -- ************************************************** * Marcus Peders?n * * System administrator * ************************************************** * Interbull Centre * * ================ * * Department of Animal Breeding & Genetics - SLU * * Box 7023, SE-750 07 * * Uppsala, Sweden * ************************************************** * Visiting address: * * Room 55614, Ulls v?g 26, Ultuna * * Uppsala * * Sweden * * * * Tel: +46-(0)18-67 1962 * * * ************************************************** * ISO 9001 Bureau Veritas No SE004561-1 * ************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180409/1c583f42/attachment.html>
Marcus, Can you share server-side gluster peer probe and client-side mount command-lines. On Tue, Apr 10, 2018 at 12:36 AM, Marcus Peders?n <marcus.pedersen at slu.se> wrote:> Hi all! > > I have setup a replicated/distributed gluster cluster 2 x (2 + 1). > > Centos 7 and gluster version 3.12.6 on server. > > All machines have two network interfaces and connected to two different > networks, > > 10.10.0.0/16 (with hostnames in /etc/hosts, gluster version 3.12.6) > > 192.168.67.0/24 (with ldap, gluster version 3.13.1) > > Gluster cluster was created on the 10.10.0.0/16 net, gluster peer > probe ...and so on. > > All nodes are available on both networks and have the same names on both > networks. > > > Now to my problem, the gluster cluster is mounted on multiple clients on > the 192.168.67.0/24 net > > and a process was running on one of the clients, reading and writing to > files. > > At the same time I mounted the cluster on a client on the 10.10.0.0/16 > net and started to create > > and edit files on the cluster. Around the same time the process on the > 192-net stopped without any > > specific errors. Started other processes on the 192-net and continued to > make changes on the 10-net > > and got the same behavior with stopping processes on the 192-net. > > > Is there any known problems with this type of setup? > > How do I proceed to figure out a solution as I need access from both > networks? > > > Following error shows a couple of times on server (systemd -> glusterd): > > [2018-04-09 11:46:46.254071] C [mem-pool.c:613:mem_pools_init_early] > 0-mem-pool: incorrect order of mem-pool initialization (init_done=3) > > > Client logs: > > Client on 192-net: > > [2018-04-09 11:35:31.402979] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > 5-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached > to remote volume '/urd-gds/gluster'. > [2018-04-09 11:35:31.403019] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > 5-urd-gds-volume-client-1: Server and Client lk-version numbers are not > same, reopening the fds > [2018-04-09 11:35:31.403051] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > 5-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, > attached to remote volume 'snapd-urd-gds-vo\ > lume'. > [2018-04-09 11:35:31.403091] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > 5-urd-gds-volume-snapd-client: Server and Client lk-version numbers are not > same, reopening the fds > [2018-04-09 11:35:31.403271] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 5-urd-gds-volume-client-3: Server lk version = 1 > [2018-04-09 11:35:31.403325] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 5-urd-gds-volume-client-4: Server lk version = 1 > [2018-04-09 11:35:31.403349] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 5-urd-gds-volume-client-0: Server lk version = 1 > [2018-04-09 11:35:31.403367] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 5-urd-gds-volume-client-2: Server lk version = 1 > [2018-04-09 11:35:31.403616] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 5-urd-gds-volume-client-1: Server lk version = 1 > [2018-04-09 11:35:31.403751] I [MSGID: 114057] [client-handshake.c:1484: > select_server_supported_programs] 5-urd-gds-volume-client-5: Using > Program GlusterFS 3.3, Num (1298437), Version (330) > [2018-04-09 11:35:31.404174] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 5-urd-gds-volume-snapd-client: Server lk version = 1 > [2018-04-09 11:35:31.405030] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > 5-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached > to remote volume '/urd-gds/gluster2'. > [2018-04-09 11:35:31.405069] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > 5-urd-gds-volume-client-5: Server and Client lk-version numbers are not > same, reopening the fds > [2018-04-09 11:35:31.405585] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 5-urd-gds-volume-client-5: Server lk version = 1 > [2018-04-09 11:42:29.622006] I [fuse-bridge.c:4835:fuse_graph_sync] > 0-fuse: switched to graph 5 > [2018-04-09 11:42:29.627533] I [MSGID: 109005] [dht-selfheal.c:2458:dht_selfheal_directory] > 5-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for > directory / > [2018-04-09 11:42:29.627935] I [MSGID: 114021] [client.c:2369:notify] > 2-urd-gds-volume-client-0: current graph is no longer active, destroying > rpc_client > [2018-04-09 11:42:29.628013] I [MSGID: 114021] [client.c:2369:notify] > 2-urd-gds-volume-client-1: current graph is no longer active, destroying > rpc_client > [2018-04-09 11:42:29.628047] I [MSGID: 114021] [client.c:2369:notify] > 2-urd-gds-volume-client-2: current graph is no longer active, destroying > rpc_client > [2018-04-09 11:42:29.628069] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-0: disconnected from urd-gds-volume-client-0. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:42:29.628077] I [MSGID: 114021] [client.c:2369:notify] > 2-urd-gds-volume-client-3: current graph is no longer active, destroying > rpc_client > [2018-04-09 11:42:29.628184] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:42:29.628191] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:42:29.628272] W [MSGID: 108001] > [afr-common.c:5370:afr_notify] 2-urd-gds-volume-replicate-0: Client-quorum > is not met > [2018-04-09 11:42:29.628299] I [MSGID: 114021] [client.c:2369:notify] > 2-urd-gds-volume-client-4: current graph is no longer active, destroying > rpc_client > [2018-04-09 11:42:29.628349] I [MSGID: 114021] [client.c:2369:notify] > 2-urd-gds-volume-client-5: current graph is no longer active, destroying > rpc_client > [2018-04-09 11:42:29.628382] I [MSGID: 114021] [client.c:2369:notify] > 2-urd-gds-volume-snapd-client: current graph is no longer active, > destroying rpc_client > [2018-04-09 11:42:29.632749] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:42:29.632804] E [MSGID: 108006] [afr-common.c:5143:__afr_handle_child_down_event] > 2-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until > atleast one of them comes back up. > [2018-04-09 11:42:29.637247] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:42:29.637294] W [MSGID: 108001] > [afr-common.c:5370:afr_notify] 2-urd-gds-volume-replicate-1: Client-quorum > is not met > [2018-04-09 11:42:29.637330] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:42:29.641674] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-snapd-client: disconnected from > urd-gds-volume-snapd-client. Client process will keep trying to connect to > glust\ > erd until brick's port is available > [2018-04-09 11:42:29.641701] E [MSGID: 108006] [afr-common.c:5143:__afr_handle_child_down_event] > 2-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until > atleast one of them comes back up. > > > Other client on 192-net: > > [2018-04-09 14:13:57.816783] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 0-urd-gds-volume-client-1: Server lk version = 1 > [2018-04-09 14:13:57.817092] I [MSGID: 114057] [client-handshake.c:1484: > select_server_supported_programs] 0-urd-gds-volume-client-3: Using > Program GlusterFS 3.3, Num (1298437), Version (330) > [2018-04-09 14:13:57.817208] I [rpc-clnt.c:1994:rpc_clnt_reconfig] > 0-urd-gds-volume-client-4: changing port to 49152 (from 0) > [2018-04-09 14:13:57.817388] W [socket.c:3216:socket_connect] > 0-urd-gds-volume-client-2: Error disabling sockopt IPV6_V6ONLY: "Protocol > not available" > [2018-04-09 14:13:57.817623] I [rpc-clnt.c:1994:rpc_clnt_reconfig] > 0-urd-gds-volume-client-5: changing port to 49153 (from 0) > [2018-04-09 14:13:57.817658] I [rpc-clnt.c:1994:rpc_clnt_reconfig] > 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0) > [2018-04-09 14:13:57.822047] W [socket.c:3216:socket_connect] > 0-urd-gds-volume-client-4: Error disabling sockopt IPV6_V6ONLY: "Protocol > not available" > [2018-04-09 14:13:57.823419] W [socket.c:3216:socket_connect] > 0-urd-gds-volume-client-5: Error disabling sockopt IPV6_V6ONLY: "Protocol > not available" > [2018-04-09 14:13:57.823613] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached > to remote volume '/urd-gds/gluster'. > [2018-04-09 14:13:57.823634] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > 0-urd-gds-volume-client-3: Server and Client lk-version numbers are not > same, reopening the fds > [2018-04-09 14:13:57.823684] I [MSGID: 108005] [afr-common.c:5066:__afr_handle_child_up_event] > 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back > up; going online. > [2018-04-09 14:13:57.825689] W [socket.c:3216:socket_connect] > 0-urd-gds-volume-snapd-client: Error disabling sockopt IPV6_V6ONLY: > "Protocol not available" > [2018-04-09 14:13:57.825845] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 0-urd-gds-volume-client-3: Server lk version = 1 > [2018-04-09 14:13:57.825873] I [MSGID: 114057] [client-handshake.c:1484: > select_server_supported_programs] 0-urd-gds-volume-client-2: Using > Program GlusterFS 3.3, Num (1298437), Version (330) > [2018-04-09 14:13:57.826270] I [MSGID: 114057] [client-handshake.c:1484: > select_server_supported_programs] 0-urd-gds-volume-client-4: Using > Program GlusterFS 3.3, Num (1298437), Version (330) > [2018-04-09 14:13:57.826414] I [MSGID: 114057] [client-handshake.c:1484: > select_server_supported_programs] 0-urd-gds-volume-client-5: Using > Program GlusterFS 3.3, Num (1298437), Version (330) > [2018-04-09 14:13:57.826562] I [MSGID: 114057] [client-handshake.c:1484: > select_server_supported_programs] 0-urd-gds-volume-snapd-client: Using > Program GlusterFS 3.3, Num (1298437), Version (330) > [2018-04-09 14:13:57.827226] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached > to remote volume '/urd-gds/gluster1'. > [2018-04-09 14:13:57.827245] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > 0-urd-gds-volume-client-2: Server and Client lk-version numbers are not > same, reopening the fds > [2018-04-09 14:13:57.827594] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached > to remote volume '/urd-gds/gluster'. > [2018-04-09 14:13:57.827630] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > 0-urd-gds-volume-client-4: Server and Client lk-version numbers are not > same, reopening the fds > [2018-04-09 14:13:57.827750] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached > to remote volume '/urd-gds/gluster2'. > [2018-04-09 14:13:57.827775] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > 0-urd-gds-volume-client-5: Server and Client lk-version numbers are not > same, reopening the fds > [2018-04-09 14:13:57.827782] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, > attached to remote volume 'snapd-urd-gds-vo\ > lume'. > [2018-04-09 14:13:57.827802] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > 0-urd-gds-volume-snapd-client: Server and Client lk-version numbers are not > same, reopening the fds > [2018-04-09 14:13:57.829136] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 0-urd-gds-volume-client-2: Server lk version = 1 > [2018-04-09 14:13:57.829173] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 0-urd-gds-volume-client-5: Server lk version = 1 > [2018-04-09 14:13:57.829180] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 0-urd-gds-volume-client-4: Server lk version = 1 > [2018-04-09 14:13:57.829210] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > 0-urd-gds-volume-snapd-client: Server lk version = 1 > [2018-04-09 14:13:57.829295] I [fuse-bridge.c:4205:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel > 7.26 > [2018-04-09 14:13:57.829320] I [fuse-bridge.c:4835:fuse_graph_sync] > 0-fuse: switched to graph 0 > [2018-04-09 14:13:57.833539] I [MSGID: 109005] [dht-selfheal.c:2458:dht_selfheal_directory] > 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for > directory / > > > Client on 10-net: > > [2018-04-09 11:35:31.113283] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:35:31.113289] W [MSGID: 108001] > [afr-common.c:5233:afr_notify] 2-urd-gds-volume-replicate-0: Client-quorum > is not met > [2018-04-09 11:35:31.113289] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:35:31.113351] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] > 2-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until > atleast one of them comes back up. > [2018-04-09 11:35:31.113367] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:35:31.113492] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:35:31.113500] W [MSGID: 108001] > [afr-common.c:5233:afr_notify] 2-urd-gds-volume-replicate-1: Client-quorum > is not met > [2018-04-09 11:35:31.113511] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 11:35:31.113554] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 2-urd-gds-volume-snapd-client: disconnected from > urd-gds-volume-snapd-client. Client process will keep trying to connect to > glust\ > erd until brick's port is available > [2018-04-09 11:35:31.113567] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] > 2-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until > atleast one of them comes back up. > [2018-04-09 12:05:35.111892] I [fuse-bridge.c:4835:fuse_graph_sync] > 0-fuse: switched to graph 5 > [2018-04-09 12:05:35.116187] I [MSGID: 114021] [client.c:2369:notify] > 0-urd-gds-volume-client-0: current graph is no longer active, destroying > rpc_client > [2018-04-09 12:05:35.116214] I [MSGID: 114021] [client.c:2369:notify] > 0-urd-gds-volume-client-1: current graph is no longer active, destroying > rpc_client > [2018-04-09 12:05:35.116223] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 0-urd-gds-volume-client-0: disconnected from urd-gds-volume-client-0. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 12:05:35.116227] I [MSGID: 114021] [client.c:2369:notify] > 0-urd-gds-volume-client-2: current graph is no longer active, destroying > rpc_client > [2018-04-09 12:05:35.116252] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 0-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 12:05:35.116257] I [MSGID: 114021] [client.c:2369:notify] > 0-urd-gds-volume-client-3: current graph is no longer active, destroying > rpc_client > [2018-04-09 12:05:35.116258] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 0-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 12:05:35.116273] I [MSGID: 114021] [client.c:2369:notify] > 0-urd-gds-volume-client-4: current graph is no longer active, destroying > rpc_client > [2018-04-09 12:05:35.116273] W [MSGID: 108001] > [afr-common.c:5233:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum > is not met > [2018-04-09 12:05:35.116288] I [MSGID: 114021] [client.c:2369:notify] > 0-urd-gds-volume-client-5: current graph is no longer active, destroying > rpc_client > [2018-04-09 12:05:35.116393] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] > 0-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until > atleast one of them comes back up. > [2018-04-09 12:05:35.116397] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 0-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 12:05:35.116574] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 0-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 12:05:35.116575] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > 0-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. > Client process will keep trying to connect to glusterd unti\ > l brick's port is available > [2018-04-09 12:05:35.116592] W [MSGID: 108001] > [afr-common.c:5233:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum > is not met > [2018-04-09 12:05:35.116646] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] > 0-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until > atleast one of them comes back up. > [2018-04-09 12:13:18.767382] I [MSGID: 109066] > [dht-rename.c:1741:dht_rename] 5-urd-gds-volume-dht: renaming > /interbull/backup/scripts/backup/gsnapshotctl.sh (hash=urd-gds-volume- > replicate-0/cache=urd-gds-volum\ > e-replicate-0) => /interbull/backup/scripts/backup/gsnapshotctl.sh~ > (hash=urd-gds-volume-replicate-1/cache=<nul>) > [2018-04-09 13:34:54.031860] I [MSGID: 109066] > [dht-rename.c:1741:dht_rename] 5-urd-gds-volume-dht: renaming > /interbull/backup/scripts/backup/bkp_gluster_to_ribston.sh > (hash=urd-gds-volume-replicate-0/cache=urd\ > -gds-volume-replicate-0) => /interbull/backup/scripts/ > backup/bkp_gluster_to_ribston.sh~ (hash=urd-gds-volume- > replicate-1/cache=urd-gds-volume-replicate-0) > > > > > Many thanks in advance!! > > > Best regards > > Marcus > > > -- > ************************************************** > * Marcus Peders?n * > * System administrator * > ************************************************** > * Interbull Centre * > * ================ * > * Department of Animal Breeding & Genetics ? SLU * > * Box 7023, SE-750 07 * > * Uppsala, Sweden * > ************************************************** > * Visiting address: * > * Room 55614, Ulls v?g 26, Ultuna * > * Uppsala * > * Sweden * > * * > * Tel: +46-(0)18-67 1962 * > * * > ************************************************** > * ISO 9001 Bureau Veritas No SE004561-1 * > ************************************************** > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-- Milind -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180410/df99bae5/attachment.html>
Yes, In first server (urd-gds-001): gluster peer probe urd-gds-000 gluster peer probe urd-gds-002 gluster peer probe urd-gds-003 gluster peer probe urd-gds-004 gluster pool list (from urd-gds-001): UUID Hostname State bdbe4622-25f9-4ef1-aad1-639ca52fc7e0 urd-gds-002 Connected 2a48a3b9-efa0-4fb7-837f-c800f04bf99f urd-gds-003 Connected ad893466-ad09-47f4-8bb4-4cea84085e5b urd-gds-004 Connected bfe05382-7e22-4b93-8816-b239b733b610 urd-gds-000 Connected 912bebfd-1a7f-44dc-b0b7-f001a20d58cd localhost Connected Client mount command (same on both sides): mount -t glusterfs urd-gds-001:/urd-gds-volume /mnt Regards Marcus On Tue, Apr 10, 2018 at 06:24:05PM +0530, Milind Changire wrote:> Marcus, > Can you share server-side gluster peer probe and client-side mount > command-lines. > > > > On Tue, Apr 10, 2018 at 12:36 AM, Marcus Peders?n <marcus.pedersen at slu.se> > wrote: > > > Hi all! > > > > I have setup a replicated/distributed gluster cluster 2 x (2 + 1). > > > > Centos 7 and gluster version 3.12.6 on server. > > > > All machines have two network interfaces and connected to two different > > networks, > > > > 10.10.0.0/16 (with hostnames in /etc/hosts, gluster version 3.12.6) > > > > 192.168.67.0/24 (with ldap, gluster version 3.13.1) > > > > Gluster cluster was created on the 10.10.0.0/16 net, gluster peer > > probe ...and so on. > > > > All nodes are available on both networks and have the same names on both > > networks. > > > > > > Now to my problem, the gluster cluster is mounted on multiple clients on > > the 192.168.67.0/24 net > > > > and a process was running on one of the clients, reading and writing to > > files. > > > > At the same time I mounted the cluster on a client on the 10.10.0.0/16 > > net and started to create > > > > and edit files on the cluster. Around the same time the process on the > > 192-net stopped without any > > > > specific errors. Started other processes on the 192-net and continued to > > make changes on the 10-net > > > > and got the same behavior with stopping processes on the 192-net. > > > > > > Is there any known problems with this type of setup? > > > > How do I proceed to figure out a solution as I need access from both > > networks? > > > > > > Following error shows a couple of times on server (systemd -> glusterd): > > > > [2018-04-09 11:46:46.254071] C [mem-pool.c:613:mem_pools_init_early] > > 0-mem-pool: incorrect order of mem-pool initialization (init_done=3) > > > > > > Client logs: > > > > Client on 192-net: > > > > [2018-04-09 11:35:31.402979] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > > 5-urd-gds-volume-client-1: Connected to urd-gds-volume-client-1, attached > > to remote volume '/urd-gds/gluster'. > > [2018-04-09 11:35:31.403019] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > > 5-urd-gds-volume-client-1: Server and Client lk-version numbers are not > > same, reopening the fds > > [2018-04-09 11:35:31.403051] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > > 5-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, > > attached to remote volume 'snapd-urd-gds-vo\ > > lume'. > > [2018-04-09 11:35:31.403091] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > > 5-urd-gds-volume-snapd-client: Server and Client lk-version numbers are not > > same, reopening the fds > > [2018-04-09 11:35:31.403271] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 5-urd-gds-volume-client-3: Server lk version = 1 > > [2018-04-09 11:35:31.403325] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 5-urd-gds-volume-client-4: Server lk version = 1 > > [2018-04-09 11:35:31.403349] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 5-urd-gds-volume-client-0: Server lk version = 1 > > [2018-04-09 11:35:31.403367] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 5-urd-gds-volume-client-2: Server lk version = 1 > > [2018-04-09 11:35:31.403616] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 5-urd-gds-volume-client-1: Server lk version = 1 > > [2018-04-09 11:35:31.403751] I [MSGID: 114057] [client-handshake.c:1484: > > select_server_supported_programs] 5-urd-gds-volume-client-5: Using > > Program GlusterFS 3.3, Num (1298437), Version (330) > > [2018-04-09 11:35:31.404174] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 5-urd-gds-volume-snapd-client: Server lk version = 1 > > [2018-04-09 11:35:31.405030] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > > 5-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached > > to remote volume '/urd-gds/gluster2'. > > [2018-04-09 11:35:31.405069] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > > 5-urd-gds-volume-client-5: Server and Client lk-version numbers are not > > same, reopening the fds > > [2018-04-09 11:35:31.405585] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 5-urd-gds-volume-client-5: Server lk version = 1 > > [2018-04-09 11:42:29.622006] I [fuse-bridge.c:4835:fuse_graph_sync] > > 0-fuse: switched to graph 5 > > [2018-04-09 11:42:29.627533] I [MSGID: 109005] [dht-selfheal.c:2458:dht_selfheal_directory] > > 5-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for > > directory / > > [2018-04-09 11:42:29.627935] I [MSGID: 114021] [client.c:2369:notify] > > 2-urd-gds-volume-client-0: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 11:42:29.628013] I [MSGID: 114021] [client.c:2369:notify] > > 2-urd-gds-volume-client-1: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 11:42:29.628047] I [MSGID: 114021] [client.c:2369:notify] > > 2-urd-gds-volume-client-2: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 11:42:29.628069] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-0: disconnected from urd-gds-volume-client-0. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:42:29.628077] I [MSGID: 114021] [client.c:2369:notify] > > 2-urd-gds-volume-client-3: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 11:42:29.628184] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:42:29.628191] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:42:29.628272] W [MSGID: 108001] > > [afr-common.c:5370:afr_notify] 2-urd-gds-volume-replicate-0: Client-quorum > > is not met > > [2018-04-09 11:42:29.628299] I [MSGID: 114021] [client.c:2369:notify] > > 2-urd-gds-volume-client-4: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 11:42:29.628349] I [MSGID: 114021] [client.c:2369:notify] > > 2-urd-gds-volume-client-5: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 11:42:29.628382] I [MSGID: 114021] [client.c:2369:notify] > > 2-urd-gds-volume-snapd-client: current graph is no longer active, > > destroying rpc_client > > [2018-04-09 11:42:29.632749] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:42:29.632804] E [MSGID: 108006] [afr-common.c:5143:__afr_handle_child_down_event] > > 2-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until > > atleast one of them comes back up. > > [2018-04-09 11:42:29.637247] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:42:29.637294] W [MSGID: 108001] > > [afr-common.c:5370:afr_notify] 2-urd-gds-volume-replicate-1: Client-quorum > > is not met > > [2018-04-09 11:42:29.637330] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:42:29.641674] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-snapd-client: disconnected from > > urd-gds-volume-snapd-client. Client process will keep trying to connect to > > glust\ > > erd until brick's port is available > > [2018-04-09 11:42:29.641701] E [MSGID: 108006] [afr-common.c:5143:__afr_handle_child_down_event] > > 2-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until > > atleast one of them comes back up. > > > > > > Other client on 192-net: > > > > [2018-04-09 14:13:57.816783] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 0-urd-gds-volume-client-1: Server lk version = 1 > > [2018-04-09 14:13:57.817092] I [MSGID: 114057] [client-handshake.c:1484: > > select_server_supported_programs] 0-urd-gds-volume-client-3: Using > > Program GlusterFS 3.3, Num (1298437), Version (330) > > [2018-04-09 14:13:57.817208] I [rpc-clnt.c:1994:rpc_clnt_reconfig] > > 0-urd-gds-volume-client-4: changing port to 49152 (from 0) > > [2018-04-09 14:13:57.817388] W [socket.c:3216:socket_connect] > > 0-urd-gds-volume-client-2: Error disabling sockopt IPV6_V6ONLY: "Protocol > > not available" > > [2018-04-09 14:13:57.817623] I [rpc-clnt.c:1994:rpc_clnt_reconfig] > > 0-urd-gds-volume-client-5: changing port to 49153 (from 0) > > [2018-04-09 14:13:57.817658] I [rpc-clnt.c:1994:rpc_clnt_reconfig] > > 0-urd-gds-volume-snapd-client: changing port to 49153 (from 0) > > [2018-04-09 14:13:57.822047] W [socket.c:3216:socket_connect] > > 0-urd-gds-volume-client-4: Error disabling sockopt IPV6_V6ONLY: "Protocol > > not available" > > [2018-04-09 14:13:57.823419] W [socket.c:3216:socket_connect] > > 0-urd-gds-volume-client-5: Error disabling sockopt IPV6_V6ONLY: "Protocol > > not available" > > [2018-04-09 14:13:57.823613] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > > 0-urd-gds-volume-client-3: Connected to urd-gds-volume-client-3, attached > > to remote volume '/urd-gds/gluster'. > > [2018-04-09 14:13:57.823634] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > > 0-urd-gds-volume-client-3: Server and Client lk-version numbers are not > > same, reopening the fds > > [2018-04-09 14:13:57.823684] I [MSGID: 108005] [afr-common.c:5066:__afr_handle_child_up_event] > > 0-urd-gds-volume-replicate-1: Subvolume 'urd-gds-volume-client-3' came back > > up; going online. > > [2018-04-09 14:13:57.825689] W [socket.c:3216:socket_connect] > > 0-urd-gds-volume-snapd-client: Error disabling sockopt IPV6_V6ONLY: > > "Protocol not available" > > [2018-04-09 14:13:57.825845] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 0-urd-gds-volume-client-3: Server lk version = 1 > > [2018-04-09 14:13:57.825873] I [MSGID: 114057] [client-handshake.c:1484: > > select_server_supported_programs] 0-urd-gds-volume-client-2: Using > > Program GlusterFS 3.3, Num (1298437), Version (330) > > [2018-04-09 14:13:57.826270] I [MSGID: 114057] [client-handshake.c:1484: > > select_server_supported_programs] 0-urd-gds-volume-client-4: Using > > Program GlusterFS 3.3, Num (1298437), Version (330) > > [2018-04-09 14:13:57.826414] I [MSGID: 114057] [client-handshake.c:1484: > > select_server_supported_programs] 0-urd-gds-volume-client-5: Using > > Program GlusterFS 3.3, Num (1298437), Version (330) > > [2018-04-09 14:13:57.826562] I [MSGID: 114057] [client-handshake.c:1484: > > select_server_supported_programs] 0-urd-gds-volume-snapd-client: Using > > Program GlusterFS 3.3, Num (1298437), Version (330) > > [2018-04-09 14:13:57.827226] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > > 0-urd-gds-volume-client-2: Connected to urd-gds-volume-client-2, attached > > to remote volume '/urd-gds/gluster1'. > > [2018-04-09 14:13:57.827245] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > > 0-urd-gds-volume-client-2: Server and Client lk-version numbers are not > > same, reopening the fds > > [2018-04-09 14:13:57.827594] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > > 0-urd-gds-volume-client-4: Connected to urd-gds-volume-client-4, attached > > to remote volume '/urd-gds/gluster'. > > [2018-04-09 14:13:57.827630] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > > 0-urd-gds-volume-client-4: Server and Client lk-version numbers are not > > same, reopening the fds > > [2018-04-09 14:13:57.827750] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > > 0-urd-gds-volume-client-5: Connected to urd-gds-volume-client-5, attached > > to remote volume '/urd-gds/gluster2'. > > [2018-04-09 14:13:57.827775] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > > 0-urd-gds-volume-client-5: Server and Client lk-version numbers are not > > same, reopening the fds > > [2018-04-09 14:13:57.827782] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] > > 0-urd-gds-volume-snapd-client: Connected to urd-gds-volume-snapd-client, > > attached to remote volume 'snapd-urd-gds-vo\ > > lume'. > > [2018-04-09 14:13:57.827802] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] > > 0-urd-gds-volume-snapd-client: Server and Client lk-version numbers are not > > same, reopening the fds > > [2018-04-09 14:13:57.829136] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 0-urd-gds-volume-client-2: Server lk version = 1 > > [2018-04-09 14:13:57.829173] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 0-urd-gds-volume-client-5: Server lk version = 1 > > [2018-04-09 14:13:57.829180] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 0-urd-gds-volume-client-4: Server lk version = 1 > > [2018-04-09 14:13:57.829210] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] > > 0-urd-gds-volume-snapd-client: Server lk version = 1 > > [2018-04-09 14:13:57.829295] I [fuse-bridge.c:4205:fuse_init] > > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel > > 7.26 > > [2018-04-09 14:13:57.829320] I [fuse-bridge.c:4835:fuse_graph_sync] > > 0-fuse: switched to graph 0 > > [2018-04-09 14:13:57.833539] I [MSGID: 109005] [dht-selfheal.c:2458:dht_selfheal_directory] > > 0-urd-gds-volume-dht: Directory selfheal failed: Unable to form layout for > > directory / > > > > > > Client on 10-net: > > > > [2018-04-09 11:35:31.113283] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:35:31.113289] W [MSGID: 108001] > > [afr-common.c:5233:afr_notify] 2-urd-gds-volume-replicate-0: Client-quorum > > is not met > > [2018-04-09 11:35:31.113289] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:35:31.113351] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] > > 2-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until > > atleast one of them comes back up. > > [2018-04-09 11:35:31.113367] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:35:31.113492] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:35:31.113500] W [MSGID: 108001] > > [afr-common.c:5233:afr_notify] 2-urd-gds-volume-replicate-1: Client-quorum > > is not met > > [2018-04-09 11:35:31.113511] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 11:35:31.113554] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 2-urd-gds-volume-snapd-client: disconnected from > > urd-gds-volume-snapd-client. Client process will keep trying to connect to > > glust\ > > erd until brick's port is available > > [2018-04-09 11:35:31.113567] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] > > 2-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until > > atleast one of them comes back up. > > [2018-04-09 12:05:35.111892] I [fuse-bridge.c:4835:fuse_graph_sync] > > 0-fuse: switched to graph 5 > > [2018-04-09 12:05:35.116187] I [MSGID: 114021] [client.c:2369:notify] > > 0-urd-gds-volume-client-0: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 12:05:35.116214] I [MSGID: 114021] [client.c:2369:notify] > > 0-urd-gds-volume-client-1: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 12:05:35.116223] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 0-urd-gds-volume-client-0: disconnected from urd-gds-volume-client-0. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 12:05:35.116227] I [MSGID: 114021] [client.c:2369:notify] > > 0-urd-gds-volume-client-2: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 12:05:35.116252] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 0-urd-gds-volume-client-1: disconnected from urd-gds-volume-client-1. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 12:05:35.116257] I [MSGID: 114021] [client.c:2369:notify] > > 0-urd-gds-volume-client-3: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 12:05:35.116258] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 0-urd-gds-volume-client-2: disconnected from urd-gds-volume-client-2. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 12:05:35.116273] I [MSGID: 114021] [client.c:2369:notify] > > 0-urd-gds-volume-client-4: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 12:05:35.116273] W [MSGID: 108001] > > [afr-common.c:5233:afr_notify] 0-urd-gds-volume-replicate-0: Client-quorum > > is not met > > [2018-04-09 12:05:35.116288] I [MSGID: 114021] [client.c:2369:notify] > > 0-urd-gds-volume-client-5: current graph is no longer active, destroying > > rpc_client > > [2018-04-09 12:05:35.116393] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] > > 0-urd-gds-volume-replicate-0: All subvolumes are down. Going offline until > > atleast one of them comes back up. > > [2018-04-09 12:05:35.116397] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 0-urd-gds-volume-client-3: disconnected from urd-gds-volume-client-3. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 12:05:35.116574] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 0-urd-gds-volume-client-4: disconnected from urd-gds-volume-client-4. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 12:05:35.116575] I [MSGID: 114018] [client.c:2285:client_rpc_notify] > > 0-urd-gds-volume-client-5: disconnected from urd-gds-volume-client-5. > > Client process will keep trying to connect to glusterd unti\ > > l brick's port is available > > [2018-04-09 12:05:35.116592] W [MSGID: 108001] > > [afr-common.c:5233:afr_notify] 0-urd-gds-volume-replicate-1: Client-quorum > > is not met > > [2018-04-09 12:05:35.116646] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] > > 0-urd-gds-volume-replicate-1: All subvolumes are down. Going offline until > > atleast one of them comes back up. > > [2018-04-09 12:13:18.767382] I [MSGID: 109066] > > [dht-rename.c:1741:dht_rename] 5-urd-gds-volume-dht: renaming > > /interbull/backup/scripts/backup/gsnapshotctl.sh (hash=urd-gds-volume- > > replicate-0/cache=urd-gds-volum\ > > e-replicate-0) => /interbull/backup/scripts/backup/gsnapshotctl.sh~ > > (hash=urd-gds-volume-replicate-1/cache=<nul>) > > [2018-04-09 13:34:54.031860] I [MSGID: 109066] > > [dht-rename.c:1741:dht_rename] 5-urd-gds-volume-dht: renaming > > /interbull/backup/scripts/backup/bkp_gluster_to_ribston.sh > > (hash=urd-gds-volume-replicate-0/cache=urd\ > > -gds-volume-replicate-0) => /interbull/backup/scripts/ > > backup/bkp_gluster_to_ribston.sh~ (hash=urd-gds-volume- > > replicate-1/cache=urd-gds-volume-replicate-0) > > > > > > > > > > Many thanks in advance!! > > > > > > Best regards > > > > Marcus > > > > > > -- > > ************************************************** > > * Marcus Peders?n * > > * System administrator * > > ************************************************** > > * Interbull Centre * > > * ================ * > > * Department of Animal Breeding & Genetics ? SLU * > > * Box 7023, SE-750 07 * > > * Uppsala, Sweden * > > ************************************************** > > * Visiting address: * > > * Room 55614, Ulls v?g 26, Ultuna * > > * Uppsala * > > * Sweden * > > * * > > * Tel: +46-(0)18-67 1962 * > > * * > > ************************************************** > > * ISO 9001 Bureau Veritas No SE004561-1 * > > ************************************************** > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > -- > Milind-- ************************************************** * Marcus Peders?n * * System administrator * ************************************************** * Interbull Centre * * ================ * * Department of Animal Breeding & Genetics ? SLU * * Box 7023, SE-750 07 * * Uppsala, Sweden * ************************************************** * Visiting address: * * Room 55614, Ulls v?g 26, Ultuna * * Uppsala * * Sweden * * * * Tel: +46-(0)18-67 1962 * * * ************************************************** * ISO 9001 Bureau Veritas No SE004561-1 * **************************************************