thr3ads.net - Gluster users - [Gluster-users] Gluster client mount fails in mid flight with signum 15 [Jun 2017]

If this information is useful, please help other people find it:
Share via:

Gabriel Lindeborg

2017-Jun-01 07:24 UTC

[Gluster-users] Gluster client mount fails in mid flight with signum 15

All four clients did run 3.10.2 as well

The volumes has been running fine until we upgraded to 3.10, when we hit some
issues with port mismatches. We restarted all the volumes, the servers and the
clients and now hit this issue.
We?ve since backed up the files, remove the volumes, removed the bricks, removed
gluster, installed glusterfs 3.7.20, created new volumes on new bricks, restored
the files and still hit the same issue at clients on the nodes that also runs
the servers. We?ve got to clients on connected to one of the volumes that has
been working fine all the time.

This is the debug logs from one of the mount as the client gets disconnected:
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht:
revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]"
repeated 26 times between [2017-05-31 13:48:51.680757] and [2017-05-31
13:50:46.325368]
/DAEMON/DEBUG [2017-05-31T15:50:50.589272+02:00] [] []
[logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out.
About to flush outstanding messages if present
/DAEMON/DEBUG [2017-05-31T15:50:50.589520+02:00] [] []
[logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting timer
now. Timeout = 120, current buf size = 5
[2017-05-31 13:50:51.908797] D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk]
0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs
cleaning]
/DAEMON/DEBUG [2017-05-31T15:51:24.592190+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:51:24.592469+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:51:26.325230+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:52:08.595536+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:52:08.595735+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:52:12.059895+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:52:12.060170+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
d
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht:
revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]"
repeated 26 times between [2017-05-31 13:50:51.908797] and [2017-05-31
13:52:46.326381]
/DAEMON/DEBUG [2017-05-31T15:52:50.598987+02:00] [] []
[logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out.
About to flush outstanding messages if present
/DAEMON/DEBUG [2017-05-31T15:52:50.599226+02:00] [] []
[logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting timer
now. Timeout = 120, current buf size = 5
[2017-05-31 13:52:52.138032] D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk]
0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs
cleaning]
/DAEMON/DEBUG [2017-05-31T15:52:54.599435+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:52:54.599718+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:52:56.325482+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:52:56.325731+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:53:38.603305+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:53:38.603533+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:53:42.226123+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:53:42.226345+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:54:24.607225+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:54:24.607479+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht:
revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]"
repeated 20 times between [2017-05-31 13:52:52.138032] and [2017-05-31
13:54:22.302116]
[2017-05-31 13:54:25.284676] W [glusterfsd.c:1251:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f36b20c6dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f36b3740915]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f36b374078b] ) 0-: received
signum (15), shutting down
/DAEMON/DEBUG [2017-05-31T15:54:25.285804+02:00] [] []
[logging.c:1766:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size
reduced. About to flush 1 extra log messages
/DAEMON/DEBUG [2017-05-31T15:54:25.286570+02:00] [] []
[logging.c:1769:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 1 extra
log messages
/DAEMON/DEBUG [2017-05-31T15:54:25.287431+02:00] [] []
[glusterfsd-mgmt.c:2361:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper
signout arguments not given
/DAEMON/INFO [2017-05-31T15:54:25.287785+02:00] [] [] [fuse-bridge.c:5720:fini]
0-fuse: Unmounting '/mnt/gluster/mule?.

Cheers
Gabbe
31 maj 2017 kl. 17:03 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu at
redhat.com<mailto:sheggodu at redhat.com>>:

Hi Gabriel,

What is the version of gluster you are running on client? Also, please share the
steps you followed to hit the issue.

Regards,
Sunil kumar Acharya

Senior Software Engineer

Red Hat

<https://www.redhat.com/>

T: +91-8067935170<http://redhatemailsignature-marketing.itos.redhat.com/>

[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]<https://red.ht/sig>
TRIED. TESTED. TRUSTED.<https://redhat.com/trusted>


On Wed, May 31, 2017 at 12:10 PM, Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se<mailto:gabriel.lindeborg at svenskaspel.se>> wrote:
Hello again

The volumes are old, since version 3.6 if I remember correctly?
These ?r det 40 last rows of the mnt-loggs go all the mounts

==> /var/log/glusterfs/mnt-gluster-alfresco.log <=/DAEMON/INFO
[2017-05-31T07:55:12.787035+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.153779+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.160995+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.167075+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.171872+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.427057+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.432239+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.439572+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.444286+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.035710+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.042366+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.049401+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.054483+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.524103+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.540314+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.543748+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.558459+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.623109+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.643979+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.648717+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.662941+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.268446+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.284765+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.287503+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.302461+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.408757+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.415831+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421736+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421887+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.431820+02:00] [] []
[2017-05-31 06:09:54.514841] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting
local read_child alfresco-client-2
/DAEMON/INFO [2017-05-31T08:11:10.179031+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.186811+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.194886+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.195062+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.205582+02:00] [] []
[2017-05-31 06:11:14.513620] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting
local read_child alfresco-client-2
[2017-05-31 06:21:21.579748] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa1d2e4cdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa1d44e4fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa1d44e4dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-31T08:21:21.580626+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581440+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-c1_2.log <= 82:     type meta
 83:     subvolumes c1
 84: end-volume
 85:
+------------------------------------------------------------------------------+
/DAEMON/ERR [2017-05-24T10:39:11.130038+02:00] [] []
[2017-05-24 08:39:11.130638] E [MSGID: 114058]
[client-handshake.c:1538:client_query_portmap_cbk] 2-c1-client-1: failed to get
the port number for remote subvolume. Please run 'gluster volume status'
on server to see if brick process is running.
[2017-05-24 08:39:11.130693] I [MSGID: 114018] [client.c:2276:client_rpc_notify]
2-c1-client-1: disconnected from c1-client-1. Client process will keep trying to
connect to glusterd until brick's port is available
[2017-05-24 08:39:11.130711] E [MSGID: 108006] [afr-common.c:4781:afr_notify]
2-c1-replicate-0: All subvolumes are down. Going offline until atleast one of
them comes back up.
/DAEMON/INFO [2017-05-24T10:39:11.471603+02:00] [] []
/DAEMON/ERR [2017-05-24T10:39:11.473783+02:00] [] []
/DAEMON/INFO [2017-05-24T10:39:14.479153+02:00] [] []
/DAEMON/ERR [2017-05-24T10:39:14.481430+02:00] [] []
[2017-05-24 08:39:14.513688] I [MSGID: 108006]
[afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up
/DAEMON/INFO [2017-05-24T10:39:14.513712+02:00] [] []
[2017-05-24 08:39:14.513858] I [MSGID: 114021] [client.c:2361:notify]
0-c1-client-2: current graph is no longer active, destroying rpc_client
[2017-05-24 08:39:14.513879] I [MSGID: 114021] [client.c:2361:notify]
0-c1-client-3: current graph is no longer active, destroying rpc_client
[2017-05-24 08:39:14.513933] I [MSGID: 108006]
[afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up
/DAEMON/WARNING [2017-05-24T10:39:14.513997+02:00] [] []
/DAEMON/WARNING [2017-05-24T10:39:14.514161+02:00] [] []
/DAEMON/INFO [2017-05-24T10:39:18.485909+02:00] [] []
/DAEMON/ERR [2017-05-24T10:39:18.488664+02:00] [] []
/DAEMON/INFO [2017-05-24T10:39:23.737962+02:00] [] []
[2017-05-24 08:39:23.740893] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 2-c1-client-0: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-24 08:39:23.742339] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-0: Connected to
c1-client-0, attached to remote volume '/gluster/c1/brick'.
[2017-05-24 08:39:23.742359] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-0: Server and Client
lk-version numbers are not same, reopening the fds
[2017-05-24 08:39:23.742437] I [MSGID: 108005] [afr-common.c:4756:afr_notify]
2-c1-replicate-0: Subvolume 'c1-client-0' came back up; going online.
[2017-05-24 08:39:23.743083] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-0: Server lk
version = 1
[2017-05-24 08:39:24.524310] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local
read_child c1-client-0
[2017-05-24 08:39:24.524414] I [MSGID: 109063]
[dht-layout.c:713:dht_layout_normalize] 2-c1-dht: Found anomalies in / (gfid =
00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
/DAEMON/INFO [2017-05-24T10:39:26.883379+02:00] [] []
[2017-05-24 08:39:26.885753] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 2-c1-client-1: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-24 08:39:26.887336] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-1: Connected to
c1-client-1, attached to remote volume '/gluster/c1/brick'.
[2017-05-24 08:39:26.887353] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-1: Server and Client
lk-version numbers are not same, reopening the fds
[2017-05-24 08:39:26.887479] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-1: Server lk
version = 1
[2017-05-24 08:39:34.515063] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local
read_child c1-client-0
The message "I [MSGID: 108006] [afr-common.c:4923:afr_local_init]
0-c1-replicate-0: no subvolumes up" repeated 4 times between [2017-05-24
08:38:24.514047] and [2017-05-24 08:39:04.514186]
/DAEMON/INFO [2017-05-24T10:40:56.356298+02:00] [] []
[2017-05-24 08:40:56.356694] W [glusterfsd.c:1329:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7feba7f55dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7feba95edf85]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7feba95eddfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-24T10:40:56.356773+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-c1.log <=/DAEMON/INFO
[2017-05-31T07:55:12.781267+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:12.786589+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.153964+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.161317+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.167226+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.172425+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.425676+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.433114+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.439753+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.444685+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.034185+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.042732+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.049094+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.053587+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.525624+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.540864+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.546015+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.546167+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.549146+02:00] [] []
[2017-05-31 05:56:14.515062] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-c1-replicate-0: selecting local
read_child c1-client-0
/DAEMON/INFO [2017-05-31T07:56:13.558847+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.622583+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.644252+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.648540+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.662541+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.269583+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.284254+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.291514+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.303324+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.408574+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.415968+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421254+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.431030+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.179194+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.187103+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.194300+02:00] [] []
[2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f7dad36fdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f7daea07fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f7daea07dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-31T08:11:10.205913+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581037+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581789+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-c2.log <= 66: end-volume
 67:
 68: volume c2-md-cache
 69:     type performance/md-cache
 70:     subvolumes c2-open-behind
 71: end-volume
 72:
 73: volume c2
 74:     type debug/io-stats
 75:     option log-level INFO
 76:     option latency-measurement off
 77:     option count-fop-hits off
 78:     subvolumes c2-md-cache
 79: end-volume
 80:
 81: volume meta-autoload
 82:     type meta
 83:     subvolumes c2
 84: end-volume
 85:
+------------------------------------------------------------------------------+
/DAEMON/INFO [2017-05-24T09:20:01.247296+02:00] [] []
[2017-05-24 07:20:01.249248] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 0-c2-client-0: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-24 07:20:01.249950] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-0: Connected to
c2-client-0, attached to remote volume '/gluster/c1/newbrick'.
[2017-05-24 07:20:01.249980] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-0: Server and Client
lk-version numbers are not same, reopening the fds
[2017-05-24 07:20:01.250050] I [MSGID: 108005] [afr-common.c:4756:afr_notify]
0-c2-replicate-0: Subvolume 'c2-client-0' came back up; going online.
[2017-05-24 07:20:01.250557] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-0: Server lk
version = 1
[2017-05-24 07:20:01.251421] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 0-c2-client-1: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-24 07:20:01.252016] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-1: Connected to
c2-client-1, attached to remote volume '/gluster/c1/newbrick'.
[2017-05-24 07:20:01.252055] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-1: Server and Client
lk-version numbers are not same, reopening the fds
[2017-05-24 07:20:01.253072] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-1: Server lk
version = 1
/DAEMON/INFO [2017-05-24T09:20:01.253273+02:00] [] []
/DAEMON/INFO [2017-05-24T09:20:01.253547+02:00] [] []
[2017-05-24 07:20:01.254056] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-c2-replicate-0: selecting local
read_child c2-client-0
[2017-05-24 07:20:01.254268] I [MSGID: 109063]
[dht-layout.c:713:dht_layout_normalize] 0-c2-dht: Found anomalies in / (gfid =
00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
/DAEMON/INFO [2017-05-24T10:39:11.116695+02:00] [] []
/DAEMON/INFO [2017-05-24T10:39:11.118327+02:00] [] []
/DAEMON/INFO [2017-05-24T10:40:47.577696+02:00] [] []
[2017-05-24 08:40:47.578001] W [glusterfsd.c:1329:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f3eaabefdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f3eac287f85]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f3eac287dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-24T10:40:47.578100+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-cdn.log <=/DAEMON/INFO
[2017-05-31T07:55:12.785775+02:00] [] []
[2017-05-31 05:55:14.515355] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local
read_child cdn-client-2
/DAEMON/INFO [2017-05-31T07:55:18.152981+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.161155+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.166602+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.171520+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.426866+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.432536+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.439418+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.443795+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.035303+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.042561+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.048928+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.053195+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.525267+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.540464+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.545703+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.559227+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.622792+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.644113+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.649031+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.649173+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.653281+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.663382+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.269801+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.283977+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.288305+02:00] [] []
[2017-05-31 05:56:24.514977] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local
read_child cdn-client-2
/DAEMON/INFO [2017-05-31T07:56:26.302883+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.408914+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.415563+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421558+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.432374+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.178859+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.186967+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.193439+02:00] [] []
[2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa420a29dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa4220c1fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa4220c1dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-31T08:11:10.204701+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581209+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581945+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-mule.log <=/DAEMON/INFO
[2017-05-31T07:55:18.154147+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.161457+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.168560+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.168765+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.172803+02:00] [] []
[2017-05-31 05:55:24.517077] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local
read_child mule-client-2
/DAEMON/INFO [2017-05-31T07:55:46.426664+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.432816+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.439270+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.443344+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.035528+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.042893+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.049247+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.054006+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.525469+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.540732+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.545856+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.559699+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.622955+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.644386+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.648877+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.663766+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.269958+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.284895+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.291677+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.291845+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.294037+02:00] [] []
[2017-05-31 05:56:34.516246] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local
read_child mule-client-2
/DAEMON/INFO [2017-05-31T07:56:26.303715+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.408417+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.416098+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421403+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.431421+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.178644+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.187364+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.193590+02:00] [] []
[2017-05-31 06:21:21.579746] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fbd1beafdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fbd1d547fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbd1d547dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-31T08:11:10.205040+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.580846+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581623+02:00] [] []

And these are 40 last rows of the brick
==> /var/log/glusterfs/bricks/gluster-alfresco-brick.log <=[2017-05-31
05:52:19.117394] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0
[2017-05-31 05:52:19.117469] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0
[2017-05-31 05:52:43.020403] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:52:43.027816] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
[2017-05-31 05:52:43.027931] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
[2017-05-31 05:56:48.604780] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:57:50.817223] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:57:50.825987] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
[2017-05-31 05:57:50.826138] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
[2017-05-31 05:58:25.746161] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:58:25.755174] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
[2017-05-31 05:58:25.755279] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
[2017-05-31 06:02:47.958784] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:02:47.976543] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
[2017-05-31 06:02:47.976683] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
[2017-05-31 06:07:53.126343] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:07:53.133904] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
[2017-05-31 06:07:53.133998] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
[2017-05-31 06:12:56.138792] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:12:56.146253] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
[2017-05-31 06:12:56.146351] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
[2017-05-31 06:16:22.552274] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
[2017-05-31 06:16:22.552428] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
[2017-05-31 06:17:54.775497] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:17:54.792370] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
[2017-05-31 06:17:54.792446] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
[2017-05-31 06:21:21.582947] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0
[2017-05-31 06:21:21.583181] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0
[2017-05-31 06:21:42.714049] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:21:42.722227] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
[2017-05-31 06:21:42.722320] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
[2017-05-31 06:22:35.445697] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:22:35.453208] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
[2017-05-31 06:22:35.453334] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
[2017-05-31 06:22:58.071308] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:22:58.078524] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
[2017-05-31 06:22:58.078608] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
[2017-05-31 06:28:06.645148] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:28:06.652828] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0
[2017-05-31 06:28:06.652942] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0

==> /var/log/glusterfs/bricks/gluster-c1-brick.log <=[2017-05-31
05:49:00.793792] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server:
Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-39066-2017/05/31-05:29:48:927855-c1-client-0-0-0
[2017-05-31 05:52:36.512431] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 05:52:36.522623] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
[2017-05-31 05:52:36.522723] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
[2017-05-31 05:56:13.544120] I [MSGID: 101190]
[event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 2
[2017-05-31 05:56:48.796772] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 05:58:39.772391] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 05:58:39.782681] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
[2017-05-31 05:58:39.782820] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
[2017-05-31 06:03:45.195471] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:03:45.205497] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
[2017-05-31 06:03:45.205610] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
[2017-05-31 06:08:45.229186] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:08:45.239068] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
[2017-05-31 06:08:45.239173] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
[2017-05-31 06:13:51.433086] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:13:51.443161] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
[2017-05-31 06:13:51.443257] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
[2017-05-31 06:16:22.552265] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
[2017-05-31 06:16:22.552418] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
[2017-05-31 06:18:55.232358] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:18:55.242110] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
[2017-05-31 06:18:55.242188] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
[2017-05-31 06:21:21.586024] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0
[2017-05-31 06:21:21.586153] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0
[2017-05-31 06:21:34.443201] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:21:34.453810] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
[2017-05-31 06:21:34.453897] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
[2017-05-31 06:23:55.706224] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:23:55.716126] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
[2017-05-31 06:23:55.716221] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
[2017-05-31 06:28:58.213831] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:28:58.223474] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
[2017-05-31 06:28:58.223557] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
[2017-05-31 06:33:07.622832] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:33:07.642391] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
[2017-05-31 06:33:07.642520] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
[2017-05-31 06:34:07.776928] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:34:07.787116] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0
[2017-05-31 06:34:07.787200] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0

==> /var/log/glusterfs/bricks/gluster-cdn-brick.log <=[2017-05-31
05:50:18.876202] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-40008-2017/05/31-05:50:18:867052-cdn-client-2-0-0
[2017-05-31 05:55:09.869166] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:55:09.877790] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
[2017-05-31 05:55:09.877898] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
[2017-05-31 05:55:20.944369] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:55:20.952275] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
[2017-05-31 05:55:20.952379] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
[2017-05-31 05:56:48.701158] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:00:20.651639] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:00:20.659659] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
[2017-05-31 06:00:20.659782] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
[2017-05-31 06:01:17.578233] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:01:17.587267] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
[2017-05-31 06:01:17.587370] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
[2017-05-31 06:05:25.961313] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:05:25.976281] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
[2017-05-31 06:05:25.976378] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
[2017-05-31 06:10:24.626906] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:10:24.634914] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
[2017-05-31 06:10:24.635016] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
[2017-05-31 06:15:21.944215] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:15:21.952554] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
[2017-05-31 06:15:21.952655] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
[2017-05-31 06:16:22.552086] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
[2017-05-31 06:16:22.552203] I [MSGID: 115013]
[server-helpers.c:296:do_fd_cleanup] 0-cdn-server: fd cleanup on
/test2/nyx/nyxclient/assets.json
[2017-05-31 06:16:22.552416] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
[2017-05-31 06:20:25.255523] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:20:25.263860] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
[2017-05-31 06:20:25.263963] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
[2017-05-31 06:21:21.584357] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0
[2017-05-31 06:21:21.584449] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0
[2017-05-31 06:21:37.944798] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:21:37.952580] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
[2017-05-31 06:21:37.952712] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
[2017-05-31 06:25:27.632381] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:25:27.640591] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
[2017-05-31 06:25:27.640721] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
[2017-05-31 06:35:42.166035] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:35:42.173662] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0
[2017-05-31 06:35:42.173752] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0

==> /var/log/glusterfs/bricks/gluster-mule-brick.log <=[2017-05-31
06:26:30.580063] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-38988-2017/05/31-06:26:30:551135-mule-client-2-0-0
[2017-05-31 06:27:31.269224] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:27:31.276755] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
[2017-05-31 06:27:31.276859] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
[2017-05-31 06:28:33.119250] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:28:33.126230] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
[2017-05-31 06:28:33.126328] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
[2017-05-31 06:29:35.271637] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:29:35.279799] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
[2017-05-31 06:29:35.279901] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
[2017-05-31 06:30:34.816144] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:30:34.824228] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
[2017-05-31 06:30:34.824319] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
[2017-05-31 06:30:36.304718] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:30:36.311898] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
[2017-05-31 06:30:36.311998] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
[2017-05-31 06:31:34.929107] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:31:34.936764] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
[2017-05-31 06:31:34.936843] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
[2017-05-31 06:31:41.145828] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:31:41.153317] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
[2017-05-31 06:31:41.153427] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
[2017-05-31 06:32:36.230320] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:32:36.239295] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
[2017-05-31 06:32:36.239420] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
[2017-05-31 06:32:36.481688] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:32:36.489207] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
[2017-05-31 06:32:36.489327] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
[2017-05-31 06:33:38.413891] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:33:38.422092] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
[2017-05-31 06:33:38.422188] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
[2017-05-31 06:33:41.618786] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:33:41.626214] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
[2017-05-31 06:33:41.626310] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
[2017-05-31 06:34:37.654479] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:34:37.661939] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
[2017-05-31 06:34:37.662043] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se/>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
[2017-05-31 06:36:12.557364] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:36:12.564751] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0
[2017-05-31 06:36:12.564861] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se/>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0


Cheers
Gabbe

30 maj 2017 kl. 14:45 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu at
redhat.com<mailto:sheggodu at redhat.com>>:

Hi Gabriel,

I am not able to hit the issue mentioned on my setup.

Please share the log files(both brick and client log files) from your setup. It
would be great if you can share the details about steps you followed to hit the
issue.


Regards,
Sunil kumar Acharya

Senior Software Engineer

Red Hat

<https://www.redhat.com/>

T: +91-8067935170<http://redhatemailsignature-marketing.itos.redhat.com/>

[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]<https://red.ht/sig>
TRIED. TESTED. TRUSTED.<https://redhat.com/trusted>


On Tue, May 30, 2017 at 3:30 PM, Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se<mailto:gabriel.lindeborg at svenskaspel.se>> wrote:
Hello

A manual mount failed the same way

Cheers
Gabbe

30 maj 2017 kl. 10:24 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu at
redhat.com<mailto:sheggodu at redhat.com>>:

Hi Gabriel,

Which gluster version are your running? Are you able to fuse mount the volume?

Please share the failure logs.

Regards,
Sunil kumar Acharya

Senior Software Engineer

Red Hat

<https://www.redhat.com/>

T: +91-8067935170<http://redhatemailsignature-marketing.itos.redhat.com/>

[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]<https://red.ht/sig>
TRIED. TESTED. TRUSTED.<https://redhat.com/trusted>


On Tue, May 30, 2017 at 1:04 PM, Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se<mailto:gabriel.lindeborg at svenskaspel.se>> wrote:
Hello All

We?ve have a problem with cluster client mounts fail in mid run with this in the
log
glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5)
[0x7f640c8b3dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
[0x7f640df4bfd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
[0x7f640df4bdfb] ) 0-: received signum (15), shutting down.

We?ve tried running debug but have not found anything suspicious happening at
the time of the failures
We?ve searched the inter web but can not find anyone else having the same
problem in mid flight

The clients have four mounts of volumes from the same server, all mounts fail
simultaneously
Peer status looks ok
Volume status looks ok
Volume info looks like this:
Volume Name: GLUSTERVOLUME
Type: Replicate
Volume ID: ca7af017-4f0f-44cc-baf6-43168eed0748
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: GLUSTERSERVER1:/gluster/GLUSTERVOLUME/brick
Brick2: GLUSTERSERVER2:/gluster/GLUSTERVOLUME/brick
Options Reconfigured:
transport.address-family: inet
cluster.self-heal-daemon: enable
nfs.disable: on
server.allow-insecure: on
client.bind-insecure: on
network.ping-timeout: 5
features.bitrot: on
features.scrub: Active
features.scrub-freq: weekly

Any ideas?

Cheers
Gabbe



AB SVENSKA SPEL
621 80 Visby
Norra Hansegatan 17, Visby
V?xel: +4610-120 00 00
https://svenskaspel.se<https://svenskaspel.se/>

Please consider the environment before printing this email
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users






-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170601/850b4bfa/attachment-0001.html>

Gabriel Lindeborg

2017-Jun-01 13:52 UTC

head link

[Gluster-users] Gluster client mount fails in mid flight with signum 15

This has been solved, as far as we can tell.

Problem was with KillUserProcesses=1 in logind.conf. This has shown to kill
mounts made using mount -a booth by root and by any user with sudo at session
logout.

Hope this will anybody else who run into this.

Thanks 4 all your help and
cheers
Gabbe

1 juni 2017 kl. 09:24 skrev Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se<mailto:gabriel.lindeborg at svenskaspel.se>>:

All four clients did run 3.10.2 as well

The volumes has been running fine until we upgraded to 3.10, when we hit some
issues with port mismatches. We restarted all the volumes, the servers and the
clients and now hit this issue.
We?ve since backed up the files, remove the volumes, removed the bricks, removed
gluster, installed glusterfs 3.7.20, created new volumes on new bricks, restored
the files and still hit the same issue at clients on the nodes that also runs
the servers. We?ve got to clients on connected to one of the volumes that has
been working fine all the time.

This is the debug logs from one of the mount as the client gets disconnected:
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht:
revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]"
repeated 26 times between [2017-05-31 13:48:51.680757] and [2017-05-31
13:50:46.325368]
/DAEMON/DEBUG [2017-05-31T15:50:50.589272+02:00] [] []
[logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out.
About to flush outstanding messages if present
/DAEMON/DEBUG [2017-05-31T15:50:50.589520+02:00] [] []
[logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting timer
now. Timeout = 120, current buf size = 5
[2017-05-31 13:50:51.908797] D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk]
0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs
cleaning]
/DAEMON/DEBUG [2017-05-31T15:51:24.592190+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:51:24.592469+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:51:26.325230+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:52:08.595536+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:52:08.595735+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:52:12.059895+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:52:12.060170+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
d
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht:
revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]"
repeated 26 times between [2017-05-31 13:50:51.908797] and [2017-05-31
13:52:46.326381]
/DAEMON/DEBUG [2017-05-31T15:52:50.598987+02:00] [] []
[logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out.
About to flush outstanding messages if present
/DAEMON/DEBUG [2017-05-31T15:52:50.599226+02:00] [] []
[logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting timer
now. Timeout = 120, current buf size = 5
[2017-05-31 13:52:52.138032] D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk]
0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs
cleaning]
/DAEMON/DEBUG [2017-05-31T15:52:54.599435+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:52:54.599718+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:52:56.325482+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:52:56.325731+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:53:38.603305+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:53:38.603533+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:53:42.226123+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:53:42.226345+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
d
/DAEMON/DEBUG [2017-05-31T15:54:24.607225+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
/DAEMON/DEBUG [2017-05-31T15:54:24.607479+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk] 0-mule-dht:
revalidate lookup of / returned with op_ret 0 [Structure needs cleaning]"
repeated 20 times between [2017-05-31 13:52:52.138032] and [2017-05-31
13:54:22.302116]
[2017-05-31 13:54:25.284676] W [glusterfsd.c:1251:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f36b20c6dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f36b3740915]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f36b374078b] ) 0-: received
signum (15), shutting down
/DAEMON/DEBUG [2017-05-31T15:54:25.285804+02:00] [] []
[logging.c:1766:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size
reduced. About to flush 1 extra log messages
/DAEMON/DEBUG [2017-05-31T15:54:25.286570+02:00] [] []
[logging.c:1769:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 1 extra
log messages
/DAEMON/DEBUG [2017-05-31T15:54:25.287431+02:00] [] []
[glusterfsd-mgmt.c:2361:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper
signout arguments not given
/DAEMON/INFO [2017-05-31T15:54:25.287785+02:00] [] [] [fuse-bridge.c:5720:fini]
0-fuse: Unmounting '/mnt/gluster/mule?.

Cheers
Gabbe
31 maj 2017 kl. 17:03 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu at
redhat.com<mailto:sheggodu at redhat.com>>:

Hi Gabriel,

What is the version of gluster you are running on client? Also, please share the
steps you followed to hit the issue.

Regards,
SUNIL KUMAR ACHARYA
SENIOR SOFTWARE ENGINEER
Red Hat

T: +91-8067935170

 [https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]
TRIED. TESTED. TRUSTED.


On Wed, May 31, 2017 at 12:10 PM, Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se> wrote:
Hello again

The volumes are old, since version 3.6 if I remember correctly?
These ?r det 40 last rows of the mnt-loggs go all the mounts

==> /var/log/glusterfs/mnt-gluster-alfresco.log <=/DAEMON/INFO
[2017-05-31T07:55:12.787035+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.153779+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.160995+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.167075+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.171872+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.427057+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.432239+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.439572+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.444286+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.035710+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.042366+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.049401+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.054483+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.524103+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.540314+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.543748+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.558459+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.623109+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.643979+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.648717+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.662941+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.268446+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.284765+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.287503+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.302461+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.408757+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.415831+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421736+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421887+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.431820+02:00] [] []
[2017-05-31 06:09:54.514841] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting
local read_child alfresco-client-2
/DAEMON/INFO [2017-05-31T08:11:10.179031+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.186811+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.194886+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.195062+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.205582+02:00] [] []
[2017-05-31 06:11:14.513620] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting
local read_child alfresco-client-2
[2017-05-31 06:21:21.579748] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa1d2e4cdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa1d44e4fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa1d44e4dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-31T08:21:21.580626+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581440+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-c1_2.log <= 82:     type meta
 83:     subvolumes c1
 84: end-volume
 85:
+------------------------------------------------------------------------------+
/DAEMON/ERR [2017-05-24T10:39:11.130038+02:00] [] []
[2017-05-24 08:39:11.130638] E [MSGID: 114058]
[client-handshake.c:1538:client_query_portmap_cbk] 2-c1-client-1: failed to get
the port number for remote subvolume. Please run 'gluster volume status'
on server to see if brick process is running.
[2017-05-24 08:39:11.130693] I [MSGID: 114018] [client.c:2276:client_rpc_notify]
2-c1-client-1: disconnected from c1-client-1. Client process will keep trying to
connect to glusterd until brick's port is available
[2017-05-24 08:39:11.130711] E [MSGID: 108006] [afr-common.c:4781:afr_notify]
2-c1-replicate-0: All subvolumes are down. Going offline until atleast one of
them comes back up.
/DAEMON/INFO [2017-05-24T10:39:11.471603+02:00] [] []
/DAEMON/ERR [2017-05-24T10:39:11.473783+02:00] [] []
/DAEMON/INFO [2017-05-24T10:39:14.479153+02:00] [] []
/DAEMON/ERR [2017-05-24T10:39:14.481430+02:00] [] []
[2017-05-24 08:39:14.513688] I [MSGID: 108006]
[afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up
/DAEMON/INFO [2017-05-24T10:39:14.513712+02:00] [] []
[2017-05-24 08:39:14.513858] I [MSGID: 114021] [client.c:2361:notify]
0-c1-client-2: current graph is no longer active, destroying rpc_client
[2017-05-24 08:39:14.513879] I [MSGID: 114021] [client.c:2361:notify]
0-c1-client-3: current graph is no longer active, destroying rpc_client
[2017-05-24 08:39:14.513933] I [MSGID: 108006]
[afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up
/DAEMON/WARNING [2017-05-24T10:39:14.513997+02:00] [] []
/DAEMON/WARNING [2017-05-24T10:39:14.514161+02:00] [] []
/DAEMON/INFO [2017-05-24T10:39:18.485909+02:00] [] []
/DAEMON/ERR [2017-05-24T10:39:18.488664+02:00] [] []
/DAEMON/INFO [2017-05-24T10:39:23.737962+02:00] [] []
[2017-05-24 08:39:23.740893] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 2-c1-client-0: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-24 08:39:23.742339] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-0: Connected to
c1-client-0, attached to remote volume '/gluster/c1/brick'.
[2017-05-24 08:39:23.742359] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-0: Server and Client
lk-version numbers are not same, reopening the fds
[2017-05-24 08:39:23.742437] I [MSGID: 108005] [afr-common.c:4756:afr_notify]
2-c1-replicate-0: Subvolume 'c1-client-0' came back up; going online.
[2017-05-24 08:39:23.743083] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-0: Server lk
version = 1
[2017-05-24 08:39:24.524310] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local
read_child c1-client-0
[2017-05-24 08:39:24.524414] I [MSGID: 109063]
[dht-layout.c:713:dht_layout_normalize] 2-c1-dht: Found anomalies in / (gfid =
00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
/DAEMON/INFO [2017-05-24T10:39:26.883379+02:00] [] []
[2017-05-24 08:39:26.885753] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 2-c1-client-1: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-24 08:39:26.887336] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-1: Connected to
c1-client-1, attached to remote volume '/gluster/c1/brick'.
[2017-05-24 08:39:26.887353] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-1: Server and Client
lk-version numbers are not same, reopening the fds
[2017-05-24 08:39:26.887479] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-1: Server lk
version = 1
[2017-05-24 08:39:34.515063] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local
read_child c1-client-0
The message "I [MSGID: 108006] [afr-common.c:4923:afr_local_init]
0-c1-replicate-0: no subvolumes up" repeated 4 times between [2017-05-24
08:38:24.514047] and [2017-05-24 08:39:04.514186]
/DAEMON/INFO [2017-05-24T10:40:56.356298+02:00] [] []
[2017-05-24 08:40:56.356694] W [glusterfsd.c:1329:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7feba7f55dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7feba95edf85]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7feba95eddfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-24T10:40:56.356773+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-c1.log <=/DAEMON/INFO
[2017-05-31T07:55:12.781267+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:12.786589+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.153964+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.161317+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.167226+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.172425+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.425676+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.433114+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.439753+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.444685+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.034185+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.042732+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.049094+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.053587+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.525624+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.540864+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.546015+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.546167+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.549146+02:00] [] []
[2017-05-31 05:56:14.515062] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-c1-replicate-0: selecting local
read_child c1-client-0
/DAEMON/INFO [2017-05-31T07:56:13.558847+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.622583+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.644252+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.648540+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.662541+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.269583+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.284254+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.291514+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.303324+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.408574+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.415968+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421254+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.431030+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.179194+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.187103+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.194300+02:00] [] []
[2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f7dad36fdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f7daea07fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f7daea07dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-31T08:11:10.205913+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581037+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581789+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-c2.log <= 66: end-volume
 67:
 68: volume c2-md-cache
 69:     type performance/md-cache
 70:     subvolumes c2-open-behind
 71: end-volume
 72:
 73: volume c2
 74:     type debug/io-stats
 75:     option log-level INFO
 76:     option latency-measurement off
 77:     option count-fop-hits off
 78:     subvolumes c2-md-cache
 79: end-volume
 80:
 81: volume meta-autoload
 82:     type meta
 83:     subvolumes c2
 84: end-volume
 85:
+------------------------------------------------------------------------------+
/DAEMON/INFO [2017-05-24T09:20:01.247296+02:00] [] []
[2017-05-24 07:20:01.249248] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 0-c2-client-0: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-24 07:20:01.249950] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-0: Connected to
c2-client-0, attached to remote volume '/gluster/c1/newbrick'.
[2017-05-24 07:20:01.249980] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-0: Server and Client
lk-version numbers are not same, reopening the fds
[2017-05-24 07:20:01.250050] I [MSGID: 108005] [afr-common.c:4756:afr_notify]
0-c2-replicate-0: Subvolume 'c2-client-0' came back up; going online.
[2017-05-24 07:20:01.250557] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-0: Server lk
version = 1
[2017-05-24 07:20:01.251421] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 0-c2-client-1: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-24 07:20:01.252016] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-1: Connected to
c2-client-1, attached to remote volume '/gluster/c1/newbrick'.
[2017-05-24 07:20:01.252055] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-1: Server and Client
lk-version numbers are not same, reopening the fds
[2017-05-24 07:20:01.253072] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-1: Server lk
version = 1
/DAEMON/INFO [2017-05-24T09:20:01.253273+02:00] [] []
/DAEMON/INFO [2017-05-24T09:20:01.253547+02:00] [] []
[2017-05-24 07:20:01.254056] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-c2-replicate-0: selecting local
read_child c2-client-0
[2017-05-24 07:20:01.254268] I [MSGID: 109063]
[dht-layout.c:713:dht_layout_normalize] 0-c2-dht: Found anomalies in / (gfid =
00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
/DAEMON/INFO [2017-05-24T10:39:11.116695+02:00] [] []
/DAEMON/INFO [2017-05-24T10:39:11.118327+02:00] [] []
/DAEMON/INFO [2017-05-24T10:40:47.577696+02:00] [] []
[2017-05-24 08:40:47.578001] W [glusterfsd.c:1329:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f3eaabefdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f3eac287f85]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f3eac287dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-24T10:40:47.578100+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-cdn.log <=/DAEMON/INFO
[2017-05-31T07:55:12.785775+02:00] [] []
[2017-05-31 05:55:14.515355] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local
read_child cdn-client-2
/DAEMON/INFO [2017-05-31T07:55:18.152981+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.161155+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.166602+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.171520+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.426866+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.432536+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.439418+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.443795+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.035303+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.042561+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.048928+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.053195+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.525267+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.540464+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.545703+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.559227+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.622792+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.644113+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.649031+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.649173+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.653281+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.663382+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.269801+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.283977+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.288305+02:00] [] []
[2017-05-31 05:56:24.514977] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local
read_child cdn-client-2
/DAEMON/INFO [2017-05-31T07:56:26.302883+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.408914+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.415563+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421558+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.432374+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.178859+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.186967+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.193439+02:00] [] []
[2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa420a29dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa4220c1fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa4220c1dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-31T08:11:10.204701+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581209+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581945+02:00] [] []

==> /var/log/glusterfs/mnt-gluster-mule.log <=/DAEMON/INFO
[2017-05-31T07:55:18.154147+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.161457+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.168560+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.168765+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:18.172803+02:00] [] []
[2017-05-31 05:55:24.517077] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local
read_child mule-client-2
/DAEMON/INFO [2017-05-31T07:55:46.426664+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.432816+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.439270+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:46.443344+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.035528+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.042893+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.049247+02:00] [] []
/DAEMON/INFO [2017-05-31T07:55:52.054006+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.525469+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.540732+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.545856+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:13.559699+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.622955+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.644386+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.648877+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:18.663766+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.269958+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.284895+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.291677+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.291845+02:00] [] []
/DAEMON/INFO [2017-05-31T07:56:26.294037+02:00] [] []
[2017-05-31 05:56:34.516246] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local
read_child mule-client-2
/DAEMON/INFO [2017-05-31T07:56:26.303715+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.408417+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.416098+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.421403+02:00] [] []
/DAEMON/INFO [2017-05-31T08:09:49.431421+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.178644+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.187364+02:00] [] []
/DAEMON/INFO [2017-05-31T08:11:10.193590+02:00] [] []
[2017-05-31 06:21:21.579746] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fbd1beafdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fbd1d547fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbd1d547dfb] ) 0-: received
signum (15), shutting down
/DAEMON/INFO [2017-05-31T08:11:10.205040+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.580846+02:00] [] []
/DAEMON/INFO [2017-05-31T08:21:21.581623+02:00] [] []

And these are 40 last rows of the brick
==> /var/log/glusterfs/bricks/gluster-alfresco-brick.log <=[2017-05-31
05:52:19.117394] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms02.horisont.svenskaspel.se-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0
[2017-05-31 05:52:19.117469] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0
[2017-05-31 05:52:43.020403] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:52:43.027816] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
[2017-05-31 05:52:43.027931] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
[2017-05-31 05:56:48.604780] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms02.horisont.svenskaspel.se-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:57:50.817223] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:57:50.825987] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
[2017-05-31 05:57:50.826138] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
[2017-05-31 05:58:25.746161] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms02.horisont.svenskaspel.se-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:58:25.755174] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms02.horisont.svenskaspel.se-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
[2017-05-31 05:58:25.755279] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
[2017-05-31 06:02:47.958784] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:02:47.976543] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
[2017-05-31 06:02:47.976683] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
[2017-05-31 06:07:53.126343] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:07:53.133904] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
[2017-05-31 06:07:53.133998] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
[2017-05-31 06:12:56.138792] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:12:56.146253] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
[2017-05-31 06:12:56.146351] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
[2017-05-31 06:16:22.552274] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms02.horisont.svenskaspel.se-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
[2017-05-31 06:16:22.552428] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
[2017-05-31 06:17:54.775497] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:17:54.792370] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
[2017-05-31 06:17:54.792446] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
[2017-05-31 06:21:21.582947] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0
[2017-05-31 06:21:21.583181] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0
[2017-05-31 06:21:42.714049] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:21:42.722227] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
[2017-05-31 06:21:42.722320] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
[2017-05-31 06:22:35.445697] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:22:35.453208] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
[2017-05-31 06:22:35.453334] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
[2017-05-31 06:22:58.071308] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:22:58.078524] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
[2017-05-31 06:22:58.078608] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
[2017-05-31 06:28:06.645148] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:28:06.652828] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms01.horisont.svenskaspel.se-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0
[2017-05-31 06:28:06.652942] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0

==> /var/log/glusterfs/bricks/gluster-c1-brick.log <=[2017-05-31
05:49:00.793792] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server:
Shutting down connection
tms02.horisont.svenskaspel.se-39066-2017/05/31-05:29:48:927855-c1-client-0-0-0
[2017-05-31 05:52:36.512431] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 05:52:36.522623] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
[2017-05-31 05:52:36.522723] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
[2017-05-31 05:56:13.544120] I [MSGID: 101190]
[event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 2
[2017-05-31 05:56:48.796772] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 05:58:39.772391] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 05:58:39.782681] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
[2017-05-31 05:58:39.782820] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
[2017-05-31 06:03:45.195471] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:03:45.205497] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
[2017-05-31 06:03:45.205610] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
[2017-05-31 06:08:45.229186] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:08:45.239068] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
[2017-05-31 06:08:45.239173] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
[2017-05-31 06:13:51.433086] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:13:51.443161] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
[2017-05-31 06:13:51.443257] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
[2017-05-31 06:16:22.552265] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
[2017-05-31 06:16:22.552418] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
[2017-05-31 06:18:55.232358] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:18:55.242110] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
[2017-05-31 06:18:55.242188] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
[2017-05-31 06:21:21.586024] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0
[2017-05-31 06:21:21.586153] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0
[2017-05-31 06:21:34.443201] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:21:34.453810] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
[2017-05-31 06:21:34.453897] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
[2017-05-31 06:23:55.706224] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:23:55.716126] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
[2017-05-31 06:23:55.716221] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
[2017-05-31 06:28:58.213831] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:28:58.223474] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
[2017-05-31 06:28:58.223557] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
[2017-05-31 06:33:07.622832] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:33:07.642391] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
[2017-05-31 06:33:07.642520] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
[2017-05-31 06:34:07.776928] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0
(version: 3.10.2)
[2017-05-31 06:34:07.787116] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0
[2017-05-31 06:34:07.787200] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0

==> /var/log/glusterfs/bricks/gluster-cdn-brick.log <=[2017-05-31
05:50:18.876202] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-40008-2017/05/31-05:50:18:867052-cdn-client-2-0-0
[2017-05-31 05:55:09.869166] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:55:09.877790] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
[2017-05-31 05:55:09.877898] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
[2017-05-31 05:55:20.944369] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 05:55:20.952275] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
[2017-05-31 05:55:20.952379] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
[2017-05-31 05:56:48.701158] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:00:20.651639] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:00:20.659659] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
[2017-05-31 06:00:20.659782] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
[2017-05-31 06:01:17.578233] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:01:17.587267] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
[2017-05-31 06:01:17.587370] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
[2017-05-31 06:05:25.961313] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:05:25.976281] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
[2017-05-31 06:05:25.976378] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
[2017-05-31 06:10:24.626906] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:10:24.634914] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
[2017-05-31 06:10:24.635016] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
[2017-05-31 06:15:21.944215] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:15:21.952554] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
[2017-05-31 06:15:21.952655] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
[2017-05-31 06:16:22.552086] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
[2017-05-31 06:16:22.552203] I [MSGID: 115013]
[server-helpers.c:296:do_fd_cleanup] 0-cdn-server: fd cleanup on
/test2/nyx/nyxclient/assets.json
[2017-05-31 06:16:22.552416] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
[2017-05-31 06:20:25.255523] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:20:25.263860] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
[2017-05-31 06:20:25.263963] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
[2017-05-31 06:21:21.584357] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0
[2017-05-31 06:21:21.584449] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0
[2017-05-31 06:21:37.944798] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms01.horisont.svenskaspel.se-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:21:37.952580] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
[2017-05-31 06:21:37.952712] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
[2017-05-31 06:25:27.632381] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:25:27.640591] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
[2017-05-31 06:25:27.640721] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
[2017-05-31 06:35:42.166035] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms01.horisont.svenskaspel.se-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:35:42.173662] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0
[2017-05-31 06:35:42.173752] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0

==> /var/log/glusterfs/bricks/gluster-mule-brick.log <=[2017-05-31
06:26:30.580063] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-38988-2017/05/31-06:26:30:551135-mule-client-2-0-0
[2017-05-31 06:27:31.269224] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:27:31.276755] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
[2017-05-31 06:27:31.276859] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
[2017-05-31 06:28:33.119250] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:28:33.126230] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
[2017-05-31 06:28:33.126328] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
[2017-05-31 06:29:35.271637] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:29:35.279799] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
[2017-05-31 06:29:35.279901] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
[2017-05-31 06:30:34.816144] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:30:34.824228] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
[2017-05-31 06:30:34.824319] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
[2017-05-31 06:30:36.304718] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:30:36.311898] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
[2017-05-31 06:30:36.311998] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
[2017-05-31 06:31:34.929107] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:31:34.936764] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
[2017-05-31 06:31:34.936843] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
[2017-05-31 06:31:41.145828] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:31:41.153317] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
[2017-05-31 06:31:41.153427] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
[2017-05-31 06:32:36.230320] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:32:36.239295] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
[2017-05-31 06:32:36.239420] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
[2017-05-31 06:32:36.481688] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:32:36.489207] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
[2017-05-31 06:32:36.489327] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
[2017-05-31 06:33:38.413891] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:33:38.422092] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
[2017-05-31 06:33:38.422188] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
[2017-05-31 06:33:41.618786] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:33:41.626214] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
[2017-05-31 06:33:41.626310] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
[2017-05-31 06:34:37.654479] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:34:37.661939] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
[2017-05-31 06:34:37.662043] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
[2017-05-31 06:36:12.557364] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0
(version: 3.10.2)
[2017-05-31 06:36:12.564751] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0
[2017-05-31 06:36:12.564861] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0


Cheers
Gabbe

30 maj 2017 kl. 14:45 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu at
redhat.com>:

Hi Gabriel,

I am not able to hit the issue mentioned on my setup.

Please share the log files(both brick and client log files) from your setup. It
would be great if you can share the details about steps you followed to hit the
issue.


Regards,
SUNIL KUMAR ACHARYA
SENIOR SOFTWARE ENGINEER
Red Hat

T: +91-8067935170

[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]
TRIED. TESTED. TRUSTED.


On Tue, May 30, 2017 at 3:30 PM, Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se> wrote:
Hello

A manual mount failed the same way

Cheers
Gabbe

30 maj 2017 kl. 10:24 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu at
redhat.com>:

Hi Gabriel,

Which gluster version are your running? Are you able to fuse mount the volume?

Please share the failure logs.

Regards,
SUNIL KUMAR ACHARYA
SENIOR SOFTWARE ENGINEER
Red Hat

T: +91-8067935170

[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]
TRIED. TESTED. TRUSTED.


On Tue, May 30, 2017 at 1:04 PM, Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se> wrote:
Hello All

We?ve have a problem with cluster client mounts fail in mid run with this in the
log
glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5)
[0x7f640c8b3dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
[0x7f640df4bfd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
[0x7f640df4bdfb] ) 0-: received signum (15), shutting down.

We?ve tried running debug but have not found anything suspicious happening at
the time of the failures
We?ve searched the inter web but can not find anyone else having the same
problem in mid flight

The clients have four mounts of volumes from the same server, all mounts fail
simultaneously
Peer status looks ok
Volume status looks ok
Volume info looks like this:
Volume Name: GLUSTERVOLUME
Type: Replicate
Volume ID: ca7af017-4f0f-44cc-baf6-43168eed0748
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: GLUSTERSERVER1:/gluster/GLUSTERVOLUME/brick
Brick2: GLUSTERSERVER2:/gluster/GLUSTERVOLUME/brick
Options Reconfigured:
transport.address-family: inet
cluster.self-heal-daemon: enable
nfs.disable: on
server.allow-insecure: on
client.bind-insecure: on
network.ping-timeout: 5
features.bitrot: on
features.scrub: Active
features.scrub-freq: weekly

Any ideas?

Cheers
Gabbe



AB SVENSKA SPEL
621 80 Visby
Norra Hansegatan 17, Visby
V?xel: +4610-120 00 00
https://svenskaspel.se

Please consider the environment before printing this email
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users







-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170601/2e2301eb/attachment.html>

Niels de Vos

2017-Jun-01 17:31 UTC

head link

[Gluster-users] Gluster client mount fails in mid flight with signum 15

On Thu, Jun 01, 2017 at 01:52:23PM +0000, Gabriel Lindeborg
wrote:> This has been solved, as far as we can tell.
> 
> Problem was with KillUserProcesses=1 in logind.conf. This has shown to
> kill mounts made using mount -a booth by root and by any user with
> sudo at session logout.
Ah, yes, that could well be the cause of the problem.
> Hope this will anybody else who run into this.
Care to share how you solved it? Just disabling the option might not be
the most suitable approach. Did you convert it to systemd.mount units,
or maybe setup automounting with x-systemd.automount or autofs? Were
there considerations that made you choose one solution over an other?

Thanks!
Niels

> Thanks 4 all your help and
> cheers
> Gabbe
> 
> 1 juni 2017 kl. 09:24 skrev Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se<mailto:gabriel.lindeborg at svenskaspel.se>>:
> 
> All four clients did run 3.10.2 as well
> 
> The volumes has been running fine until we upgraded to 3.10, when we hit
some issues with port mismatches. We restarted all the volumes, the servers and
the clients and now hit this issue.
> We?ve since backed up the files, remove the volumes, removed the bricks,
removed gluster, installed glusterfs 3.7.20, created new volumes on new bricks,
restored the files and still hit the same issue at clients on the nodes that
also runs the servers. We?ve got to clients on connected to one of the volumes
that has been working fine all the time.
> 
> This is the debug logs from one of the mount as the client gets
disconnected:
> The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk]
0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs
cleaning]" repeated 26 times between [2017-05-31 13:48:51.680757] and
[2017-05-31 13:50:46.325368]
> /DAEMON/DEBUG [2017-05-31T15:50:50.589272+02:00] [] []
[logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out.
About to flush outstanding messages if present
> /DAEMON/DEBUG [2017-05-31T15:50:50.589520+02:00] [] []
[logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting timer
now. Timeout = 120, current buf size = 5
> [2017-05-31 13:50:51.908797] D [MSGID: 0]
[dht-common.c:979:dht_revalidate_cbk] 0-mule-dht: revalidate lookup of /
returned with op_ret 0 [Structure needs cleaning]
> /DAEMON/DEBUG [2017-05-31T15:51:24.592190+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:51:24.592469+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:51:26.324867+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
> 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
> d
> /DAEMON/DEBUG [2017-05-31T15:51:26.325230+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
> 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
> d
> /DAEMON/DEBUG [2017-05-31T15:52:08.595536+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:52:08.595735+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:52:12.059895+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
> 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
> d
> /DAEMON/DEBUG [2017-05-31T15:52:12.060170+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
> 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
> d
> The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk]
0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs
cleaning]" repeated 26 times between [2017-05-31 13:50:51.908797] and
[2017-05-31 13:52:46.326381]
> /DAEMON/DEBUG [2017-05-31T15:52:50.598987+02:00] [] []
[logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out.
About to flush outstanding messages if present
> /DAEMON/DEBUG [2017-05-31T15:52:50.599226+02:00] [] []
[logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting timer
now. Timeout = 120, current buf size = 5
> [2017-05-31 13:52:52.138032] D [MSGID: 0]
[dht-common.c:979:dht_revalidate_cbk] 0-mule-dht: revalidate lookup of /
returned with op_ret 0 [Structure needs cleaning]
> /DAEMON/DEBUG [2017-05-31T15:52:54.599435+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:52:54.599718+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:52:56.325482+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
> 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
> d
> /DAEMON/DEBUG [2017-05-31T15:52:56.325731+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
> 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
> d
> /DAEMON/DEBUG [2017-05-31T15:53:38.603305+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:53:38.603533+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:53:42.226123+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
> 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.179:49155: ping timer event already remove
> d
> /DAEMON/DEBUG [2017-05-31T15:53:42.226345+02:00] [] []
[rpc-clnt-ping.c:98:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f36b3260192] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f36b302f9db]
(--> /lib
> 64/libgfrpc.so.0(+0x13fd4)[0x7f36b302ffd4] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f36b302cf01] (-->
/usr/lib64/glusterfs/3.7.20/xlator/protocol/client.so(client_submit_request+0x1fc)[0x7f36a599c33c]
))))) 0-: 10.3.48.180:49155: ping timer event already remove
> d
> /DAEMON/DEBUG [2017-05-31T15:54:24.607225+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
> /DAEMON/DEBUG [2017-05-31T15:54:24.607479+02:00] [] []
[rpc-clnt-ping.c:300:rpc_clnt_start_ping] 0-mule-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
> The message "D [MSGID: 0] [dht-common.c:979:dht_revalidate_cbk]
0-mule-dht: revalidate lookup of / returned with op_ret 0 [Structure needs
cleaning]" repeated 20 times between [2017-05-31 13:52:52.138032] and
[2017-05-31 13:54:22.302116]
> [2017-05-31 13:54:25.284676] W [glusterfsd.c:1251:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f36b20c6dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f36b3740915]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f36b374078b] ) 0-: received
signum (15), shutting down
> /DAEMON/DEBUG [2017-05-31T15:54:25.285804+02:00] [] []
[logging.c:1766:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size
reduced. About to flush 1 extra log messages
> /DAEMON/DEBUG [2017-05-31T15:54:25.286570+02:00] [] []
[logging.c:1769:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 1 extra
log messages
> /DAEMON/DEBUG [2017-05-31T15:54:25.287431+02:00] [] []
[glusterfsd-mgmt.c:2361:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper
signout arguments not given
> /DAEMON/INFO [2017-05-31T15:54:25.287785+02:00] [] []
[fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/mnt/gluster/mule?.
> 
> Cheers
> Gabbe
> 31 maj 2017 kl. 17:03 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu
at redhat.com<mailto:sheggodu at redhat.com>>:
> 
> Hi Gabriel,
> 
> What is the version of gluster you are running on client? Also, please
share the steps you followed to hit the issue.
> 
> Regards,
> SUNIL KUMAR ACHARYA
> SENIOR SOFTWARE ENGINEER
> Red Hat
> 
> T: +91-8067935170
> 
> 
[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]
> TRIED. TESTED. TRUSTED.
> 
> 
> On Wed, May 31, 2017 at 12:10 PM, Gabriel Lindeborg <gabriel.lindeborg
at svenskaspel.se> wrote:
> Hello again
> 
> The volumes are old, since version 3.6 if I remember correctly?
> These ?r det 40 last rows of the mnt-loggs go all the mounts
> 
> ==> /var/log/glusterfs/mnt-gluster-alfresco.log <=> /DAEMON/INFO
[2017-05-31T07:55:12.787035+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.153779+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.160995+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.167075+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.171872+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.427057+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.432239+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.439572+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.444286+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.035710+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.042366+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.049401+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.054483+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.524103+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.540314+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.543748+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.558459+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.623109+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.643979+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.648717+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.662941+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.268446+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.284765+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.287503+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.302461+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.408757+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.415831+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.421736+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.421887+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.431820+02:00] [] []
> [2017-05-31 06:09:54.514841] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting
local read_child alfresco-client-2
> /DAEMON/INFO [2017-05-31T08:11:10.179031+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.186811+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.194886+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.195062+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.205582+02:00] [] []
> [2017-05-31 06:11:14.513620] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting
local read_child alfresco-client-2
> [2017-05-31 06:21:21.579748] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa1d2e4cdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa1d44e4fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa1d44e4dfb] ) 0-: received
signum (15), shutting down
> /DAEMON/INFO [2017-05-31T08:21:21.580626+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:21:21.581440+02:00] [] []
> 
> ==> /var/log/glusterfs/mnt-gluster-c1_2.log <=>  82:     type meta
>  83:     subvolumes c1
>  84: end-volume
>  85:
>
+------------------------------------------------------------------------------+
> /DAEMON/ERR [2017-05-24T10:39:11.130038+02:00] [] []
> [2017-05-24 08:39:11.130638] E [MSGID: 114058]
[client-handshake.c:1538:client_query_portmap_cbk] 2-c1-client-1: failed to get
the port number for remote subvolume. Please run 'gluster volume status'
on server to see if brick process is running.
> [2017-05-24 08:39:11.130693] I [MSGID: 114018]
[client.c:2276:client_rpc_notify] 2-c1-client-1: disconnected from c1-client-1.
Client process will keep trying to connect to glusterd until brick's port is
available
> [2017-05-24 08:39:11.130711] E [MSGID: 108006]
[afr-common.c:4781:afr_notify] 2-c1-replicate-0: All subvolumes are down. Going
offline until atleast one of them comes back up.
> /DAEMON/INFO [2017-05-24T10:39:11.471603+02:00] [] []
> /DAEMON/ERR [2017-05-24T10:39:11.473783+02:00] [] []
> /DAEMON/INFO [2017-05-24T10:39:14.479153+02:00] [] []
> /DAEMON/ERR [2017-05-24T10:39:14.481430+02:00] [] []
> [2017-05-24 08:39:14.513688] I [MSGID: 108006]
[afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up
> /DAEMON/INFO [2017-05-24T10:39:14.513712+02:00] [] []
> [2017-05-24 08:39:14.513858] I [MSGID: 114021] [client.c:2361:notify]
0-c1-client-2: current graph is no longer active, destroying rpc_client
> [2017-05-24 08:39:14.513879] I [MSGID: 114021] [client.c:2361:notify]
0-c1-client-3: current graph is no longer active, destroying rpc_client
> [2017-05-24 08:39:14.513933] I [MSGID: 108006]
[afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up
> /DAEMON/WARNING [2017-05-24T10:39:14.513997+02:00] [] []
> /DAEMON/WARNING [2017-05-24T10:39:14.514161+02:00] [] []
> /DAEMON/INFO [2017-05-24T10:39:18.485909+02:00] [] []
> /DAEMON/ERR [2017-05-24T10:39:18.488664+02:00] [] []
> /DAEMON/INFO [2017-05-24T10:39:23.737962+02:00] [] []
> [2017-05-24 08:39:23.740893] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 2-c1-client-0: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
> [2017-05-24 08:39:23.742339] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-0: Connected to
c1-client-0, attached to remote volume '/gluster/c1/brick'.
> [2017-05-24 08:39:23.742359] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-0: Server and Client
lk-version numbers are not same, reopening the fds
> [2017-05-24 08:39:23.742437] I [MSGID: 108005]
[afr-common.c:4756:afr_notify] 2-c1-replicate-0: Subvolume 'c1-client-0'
came back up; going online.
> [2017-05-24 08:39:23.743083] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-0: Server lk
version = 1
> [2017-05-24 08:39:24.524310] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local
read_child c1-client-0
> [2017-05-24 08:39:24.524414] I [MSGID: 109063]
[dht-layout.c:713:dht_layout_normalize] 2-c1-dht: Found anomalies in / (gfid =
00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
> /DAEMON/INFO [2017-05-24T10:39:26.883379+02:00] [] []
> [2017-05-24 08:39:26.885753] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 2-c1-client-1: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
> [2017-05-24 08:39:26.887336] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-1: Connected to
c1-client-1, attached to remote volume '/gluster/c1/brick'.
> [2017-05-24 08:39:26.887353] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-1: Server and Client
lk-version numbers are not same, reopening the fds
> [2017-05-24 08:39:26.887479] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-1: Server lk
version = 1
> [2017-05-24 08:39:34.515063] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local
read_child c1-client-0
> The message "I [MSGID: 108006] [afr-common.c:4923:afr_local_init]
0-c1-replicate-0: no subvolumes up" repeated 4 times between [2017-05-24
08:38:24.514047] and [2017-05-24 08:39:04.514186]
> /DAEMON/INFO [2017-05-24T10:40:56.356298+02:00] [] []
> [2017-05-24 08:40:56.356694] W [glusterfsd.c:1329:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7feba7f55dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7feba95edf85]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7feba95eddfb] ) 0-: received
signum (15), shutting down
> /DAEMON/INFO [2017-05-24T10:40:56.356773+02:00] [] []
> 
> ==> /var/log/glusterfs/mnt-gluster-c1.log <=> /DAEMON/INFO
[2017-05-31T07:55:12.781267+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:12.786589+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.153964+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.161317+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.167226+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.172425+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.425676+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.433114+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.439753+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.444685+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.034185+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.042732+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.049094+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.053587+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.525624+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.540864+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.546015+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.546167+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.549146+02:00] [] []
> [2017-05-31 05:56:14.515062] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-c1-replicate-0: selecting local
read_child c1-client-0
> /DAEMON/INFO [2017-05-31T07:56:13.558847+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.622583+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.644252+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.648540+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.662541+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.269583+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.284254+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.291514+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.303324+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.408574+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.415968+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.421254+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.431030+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.179194+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.187103+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.194300+02:00] [] []
> [2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f7dad36fdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f7daea07fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f7daea07dfb] ) 0-: received
signum (15), shutting down
> /DAEMON/INFO [2017-05-31T08:11:10.205913+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:21:21.581037+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:21:21.581789+02:00] [] []
> 
> ==> /var/log/glusterfs/mnt-gluster-c2.log <=>  66: end-volume
>  67:
>  68: volume c2-md-cache
>  69:     type performance/md-cache
>  70:     subvolumes c2-open-behind
>  71: end-volume
>  72:
>  73: volume c2
>  74:     type debug/io-stats
>  75:     option log-level INFO
>  76:     option latency-measurement off
>  77:     option count-fop-hits off
>  78:     subvolumes c2-md-cache
>  79: end-volume
>  80:
>  81: volume meta-autoload
>  82:     type meta
>  83:     subvolumes c2
>  84: end-volume
>  85:
>
+------------------------------------------------------------------------------+
> /DAEMON/INFO [2017-05-24T09:20:01.247296+02:00] [] []
> [2017-05-24 07:20:01.249248] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 0-c2-client-0: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
> [2017-05-24 07:20:01.249950] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-0: Connected to
c2-client-0, attached to remote volume '/gluster/c1/newbrick'.
> [2017-05-24 07:20:01.249980] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-0: Server and Client
lk-version numbers are not same, reopening the fds
> [2017-05-24 07:20:01.250050] I [MSGID: 108005]
[afr-common.c:4756:afr_notify] 0-c2-replicate-0: Subvolume 'c2-client-0'
came back up; going online.
> [2017-05-24 07:20:01.250557] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-0: Server lk
version = 1
> [2017-05-24 07:20:01.251421] I [MSGID: 114057]
[client-handshake.c:1451:select_server_supported_programs] 0-c2-client-1: Using
Program GlusterFS 3.3, Num (1298437), Version (330)
> [2017-05-24 07:20:01.252016] I [MSGID: 114046]
[client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-1: Connected to
c2-client-1, attached to remote volume '/gluster/c1/newbrick'.
> [2017-05-24 07:20:01.252055] I [MSGID: 114047]
[client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-1: Server and Client
lk-version numbers are not same, reopening the fds
> [2017-05-24 07:20:01.253072] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-1: Server lk
version = 1
> /DAEMON/INFO [2017-05-24T09:20:01.253273+02:00] [] []
> /DAEMON/INFO [2017-05-24T09:20:01.253547+02:00] [] []
> [2017-05-24 07:20:01.254056] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-c2-replicate-0: selecting local
read_child c2-client-0
> [2017-05-24 07:20:01.254268] I [MSGID: 109063]
[dht-layout.c:713:dht_layout_normalize] 0-c2-dht: Found anomalies in / (gfid =
00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
> /DAEMON/INFO [2017-05-24T10:39:11.116695+02:00] [] []
> /DAEMON/INFO [2017-05-24T10:39:11.118327+02:00] [] []
> /DAEMON/INFO [2017-05-24T10:40:47.577696+02:00] [] []
> [2017-05-24 08:40:47.578001] W [glusterfsd.c:1329:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7f3eaabefdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f3eac287f85]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f3eac287dfb] ) 0-: received
signum (15), shutting down
> /DAEMON/INFO [2017-05-24T10:40:47.578100+02:00] [] []
> 
> ==> /var/log/glusterfs/mnt-gluster-cdn.log <=> /DAEMON/INFO
[2017-05-31T07:55:12.785775+02:00] [] []
> [2017-05-31 05:55:14.515355] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local
read_child cdn-client-2
> /DAEMON/INFO [2017-05-31T07:55:18.152981+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.161155+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.166602+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.171520+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.426866+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.432536+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.439418+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.443795+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.035303+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.042561+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.048928+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.053195+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.525267+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.540464+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.545703+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.559227+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.622792+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.644113+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.649031+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.649173+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.653281+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.663382+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.269801+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.283977+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.288305+02:00] [] []
> [2017-05-31 05:56:24.514977] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local
read_child cdn-client-2
> /DAEMON/INFO [2017-05-31T07:56:26.302883+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.408914+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.415563+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.421558+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.432374+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.178859+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.186967+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.193439+02:00] [] []
> [2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa420a29dc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa4220c1fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa4220c1dfb] ) 0-: received
signum (15), shutting down
> /DAEMON/INFO [2017-05-31T08:11:10.204701+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:21:21.581209+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:21:21.581945+02:00] [] []
> 
> ==> /var/log/glusterfs/mnt-gluster-mule.log <=> /DAEMON/INFO
[2017-05-31T07:55:18.154147+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.161457+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.168560+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.168765+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:18.172803+02:00] [] []
> [2017-05-31 05:55:24.517077] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local
read_child mule-client-2
> /DAEMON/INFO [2017-05-31T07:55:46.426664+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.432816+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.439270+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:46.443344+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.035528+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.042893+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.049247+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:55:52.054006+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.525469+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.540732+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.545856+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:13.559699+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.622955+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.644386+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.648877+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:18.663766+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.269958+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.284895+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.291677+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.291845+02:00] [] []
> /DAEMON/INFO [2017-05-31T07:56:26.294037+02:00] [] []
> [2017-05-31 05:56:34.516246] I [MSGID: 108031]
[afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local
read_child mule-client-2
> /DAEMON/INFO [2017-05-31T07:56:26.303715+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.408417+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.416098+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.421403+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:09:49.431421+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.178644+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.187364+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:11:10.193590+02:00] [] []
> [2017-05-31 06:21:21.579746] W [glusterfsd.c:1332:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dc5) [0x7fbd1beafdc5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fbd1d547fd5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbd1d547dfb] ) 0-: received
signum (15), shutting down
> /DAEMON/INFO [2017-05-31T08:11:10.205040+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:21:21.580846+02:00] [] []
> /DAEMON/INFO [2017-05-31T08:21:21.581623+02:00] [] []
> 
> And these are 40 last rows of the brick
> ==> /var/log/glusterfs/bricks/gluster-alfresco-brick.log <=>
[2017-05-31 05:52:19.117394] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-alfresco-server: disconnecting connection from
tms02.horisont.svenskaspel.se-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0
> [2017-05-31 05:52:19.117469] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0
> [2017-05-31 05:52:43.020403] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 05:52:43.027816] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
> [2017-05-31 05:52:43.027931] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0
> [2017-05-31 05:56:48.604780] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms02.horisont.svenskaspel.se-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 05:57:50.817223] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 05:57:50.825987] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
> [2017-05-31 05:57:50.826138] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0
> [2017-05-31 05:58:25.746161] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms02.horisont.svenskaspel.se-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 05:58:25.755174] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms02.horisont.svenskaspel.se-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
> [2017-05-31 05:58:25.755279] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0
> [2017-05-31 06:02:47.958784] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:02:47.976543] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
> [2017-05-31 06:02:47.976683] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0
> [2017-05-31 06:07:53.126343] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:07:53.133904] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
> [2017-05-31 06:07:53.133998] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0
> [2017-05-31 06:12:56.138792] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:12:56.146253] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
> [2017-05-31 06:12:56.146351] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0
> [2017-05-31 06:16:22.552274] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms02.horisont.svenskaspel.se-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
> [2017-05-31 06:16:22.552428] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms02.horisont.svenskaspel.se-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0
> [2017-05-31 06:17:54.775497] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:17:54.792370] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
> [2017-05-31 06:17:54.792446] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0
> [2017-05-31 06:21:21.582947] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0
> [2017-05-31 06:21:21.583181] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0
> [2017-05-31 06:21:42.714049] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:21:42.722227] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
> [2017-05-31 06:21:42.722320] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0
> [2017-05-31 06:22:35.445697] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:22:35.453208] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
> [2017-05-31 06:22:35.453334] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0
> [2017-05-31 06:22:58.071308] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:22:58.078524] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
> [2017-05-31 06:22:58.078608] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0
> [2017-05-31 06:28:06.645148] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client
from
tms01.horisont.svenskaspel.se-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:28:06.652828] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection
from
tms01.horisont.svenskaspel.se-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0
> [2017-05-31 06:28:06.652942] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection
tms01.horisont.svenskaspel.se-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0
> 
> ==> /var/log/glusterfs/bricks/gluster-c1-brick.log <=> [2017-05-31
05:49:00.793792] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server:
Shutting down connection
tms02.horisont.svenskaspel.se-39066-2017/05/31-05:29:48:927855-c1-client-0-0-0
> [2017-05-31 05:52:36.512431] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 05:52:36.522623] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
> [2017-05-31 05:52:36.522723] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0
> [2017-05-31 05:56:13.544120] I [MSGID: 101190]
[event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 2
> [2017-05-31 05:56:48.796772] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 05:58:39.772391] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 05:58:39.782681] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
> [2017-05-31 05:58:39.782820] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0
> [2017-05-31 06:03:45.195471] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:03:45.205497] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
> [2017-05-31 06:03:45.205610] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0
> [2017-05-31 06:08:45.229186] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:08:45.239068] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
> [2017-05-31 06:08:45.239173] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0
> [2017-05-31 06:13:51.433086] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:13:51.443161] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
> [2017-05-31 06:13:51.443257] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0
> [2017-05-31 06:16:22.552265] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
> [2017-05-31 06:16:22.552418] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0
> [2017-05-31 06:18:55.232358] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:18:55.242110] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
> [2017-05-31 06:18:55.242188] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0
> [2017-05-31 06:21:21.586024] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0
> [2017-05-31 06:21:21.586153] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0
> [2017-05-31 06:21:34.443201] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:21:34.453810] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
> [2017-05-31 06:21:34.453897] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0
> [2017-05-31 06:23:55.706224] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:23:55.716126] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
> [2017-05-31 06:23:55.716221] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0
> [2017-05-31 06:28:58.213831] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:28:58.223474] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
> [2017-05-31 06:28:58.223557] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0
> [2017-05-31 06:33:07.622832] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms01.horisont.svenskaspel.se-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:33:07.642391] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms01.horisont.svenskaspel.se-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
> [2017-05-31 06:33:07.642520] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms01.horisont.svenskaspel.se-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0
> [2017-05-31 06:34:07.776928] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from
tms02.horisont.svenskaspel.se-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0
(version: 3.10.2)
> [2017-05-31 06:34:07.787116] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from
tms02.horisont.svenskaspel.se-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0
> [2017-05-31 06:34:07.787200] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection
tms02.horisont.svenskaspel.se-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0
> 
> ==> /var/log/glusterfs/bricks/gluster-cdn-brick.log <=>
[2017-05-31 05:50:18.876202] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-40008-2017/05/31-05:50:18:867052-cdn-client-2-0-0
> [2017-05-31 05:55:09.869166] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 05:55:09.877790] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
> [2017-05-31 05:55:09.877898] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0
> [2017-05-31 05:55:20.944369] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 05:55:20.952275] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
> [2017-05-31 05:55:20.952379] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0
> [2017-05-31 05:56:48.701158] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:00:20.651639] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:00:20.659659] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
> [2017-05-31 06:00:20.659782] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0
> [2017-05-31 06:01:17.578233] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:01:17.587267] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
> [2017-05-31 06:01:17.587370] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0
> [2017-05-31 06:05:25.961313] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:05:25.976281] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
> [2017-05-31 06:05:25.976378] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0
> [2017-05-31 06:10:24.626906] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:10:24.634914] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
> [2017-05-31 06:10:24.635016] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0
> [2017-05-31 06:15:21.944215] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:15:21.952554] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
> [2017-05-31 06:15:21.952655] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0
> [2017-05-31 06:16:22.552086] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
> [2017-05-31 06:16:22.552203] I [MSGID: 115013]
[server-helpers.c:296:do_fd_cleanup] 0-cdn-server: fd cleanup on
/test2/nyx/nyxclient/assets.json
> [2017-05-31 06:16:22.552416] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0
> [2017-05-31 06:20:25.255523] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:20:25.263860] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
> [2017-05-31 06:20:25.263963] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0
> [2017-05-31 06:21:21.584357] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0
> [2017-05-31 06:21:21.584449] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0
> [2017-05-31 06:21:37.944798] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms01.horisont.svenskaspel.se-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:21:37.952580] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
> [2017-05-31 06:21:37.952712] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0
> [2017-05-31 06:25:27.632381] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms02.horisont.svenskaspel.se-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:25:27.640591] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms02.horisont.svenskaspel.se-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
> [2017-05-31 06:25:27.640721] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms02.horisont.svenskaspel.se-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0
> [2017-05-31 06:35:42.166035] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from
tms01.horisont.svenskaspel.se-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:35:42.173662] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from
tms01.horisont.svenskaspel.se-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0
> [2017-05-31 06:35:42.173752] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection
tms01.horisont.svenskaspel.se-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0
> 
> ==> /var/log/glusterfs/bricks/gluster-mule-brick.log <=>
[2017-05-31 06:26:30.580063] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-38988-2017/05/31-06:26:30:551135-mule-client-2-0-0
> [2017-05-31 06:27:31.269224] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:27:31.276755] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
> [2017-05-31 06:27:31.276859] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0
> [2017-05-31 06:28:33.119250] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:28:33.126230] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
> [2017-05-31 06:28:33.126328] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0
> [2017-05-31 06:29:35.271637] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:29:35.279799] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
> [2017-05-31 06:29:35.279901] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0
> [2017-05-31 06:30:34.816144] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:30:34.824228] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
> [2017-05-31 06:30:34.824319] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0
> [2017-05-31 06:30:36.304718] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:30:36.311898] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
> [2017-05-31 06:30:36.311998] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0
> [2017-05-31 06:31:34.929107] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:31:34.936764] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
> [2017-05-31 06:31:34.936843] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0
> [2017-05-31 06:31:41.145828] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:31:41.153317] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
> [2017-05-31 06:31:41.153427] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0
> [2017-05-31 06:32:36.230320] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:32:36.239295] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
> [2017-05-31 06:32:36.239420] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0
> [2017-05-31 06:32:36.481688] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:32:36.489207] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
> [2017-05-31 06:32:36.489327] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0
> [2017-05-31 06:33:38.413891] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:33:38.422092] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
> [2017-05-31 06:33:38.422188] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0
> [2017-05-31 06:33:41.618786] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:33:41.626214] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
> [2017-05-31 06:33:41.626310] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0
> [2017-05-31 06:34:37.654479] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms01.horisont.svenskaspel.se-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:34:37.661939] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms01.horisont.svenskaspel.se-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
> [2017-05-31 06:34:37.662043] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms01.horisont.svenskaspel.se-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0
> [2017-05-31 06:36:12.557364] I [MSGID: 115029]
[server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from
tms02.horisont.svenskaspel.se-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0
(version: 3.10.2)
> [2017-05-31 06:36:12.564751] I [MSGID: 115036]
[server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from
tms02.horisont.svenskaspel.se-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0
> [2017-05-31 06:36:12.564861] I [MSGID: 101055]
[client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection
tms02.horisont.svenskaspel.se-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0
> 
> 
> Cheers
> Gabbe
> 
> 30 maj 2017 kl. 14:45 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu
at redhat.com>:
> 
> Hi Gabriel,
> 
> I am not able to hit the issue mentioned on my setup.
> 
> Please share the log files(both brick and client log files) from your
setup. It would be great if you can share the details about steps you followed
to hit the issue.
> 
> 
> Regards,
> SUNIL KUMAR ACHARYA
> SENIOR SOFTWARE ENGINEER
> Red Hat
> 
> T: +91-8067935170
> 
>
[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]
> TRIED. TESTED. TRUSTED.
> 
> 
> On Tue, May 30, 2017 at 3:30 PM, Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se> wrote:
> Hello
> 
> A manual mount failed the same way
> 
> Cheers
> Gabbe
> 
> 30 maj 2017 kl. 10:24 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu
at redhat.com>:
> 
> Hi Gabriel,
> 
> Which gluster version are your running? Are you able to fuse mount the
volume?
> 
> Please share the failure logs.
> 
> Regards,
> SUNIL KUMAR ACHARYA
> SENIOR SOFTWARE ENGINEER
> Red Hat
> 
> T: +91-8067935170
> 
>
[https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]
> TRIED. TESTED. TRUSTED.
> 
> 
> On Tue, May 30, 2017 at 1:04 PM, Gabriel Lindeborg <gabriel.lindeborg at
svenskaspel.se> wrote:
> Hello All
> 
> We?ve have a problem with cluster client mounts fail in mid run with this
in the log
> glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5)
[0x7f640c8b3dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
[0x7f640df4bfd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
[0x7f640df4bdfb] ) 0-: received signum (15), shutting down.
> 
> We?ve tried running debug but have not found anything suspicious happening
at the time of the failures
> We?ve searched the inter web but can not find anyone else having the same
problem in mid flight
> 
> The clients have four mounts of volumes from the same server, all mounts
fail simultaneously
> Peer status looks ok
> Volume status looks ok
> Volume info looks like this:
> Volume Name: GLUSTERVOLUME
> Type: Replicate
> Volume ID: ca7af017-4f0f-44cc-baf6-43168eed0748
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: GLUSTERSERVER1:/gluster/GLUSTERVOLUME/brick
> Brick2: GLUSTERSERVER2:/gluster/GLUSTERVOLUME/brick
> Options Reconfigured:
> transport.address-family: inet
> cluster.self-heal-daemon: enable
> nfs.disable: on
> server.allow-insecure: on
> client.bind-insecure: on
> network.ping-timeout: 5
> features.bitrot: on
> features.scrub: Active
> features.scrub-freq: weekly
> 
> Any ideas?
> 
> Cheers
> Gabbe
> 
> 
> 
> AB SVENSKA SPEL
> 621 80 Visby
> Norra Hansegatan 17, Visby
> V?xel: +4610-120 00 00
> https://svenskaspel.se
> 
> Please consider the environment before printing this email
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170601/300f30cb/attachment.sig>

Maybe Matching Threads

Search for more possibly parallel threads

Gluster users - Jun 2017 - Gluster client mount fails in mid flight with signum 15

[Gluster-users] Gluster client mount fails in mid flight with signum 15

[Gluster-users] Gluster client mount fails in mid flight with signum 15

[Gluster-users] Gluster client mount fails in mid flight with signum 15

Maybe Matching Threads