Sunil Kumar Heggodu Gopala Acharya
2017-May-30 12:45 UTC
[Gluster-users] Gluster client mount fails in mid flight with signum 15
Hi Gabriel, I am not able to hit the issue mentioned on my setup. Please share the log files(both brick and client log files) from your setup. It would be great if you can share the details about steps you followed to hit the issue. Regards, Sunil kumar Acharya Senior Software Engineer Red Hat <https://www.redhat.com> T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> On Tue, May 30, 2017 at 3:30 PM, Gabriel Lindeborg < gabriel.lindeborg at svenskaspel.se> wrote:> Hello > > A manual mount failed the same way > > Cheers > Gabbe > > 30 maj 2017 kl. 10:24 skrev Sunil Kumar Heggodu Gopala Acharya < > sheggodu at redhat.com>: > > Hi Gabriel, > > Which gluster version are your running? Are you able to fuse mount the > volume? > > Please share the failure logs. > > Regards, > Sunil kumar Acharya > > Senior Software Engineer > Red Hat > > <https://www.redhat.com/> > > T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/> > > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > > On Tue, May 30, 2017 at 1:04 PM, Gabriel Lindeborg <gabriel.lindeborg@ > svenskaspel.se> wrote: > >> Hello All >> >> We?ve have a problem with cluster client mounts fail in mid run with this >> in the log >> glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) >> [0x7f640c8b3dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) >> [0x7f640df4bfd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) >> [0x7f640df4bdfb] ) 0-: received signum (15), shutting down. >> >> We?ve tried running debug but have not found anything suspicious >> happening at the time of the failures >> We?ve searched the inter web but can not find anyone else having the same >> problem in mid flight >> >> The clients have four mounts of volumes from the same server, all mounts >> fail simultaneously >> Peer status looks ok >> Volume status looks ok >> Volume info looks like this: >> Volume Name: GLUSTERVOLUME >> Type: Replicate >> Volume ID: ca7af017-4f0f-44cc-baf6-43168eed0748 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: GLUSTERSERVER1:/gluster/GLUSTERVOLUME/brick >> Brick2: GLUSTERSERVER2:/gluster/GLUSTERVOLUME/brick >> Options Reconfigured: >> transport.address-family: inet >> cluster.self-heal-daemon: enable >> nfs.disable: on >> server.allow-insecure: on >> client.bind-insecure: on >> network.ping-timeout: 5 >> features.bitrot: on >> features.scrub: Active >> features.scrub-freq: weekly >> >> Any ideas? >> >> Cheers >> Gabbe >> >> >> >> AB SVENSKA SPEL >> 621 80 Visby >> Norra Hansegatan 17, Visby >> V?xel: +4610-120 00 00 >> https://svenskaspel.se >> >> Please consider the environment before printing this email >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170530/58cb7209/attachment.html>
Gabriel Lindeborg
2017-May-31 06:40 UTC
[Gluster-users] Gluster client mount fails in mid flight with signum 15
Hello again The volumes are old, since version 3.6 if I remember correctly? These ?r det 40 last rows of the mnt-loggs go all the mounts ==> /var/log/glusterfs/mnt-gluster-alfresco.log <=/DAEMON/INFO [2017-05-31T07:55:12.787035+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.153779+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.160995+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.167075+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.171872+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.427057+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.432239+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.439572+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.444286+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.035710+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.042366+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.049401+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.054483+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.524103+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.540314+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.543748+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.558459+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.623109+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.643979+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.648717+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.662941+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.268446+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.284765+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.287503+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.302461+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.408757+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.415831+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.421736+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.421887+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.431820+02:00] [] [] [2017-05-31 06:09:54.514841] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting local read_child alfresco-client-2 /DAEMON/INFO [2017-05-31T08:11:10.179031+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.186811+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.194886+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.195062+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.205582+02:00] [] [] [2017-05-31 06:11:14.513620] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-alfresco-replicate-0: selecting local read_child alfresco-client-2 [2017-05-31 06:21:21.579748] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa1d2e4cdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa1d44e4fd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa1d44e4dfb] ) 0-: received signum (15), shutting down /DAEMON/INFO [2017-05-31T08:21:21.580626+02:00] [] [] /DAEMON/INFO [2017-05-31T08:21:21.581440+02:00] [] [] ==> /var/log/glusterfs/mnt-gluster-c1_2.log <= 82: type meta 83: subvolumes c1 84: end-volume 85: +------------------------------------------------------------------------------+ /DAEMON/ERR [2017-05-24T10:39:11.130038+02:00] [] [] [2017-05-24 08:39:11.130638] E [MSGID: 114058] [client-handshake.c:1538:client_query_portmap_cbk] 2-c1-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2017-05-24 08:39:11.130693] I [MSGID: 114018] [client.c:2276:client_rpc_notify] 2-c1-client-1: disconnected from c1-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2017-05-24 08:39:11.130711] E [MSGID: 108006] [afr-common.c:4781:afr_notify] 2-c1-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. /DAEMON/INFO [2017-05-24T10:39:11.471603+02:00] [] [] /DAEMON/ERR [2017-05-24T10:39:11.473783+02:00] [] [] /DAEMON/INFO [2017-05-24T10:39:14.479153+02:00] [] [] /DAEMON/ERR [2017-05-24T10:39:14.481430+02:00] [] [] [2017-05-24 08:39:14.513688] I [MSGID: 108006] [afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up /DAEMON/INFO [2017-05-24T10:39:14.513712+02:00] [] [] [2017-05-24 08:39:14.513858] I [MSGID: 114021] [client.c:2361:notify] 0-c1-client-2: current graph is no longer active, destroying rpc_client [2017-05-24 08:39:14.513879] I [MSGID: 114021] [client.c:2361:notify] 0-c1-client-3: current graph is no longer active, destroying rpc_client [2017-05-24 08:39:14.513933] I [MSGID: 108006] [afr-common.c:4923:afr_local_init] 2-c1-replicate-0: no subvolumes up /DAEMON/WARNING [2017-05-24T10:39:14.513997+02:00] [] [] /DAEMON/WARNING [2017-05-24T10:39:14.514161+02:00] [] [] /DAEMON/INFO [2017-05-24T10:39:18.485909+02:00] [] [] /DAEMON/ERR [2017-05-24T10:39:18.488664+02:00] [] [] /DAEMON/INFO [2017-05-24T10:39:23.737962+02:00] [] [] [2017-05-24 08:39:23.740893] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 2-c1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-24 08:39:23.742339] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-0: Connected to c1-client-0, attached to remote volume '/gluster/c1/brick'. [2017-05-24 08:39:23.742359] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-0: Server and Client lk-version numbers are not same, reopening the fds [2017-05-24 08:39:23.742437] I [MSGID: 108005] [afr-common.c:4756:afr_notify] 2-c1-replicate-0: Subvolume 'c1-client-0' came back up; going online. [2017-05-24 08:39:23.743083] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-0: Server lk version = 1 [2017-05-24 08:39:24.524310] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local read_child c1-client-0 [2017-05-24 08:39:24.524414] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-c1-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 /DAEMON/INFO [2017-05-24T10:39:26.883379+02:00] [] [] [2017-05-24 08:39:26.885753] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 2-c1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-24 08:39:26.887336] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 2-c1-client-1: Connected to c1-client-1, attached to remote volume '/gluster/c1/brick'. [2017-05-24 08:39:26.887353] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 2-c1-client-1: Server and Client lk-version numbers are not same, reopening the fds [2017-05-24 08:39:26.887479] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 2-c1-client-1: Server lk version = 1 [2017-05-24 08:39:34.515063] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 2-c1-replicate-0: selecting local read_child c1-client-0 The message "I [MSGID: 108006] [afr-common.c:4923:afr_local_init] 0-c1-replicate-0: no subvolumes up" repeated 4 times between [2017-05-24 08:38:24.514047] and [2017-05-24 08:39:04.514186] /DAEMON/INFO [2017-05-24T10:40:56.356298+02:00] [] [] [2017-05-24 08:40:56.356694] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7feba7f55dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7feba95edf85] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7feba95eddfb] ) 0-: received signum (15), shutting down /DAEMON/INFO [2017-05-24T10:40:56.356773+02:00] [] [] ==> /var/log/glusterfs/mnt-gluster-c1.log <=/DAEMON/INFO [2017-05-31T07:55:12.781267+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:12.786589+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.153964+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.161317+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.167226+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.172425+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.425676+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.433114+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.439753+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.444685+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.034185+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.042732+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.049094+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.053587+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.525624+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.540864+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.546015+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.546167+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.549146+02:00] [] [] [2017-05-31 05:56:14.515062] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-c1-replicate-0: selecting local read_child c1-client-0 /DAEMON/INFO [2017-05-31T07:56:13.558847+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.622583+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.644252+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.648540+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.662541+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.269583+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.284254+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.291514+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.303324+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.408574+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.415968+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.421254+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.431030+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.179194+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.187103+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.194300+02:00] [] [] [2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f7dad36fdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f7daea07fd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f7daea07dfb] ) 0-: received signum (15), shutting down /DAEMON/INFO [2017-05-31T08:11:10.205913+02:00] [] [] /DAEMON/INFO [2017-05-31T08:21:21.581037+02:00] [] [] /DAEMON/INFO [2017-05-31T08:21:21.581789+02:00] [] [] ==> /var/log/glusterfs/mnt-gluster-c2.log <= 66: end-volume 67: 68: volume c2-md-cache 69: type performance/md-cache 70: subvolumes c2-open-behind 71: end-volume 72: 73: volume c2 74: type debug/io-stats 75: option log-level INFO 76: option latency-measurement off 77: option count-fop-hits off 78: subvolumes c2-md-cache 79: end-volume 80: 81: volume meta-autoload 82: type meta 83: subvolumes c2 84: end-volume 85: +------------------------------------------------------------------------------+ /DAEMON/INFO [2017-05-24T09:20:01.247296+02:00] [] [] [2017-05-24 07:20:01.249248] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-c2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-24 07:20:01.249950] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-0: Connected to c2-client-0, attached to remote volume '/gluster/c1/newbrick'. [2017-05-24 07:20:01.249980] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-0: Server and Client lk-version numbers are not same, reopening the fds [2017-05-24 07:20:01.250050] I [MSGID: 108005] [afr-common.c:4756:afr_notify] 0-c2-replicate-0: Subvolume 'c2-client-0' came back up; going online. [2017-05-24 07:20:01.250557] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-0: Server lk version = 1 [2017-05-24 07:20:01.251421] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-c2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-24 07:20:01.252016] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-c2-client-1: Connected to c2-client-1, attached to remote volume '/gluster/c1/newbrick'. [2017-05-24 07:20:01.252055] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-c2-client-1: Server and Client lk-version numbers are not same, reopening the fds [2017-05-24 07:20:01.253072] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-c2-client-1: Server lk version = 1 /DAEMON/INFO [2017-05-24T09:20:01.253273+02:00] [] [] /DAEMON/INFO [2017-05-24T09:20:01.253547+02:00] [] [] [2017-05-24 07:20:01.254056] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-c2-replicate-0: selecting local read_child c2-client-0 [2017-05-24 07:20:01.254268] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-c2-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 /DAEMON/INFO [2017-05-24T10:39:11.116695+02:00] [] [] /DAEMON/INFO [2017-05-24T10:39:11.118327+02:00] [] [] /DAEMON/INFO [2017-05-24T10:40:47.577696+02:00] [] [] [2017-05-24 08:40:47.578001] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f3eaabefdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f3eac287f85] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f3eac287dfb] ) 0-: received signum (15), shutting down /DAEMON/INFO [2017-05-24T10:40:47.578100+02:00] [] [] ==> /var/log/glusterfs/mnt-gluster-cdn.log <=/DAEMON/INFO [2017-05-31T07:55:12.785775+02:00] [] [] [2017-05-31 05:55:14.515355] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local read_child cdn-client-2 /DAEMON/INFO [2017-05-31T07:55:18.152981+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.161155+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.166602+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.171520+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.426866+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.432536+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.439418+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.443795+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.035303+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.042561+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.048928+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.053195+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.525267+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.540464+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.545703+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.559227+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.622792+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.644113+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.649031+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.649173+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.653281+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.663382+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.269801+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.283977+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.288305+02:00] [] [] [2017-05-31 05:56:24.514977] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-cdn-replicate-0: selecting local read_child cdn-client-2 /DAEMON/INFO [2017-05-31T07:56:26.302883+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.408914+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.415563+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.421558+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.432374+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.178859+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.186967+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.193439+02:00] [] [] [2017-05-31 06:21:21.579784] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fa420a29dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fa4220c1fd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fa4220c1dfb] ) 0-: received signum (15), shutting down /DAEMON/INFO [2017-05-31T08:11:10.204701+02:00] [] [] /DAEMON/INFO [2017-05-31T08:21:21.581209+02:00] [] [] /DAEMON/INFO [2017-05-31T08:21:21.581945+02:00] [] [] ==> /var/log/glusterfs/mnt-gluster-mule.log <=/DAEMON/INFO [2017-05-31T07:55:18.154147+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.161457+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.168560+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.168765+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:18.172803+02:00] [] [] [2017-05-31 05:55:24.517077] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local read_child mule-client-2 /DAEMON/INFO [2017-05-31T07:55:46.426664+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.432816+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.439270+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:46.443344+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.035528+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.042893+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.049247+02:00] [] [] /DAEMON/INFO [2017-05-31T07:55:52.054006+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.525469+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.540732+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.545856+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:13.559699+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.622955+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.644386+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.648877+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:18.663766+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.269958+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.284895+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.291677+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.291845+02:00] [] [] /DAEMON/INFO [2017-05-31T07:56:26.294037+02:00] [] [] [2017-05-31 05:56:34.516246] I [MSGID: 108031] [afr-common.c:2340:afr_local_discovery_cbk] 0-mule-replicate-0: selecting local read_child mule-client-2 /DAEMON/INFO [2017-05-31T07:56:26.303715+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.408417+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.416098+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.421403+02:00] [] [] /DAEMON/INFO [2017-05-31T08:09:49.431421+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.178644+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.187364+02:00] [] [] /DAEMON/INFO [2017-05-31T08:11:10.193590+02:00] [] [] [2017-05-31 06:21:21.579746] W [glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fbd1beafdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fbd1d547fd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbd1d547dfb] ) 0-: received signum (15), shutting down /DAEMON/INFO [2017-05-31T08:11:10.205040+02:00] [] [] /DAEMON/INFO [2017-05-31T08:21:21.580846+02:00] [] [] /DAEMON/INFO [2017-05-31T08:21:21.581623+02:00] [] [] And these are 40 last rows of the brick ==> /var/log/glusterfs/bricks/gluster-alfresco-brick.log <=[2017-05-31 05:52:19.117394] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0 [2017-05-31 05:52:19.117469] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-3219-2017/05/31-05:52:19:118674-alfresco-client-2-0-0 [2017-05-31 05:52:43.020403] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 05:52:43.027816] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0 [2017-05-31 05:52:43.027931] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-22656-2017/05/31-05:52:43:701-alfresco-client-2-0-0 [2017-05-31 05:56:48.604780] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 05:57:50.817223] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 05:57:50.825987] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0 [2017-05-31 05:57:50.826138] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-30700-2017/05/31-05:57:50:797566-alfresco-client-2-0-0 [2017-05-31 05:58:25.746161] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 05:58:25.755174] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0 [2017-05-31 05:58:25.755279] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-14439-2017/05/31-05:58:25:759341-alfresco-client-2-0-0 [2017-05-31 06:02:47.958784] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 06:02:47.976543] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0 [2017-05-31 06:02:47.976683] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-38293-2017/05/31-06:02:47:939035-alfresco-client-2-0-0 [2017-05-31 06:07:53.126343] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 06:07:53.133904] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0 [2017-05-31 06:07:53.133998] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-5812-2017/05/31-06:07:53:106035-alfresco-client-2-0-0 [2017-05-31 06:12:56.138792] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 06:12:56.146253] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0 [2017-05-31 06:12:56.146351] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-15398-2017/05/31-06:12:56:119113-alfresco-client-2-0-0 [2017-05-31 06:16:22.552274] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0 [2017-05-31 06:16:22.552428] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11625-2017/05/31-05:56:48:609743-alfresco-client-2-0-0 [2017-05-31 06:17:54.775497] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 06:17:54.792370] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0 [2017-05-31 06:17:54.792446] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-23800-2017/05/31-06:17:54:755940-alfresco-client-2-0-0 [2017-05-31 06:21:21.582947] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0 [2017-05-31 06:21:21.583181] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-24520-2017/05/31-05:30:40:506115-alfresco-client-2-0-0 [2017-05-31 06:21:42.714049] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 06:21:42.722227] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0 [2017-05-31 06:21:42.722320] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31327-2017/05/31-06:21:42:693446-alfresco-client-2-0-0 [2017-05-31 06:22:35.445697] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 06:22:35.453208] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0 [2017-05-31 06:22:35.453334] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-32602-2017/05/31-06:22:35:424282-alfresco-client-2-0-0 [2017-05-31 06:22:58.071308] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 06:22:58.078524] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0 [2017-05-31 06:22:58.078608] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-33322-2017/05/31-06:22:58:50528-alfresco-client-2-0-0 [2017-05-31 06:28:06.645148] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-alfresco-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0 (version: 3.10.2) [2017-05-31 06:28:06.652828] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-alfresco-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0 [2017-05-31 06:28:06.652942] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-alfresco-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-2949-2017/05/31-06:28:06:624729-alfresco-client-2-0-0 ==> /var/log/glusterfs/bricks/gluster-c1-brick.log <=[2017-05-31 05:49:00.793792] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-39066-2017/05/31-05:29:48:927855-c1-client-0-0-0 [2017-05-31 05:52:36.512431] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 05:52:36.522623] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0 [2017-05-31 05:52:36.522723] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-3773-2017/05/31-05:52:36:525116-c1-client-0-0-0 [2017-05-31 05:56:13.544120] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2017-05-31 05:56:48.796772] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 05:58:39.772391] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 05:58:39.782681] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0 [2017-05-31 05:58:39.782820] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31756-2017/05/31-05:58:39:755387-c1-client-0-0-0 [2017-05-31 06:03:45.195471] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:03:45.205497] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0 [2017-05-31 06:03:45.205610] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-23437-2017/05/31-06:03:45:209999-c1-client-0-0-0 [2017-05-31 06:08:45.229186] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:08:45.239068] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0 [2017-05-31 06:08:45.239173] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-31667-2017/05/31-06:08:45:244141-c1-client-0-0-0 [2017-05-31 06:13:51.433086] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:13:51.443161] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0 [2017-05-31 06:13:51.443257] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2422-2017/05/31-06:13:51:449799-c1-client-0-0-0 [2017-05-31 06:16:22.552265] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0 [2017-05-31 06:16:22.552418] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11767-2017/05/31-05:56:48:807559-c1-client-0-0-0 [2017-05-31 06:18:55.232358] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:18:55.242110] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0 [2017-05-31 06:18:55.242188] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-13173-2017/05/31-06:18:55:249928-c1-client-0-0-0 [2017-05-31 06:21:21.586024] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0 [2017-05-31 06:21:21.586153] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-24704-2017/05/31-05:30:40:701274-c1-client-0-0-0 [2017-05-31 06:21:34.443201] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:21:34.453810] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0 [2017-05-31 06:21:34.453897] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-30927-2017/05/31-06:21:34:425675-c1-client-0-0-0 [2017-05-31 06:23:55.706224] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:23:55.716126] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0 [2017-05-31 06:23:55.716221] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-23492-2017/05/31-06:23:55:724580-c1-client-0-0-0 [2017-05-31 06:28:58.213831] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:28:58.223474] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0 [2017-05-31 06:28:58.223557] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-34879-2017/05/31-06:28:58:233583-c1-client-0-0-0 [2017-05-31 06:33:07.622832] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:33:07.642391] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0 [2017-05-31 06:33:07.642520] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-12268-2017/05/31-06:33:07:606256-c1-client-0-0-0 [2017-05-31 06:34:07.776928] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-c1-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0 (version: 3.10.2) [2017-05-31 06:34:07.787116] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-c1-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0 [2017-05-31 06:34:07.787200] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-c1-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-5896-2017/05/31-06:34:07:795608-c1-client-0-0-0 ==> /var/log/glusterfs/bricks/gluster-cdn-brick.log <=[2017-05-31 05:50:18.876202] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-40008-2017/05/31-05:50:18:867052-cdn-client-2-0-0 [2017-05-31 05:55:09.869166] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 05:55:09.877790] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0 [2017-05-31 05:55:09.877898] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-8743-2017/05/31-05:55:09:880982-cdn-client-2-0-0 [2017-05-31 05:55:20.944369] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 05:55:20.952275] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0 [2017-05-31 05:55:20.952379] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-9021-2017/05/31-05:55:20:957702-cdn-client-2-0-0 [2017-05-31 05:56:48.701158] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:00:20.651639] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:00:20.659659] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0 [2017-05-31 06:00:20.659782] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-18151-2017/05/31-06:00:20:662693-cdn-client-2-0-0 [2017-05-31 06:01:17.578233] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:01:17.587267] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0 [2017-05-31 06:01:17.587370] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-19612-2017/05/31-06:01:17:591046-cdn-client-2-0-0 [2017-05-31 06:05:25.961313] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:05:25.976281] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0 [2017-05-31 06:05:25.976378] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-25703-2017/05/31-06:05:25:974773-cdn-client-2-0-0 [2017-05-31 06:10:24.626906] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:10:24.634914] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0 [2017-05-31 06:10:24.635016] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-34985-2017/05/31-06:10:24:640146-cdn-client-2-0-0 [2017-05-31 06:15:21.944215] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:15:21.952554] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0 [2017-05-31 06:15:21.952655] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-5449-2017/05/31-06:15:21:960278-cdn-client-2-0-0 [2017-05-31 06:16:22.552086] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0 [2017-05-31 06:16:22.552203] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cdn-server: fd cleanup on /test2/nyx/nyxclient/assets.json [2017-05-31 06:16:22.552416] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-11697-2017/05/31-05:56:48:710016-cdn-client-2-0-0 [2017-05-31 06:20:25.255523] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:20:25.263860] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0 [2017-05-31 06:20:25.263963] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-16619-2017/05/31-06:20:25:272616-cdn-client-2-0-0 [2017-05-31 06:21:21.584357] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0 [2017-05-31 06:21:21.584449] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-24622-2017/05/31-05:30:40:603607-cdn-client-2-0-0 [2017-05-31 06:21:37.944798] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:21:37.952580] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0 [2017-05-31 06:21:37.952712] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-31121-2017/05/31-06:21:37:924507-cdn-client-2-0-0 [2017-05-31 06:25:27.632381] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:25:27.640591] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0 [2017-05-31 06:25:27.640721] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-25859-2017/05/31-06:25:27:650357-cdn-client-2-0-0 [2017-05-31 06:35:42.166035] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-cdn-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0 (version: 3.10.2) [2017-05-31 06:35:42.173662] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-cdn-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0 [2017-05-31 06:35:42.173752] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-cdn-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-16416-2017/05/31-06:35:42:142067-cdn-client-2-0-0 ==> /var/log/glusterfs/bricks/gluster-mule-brick.log <=[2017-05-31 06:26:30.580063] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-38988-2017/05/31-06:26:30:551135-mule-client-2-0-0 [2017-05-31 06:27:31.269224] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:27:31.276755] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0 [2017-05-31 06:27:31.276859] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-31024-2017/05/31-06:27:31:285492-mule-client-2-0-0 [2017-05-31 06:28:33.119250] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:28:33.126230] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0 [2017-05-31 06:28:33.126328] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-4225-2017/05/31-06:28:33:98956-mule-client-2-0-0 [2017-05-31 06:29:35.271637] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:29:35.279799] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0 [2017-05-31 06:29:35.279901] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-6043-2017/05/31-06:29:35:251515-mule-client-2-0-0 [2017-05-31 06:30:34.816144] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:30:34.824228] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0 [2017-05-31 06:30:34.824319] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-38979-2017/05/31-06:30:34:832667-mule-client-2-0-0 [2017-05-31 06:30:36.304718] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:30:36.311898] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0 [2017-05-31 06:30:36.311998] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-39200-2017/05/31-06:30:36:323475-mule-client-2-0-0 [2017-05-31 06:31:34.929107] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:31:34.936764] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0 [2017-05-31 06:31:34.936843] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-9816-2017/05/31-06:31:34:908732-mule-client-2-0-0 [2017-05-31 06:31:41.145828] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:31:41.153317] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0 [2017-05-31 06:31:41.153427] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-10070-2017/05/31-06:31:41:126623-mule-client-2-0-0 [2017-05-31 06:32:36.230320] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:32:36.239295] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0 [2017-05-31 06:32:36.239420] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2292-2017/05/31-06:32:36:246972-mule-client-2-0-0 [2017-05-31 06:32:36.481688] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:32:36.489207] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0 [2017-05-31 06:32:36.489327] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-2387-2017/05/31-06:32:36:500604-mule-client-2-0-0 [2017-05-31 06:33:38.413891] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:33:38.422092] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0 [2017-05-31 06:33:38.422188] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-4613-2017/05/31-06:33:38:431298-mule-client-2-0-0 [2017-05-31 06:33:41.618786] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:33:41.626214] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0 [2017-05-31 06:33:41.626310] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-13210-2017/05/31-06:33:41:599540-mule-client-2-0-0 [2017-05-31 06:34:37.654479] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:34:37.661939] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0 [2017-05-31 06:34:37.662043] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms01.horisont.svenskaspel.se<http://tms01.horisont.svenskaspel.se>-14570-2017/05/31-06:34:37:634424-mule-client-2-0-0 [2017-05-31 06:36:12.557364] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-mule-server: accepted client from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0 (version: 3.10.2) [2017-05-31 06:36:12.564751] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-mule-server: disconnecting connection from tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0 [2017-05-31 06:36:12.564861] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-mule-server: Shutting down connection tms02.horisont.svenskaspel.se<http://tms02.horisont.svenskaspel.se>-9518-2017/05/31-06:36:12:574270-mule-client-2-0-0 Cheers Gabbe 30 maj 2017 kl. 14:45 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu at redhat.com<mailto:sheggodu at redhat.com>>: Hi Gabriel, I am not able to hit the issue mentioned on my setup. Please share the log files(both brick and client log files) from your setup. It would be great if you can share the details about steps you followed to hit the issue. Regards, Sunil kumar Acharya Senior Software Engineer Red Hat <https://www.redhat.com/> T: +91-8067935170<http://redhatemailsignature-marketing.itos.redhat.com/> [https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]<https://red.ht/sig> TRIED. TESTED. TRUSTED.<https://redhat.com/trusted> On Tue, May 30, 2017 at 3:30 PM, Gabriel Lindeborg <gabriel.lindeborg at svenskaspel.se<mailto:gabriel.lindeborg at svenskaspel.se>> wrote: Hello A manual mount failed the same way Cheers Gabbe 30 maj 2017 kl. 10:24 skrev Sunil Kumar Heggodu Gopala Acharya <sheggodu at redhat.com<mailto:sheggodu at redhat.com>>: Hi Gabriel, Which gluster version are your running? Are you able to fuse mount the volume? Please share the failure logs. Regards, Sunil kumar Acharya Senior Software Engineer Red Hat <https://www.redhat.com/> T: +91-8067935170<http://redhatemailsignature-marketing.itos.redhat.com/> [https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png]<https://red.ht/sig> TRIED. TESTED. TRUSTED.<https://redhat.com/trusted> On Tue, May 30, 2017 at 1:04 PM, Gabriel Lindeborg <gabriel.lindeborg at svenskaspel.se<mailto:gabriel.lindeborg at svenskaspel.se>> wrote: Hello All We?ve have a problem with cluster client mounts fail in mid run with this in the log glusterfsd.c:1332:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f640c8b3dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f640df4bfd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f640df4bdfb] ) 0-: received signum (15), shutting down. We?ve tried running debug but have not found anything suspicious happening at the time of the failures We?ve searched the inter web but can not find anyone else having the same problem in mid flight The clients have four mounts of volumes from the same server, all mounts fail simultaneously Peer status looks ok Volume status looks ok Volume info looks like this: Volume Name: GLUSTERVOLUME Type: Replicate Volume ID: ca7af017-4f0f-44cc-baf6-43168eed0748 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: GLUSTERSERVER1:/gluster/GLUSTERVOLUME/brick Brick2: GLUSTERSERVER2:/gluster/GLUSTERVOLUME/brick Options Reconfigured: transport.address-family: inet cluster.self-heal-daemon: enable nfs.disable: on server.allow-insecure: on client.bind-insecure: on network.ping-timeout: 5 features.bitrot: on features.scrub: Active features.scrub-freq: weekly Any ideas? Cheers Gabbe AB SVENSKA SPEL 621 80 Visby Norra Hansegatan 17, Visby V?xel: +4610-120 00 00 https://svenskaspel.se<https://svenskaspel.se/> Please consider the environment before printing this email _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170531/e245fa2e/attachment.html>