I don't think it is similar to Xavi fixed in
https://review.gluster.org/#/c/glusterfs/+/24099/.
Is it possible to share the output "thread apply all bt full" after
attaching the core with gdb?
Regards,
Mohit Agrawal
On Sat, Feb 15, 2020 at 7:25 AM Amar Tumballi <amar at kadalu.io> wrote:
> Is this crash seen already ? Does
> https://review.gluster.org/#/c/glusterfs/+/24099/ fix this?
>
> On Sat, Feb 15, 2020 at 4:32 AM Joe Julian <joe at julianfamily.org>
wrote:
>
>> These crashes have been happening almost daily. Any thoughts on how to
>> stabilize this?
>>
>>
>> [2020-02-14 19:02:13.932178] I [MSGID: 100030] [glusterfsd.c:2865:main]
>> 0-/usr/bin/glusterfs: Started running /usr/bin/glusterfs version 7.0
>> (args: /usr/bin/glusterfs --process-name fuse --volfile-server=gluster
>> --volfile-id=kube /tmp/hostpath_pv)
>> [2020-02-14 19:02:13.939208] I [glusterfsd.c:2593:daemonize]
>> 0-glusterfs: Pid of current running process is 1828083
>> [2020-02-14 19:02:13.970829] I [MSGID: 101190]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 0
>> [2020-02-14 19:02:13.970870] I [MSGID: 101190]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 1
>> [2020-02-14 19:02:14.026097] W [MSGID: 108003]
>> [afr.c:98:fix_quorum_options] 0-kube-replicate-0: quorum-type none
>> overriding quorum-count 1
>> [2020-02-14 19:02:14.027753] I [MSGID: 101190]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 2
>> [2020-02-14 19:02:14.027871] I [MSGID: 101190]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 4
>> [2020-02-14 19:02:14.027939] I [MSGID: 101190]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 5
>> [2020-02-14 19:02:14.027984] I [MSGID: 101190]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 3
>> [2020-02-14 19:02:14.028010] I [MSGID: 101190]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 6
>> [2020-02-14 19:02:14.028410] I [MSGID: 101190]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 7
>> [2020-02-14 19:02:14.028826] I [MSGID: 114020] [client.c:2434:notify]
>> 0-kube-client-0: parent translators are ready, attempting connect on
>> transport
>> [2020-02-14 19:02:14.031361] I [MSGID: 114020] [client.c:2434:notify]
>> 0-kube-client-1: parent translators are ready, attempting connect on
>> transport
>> [2020-02-14 19:02:14.031840] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>> 0-kube-client-0: changing port to 49159 (from 0)
>> [2020-02-14 19:02:14.031880] I [socket.c:863:__socket_shutdown]
>> 0-kube-client-0: intentional socket shutdown(12)
>> Final graph:
>>
>>
+------------------------------------------------------------------------------+
>> 1: volume kube-client-0
>> 2: type protocol/client
>> 3: option ping-timeout 42
>> 4: option remote-host strabo
>> 5: option remote-subvolume /data/gluster/kube
>> 6: option transport-type socket
>> 7: option transport.address-family inet
>> 8: option username 51b95740-bff7-4744-bbfc-dbae17670997
>> 9: option password 021da395-9404-43ac-b358-2f5dfc5acc71
>> 10: option transport.socket.ssl-enabled off
>> 11: option filter-O_DIRECT on
>> 12: option event-threads 8
>> 13: option transport.tcp-user-timeout 0
>> 14: option transport.socket.keepalive-time 20
>> 15: option transport.socket.keepalive-interval 2
>> 16: option transport.socket.keepalive-count 9
>> 17: option send-gids true
>> 18: end-volume
>> 19:
>> 20: volume kube-client-1
>> 21: type protocol/client
>> 22: option ping-timeout 42
>> 23: option remote-host nightshade
>> 24: option remote-subvolume /data/gluster/kube
>> 25: option transport-type socket
>> 26: option transport.address-family inet
>> 27: option username 51b95740-bff7-4744-bbfc-dbae17670997
>> 28: option password 021da395-9404-43ac-b358-2f5dfc5acc71
>> 29: option transport.socket.ssl-enabled off
>> 30: option filter-O_DIRECT on
>> 31: option event-threads 8
>> 32: option transport.tcp-user-timeout 0
>> 33: option transport.socket.keepalive-time 20
>> 34: option transport.socket.keepalive-interval 2
>> 35: option transport.socket.keepalive-count 9
>> 36: option send-gids true
>> 37: end-volume
>> 38:
>> 39: volume kube-replicate-0
>> 40: type cluster/replicate
>> 41: option afr-pending-xattr kube-client-0,kube-client-1
>> 42: option quorum-count 1
>> 43: option use-compound-fops off
>> 44: subvolumes kube-client-0 kube-client-1
>> 45: end-volume
>> 46:
>> 47: volume kube-dht
>> 48: type cluster/distribute
>> 49: option readdir-optimize on
>> 50: option lock-migration off
>> 51: option force-migration off
>> 52: subvolumes kube-replicate-0
>> 53: end-volume
>> 54:
>> 55: volume kube-io-cache
>> 56: type performance/io-cache
>> 57: option cache-timeout 30
>> 58: subvolumes kube-dht
>> 59: end-volume
>> 60:
>> 61: volume kube-open-behind
>> 62: type performance/open-behind
>> 63: subvolumes kube-io-cache
>> 64: end-volume
>> 65:
>> 66: volume kube-md-cache
>> 67: type performance/md-cache
>> 68: option cache-invalidation on
>> 69: subvolumes kube-open-behind
>> 70: end-volume
>> 71:
>> 72: volume kube-io-threads
>> 73: type performance/io-threads
>> 74: subvolumes kube-md-cache
>> 75: end-volume
>> 76:
>> 77: volume kube
>> 78: type debug/io-stats
>> 79: option log-level INFO
>> 80: option threads 16
>> 81: option latency-measurement off
>> 82: option count-fop-hits off
>> 83: option global-threading off
>> 84: subvolumes kube-io-threads
>> 85: end-volume
>> 86:
>> 87: volume meta-autoload
>> 88: type meta
>> 89: subvolumes kube
>> 90: end-volume
>> 91:
>>
>>
+------------------------------------------------------------------------------+
>> [2020-02-14 19:02:14.052685] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>> 0-kube-client-1: changing port to 49159 (from 0)
>> [2020-02-14 19:02:14.052792] I [socket.c:863:__socket_shutdown]
>> 0-kube-client-1: intentional socket shutdown(13)
>> [2020-02-14 19:02:14.056240] I [MSGID: 114057]
>> [client-handshake.c:1373:select_server_supported_programs]
>> 0-kube-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version
>> (400)
>> [2020-02-14 19:02:14.076058] I [MSGID: 114046]
>> [client-handshake.c:1104:client_setvolume_cbk] 0-kube-client-1:
>> Connected to kube-client-1, attached to remote volume
>> '/data/gluster/kube'.
>> [2020-02-14 19:02:14.076113] I [MSGID: 108005]
>> [afr-common.c:5277:__afr_handle_child_up_event] 0-kube-replicate-0:
>> Subvolume 'kube-client-1' came back up; going online.
>> [2020-02-14 19:02:24.031553] I [fuse-bridge.c:5162:fuse_init]
>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24
>> kernel 7.31
>> [2020-02-14 19:02:24.031605] I [fuse-bridge.c:5777:fuse_graph_sync]
>> 0-fuse: switched to graph 0
>> [2020-02-14 19:04:27.205776] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>> 0-kube-client-0: changing port to 49159 (from 0)
>> [2020-02-14 19:06:40.325688] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>> 0-kube-client-0: changing port to 49159 (from 0)
>> [2020-02-14 19:08:53.444994] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>> 0-kube-client-0: changing port to 49159 (from 0)
>> [2020-02-14 19:11:06.565527] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>> 0-kube-client-0: changing port to 49159 (from 0)
>> pending frames:
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(FLUSH)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(FLUSH)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> frame : type(1) op(LOOKUP)
>> patchset: git://git.gluster.org/glusterfs.git
>> signal received: 11
>> time of crash:
>> 2020-02-14 19:13:00
>> configuration details:
>> argp 1
>> backtrace 1
>> dlfcn 1
>> libpthread 1
>> llistxattr 1
>> setfsid 1
>> spinlock 1
>> epoll.h 1
>> xattr.h 1
>> st_atim.tv_nsec 1
>> package-string: glusterfs 7.0
>> /usr/lib/libglusterfs.so.0(+0x25e12)[0x7fc026b55e12]
>> /usr/lib/libglusterfs.so.0(gf_print_trace+0x354)[0x7fc026b60b74]
>> /usr/lib/libc.so.6(+0x3bfb0)[0x7fc0268e3fb0]
>> /usr/lib/libglusterfs.so.0(mem_get+0xf0)[0x7fc026b81850]
>> /usr/lib/libglusterfs.so.0(+0x1a4cd)[0x7fc026b4a4cd]
>> /usr/lib/libglusterfs.so.0(dict_setn+0x1f1)[0x7fc026b4a941]
>> /usr/lib/libglusterfs.so.0(dict_set_dynptr+0x32)[0x7fc026b4e342]
>>
/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x3b318)[0x7fc0210cb318]
>>
/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x56d03)[0x7fc0210e6d03]
>> /usr/lib/libgfrpc.so.0(+0xe957)[0x7fc026afe957]
>> /usr/lib/libgfrpc.so.0(+0xecc9)[0x7fc026afecc9]
>> /usr/lib/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7fc026afb816]
>> /usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0x4629)[0x7fc022294629]
>> /usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0xc45c)[0x7fc02229c45c]
>> /usr/lib/libglusterfs.so.0(+0x8cedc)[0x7fc026bbcedc]
>> /usr/lib/libpthread.so.0(+0x94cf)[0x7fc026a794cf]
>> /usr/lib/libc.so.6(clone+0x43)[0x7fc0269a72d3]
>> ---------
>>
>> ________
>>
>> Community Meeting Calendar:
>>
>> APAC Schedule -
>> Every 2nd and 4th Tuesday at 11:30 AM IST
>> Bridge: https://bluejeans.com/441850968
>>
>> NA/EMEA Schedule -
>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> --
> --
> https://kadalu.io
> Container Storage made easy!
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200215/4f8a9951/attachment.html>