Dang Zhiqiang
2015-May-25 10:00 UTC
[Gluster-users] gluster 3.4.5，gluster client process was core dump
Hi,
Why this is and how to fix it?
Thanks.
client log:
data1.log:20695:[2015-05-25 03:12:31.084149] W
[dht-common.c:2016:dht_getxattr_cbk]
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x346d80d6f5]
(-->/usr/lib64/glusterfs/3.4.5/xlator/protocol/client.so(client3_3_getxattr_cbk+0x178)
[0x7f81fcf33ad8]
(-->/usr/lib64/glusterfs/3.4.5/xlator/cluster/replicate.so(afr_getxattr_cbk+0xf8)
[0x7f81fccc5168]))) 0-dfs-dht: invalid argument: frame->local
core dump info:
Core was generated by `/usr/sbin/glusterfs --volfile-id=dfs
--volfile-server=node1 /dat'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f81fca87354 in dht_getxattr_cbk (frame=0x7f82009efe34,
cookie=<value optimized out>, this=<value optimized out>,
op_ret=<value optimized out>, op_errno=0,
xattr=<value optimized out>, xdata=0x0) at dht-common.c:2043
2043 DHT_STACK_UNWIND (getxattr, frame, local->op_ret,
op_errno,
Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.132.el6_5.4.x86_64 keyutils-libs-1.4-4.el6.x86_64
krb5-libs-1.10.3-15.el6_5.1.x86_64 libcom_err-1.41.12-18.el6_5.1.x86_64
libgcc-4.4.7-4.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64
openssl-1.0.1e-16.el6_5.7.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f81fca87354 in dht_getxattr_cbk (frame=0x7f82009efe34,
cookie=<value optimized out>, this=<value optimized out>,
op_ret=<value optimized out>, op_errno=0,
xattr=<value optimized out>, xdata=0x0) at dht-common.c:2043
#1 0x00007f81fccc5168 in afr_getxattr_cbk (frame=0x7f8200a0d32c,
cookie=<value optimized out>, this=<value optimized out>, op_ret=0,
op_errno=0, dict=0x7f82003a768c,
xdata=0x0) at afr-inode-read.c:618
#2 0x00007f81fcf33ad8 in client3_3_getxattr_cbk (req=<value optimized
out>, iov=<value optimized out>, count=<value optimized out>,
myframe=0x7f82009a58fc)
at client-rpc-fops.c:1115
#3 0x000000346d80d6f5 in rpc_clnt_handle_reply (clnt=0x232cb40,
pollin=0x1173ac10) at rpc-clnt.c:771
#4 0x000000346d80ec6f in rpc_clnt_notify (trans=<value optimized out>,
mydata=0x232cb70, event=<value optimized out>, data=<value optimized
out>) at rpc-clnt.c:891
#5 0x000000346d80a4e8 in rpc_transport_notify (this=<value optimized
out>, event=<value optimized out>, data=<value optimized out>) at
rpc-transport.c:497
#6 0x00007f81fdf7f216 in socket_event_poll_in (this=0x233c5a0) at socket.c:2118
#7 0x00007f81fdf80c3d in socket_event_handler (fd=<value optimized out>,
idx=<value optimized out>, data=0x233c5a0, poll_in=1, poll_out=0,
poll_err=0) at socket.c:2230
#8 0x000000346d45e907 in event_dispatch_epoll_handler (event_pool=0x228be90) at
event-epoll.c:384
#9 event_dispatch_epoll (event_pool=0x228be90) at event-epoll.c:445
#10 0x0000000000406818 in main (argc=4, argv=0x7fff9e2e4898) at
glusterfsd.c:1934
(gdb) print ((call_frame_t *)0x7f82009efe34)->local
$2 = (void *) 0x0
(gdb) l *0x00007f81fca87354
0x7f81fca87354 is in dht_getxattr_cbk (dht-common.c:2043).
2038 dht_aggregate_xattr (xattr, local->xattr);
2039 local->xattr = dict_copy (xattr, local->xattr);
2040 }
2041out:
2042 if (is_last_call (this_call_cnt)) {
2043 DHT_STACK_UNWIND (getxattr, frame, local->op_ret,
op_errno,
2044 local->xattr, NULL);
2045 }
2046 return 0;
2047}
jump code:
2016 VALIDATE_OR_GOTO (frame->local, out);
volume info:
# gluster v info
Volume Name: dfs
Type: Distributed-Replicate
Volume ID: 1848afb0-44ef-418c-a58f-8d7159ec5d1e
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: node1:/data/vol/dfs
Brick2: node2:/data/vol/dfs
Brick3: node3:/data/vol/dfs
Brick4: node4:/data/vol/dfs
Options Reconfigured:
diagnostics.client-log-level: WARNING
diagnostics.brick-log-level: WARNING
nfs.disable: on
features.quota: on
features.limit-usage:
/video/CLOUD:200TB,/video/YINGSHIKU:200TB,/video/LIVENEW:200TB,/video/SOCIAL:200TB,/video/mini:200TB,/video/2013:200TB,/video:200TB
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150525/20844646/attachment.html>
Susant Palai
2015-May-25 10:17 UTC
[Gluster-users] gluster 3.4.5，gluster client process was core dump
We found a similar crash and the fix for the same is here http://review.gluster.org/#/c/10389/. You can find the RCA in the commit message. Regards, Susant ----- Original Message -----> From: "Dang Zhiqiang" <dzq008 at 163.com> > To: gluster-users at gluster.org > Sent: Monday, 25 May, 2015 3:30:16 PM > Subject: [Gluster-users] gluster 3.4.5?gluster client process was core dump > > Hi, > > Why this is and how to fix it? > Thanks. > > client log: > data1.log:20695:[2015-05-25 03:12:31.084149] W > [dht-common.c:2016:dht_getxattr_cbk] > (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x346d80d6f5] > (-->/usr/lib64/glusterfs/3.4.5/xlator/protocol/client.so(client3_3_getxattr_cbk+0x178) > [0x7f81fcf33ad8] > (-->/usr/lib64/glusterfs/3.4.5/xlator/cluster/replicate.so(afr_getxattr_cbk+0xf8) > [0x7f81fccc5168]))) 0-dfs-dht: invalid argument: frame->local > > core dump info: > Core was generated by `/usr/sbin/glusterfs --volfile-id=dfs > --volfile-server=node1 /dat'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007f81fca87354 in dht_getxattr_cbk (frame=0x7f82009efe34, > cookie=<value optimized out>, this=<value optimized out>, op_ret=<value > optimized out>, op_errno=0, > xattr=<value optimized out>, xdata=0x0) at dht-common.c:2043 > 2043 DHT_STACK_UNWIND (getxattr, frame, local->op_ret, op_errno, > Missing separate debuginfos, use: debuginfo-install > glibc-2.12-1.132.el6_5.4.x86_64 keyutils-libs-1.4-4.el6.x86_64 > krb5-libs-1.10.3-15.el6_5.1.x86_64 libcom_err-1.41.12-18.el6_5.1.x86_64 > libgcc-4.4.7-4.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 > openssl-1.0.1e-16.el6_5.7.x86_64 zlib-1.2.3-29.el6.x86_64 > (gdb) bt > #0 0x00007f81fca87354 in dht_getxattr_cbk (frame=0x7f82009efe34, > cookie=<value optimized out>, this=<value optimized out>, op_ret=<value > optimized out>, op_errno=0, > xattr=<value optimized out>, xdata=0x0) at dht-common.c:2043 > #1 0x00007f81fccc5168 in afr_getxattr_cbk (frame=0x7f8200a0d32c, > cookie=<value optimized out>, this=<value optimized out>, op_ret=0, > op_errno=0, dict=0x7f82003a768c, > xdata=0x0) at afr-inode-read.c:618 > #2 0x00007f81fcf33ad8 in client3_3_getxattr_cbk (req=<value optimized out>, > iov=<value optimized out>, count=<value optimized out>, > myframe=0x7f82009a58fc) > at client-rpc-fops.c:1115 > #3 0x000000346d80d6f5 in rpc_clnt_handle_reply (clnt=0x232cb40, > pollin=0x1173ac10) at rpc-clnt.c:771 > #4 0x000000346d80ec6f in rpc_clnt_notify (trans=<value optimized out>, > mydata=0x232cb70, event=<value optimized out>, data=<value optimized out>) > at rpc-clnt.c:891 > #5 0x000000346d80a4e8 in rpc_transport_notify (this=<value optimized out>, > event=<value optimized out>, data=<value optimized out>) at > rpc-transport.c:497 > #6 0x00007f81fdf7f216 in socket_event_poll_in (this=0x233c5a0) at > socket.c:2118 > #7 0x00007f81fdf80c3d in socket_event_handler (fd=<value optimized out>, > idx=<value optimized out>, data=0x233c5a0, poll_in=1, poll_out=0, > poll_err=0) at socket.c:2230 > #8 0x000000346d45e907 in event_dispatch_epoll_handler (event_pool=0x228be90) > at event-epoll.c:384 > #9 event_dispatch_epoll (event_pool=0x228be90) at event-epoll.c:445 > #10 0x0000000000406818 in main (argc=4, argv=0x7fff9e2e4898) at > glusterfsd.c:1934 > (gdb) print ((call_frame_t *)0x7f82009efe34)->local > $2 = (void *) 0x0 > (gdb) l *0x00007f81fca87354 > 0x7f81fca87354 is in dht_getxattr_cbk (dht-common.c:2043). > 2038 dht_aggregate_xattr (xattr, local->xattr); > 2039 local->xattr = dict_copy (xattr, local->xattr); > 2040 } > 2041 out: > 2042 if (is_last_call (this_call_cnt)) { > 2043 DHT_STACK_UNWIND (getxattr, frame, local->op_ret, op_errno, > 2044 local->xattr, NULL); > 2045 } > 2046 return 0; > 2047 } > > jump code: > 2016 VALIDATE_OR_GOTO (frame->local, out); > > > volume info: > # gluster v info > Volume Name: dfs > Type: Distributed-Replicate > Volume ID: 1848afb0-44ef-418c-a58f-8d7159ec5d1e > Status: Started > Number of Bricks: 2 x 2 = 4 > Transport-type: tcp > Bricks: > Brick1: node1:/data/vol/dfs > Brick2: node2:/data/vol/dfs > Brick3: node3:/data/vol/dfs > Brick4: node4:/data/vol/dfs > Options Reconfigured: > diagnostics.client-log-level: WARNING > diagnostics.brick-log-level: WARNING > nfs.disable: on > features.quota: on > features.limit-usage: > /video/CLOUD:200TB,/video/YINGSHIKU:200TB,/video/LIVENEW:200TB,/video/SOCIAL:200TB,/video/mini:200TB,/video/2013:200TB,/video:200TB > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users