+Anoop, Jiffin On Mon, Jun 27, 2016 at 1:50 PM, Yann LEMARIE <ylemarie at iraiser.eu> wrote:> Thanks Atin, > > Here is the log with "backtrace" > > time of crash: > 2016-06-26 09:27:44 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 3.7.11 > > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f2626e7a5e2] > > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f2626e966cd] > /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f2626278cb0] > /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a)[0x7f26262ca9da] > > /usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_stat_cbk+0x279)[0x7f2620c7a909] > > > I have lot of message about [File exist] or [No such file or directory] > within the .trash directory, is ther any link with the crash ? > > [2016-06-26 09:27:37.965461] E [MSGID: 113027] [posix.c:1427:posix_mkdir] > 0-cdn-posix: mkdir of /srv/cdn/ failed [File exists] > [2016-06-26 09:27:37.965479] E [MSGID: 113027] [posix.c:1427:posix_mkdir] > 0-cdn-posix: mkdir of /srv/cdn/.trashcan failed [File exists] > [2016-06-26 09:27:37.965495] E [MSGID: 113027] [posix.c:1427:posix_mkdir] > 0-cdn-posix: mkdir of /srv/cdn/.trashcan/ failed [File exists] > [2016-06-26 09:27:37.965512] E [MSGID: 113027] [posix.c:1427:posix_mkdir] > 0-cdn-posix: mkdir of /srv/cdn/.trashcan//medias failed [File exists] > [2016-06-26 09:27:37.965570] E [MSGID: 113001] [posix.c:1451:posix_mkdir] > 0-cdn-posix: setting xattrs on > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yrl8ZiGVIvdiyBD5u1pJUXBzYJ5b > failed [No such file or directory] > [2016-06-26 09:27:37.965615] E [MSGID: 113001] [posix.c:1451:posix_mkdir] > 0-cdn-posix: setting xattrs on > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yrl8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive > failed [No such file or directory] > [2016-06-26 09:27:37.965659] E [MSGID: 113001] [posix.c:1451:posix_mkdir] > 0-cdn-posix: setting xattrs on > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yrl8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin > failed [No such file or directory] > [2016-06-26 09:27:37.965683] E [MSGID: 113027] [posix.c:1427:posix_mkdir] > 0-cdn-posix: mkdir of > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yrl8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin > failed [File exists] > [2016-06-26 09:27:37.965720] E [MSGID: 113001] [posix.c:1451:posix_mkdir] > 0-cdn-posix: setting xattrs on > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yrl8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin/tmp > failed [No such file or directory] > [2016-06-26 09:27:37.965767] E [MSGID: 113001] [posix.c:2644:posix_create] > 0-cdn-posix: setting xattrs on > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yrl8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin/tmp/IMG3920.JPG_2016-06-26_092737 > failed [No such file or directory] > [2016-06-26 09:27:37.965955] E [posix.c:2996:_fill_writev_xdata] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_readv_cbk+0x16c) > [0x7f2620c6fc4c] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/storage/posix.so(posix_writev+0x1dc) > [0x7f26214ae37c] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/storage/posix.so(_fill_writev_xdata+0x1ff) > [0x7f26214ae16f] ) 0-cdn-posix: fd: 0x7f26140057fc inode: > 0x7f2600932484gfid:00000000-0000-0000-0000-000000000000 [Invalid argument] > [2016-06-26 09:27:37.966096] E [posix.c:2996:_fill_writev_xdata] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_readv_cbk+0x16c) > [0x7f2620c6fc4c] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/storage/posix.so(posix_writev+0x1dc) > [0x7f26214ae37c] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/storage/posix.so(_fill_writev_xdata+0x1ff) > [0x7f26214ae16f] ) 0-cdn-posix: fd: 0x7f26140057fc inode: > 0x7f2600932484gfid:00000000-0000-0000-0000-000000000000 [Invalid argument] > > > Yann > > > Le 27/06/2016 10:02, Atin Mukherjee a ?crit : > > Grep for "backtrace" in /var/log/glusterfs, that should give you a > particular log file name by which you can determine which process crashed. > You should also be able to see a coredump. Why don't you attach the same > for further analysis? > > ~Atin > > On Mon, Jun 27, 2016 at 1:17 PM, Yann LEMARIE <ylemarie at iraiser.eu> wrote: > >> Hi, >> >> I'm using GlusterFS since many years and never see this problem, but this >> is the second time in one week ... >> >> I have 3 volumes with 2 bricks and 1 volume crash with no reason, I just >> have to stop/start the volume to make it up again. >> The only logs I can find are in syslog : >> >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames: >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10) >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset: git:// >> git.gluster.com/glusterfs.git >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash: >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details: >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string: glusterfs 3.7.11 >> Jun 26 11:27:44 nfs05 srv-cdn[7879]: --------- >> >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames: >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10) >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset: git:// >> git.gluster.com/glusterfs.git >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash: >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details: >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string: glusterfs 3.7.11 >> Jun 26 11:27:44 nfs06 srv-cdn[1787]: --------- >> >> >> Thanks for your help >> >> >> Regards >> -- >> Yann Lemari? >> iRaiser - Support Technique >> [image: iRaiser Logotype] <http://www.iraiser.eu> >> <ylemarie at iraiser.eu>ylemarie at iraiser.eu >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > -- > Yann Lemari? > iRaiser - Support Technique > [image: iRaiser Logotype] <http://www.iraiser.eu> > <ylemarie at iraiser.eu>ylemarie at iraiser.eu >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160627/7dbf81f1/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: dcfhgbch.png Type: image/png Size: 1842 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160627/7dbf81f1/attachment.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 1842 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160627/7dbf81f1/attachment-0001.png>
On Mon, 2016-06-27 at 13:54 +0530, Atin Mukherjee wrote:> +Anoop, Jiffin > > On Mon, Jun 27, 2016 at 1:50 PM, Yann LEMARIE <ylemarie at iraiser.eu> > wrote: > > Thanks Atin, > > > > Here is the log with "backtrace" > > > > > time of crash: > > > 2016-06-26 09:27:44 > > > configuration details: > > > argp 1 > > > backtrace 1 > > > dlfcn 1 > > > libpthread 1 > > > llistxattr 1 > > > setfsid 1 > > > spinlock 1 > > > epoll.h 1 > > > xattr.h 1 > > > st_atim.tv_nsec 1 > > > package-string: glusterfs 3.7.11 > > > /usr/lib/x86_64-linux- > > > gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f2626e7a5e > > > 2] > > > /usr/lib/x86_64-linux- > > > gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f2626e966cd] > > > /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f2626278cb0] > > > /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a)[0x7f26262ca9da] > > > /usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_stat > > > _cbk+0x279)[0x7f2620c7a909] > >The crash has been observed from trash translator. If possible can you please send us the core dump file so as to debug the issue in detail?> > I have lot of message about [File exist] or [No such file or > > directory] within the .trash directory, is ther any link with the > > crash ? > > > > > [2016-06-26 09:27:37.965461] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of /srv/cdn/ failed > > > [File exists] > > > [2016-06-26 09:27:37.965479] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of > > > /srv/cdn/.trashcan failed [File exists] > > > [2016-06-26 09:27:37.965495] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of > > > /srv/cdn/.trashcan/ failed [File exists] > > > [2016-06-26 09:27:37.965512] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of > > > /srv/cdn/.trashcan//medias failed [File exists] > > > [2016-06-26 09:27:37.965570] E [MSGID: 113001] > > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b failed [No such file or directory] > > > [2016-06-26 09:27:37.965615] E [MSGID: 113001] > > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive failed [No such file or > > > directory] > > > [2016-06-26 09:27:37.965659] E [MSGID: 113001] > > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin failed [No such > > > file or directory] > > > [2016-06-26 09:27:37.965683] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin???????failed [File > > > exists] > > > [2016-06-26 09:27:37.965720] E [MSGID: 113001] > > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin/tmp failed [No such > > > file or directory] > > > [2016-06-26 09:27:37.965767] E [MSGID: 113001] > > > [posix.c:2644:posix_create] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin/tmp/IMG3920.JPG_201 > > > 6-06-26_092737 failed? [No such file or directory] > > > [2016-06-26 09:27:37.965955] E [posix.c:2996:_fill_writev_xdata] > > > (-->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_read > > > v_cbk+0x16c) [0x7f2620c6fc4c] -->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(posix_writev+0x1dc) > > > [0x7f26214ae37c] -->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(_fill_writev_xdata+0 > > > x1ff) [0x7f26214ae16f] ) 0-cdn-posix: fd: 0x7f26140057fc inode: > > > 0x7f2600932484gfid:00000000-0000-0000-0000-000000000000 [Invalid > > > argument] > > > [2016-06-26 09:27:37.966096] E [posix.c:2996:_fill_writev_xdata] > > > (-->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_read > > > v_cbk+0x16c) [0x7f2620c6fc4c] -->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(posix_writev+0x1dc) > > > [0x7f26214ae37c] -->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(_fill_writev_xdata+0 > > > x1ff) [0x7f26214ae16f] ) 0-cdn-posix: fd: 0x7f26140057fc inode: > > > 0x7f2600932484gfid:00000000-0000-0000-0000-000000000000 [Invalid > > > argument] > > ? > > Yann > > > > > > Le 27/06/2016 10:02, Atin Mukherjee a ?crit?: > > > Grep for "backtrace" in /var/log/glusterfs, that should give you > > > a particular log file name by which you can determine which > > > process crashed. You should also be able to see a coredump. Why > > > don't you attach the same for further analysis? > > > > > > ~Atin > > > > > > On Mon, Jun 27, 2016 at 1:17 PM, Yann LEMARIE <ylemarie at iraiser.e > > > u> wrote: > > > > Hi, > > > > > > > > I'm using GlusterFS since many years and never see this > > > > problem, but this is the second time in one week ... > > > > > > > > I have 3 volumes with 2 bricks and 1 volume crash with no > > > > reason, I just have to stop/start the volume to make it up > > > > again. > > > > The only logs I can find are in syslog : > > > > > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames: > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10) > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset: > > > > > git://git.gluster.com/glusterfs.git > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash: > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details: > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string: > > > > > glusterfs 3.7.11 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: --------- > > > > > > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames: > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10) > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset: > > > > > git://git.gluster.com/glusterfs.git > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash: > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details: > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string: > > > > > glusterfs 3.7.11 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: --------- > > > > > > > > > ? > > > > Thanks for your help > > > > > > > > > > > > Regards > > > > --? > > > > Yann Lemari? > > > > iRaiser - Support Technique > > > > ? > > > > ylemarie at iraiser.eu > > > > > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users at gluster.org > > > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > ? > > --? > > Yann Lemari? > > iRaiser - Support Technique > > ? > > ylemarie at iraiser.eu > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
On Mon, 2016-06-27 at 13:54 +0530, Atin Mukherjee wrote:> +Anoop, Jiffin > > On Mon, Jun 27, 2016 at 1:50 PM, Yann LEMARIE <ylemarie at iraiser.eu> > wrote: > > Thanks Atin, > > > > Here is the log with "backtrace" > > > > > time of crash: > > > 2016-06-26 09:27:44 > > > configuration details: > > > argp 1 > > > backtrace 1 > > > dlfcn 1 > > > libpthread 1 > > > llistxattr 1 > > > setfsid 1 > > > spinlock 1 > > > epoll.h 1 > > > xattr.h 1 > > > st_atim.tv_nsec 1 > > > package-string: glusterfs 3.7.11 > > > /usr/lib/x86_64-linux- > > > gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f2626e7a5e > > > 2] > > > /usr/lib/x86_64-linux- > > > gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f2626e966cd] > > > /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f2626278cb0] > > > /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a)[0x7f26262ca9da] > > > /usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_stat > > > _cbk+0x279)[0x7f2620c7a909] > > ? > > I have lot of message about [File exist] or [No such file or > > directory] within the .trash directory, is ther any link with the > > crash ? > > > > > [2016-06-26 09:27:37.965461] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of /srv/cdn/ failed > > > [File exists] > > > [2016-06-26 09:27:37.965479] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of > > > /srv/cdn/.trashcan failed [File exists] > > > [2016-06-26 09:27:37.965495] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of > > > /srv/cdn/.trashcan/ failed [File exists] > > > [2016-06-26 09:27:37.965512] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of > > > /srv/cdn/.trashcan//medias failed [File exists] > > > [2016-06-26 09:27:37.965570] E [MSGID: 113001] > > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b failed [No such file or directory] > > > [2016-06-26 09:27:37.965615] E [MSGID: 113001] > > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive failed [No such file or > > > directory] > > > [2016-06-26 09:27:37.965659] E [MSGID: 113001] > > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin failed [No such > > > file or directory] > > > [2016-06-26 09:27:37.965683] E [MSGID: 113027] > > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin???????failed [File > > > exists] > > > [2016-06-26 09:27:37.965720] E [MSGID: 113001] > > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin/tmp failed [No such > > > file or directory] > > > [2016-06-26 09:27:37.965767] E [MSGID: 113001] > > > [posix.c:2644:posix_create] 0-cdn-posix: setting xattrs on > > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr > > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin/tmp/IMG3920.JPG_201 > > > 6-06-26_092737 failed? [No such file or directory] > > > [2016-06-26 09:27:37.965955] E [posix.c:2996:_fill_writev_xdata] > > > (-->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_read > > > v_cbk+0x16c) [0x7f2620c6fc4c] -->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(posix_writev+0x1dc) > > > [0x7f26214ae37c] -->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(_fill_writev_xdata+0 > > > x1ff) [0x7f26214ae16f] ) 0-cdn-posix: fd: 0x7f26140057fc inode: > > > 0x7f2600932484gfid:00000000-0000-0000-0000-000000000000 [Invalid > > > argument] > > > [2016-06-26 09:27:37.966096] E [posix.c:2996:_fill_writev_xdata] > > > (-->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_read > > > v_cbk+0x16c) [0x7f2620c6fc4c] -->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(posix_writev+0x1dc) > > > [0x7f26214ae37c] -->/usr/lib/x86_64-linux- > > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(_fill_writev_xdata+0 > > > x1ff) [0x7f26214ae16f] ) 0-cdn-posix: fd: 0x7f26140057fc inode: > > > 0x7f2600932484gfid:00000000-0000-0000-0000-000000000000 [Invalid > > > argument] > >Can you confirm whether truncated files are present in .trashcan directory or not? From the above brick logs check whether?medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yrl8ZiGVIvdiyBD5u1pJ UXBzYJ5b/Dante_Olive/origin/tmp/IMG3920.JPG_2016-06-26_092737 exists under .trashcan (from mount or directly on bricks)?> > Yann > > > > > > Le 27/06/2016 10:02, Atin Mukherjee a ?crit?: > > > Grep for "backtrace" in /var/log/glusterfs, that should give you > > > a particular log file name by which you can determine which > > > process crashed. You should also be able to see a coredump. Why > > > don't you attach the same for further analysis? > > > > > > ~Atin > > > > > > On Mon, Jun 27, 2016 at 1:17 PM, Yann LEMARIE <ylemarie at iraiser.e > > > u> wrote: > > > > Hi, > > > > > > > > I'm using GlusterFS since many years and never see this > > > > problem, but this is the second time in one week ... > > > > > > > > I have 3 volumes with 2 bricks and 1 volume crash with no > > > > reason, I just have to stop/start the volume to make it up > > > > again. > > > > The only logs I can find are in syslog : > > > > > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames: > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10) > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset: > > > > > git://git.gluster.com/glusterfs.git > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash: > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details: > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string: > > > > > glusterfs 3.7.11 > > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: --------- > > > > > > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames: > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10) > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset: > > > > > git://git.gluster.com/glusterfs.git > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash: > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details: > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string: > > > > > glusterfs 3.7.11 > > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: --------- > > > > > > > > > ? > > > > Thanks for your help > > > > > > > > > > > > Regards > > > > --? > > > > Yann Lemari? > > > > iRaiser - Support Technique > > > > ? > > > > ylemarie at iraiser.eu > > > > > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users at gluster.org > > > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > ? > > --? > > Yann Lemari? > > iRaiser - Support Technique > > ? > > ylemarie at iraiser.eu > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users