Hi Soumya, We just had the opportunity to try the option of disabling the kernel-NFS and restarting glusterd to start gNFS. However the gluster demon crashes immediately on startup. What additional information besides what we provide below would help debugging this? Thanks, Pat -------- Forwarded Message -------- Subject: gluster-nfs crashing on start Date: Mon, 7 Aug 2017 16:05:09 +0000 From: Steve Postma <SPostma at ztechnet.com> To: Pat Haley <phaley at mit.edu> *To disable kernal-nfs and enable nfs through Gluster we:* gluster volume set data-volume nfs.export-volumes on gluster volume set data-volume nfs.disable off /etc/init.d/glusterd stop service nfslock stop service rpcgssd stop service rpcidmapd stop service portmap stop service nfs stop /etc/init.d/glusterd stop *the /var/log/glusterfs/nfs.log immediately reports a crash:* * * [root at mseas-data2 glusterfs]# cat nfs.log [2017-08-07 15:20:16.327026] I [MSGID: 100030] [glusterfsd.c:2332:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.11 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/7db74f19472511d20849e471bf224c1a.socket) [2017-08-07 15:20:16.345166] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-08-07 15:20:16.351290] I [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 pending frames: frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2017-08-07 15:20:17 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.7.11 /usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb8)[0x3889625a18] /usr/lib64/libglusterfs.so.0(gf_print_trace+0x32f)[0x38896456af] /lib64/libc.so.6[0x34a1c32660] /lib64/libc.so.6[0x34a1d3382f] /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(+0x53307)[0x7f8d071b3307] /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(exp_file_parse+0x302)[0x7f8d071b3742] /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(mnt3_auth_set_exports_auth+0x45)[0x7f8d071b47a5] /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(_mnt3_init_auth_params+0x91)[0x7f8d07183e41] /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(mnt3svc_init+0x218)[0x7f8d07184228] /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(nfs_init_versions+0xd7)[0x7f8d07174a37] /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(init+0x77)[0x7f8d071767c7] /usr/lib64/libglusterfs.so.0(xlator_init+0x52)[0x3889622a82] /usr/lib64/libglusterfs.so.0(glusterfs_graph_init+0x31)[0x3889669aa1] /usr/lib64/libglusterfs.so.0(glusterfs_graph_activate+0x57)[0x3889669bd7] /usr/sbin/glusterfs(glusterfs_process_volfp+0xed)[0x405c0d] /usr/sbin/glusterfs(mgmt_getspec_cbk+0x312)[0x40dbd2] /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x3889e0f7b5] /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a1)[0x3889e10891] /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x3889e0bbd8] /usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so(+0x94cd)[0x7f8d088e04cd] /usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so(+0xa79d)[0x7f8d088e179d] /usr/lib64/libglusterfs.so.0[0x388968b2f0] /lib64/libpthread.so.0[0x34a2007aa1] /lib64/libc.so.6(clone+0x6d)[0x34a1ce8aad] [root at mseas-data2 glusterfs]# gluster volume info Volume Name: data-volume Type: Distribute Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: mseas-data2:/mnt/brick1 Brick2: mseas-data2:/mnt/brick2 Options Reconfigured: nfs.export-volumes: on nfs.disable: off performance.readdir-ahead: on diagnostics.brick-sys-log-level: WARNING nfs.exports-auth-enable: on "/var/lib/glusterd/nfs/exports" /gdata 172.16.1.0/255.255.255.0(rw) *What else can we do to identify why this is failing?** * ** ** -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170807/fdb6c81b/attachment.html>
----- Original Message -----> From: "Pat Haley" <phaley at mit.edu> > To: "Soumya Koduri" <skoduri at redhat.com>, gluster-users at gluster.org, "Pranith Kumar Karampuri" <pkarampu at redhat.com> > Cc: "Ben Turner" <bturner at redhat.com>, "Ravishankar N" <ravishankar at redhat.com>, "Raghavendra Gowdappa" > <rgowdapp at redhat.com>, "Niels de Vos" <ndevos at redhat.com>, "Steve Postma" <SPostma at ztechnet.com> > Sent: Monday, August 7, 2017 9:52:48 PM > Subject: Re: [Gluster-users] Slow write times to gluster disk > > > Hi Soumya, > > We just had the opportunity to try the option of disabling the > kernel-NFS and restarting glusterd to start gNFS. However the gluster > demon crashes immediately on startup. What additional information > besides what we provide below would help debugging this? >Which version of glusterfs are you using? There were few regressions caused (all fixed now in master branch atleast) by recent changes in mount codepath. Request Niels to comment. Thanks, Soumya> Thanks, > > Pat > > > -------- Forwarded Message -------- > Subject: gluster-nfs crashing on start > Date: Mon, 7 Aug 2017 16:05:09 +0000 > From: Steve Postma <SPostma at ztechnet.com> > To: Pat Haley <phaley at mit.edu> > > > > *To disable kernal-nfs and enable nfs through Gluster we:* > > > gluster volume set data-volume nfs.export-volumes on > gluster volume set data-volume nfs.disable off > > /etc/init.d/glusterd stop > > > service nfslock stop > > service rpcgssd stop > > service rpcidmapd stop > > service portmap stop > > service nfs stop > > > /etc/init.d/glusterd stop > > > > > *the /var/log/glusterfs/nfs.log immediately reports a crash:* > > * > * > > [root at mseas-data2 glusterfs]# cat nfs.log > > [2017-08-07 15:20:16.327026] I [MSGID: 100030] [glusterfsd.c:2332:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version > 3.7.11 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs > -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S > /var/run/gluster/7db74f19472511d20849e471bf224c1a.socket) > > [2017-08-07 15:20:16.345166] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > > [2017-08-07 15:20:16.351290] I > [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: > Configured rpc.outstanding-rpc-limit with value 16 > > pending frames: > > frame : type(0) op(0) > > patchset: git://git.gluster.com/glusterfs.git > > signal received: 11 > > time of crash: > > 2017-08-07 15:20:17 > > configuration details: > > argp 1 > > backtrace 1 > > dlfcn 1 > > libpthread 1 > > llistxattr 1 > > setfsid 1 > > spinlock 1 > > epoll.h 1 > > xattr.h 1 > > st_atim.tv_nsec 1 > > package-string: glusterfs 3.7.11 > > /usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb8)[0x3889625a18] > > /usr/lib64/libglusterfs.so.0(gf_print_trace+0x32f)[0x38896456af] > > /lib64/libc.so.6[0x34a1c32660] > > /lib64/libc.so.6[0x34a1d3382f] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(+0x53307)[0x7f8d071b3307] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(exp_file_parse+0x302)[0x7f8d071b3742] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(mnt3_auth_set_exports_auth+0x45)[0x7f8d071b47a5] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(_mnt3_init_auth_params+0x91)[0x7f8d07183e41] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(mnt3svc_init+0x218)[0x7f8d07184228] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(nfs_init_versions+0xd7)[0x7f8d07174a37] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(init+0x77)[0x7f8d071767c7] > > /usr/lib64/libglusterfs.so.0(xlator_init+0x52)[0x3889622a82] > > /usr/lib64/libglusterfs.so.0(glusterfs_graph_init+0x31)[0x3889669aa1] > > /usr/lib64/libglusterfs.so.0(glusterfs_graph_activate+0x57)[0x3889669bd7] > > /usr/sbin/glusterfs(glusterfs_process_volfp+0xed)[0x405c0d] > > /usr/sbin/glusterfs(mgmt_getspec_cbk+0x312)[0x40dbd2] > > /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x3889e0f7b5] > > /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a1)[0x3889e10891] > > /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x3889e0bbd8] > > /usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so(+0x94cd)[0x7f8d088e04cd] > > /usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so(+0xa79d)[0x7f8d088e179d] > > /usr/lib64/libglusterfs.so.0[0x388968b2f0] > > /lib64/libpthread.so.0[0x34a2007aa1] > > /lib64/libc.so.6(clone+0x6d)[0x34a1ce8aad] > > > > > > > [root at mseas-data2 glusterfs]# gluster volume info > > Volume Name: data-volume > > Type: Distribute > > Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 > > Status: Started > > Number of Bricks: 2 > > Transport-type: tcp > > Bricks: > > Brick1: mseas-data2:/mnt/brick1 > > Brick2: mseas-data2:/mnt/brick2 > > Options Reconfigured: > > nfs.export-volumes: on > > nfs.disable: off > > performance.readdir-ahead: on > > diagnostics.brick-sys-log-level: WARNING > > nfs.exports-auth-enable: on > > > > "/var/lib/glusterd/nfs/exports" > > /gdata 172.16.1.0/255.255.255.0(rw) > > > > > *What else can we do to identify why this is failing?** > * > > ** > ** > >
Soumya, its [root at mseas-data2 ~]# glusterfs --version glusterfs 3.7.11 built on Apr 27 2016 14:09:20 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root at mseas-data2 ~]# Thanks, Steve ________________________________ From: Soumya Koduri <skoduri at redhat.com> Sent: Tuesday, August 8, 2017 1:37 AM To: Pat Haley; Niels de Vos Cc: gluster-users at gluster.org; Pranith Kumar Karampuri; Ben Turner; Ravishankar N; Raghavendra Gowdappa; Niels de Vos; Steve Postma Subject: Re: [Gluster-users] Slow write times to gluster disk ----- Original Message -----> From: "Pat Haley" <phaley at mit.edu><mailto:phaley at mit.edu>> > To: "Soumya Koduri" <skoduri at redhat.com><mailto:skoduri at redhat.com>>, gluster-users at gluster.org<mailto:gluster-users at gluster.org>, "Pranith Kumar Karampuri" <pkarampu at redhat.com><mailto:pkarampu at redhat.com>> > Cc: "Ben Turner" <bturner at redhat.com><mailto:bturner at redhat.com>>, "Ravishankar N" <ravishankar at redhat.com><mailto:ravishankar at redhat.com>>, "Raghavendra Gowdappa" > <rgowdapp at redhat.com><mailto:rgowdapp at redhat.com>>, "Niels de Vos" <ndevos at redhat.com><mailto:ndevos at redhat.com>>, "Steve Postma" <SPostma at ztechnet.com><mailto:SPostma at ztechnet.com>> > Sent: Monday, August 7, 2017 9:52:48 PM > Subject: Re: [Gluster-users] Slow write times to gluster disk > > > Hi Soumya, > > We just had the opportunity to try the option of disabling the > kernel-NFS and restarting glusterd to start gNFS. However the gluster > demon crashes immediately on startup. What additional information > besides what we provide below would help debugging this? >Which version of glusterfs are you using? There were few regressions caused (all fixed now in master branch atleast) by recent changes in mount codepath. Request Niels to comment. Thanks, Soumya> Thanks, > > Pat > > > -------- Forwarded Message -------- > Subject: gluster-nfs crashing on start > Date: Mon, 7 Aug 2017 16:05:09 +0000 > From: Steve Postma <SPostma at ztechnet.com><mailto:SPostma at ztechnet.com>> > To: Pat Haley <phaley at mit.edu><mailto:phaley at mit.edu>> > > > > *To disable kernal-nfs and enable nfs through Gluster we:* > > > gluster volume set data-volume nfs.export-volumes on > gluster volume set data-volume nfs.disable off > > /etc/init.d/glusterd stop > > > service nfslock stop > > service rpcgssd stop > > service rpcidmapd stop > > service portmap stop > > service nfs stop > > > /etc/init.d/glusterd stop > > > > > *the /var/log/glusterfs/nfs.log immediately reports a crash:* > > * > * > > [root at mseas-data2<mailto:root at mseas-data2> glusterfs]# cat nfs.log > > [2017-08-07 15:20:16.327026] I [MSGID: 100030] [glusterfsd.c:2332:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version > 3.7.11 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs > -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S > /var/run/gluster/7db74f19472511d20849e471bf224c1a.socket) > > [2017-08-07 15:20:16.345166] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > > [2017-08-07 15:20:16.351290] I > [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: > Configured rpc.outstanding-rpc-limit with value 16 > > pending frames: > > frame : type(0) op(0) > > patchset: git://git.gluster.com/glusterfs.git > > signal received: 11 > > time of crash: > > 2017-08-07 15:20:17 > > configuration details: > > argp 1 > > backtrace 1 > > dlfcn 1 > > libpthread 1 > > llistxattr 1 > > setfsid 1 > > spinlock 1 > > epoll.h 1 > > xattr.h 1 > > st_atim.tv_nsec 1 > > package-string: glusterfs 3.7.11 > > /usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb8)[0x3889625a18] > > /usr/lib64/libglusterfs.so.0(gf_print_trace+0x32f)[0x38896456af] > > /lib64/libc.so.6[0x34a1c32660] > > /lib64/libc.so.6[0x34a1d3382f] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(+0x53307)[0x7f8d071b3307] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(exp_file_parse+0x302)[0x7f8d071b3742] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(mnt3_auth_set_exports_auth+0x45)[0x7f8d071b47a5] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(_mnt3_init_auth_params+0x91)[0x7f8d07183e41] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(mnt3svc_init+0x218)[0x7f8d07184228] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(nfs_init_versions+0xd7)[0x7f8d07174a37] > > /usr/lib64/glusterfs/3.7.11/xlator/nfs/server.so(init+0x77)[0x7f8d071767c7] > > /usr/lib64/libglusterfs.so.0(xlator_init+0x52)[0x3889622a82] > > /usr/lib64/libglusterfs.so.0(glusterfs_graph_init+0x31)[0x3889669aa1] > > /usr/lib64/libglusterfs.so.0(glusterfs_graph_activate+0x57)[0x3889669bd7] > > /usr/sbin/glusterfs(glusterfs_process_volfp+0xed)[0x405c0d] > > /usr/sbin/glusterfs(mgmt_getspec_cbk+0x312)[0x40dbd2] > > /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x3889e0f7b5] > > /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a1)[0x3889e10891] > > /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x3889e0bbd8] > > /usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so(+0x94cd)[0x7f8d088e04cd] > > /usr/lib64/glusterfs/3.7.11/rpc-transport/socket.so(+0xa79d)[0x7f8d088e179d] > > /usr/lib64/libglusterfs.so.0[0x388968b2f0] > > /lib64/libpthread.so.0[0x34a2007aa1] > > /lib64/libc.so.6(clone+0x6d)[0x34a1ce8aad] > > > > > > > [root at mseas-data2<mailto:root at mseas-data2> glusterfs]# gluster volume info > > Volume Name: data-volume > > Type: Distribute > > Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 > > Status: Started > > Number of Bricks: 2 > > Transport-type: tcp > > Bricks: > > Brick1: mseas-data2:/mnt/brick1 > > Brick2: mseas-data2:/mnt/brick2 > > Options Reconfigured: > > nfs.export-volumes: on > > nfs.disable: off > > performance.readdir-ahead: on > > diagnostics.brick-sys-log-level: WARNING > > nfs.exports-auth-enable: on > > > > "/var/lib/glusterd/nfs/exports" > > /gdata 172.16.1.0/255.255.255.0(rw) > > > > > *What else can we do to identify why this is failing?** > * > > ** > ** > >________________________________ Total Control Panel Login<https://asp.reflexion.net/login?domain=ztechnet.com> To: spostma at ztechnet.com<https://asp.reflexion.net/address-properties?aID=11541174&domain=ztechnet.com> From: skoduri at redhat.com Message Score: 1 High (60): Pass My Spam Blocking Level: Medium Medium (75): Pass Low (90): Pass Block<https://asp.reflexion.net/FooterAction?ver=3&bl-sender-address=1&hID=22715781184&domain=ztechnet.com> this sender / Block<https://asp.reflexion.net/FooterAction?ver=3&ent=1&bl-sender-address=1&hID=22715781184&domain=ztechnet.com> this sender enterprise-wide Block<https://asp.reflexion.net/FooterAction?ver=3&bl-sender-domain=1&hID=22715781184&domain=ztechnet.com> redhat.com / Block<https://asp.reflexion.net/FooterAction?ver=3&ent=1&bl-sender-domain=1&hID=22715781184&domain=ztechnet.com> redhat.com enterprise-wide This message was delivered because the content filter score did not exceed your filter level. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170808/e7e746d7/attachment.html>