Yaroslav Molochko
2015-Sep-13 04:08 UTC
[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers
I wish this could be that simple: root at PSC01SERV008:/var/lib# netstat -nap | grep 38465 root at PSC01SERV008:/var/lib# ss -n | grep 38465 root at PSC01SERV008:/var/lib# 2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83 at gmail.com>:> By any chance is your Gluster NFS server is already running? Output of > netstat -nap | grep 38465 might give some clue? > > -Atin > Sent from one plus one > On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua at gmail.com> wrote: > >> Hello, >> >> I have a problem reported in logs: >> =================>> [2015-09-12 13:56:06.271644] I [MSGID: 100030] [glusterfsd.c:2301:main] >> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4 >> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p >> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S >> /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket) >> [2015-09-12 13:56:06.277921] I [MSGID: 101190] >> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 1 >> [2015-09-12 13:56:07.284888] I >> [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured >> rpc.outstanding-rpc-limit with value 16 >> [2015-09-12 13:56:07.292484] W [MSGID: 112153] >> [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled! >> [2015-09-12 13:56:07.294357] E >> [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service: Could not >> register with portmap 100005 3 38465 >> [2015-09-12 13:56:07.294398] E [MSGID: 112088] >> [nfs.c:341:nfs_init_versions] 0-nfs: Required program MOUNT3 registration >> failed >> [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init] 0-nfs: >> Failed to initialize protocols >> [2015-09-12 13:56:07.294426] E [MSGID: 101019] [xlator.c:428:xlator_init] >> 0-nfs-server: Initialization of volume 'nfs-server' failed, review your >> volfile again >> [2015-09-12 13:56:07.294438] E [graph.c:322:glusterfs_graph_init] >> 0-nfs-server: initializing translator failed >> [2015-09-12 13:56:07.294448] E [graph.c:661:glusterfs_graph_activate] >> 0-graph: init failed >> [2015-09-12 13:56:07.294781] W [glusterfsd.c:1219:cleanup_and_exit] >> (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a] >> -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123) [0x7fbe9c74fcb3] >> -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) [0x7fbe9c74f329] ) 0-: >> received signum (0), shutting down >> ==================>> >> I've checked the page: >> >> http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions >> >> I've found report in RedHat that it's necessary to remove -w from rpcbind >> becuse some times it causes problems. >> I did all that but still no luck on one of the servers, what is >> interesting, the other server (peered) is working fine without any >> problems. >> >> root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs >> ? nfs.service >> Loaded: not-found (Reason: No such file or directory) >> Active: inactive (dead) >> >> root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status rpcbind >> ? rpcbind.service - RPC bind portmap service >> Loaded: loaded (/etc/systemd/system/rpcbind.service; enabled; vendor >> preset: enabled) >> Drop-In: /run/systemd/generator/rpcbind.service.d >> ??50-rpcbind-$portmap.conf >> Active: active (running) since Sat 2015-09-12 13:55:07 UTC; 6min ago >> Main PID: 9796 (rpcbind) >> Memory: 428.0K >> CGroup: /system.slice/rpcbind.service >> ??9796 /sbin/rpcbind >> >> Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind portmap >> service... >> Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind portmap service. >> root at PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p >> program vers proto port service >> 100000 4 tcp 111 portmapper >> 100000 3 tcp 111 portmapper >> 100000 2 tcp 111 portmapper >> 100000 4 udp 111 portmapper >> 100000 3 udp 111 portmapper >> 100000 2 udp 111 portmapper >> >> I've tried to reinstall it agaon and again - but there is no luck. >> >> What I have: >> cat /etc/lsb-release >> DISTRIB_ID=Ubuntu >> DISTRIB_RELEASE=15.04 >> DISTRIB_CODENAME=vivid >> DISTRIB_DESCRIPTION="Ubuntu 15.04" >> >> ii glusterfs-client 3.7.4-ubuntu1~vivid1 >> amd64 clustered file-system (client package) >> ii glusterfs-common 3.7.4-ubuntu1~vivid1 >> amd64 GlusterFS common libraries and translator modules >> ii glusterfs-server 3.7.4-ubuntu1~vivid1 >> amd64 clustered file-system (server package) >> >> What else can I check? How can I fix it, what is most important :) >> Thanks in advance! >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150913/b42db11b/attachment.html>
Atin Mukherjee
2015-Sep-13 08:19 UTC
[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers
I just had my luck tried there. In that case I would ask NFS team too look at it. -Atin Sent from one plus one On Sep 13, 2015 9:38 AM, "Yaroslav Molochko" <onorua at gmail.com> wrote:> I wish this could be that simple: > root at PSC01SERV008:/var/lib# netstat -nap | grep 38465 > root at PSC01SERV008:/var/lib# ss -n | grep 38465 > root at PSC01SERV008:/var/lib# > > 2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83 at gmail.com>: > >> By any chance is your Gluster NFS server is already running? Output of >> netstat -nap | grep 38465 might give some clue? >> >> -Atin >> Sent from one plus one >> On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua at gmail.com> wrote: >> >>> Hello, >>> >>> I have a problem reported in logs: >>> =================>>> [2015-09-12 13:56:06.271644] I [MSGID: 100030] [glusterfsd.c:2301:main] >>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.4 >>> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p >>> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S >>> /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket) >>> [2015-09-12 13:56:06.277921] I [MSGID: 101190] >>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread >>> with index 1 >>> [2015-09-12 13:56:07.284888] I >>> [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured >>> rpc.outstanding-rpc-limit with value 16 >>> [2015-09-12 13:56:07.292484] W [MSGID: 112153] >>> [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled! >>> [2015-09-12 13:56:07.294357] E >>> [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service: Could not >>> register with portmap 100005 3 38465 >>> [2015-09-12 13:56:07.294398] E [MSGID: 112088] >>> [nfs.c:341:nfs_init_versions] 0-nfs: Required program MOUNT3 registration >>> failed >>> [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init] 0-nfs: >>> Failed to initialize protocols >>> [2015-09-12 13:56:07.294426] E [MSGID: 101019] >>> [xlator.c:428:xlator_init] 0-nfs-server: Initialization of volume >>> 'nfs-server' failed, review your volfile again >>> [2015-09-12 13:56:07.294438] E [graph.c:322:glusterfs_graph_init] >>> 0-nfs-server: initializing translator failed >>> [2015-09-12 13:56:07.294448] E [graph.c:661:glusterfs_graph_activate] >>> 0-graph: init failed >>> [2015-09-12 13:56:07.294781] W [glusterfsd.c:1219:cleanup_and_exit] >>> (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a] >>> -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123) [0x7fbe9c74fcb3] >>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) [0x7fbe9c74f329] ) 0-: >>> received signum (0), shutting down >>> ==================>>> >>> I've checked the page: >>> >>> http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions >>> >>> I've found report in RedHat that it's necessary to remove -w from >>> rpcbind becuse some times it causes problems. >>> I did all that but still no luck on one of the servers, what is >>> interesting, the other server (peered) is working fine without any >>> problems. >>> >>> root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs >>> ? nfs.service >>> Loaded: not-found (Reason: No such file or directory) >>> Active: inactive (dead) >>> >>> root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status rpcbind >>> ? rpcbind.service - RPC bind portmap service >>> Loaded: loaded (/etc/systemd/system/rpcbind.service; enabled; vendor >>> preset: enabled) >>> Drop-In: /run/systemd/generator/rpcbind.service.d >>> ??50-rpcbind-$portmap.conf >>> Active: active (running) since Sat 2015-09-12 13:55:07 UTC; 6min ago >>> Main PID: 9796 (rpcbind) >>> Memory: 428.0K >>> CGroup: /system.slice/rpcbind.service >>> ??9796 /sbin/rpcbind >>> >>> Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind portmap >>> service... >>> Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind portmap >>> service. >>> root at PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p >>> program vers proto port service >>> 100000 4 tcp 111 portmapper >>> 100000 3 tcp 111 portmapper >>> 100000 2 tcp 111 portmapper >>> 100000 4 udp 111 portmapper >>> 100000 3 udp 111 portmapper >>> 100000 2 udp 111 portmapper >>> >>> I've tried to reinstall it agaon and again - but there is no luck. >>> >>> What I have: >>> cat /etc/lsb-release >>> DISTRIB_ID=Ubuntu >>> DISTRIB_RELEASE=15.04 >>> DISTRIB_CODENAME=vivid >>> DISTRIB_DESCRIPTION="Ubuntu 15.04" >>> >>> ii glusterfs-client >>> 3.7.4-ubuntu1~vivid1 amd64 clustered file-system >>> (client package) >>> ii glusterfs-common >>> 3.7.4-ubuntu1~vivid1 amd64 GlusterFS common libraries >>> and translator modules >>> ii glusterfs-server >>> 3.7.4-ubuntu1~vivid1 amd64 clustered file-system >>> (server package) >>> >>> What else can I check? How can I fix it, what is most important :) >>> Thanks in advance! >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150913/2e9a9e7f/attachment.html>
Soumya Koduri
2015-Sep-13 11:00 UTC
[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers
On 09/13/2015 09:38 AM, Yaroslav Molochko wrote:> I wish this could be that simple: > root at PSC01SERV008:/var/lib# netstat -nap | grep 38465 > root at PSC01SERV008:/var/lib# ss -n | grep 38465 > root at PSC01SERV008:/var/lib# > > 2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83 at gmail.com > <mailto:atin.mukherjee83 at gmail.com>>: > > By any chance is your Gluster NFS server is already running? Output > of netstat -nap | grep 38465 might give some clue? > > -Atin > Sent from one plus one > > On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua at gmail.com > <mailto:onorua at gmail.com>> wrote: > > Hello, > > I have a problem reported in logs: > =================> [2015-09-12 13:56:06.271644] I [MSGID: 100030] > [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running > /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs -s > localhost --volfile-id gluster/nfs -p > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log > -S /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket) > [2015-09-12 13:56:06.277921] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 1 > [2015-09-12 13:56:07.284888] I > [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: > Configured rpc.outstanding-rpc-limit with value 16 > [2015-09-12 13:56:07.292484] W [MSGID: 112153] > [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been > disabled! > [2015-09-12 13:56:07.294357] E > [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service: > Could not register with portmap 100005 3 38465Port registration failed. Could you check '/var/log/messages' and dmesg to see if there are any errors logged? Is firewalld running on your system. Verify if the port is open to be used. Thanks, Soumya> [2015-09-12 13:56:07.294398] E [MSGID: 112088] > [nfs.c:341:nfs_init_versions] 0-nfs: Required program MOUNT3 > registration failed > [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init] > 0-nfs: Failed to initialize protocols > [2015-09-12 13:56:07.294426] E [MSGID: 101019] > [xlator.c:428:xlator_init] 0-nfs-server: Initialization of > volume 'nfs-server' failed, review your volfile again > [2015-09-12 13:56:07.294438] E > [graph.c:322:glusterfs_graph_init] 0-nfs-server: initializing > translator failed > [2015-09-12 13:56:07.294448] E > [graph.c:661:glusterfs_graph_activate] 0-graph: init failed > [2015-09-12 13:56:07.294781] W > [glusterfsd.c:1219:cleanup_and_exit] > (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a] > -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123) > [0x7fbe9c74fcb3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) > [0x7fbe9c74f329] ) 0-: received signum (0), shutting down > ==================> > I've checked the page: > http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions > > I've found report in RedHat that it's necessary to remove -w > from rpcbind becuse some times it causes problems. > I did all that but still no luck on one of the servers, what is > interesting, the other server (peered) is working fine without > any problems. > > root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs > ? nfs.service > Loaded: not-found (Reason: No such file or directory) > Active: inactive (dead) > > root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status rpcbind > ? rpcbind.service - RPC bind portmap service > Loaded: loaded (/etc/systemd/system/rpcbind.service; > enabled; vendor preset: enabled) > Drop-In: /run/systemd/generator/rpcbind.service.d > ??50-rpcbind-$portmap.conf > Active: active (running) since Sat 2015-09-12 13:55:07 UTC; > 6min ago > Main PID: 9796 (rpcbind) > Memory: 428.0K > CGroup: /system.slice/rpcbind.service > ??9796 /sbin/rpcbind > > Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind > portmap service... > Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind > portmap service. > root at PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p > program vers proto port service > 100000 4 tcp 111 portmapper > 100000 3 tcp 111 portmapper > 100000 2 tcp 111 portmapper > 100000 4 udp 111 portmapper > 100000 3 udp 111 portmapper > 100000 2 udp 111 portmapper > > I've tried to reinstall it agaon and again - but there is no luck. > > What I have: > cat /etc/lsb-release > DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=15.04 > DISTRIB_CODENAME=vivid > DISTRIB_DESCRIPTION="Ubuntu 15.04" > > ii glusterfs-client > 3.7.4-ubuntu1~vivid1 amd64 clustered > file-system (client package) > ii glusterfs-common > 3.7.4-ubuntu1~vivid1 amd64 GlusterFS common > libraries and translator modules > ii glusterfs-server > 3.7.4-ubuntu1~vivid1 amd64 clustered > file-system (server package) > > What else can I check? How can I fix it, what is most important :) > Thanks in advance! > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >