Soumya Koduri
2015-Sep-13 11:00 UTC
[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers
On 09/13/2015 09:38 AM, Yaroslav Molochko wrote:> I wish this could be that simple: > root at PSC01SERV008:/var/lib# netstat -nap | grep 38465 > root at PSC01SERV008:/var/lib# ss -n | grep 38465 > root at PSC01SERV008:/var/lib# > > 2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83 at gmail.com > <mailto:atin.mukherjee83 at gmail.com>>: > > By any chance is your Gluster NFS server is already running? Output > of netstat -nap | grep 38465 might give some clue? > > -Atin > Sent from one plus one > > On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua at gmail.com > <mailto:onorua at gmail.com>> wrote: > > Hello, > > I have a problem reported in logs: > =================> [2015-09-12 13:56:06.271644] I [MSGID: 100030] > [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running > /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs -s > localhost --volfile-id gluster/nfs -p > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log > -S /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket) > [2015-09-12 13:56:06.277921] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 1 > [2015-09-12 13:56:07.284888] I > [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: > Configured rpc.outstanding-rpc-limit with value 16 > [2015-09-12 13:56:07.292484] W [MSGID: 112153] > [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been > disabled! > [2015-09-12 13:56:07.294357] E > [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service: > Could not register with portmap 100005 3 38465Port registration failed. Could you check '/var/log/messages' and dmesg to see if there are any errors logged? Is firewalld running on your system. Verify if the port is open to be used. Thanks, Soumya> [2015-09-12 13:56:07.294398] E [MSGID: 112088] > [nfs.c:341:nfs_init_versions] 0-nfs: Required program MOUNT3 > registration failed > [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init] > 0-nfs: Failed to initialize protocols > [2015-09-12 13:56:07.294426] E [MSGID: 101019] > [xlator.c:428:xlator_init] 0-nfs-server: Initialization of > volume 'nfs-server' failed, review your volfile again > [2015-09-12 13:56:07.294438] E > [graph.c:322:glusterfs_graph_init] 0-nfs-server: initializing > translator failed > [2015-09-12 13:56:07.294448] E > [graph.c:661:glusterfs_graph_activate] 0-graph: init failed > [2015-09-12 13:56:07.294781] W > [glusterfsd.c:1219:cleanup_and_exit] > (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a] > -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123) > [0x7fbe9c74fcb3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) > [0x7fbe9c74f329] ) 0-: received signum (0), shutting down > ==================> > I've checked the page: > http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions > > I've found report in RedHat that it's necessary to remove -w > from rpcbind becuse some times it causes problems. > I did all that but still no luck on one of the servers, what is > interesting, the other server (peered) is working fine without > any problems. > > root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs > ? nfs.service > Loaded: not-found (Reason: No such file or directory) > Active: inactive (dead) > > root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status rpcbind > ? rpcbind.service - RPC bind portmap service > Loaded: loaded (/etc/systemd/system/rpcbind.service; > enabled; vendor preset: enabled) > Drop-In: /run/systemd/generator/rpcbind.service.d > ??50-rpcbind-$portmap.conf > Active: active (running) since Sat 2015-09-12 13:55:07 UTC; > 6min ago > Main PID: 9796 (rpcbind) > Memory: 428.0K > CGroup: /system.slice/rpcbind.service > ??9796 /sbin/rpcbind > > Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind > portmap service... > Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind > portmap service. > root at PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p > program vers proto port service > 100000 4 tcp 111 portmapper > 100000 3 tcp 111 portmapper > 100000 2 tcp 111 portmapper > 100000 4 udp 111 portmapper > 100000 3 udp 111 portmapper > 100000 2 udp 111 portmapper > > I've tried to reinstall it agaon and again - but there is no luck. > > What I have: > cat /etc/lsb-release > DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=15.04 > DISTRIB_CODENAME=vivid > DISTRIB_DESCRIPTION="Ubuntu 15.04" > > ii glusterfs-client > 3.7.4-ubuntu1~vivid1 amd64 clustered > file-system (client package) > ii glusterfs-common > 3.7.4-ubuntu1~vivid1 amd64 GlusterFS common > libraries and translator modules > ii glusterfs-server > 3.7.4-ubuntu1~vivid1 amd64 clustered > file-system (server package) > > What else can I check? How can I fix it, what is most important :) > Thanks in advance! > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >
Yaroslav Molochko
2015-Sep-13 12:41 UTC
[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers
So, I've done: root at PSC01SERV008:/var/log# tail -f syslog | grep -Ev 'docker|kubelet|kube-proxy' Sep 13 12:18:16 psc01serv008 systemd[1]: Stopped GlusterFS an clustered file-system server. Sep 13 12:19:21 psc01serv008 systemd[1]: Reloading. Sep 13 12:19:35 psc01serv008 systemd[1]: message repeated 3 times: [ Reloading.] Sep 13 12:20:10 psc01serv008 systemd[1]: Starting GlusterFS an clustered file-system server... Sep 13 12:20:12 psc01serv008 systemd[1]: Started GlusterFS an clustered file-system server. and stopped glusterfs, it said it was stopped but processes where there, and I killed them manually, maybe something wrong with the system unit file, but it was working with 3.5, so don't know. Then I disabled my "homemade" glusterfs service and enabled /etc/init.d/glusterfs-server and it got the same problem, I could not restart the glusterfs processes from "init" doesn't matter what init file I try. And when I kill processes by hands, it starts up but, as you can see there is no reports of any problems with starting up the NFS or any blocking port. There is no firewalld running on my host, and the problem is that I have 2 hosts identical to peer with, one is working and one is not. dmesg is attached as well as my "handmade" glusterfs systemd service, just in case I start it wrongly. 2015-09-13 19:00 GMT+08:00 Soumya Koduri <skoduri at redhat.com>:> > > On 09/13/2015 09:38 AM, Yaroslav Molochko wrote: > >> I wish this could be that simple: >> root at PSC01SERV008:/var/lib# netstat -nap | grep 38465 >> root at PSC01SERV008:/var/lib# ss -n | grep 38465 >> root at PSC01SERV008:/var/lib# >> >> 2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83 at gmail.com >> <mailto:atin.mukherjee83 at gmail.com>>: >> >> By any chance is your Gluster NFS server is already running? Output >> of netstat -nap | grep 38465 might give some clue? >> >> -Atin >> Sent from one plus one >> >> On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua at gmail.com >> <mailto:onorua at gmail.com>> wrote: >> >> Hello, >> >> I have a problem reported in logs: >> =================>> [2015-09-12 13:56:06.271644] I [MSGID: 100030] >> [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running >> /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs -s >> localhost --volfile-id gluster/nfs -p >> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log >> -S /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket) >> [2015-09-12 13:56:06.277921] I [MSGID: 101190] >> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started >> thread with index 1 >> [2015-09-12 13:56:07.284888] I >> [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: >> Configured rpc.outstanding-rpc-limit with value 16 >> [2015-09-12 13:56:07.292484] W [MSGID: 112153] >> [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been >> disabled! >> [2015-09-12 13:56:07.294357] E >> [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service: >> Could not register with portmap 100005 3 38465 >> > > Port registration failed. Could you check '/var/log/messages' and dmesg to > see if there are any errors logged? Is firewalld running on your system. > Verify if the port is open to be used. > > Thanks, > Soumya > >> [2015-09-12 13:56:07.294398] E [MSGID: 112088] >> [nfs.c:341:nfs_init_versions] 0-nfs: Required program MOUNT3 >> registration failed >> [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init] >> 0-nfs: Failed to initialize protocols >> [2015-09-12 13:56:07.294426] E [MSGID: 101019] >> [xlator.c:428:xlator_init] 0-nfs-server: Initialization of >> volume 'nfs-server' failed, review your volfile again >> [2015-09-12 13:56:07.294438] E >> [graph.c:322:glusterfs_graph_init] 0-nfs-server: initializing >> translator failed >> [2015-09-12 13:56:07.294448] E >> [graph.c:661:glusterfs_graph_activate] 0-graph: init failed >> [2015-09-12 13:56:07.294781] W >> [glusterfsd.c:1219:cleanup_and_exit] >> (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a] >> -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123) >> [0x7fbe9c74fcb3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) >> [0x7fbe9c74f329] ) 0-: received signum (0), shutting down >> ==================>> >> I've checked the page: >> >> http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions >> >> I've found report in RedHat that it's necessary to remove -w >> from rpcbind becuse some times it causes problems. >> I did all that but still no luck on one of the servers, what is >> interesting, the other server (peered) is working fine without >> any problems. >> >> root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs >> ? nfs.service >> Loaded: not-found (Reason: No such file or directory) >> Active: inactive (dead) >> >> root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status rpcbind >> ? rpcbind.service - RPC bind portmap service >> Loaded: loaded (/etc/systemd/system/rpcbind.service; >> enabled; vendor preset: enabled) >> Drop-In: /run/systemd/generator/rpcbind.service.d >> ??50-rpcbind-$portmap.conf >> Active: active (running) since Sat 2015-09-12 13:55:07 UTC; >> 6min ago >> Main PID: 9796 (rpcbind) >> Memory: 428.0K >> CGroup: /system.slice/rpcbind.service >> ??9796 /sbin/rpcbind >> >> Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind >> portmap service... >> Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind >> portmap service. >> root at PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p >> program vers proto port service >> 100000 4 tcp 111 portmapper >> 100000 3 tcp 111 portmapper >> 100000 2 tcp 111 portmapper >> 100000 4 udp 111 portmapper >> 100000 3 udp 111 portmapper >> 100000 2 udp 111 portmapper >> >> I've tried to reinstall it agaon and again - but there is no luck. >> >> What I have: >> cat /etc/lsb-release >> DISTRIB_ID=Ubuntu >> DISTRIB_RELEASE=15.04 >> DISTRIB_CODENAME=vivid >> DISTRIB_DESCRIPTION="Ubuntu 15.04" >> >> ii glusterfs-client >> 3.7.4-ubuntu1~vivid1 amd64 clustered >> file-system (client package) >> ii glusterfs-common >> 3.7.4-ubuntu1~vivid1 amd64 GlusterFS common >> libraries and translator modules >> ii glusterfs-server >> 3.7.4-ubuntu1~vivid1 amd64 clustered >> file-system (server package) >> >> What else can I check? How can I fix it, what is most important :) >> Thanks in advance! >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> http://www.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150913/545898be/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: dmesg.log Type: application/octet-stream Size: 110382 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150913/545898be/attachment.obj> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterfs.service Type: application/octet-stream Size: 278 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150913/545898be/attachment-0001.obj>
Niels de Vos
2015-Sep-15 20:37 UTC
[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers
On Sun, Sep 13, 2015 at 04:30:43PM +0530, Soumya Koduri wrote:> > > On 09/13/2015 09:38 AM, Yaroslav Molochko wrote: > >I wish this could be that simple: > >root at PSC01SERV008:/var/lib# netstat -nap | grep 38465 > >root at PSC01SERV008:/var/lib# ss -n | grep 38465 > >root at PSC01SERV008:/var/lib# > > > >2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83 at gmail.com > ><mailto:atin.mukherjee83 at gmail.com>>: > > > > By any chance is your Gluster NFS server is already running? Output > > of netstat -nap | grep 38465 might give some clue? > > > > -Atin > > Sent from one plus one > > > > On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua at gmail.com > > <mailto:onorua at gmail.com>> wrote: > > > > Hello, > > > > I have a problem reported in logs: > > =================> > [2015-09-12 13:56:06.271644] I [MSGID: 100030] > > [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running > > /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs -s > > localhost --volfile-id gluster/nfs -p > > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log > > -S /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket) > > [2015-09-12 13:56:06.277921] I [MSGID: 101190] > > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > > thread with index 1 > > [2015-09-12 13:56:07.284888] I > > [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: > > Configured rpc.outstanding-rpc-limit with value 16 > > [2015-09-12 13:56:07.292484] W [MSGID: 112153] > > [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been > > disabled! > > [2015-09-12 13:56:07.294357] E > > [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service: > > Could not register with portmap 100005 3 38465 > > Port registration failed. Could you check '/var/log/messages' and dmesg to > see if there are any errors logged? Is firewalld running on your system. > Verify if the port is open to be used.Registration of services at portmap can also fail because there is a service with that program number and version registered already. Could you check if that is the case? $ rpcinfo -p | grep 100005 100005 3 tcp 38465 mountd 100005 1 tcp 38466 mountd If you see a similar output, check if you have standard NFS services running. The rpc.mountd process will also register itself at the portmapper, but it will conflict with the NFS-services that Gluster provides. Make sure all NFS services (server and client) have been disabled and stopped. After that, check with the 'rpcinfo' command if any of nlockmgr, mount, status or nfs are registered. If that is the case, you can unregister them one-by-one with commands like this: # rpcinfo -d 100005 1 # rpcinfo -d 100005 3 ... After unregistering the services at the portmapper, you should be able to start the Gluster-NFS service by restarting glusterd. HTH, Niels> > Thanks, > Soumya > > [2015-09-12 13:56:07.294398] E [MSGID: 112088] > > [nfs.c:341:nfs_init_versions] 0-nfs: Required program MOUNT3 > > registration failed > > [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init] > > 0-nfs: Failed to initialize protocols > > [2015-09-12 13:56:07.294426] E [MSGID: 101019] > > [xlator.c:428:xlator_init] 0-nfs-server: Initialization of > > volume 'nfs-server' failed, review your volfile again > > [2015-09-12 13:56:07.294438] E > > [graph.c:322:glusterfs_graph_init] 0-nfs-server: initializing > > translator failed > > [2015-09-12 13:56:07.294448] E > > [graph.c:661:glusterfs_graph_activate] 0-graph: init failed > > [2015-09-12 13:56:07.294781] W > > [glusterfsd.c:1219:cleanup_and_exit] > > (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a] > > -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123) > > [0x7fbe9c74fcb3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) > > [0x7fbe9c74f329] ) 0-: received signum (0), shutting down > > ==================> > > > I've checked the page: > > http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions > > > > I've found report in RedHat that it's necessary to remove -w > > from rpcbind becuse some times it causes problems. > > I did all that but still no luck on one of the servers, what is > > interesting, the other server (peered) is working fine without > > any problems. > > > > root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs > > ? nfs.service > > Loaded: not-found (Reason: No such file or directory) > > Active: inactive (dead) > > > > root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status rpcbind > > ? rpcbind.service - RPC bind portmap service > > Loaded: loaded (/etc/systemd/system/rpcbind.service; > > enabled; vendor preset: enabled) > > Drop-In: /run/systemd/generator/rpcbind.service.d > > ??50-rpcbind-$portmap.conf > > Active: active (running) since Sat 2015-09-12 13:55:07 UTC; > > 6min ago > > Main PID: 9796 (rpcbind) > > Memory: 428.0K > > CGroup: /system.slice/rpcbind.service > > ??9796 /sbin/rpcbind > > > > Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind > > portmap service... > > Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind > > portmap service. > > root at PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p > > program vers proto port service > > 100000 4 tcp 111 portmapper > > 100000 3 tcp 111 portmapper > > 100000 2 tcp 111 portmapper > > 100000 4 udp 111 portmapper > > 100000 3 udp 111 portmapper > > 100000 2 udp 111 portmapper > > > > I've tried to reinstall it agaon and again - but there is no luck. > > > > What I have: > > cat /etc/lsb-release > > DISTRIB_ID=Ubuntu > > DISTRIB_RELEASE=15.04 > > DISTRIB_CODENAME=vivid > > DISTRIB_DESCRIPTION="Ubuntu 15.04" > > > > ii glusterfs-client > > 3.7.4-ubuntu1~vivid1 amd64 clustered > > file-system (client package) > > ii glusterfs-common > > 3.7.4-ubuntu1~vivid1 amd64 GlusterFS common > > libraries and translator modules > > ii glusterfs-server > > 3.7.4-ubuntu1~vivid1 amd64 clustered > > file-system (server package) > > > > What else can I check? How can I fix it, what is most important :) > > Thanks in advance! > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > >_______________________________________________ > >Gluster-users mailing list > >Gluster-users at gluster.org > >http://www.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150915/8101b129/attachment.sig>