Yaroslav Molochko
2015-Sep-14 17:09 UTC
[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers
any ideas? 2015-09-13 20:41 GMT+08:00 Yaroslav Molochko <onorua at gmail.com>:> So, I've done: > root at PSC01SERV008:/var/log# tail -f syslog | grep -Ev > 'docker|kubelet|kube-proxy' > Sep 13 12:18:16 psc01serv008 systemd[1]: Stopped GlusterFS an clustered > file-system server. > Sep 13 12:19:21 psc01serv008 systemd[1]: Reloading. > Sep 13 12:19:35 psc01serv008 systemd[1]: message repeated 3 times: [ > Reloading.] > Sep 13 12:20:10 psc01serv008 systemd[1]: Starting GlusterFS an clustered > file-system server... > Sep 13 12:20:12 psc01serv008 systemd[1]: Started GlusterFS an clustered > file-system server. > > and stopped glusterfs, it said it was stopped but processes where there, > and I killed them manually, maybe something wrong with the system unit > file, but it was working with 3.5, so don't know. Then I disabled my > "homemade" glusterfs service and enabled /etc/init.d/glusterfs-server and > it got the same problem, I could not restart the glusterfs processes from > "init" doesn't matter what init file I try. > And when I kill processes by hands, it starts up but, as you can see there > is no reports of any problems with starting up the NFS or any blocking > port. There is no firewalld running on my host, and the problem is that I > have 2 hosts identical to peer with, one is working and one is not. > dmesg is attached as well as my "handmade" glusterfs systemd service, just > in case I start it wrongly. > > 2015-09-13 19:00 GMT+08:00 Soumya Koduri <skoduri at redhat.com>: > >> >> >> On 09/13/2015 09:38 AM, Yaroslav Molochko wrote: >> >>> I wish this could be that simple: >>> root at PSC01SERV008:/var/lib# netstat -nap | grep 38465 >>> root at PSC01SERV008:/var/lib# ss -n | grep 38465 >>> root at PSC01SERV008:/var/lib# >>> >>> 2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83 at gmail.com >>> <mailto:atin.mukherjee83 at gmail.com>>: >>> >>> By any chance is your Gluster NFS server is already running? Output >>> of netstat -nap | grep 38465 might give some clue? >>> >>> -Atin >>> Sent from one plus one >>> >>> On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua at gmail.com >>> <mailto:onorua at gmail.com>> wrote: >>> >>> Hello, >>> >>> I have a problem reported in logs: >>> =================>>> [2015-09-12 13:56:06.271644] I [MSGID: 100030] >>> [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running >>> /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs -s >>> localhost --volfile-id gluster/nfs -p >>> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log >>> -S /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket) >>> [2015-09-12 13:56:06.277921] I [MSGID: 101190] >>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started >>> thread with index 1 >>> [2015-09-12 13:56:07.284888] I >>> [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: >>> Configured rpc.outstanding-rpc-limit with value 16 >>> [2015-09-12 13:56:07.292484] W [MSGID: 112153] >>> [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been >>> disabled! >>> [2015-09-12 13:56:07.294357] E >>> [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service: >>> Could not register with portmap 100005 3 38465 >>> >> >> Port registration failed. Could you check '/var/log/messages' and dmesg >> to see if there are any errors logged? Is firewalld running on your system. >> Verify if the port is open to be used. >> >> Thanks, >> Soumya >> >>> [2015-09-12 13:56:07.294398] E [MSGID: 112088] >>> [nfs.c:341:nfs_init_versions] 0-nfs: Required program MOUNT3 >>> registration failed >>> [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init] >>> 0-nfs: Failed to initialize protocols >>> [2015-09-12 13:56:07.294426] E [MSGID: 101019] >>> [xlator.c:428:xlator_init] 0-nfs-server: Initialization of >>> volume 'nfs-server' failed, review your volfile again >>> [2015-09-12 13:56:07.294438] E >>> [graph.c:322:glusterfs_graph_init] 0-nfs-server: initializing >>> translator failed >>> [2015-09-12 13:56:07.294448] E >>> [graph.c:661:glusterfs_graph_activate] 0-graph: init failed >>> [2015-09-12 13:56:07.294781] W >>> [glusterfsd.c:1219:cleanup_and_exit] >>> (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a] >>> -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123) >>> [0x7fbe9c74fcb3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) >>> [0x7fbe9c74f329] ) 0-: received signum (0), shutting down >>> ==================>>> >>> I've checked the page: >>> >>> http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions >>> >>> I've found report in RedHat that it's necessary to remove -w >>> from rpcbind becuse some times it causes problems. >>> I did all that but still no luck on one of the servers, what is >>> interesting, the other server (peered) is working fine without >>> any problems. >>> >>> root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs >>> ? nfs.service >>> Loaded: not-found (Reason: No such file or directory) >>> Active: inactive (dead) >>> >>> root at PSC01SERV008:/var/lib/glusterd/nfs# systemctl status >>> rpcbind >>> ? rpcbind.service - RPC bind portmap service >>> Loaded: loaded (/etc/systemd/system/rpcbind.service; >>> enabled; vendor preset: enabled) >>> Drop-In: /run/systemd/generator/rpcbind.service.d >>> ??50-rpcbind-$portmap.conf >>> Active: active (running) since Sat 2015-09-12 13:55:07 UTC; >>> 6min ago >>> Main PID: 9796 (rpcbind) >>> Memory: 428.0K >>> CGroup: /system.slice/rpcbind.service >>> ??9796 /sbin/rpcbind >>> >>> Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind >>> portmap service... >>> Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind >>> portmap service. >>> root at PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p >>> program vers proto port service >>> 100000 4 tcp 111 portmapper >>> 100000 3 tcp 111 portmapper >>> 100000 2 tcp 111 portmapper >>> 100000 4 udp 111 portmapper >>> 100000 3 udp 111 portmapper >>> 100000 2 udp 111 portmapper >>> >>> I've tried to reinstall it agaon and again - but there is no >>> luck. >>> >>> What I have: >>> cat /etc/lsb-release >>> DISTRIB_ID=Ubuntu >>> DISTRIB_RELEASE=15.04 >>> DISTRIB_CODENAME=vivid >>> DISTRIB_DESCRIPTION="Ubuntu 15.04" >>> >>> ii glusterfs-client >>> 3.7.4-ubuntu1~vivid1 amd64 clustered >>> file-system (client package) >>> ii glusterfs-common >>> 3.7.4-ubuntu1~vivid1 amd64 GlusterFS common >>> libraries and translator modules >>> ii glusterfs-server >>> 3.7.4-ubuntu1~vivid1 amd64 clustered >>> file-system (server package) >>> >>> What else can I check? How can I fix it, what is most important >>> :) >>> Thanks in advance! >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150915/b6d758a3/attachment.html>
Soumya Koduri
2015-Sep-14 17:55 UTC
[Gluster-users] After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers
Could you try * disabling iptables (& firewalld if enabled) * restart rpcbind service * restart glusterd If this doesn't work, (mentioned in one of the forums) Add below line in '/etc/hosts.allow' file. ALL: 127.0.0.1 : ALLOW Restart rpcbind and glusterd services. Thanks, Soumya On 09/14/2015 10:39 PM, Yaroslav Molochko wrote:> Could not register with portmap