Adam Ru
2017-May-03 09:09 UTC
[Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot
Hi Soumya, thank you very much for your reply. I enabled pcsd during setup and after reboot during troubleshooting I manually started it and checked resources (pcs status). They were not running. I didn?t find what was wrong but I?m going to try it again. I?ve thoroughly checked http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/ and I can confirm that I followed all steps with one exception. I installed following RPMs: glusterfs-server glusterfs-fuse glusterfs-cli glusterfs-ganesha nfs-ganesha-xfs and the guide referenced above specifies: glusterfs-server glusterfs-api glusterfs-ganesha glusterfs-api is a dependency of one of RPMs that I installed so this is not a problem. But I cannot find any mention to install nfs-ganesha-xfs. I?ll try to setup the whole environment again without installing nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries). Again, thank you for you time to answer my previous message. Kind regards, Adam On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri <skoduri at redhat.com> wrote:> Hi, > > On 05/02/2017 01:34 AM, Rudolf wrote: > >> Hi Gluster users, >> >> First, I'd like to thank you all for this amazing open-source! Thank you! >> >> I'm working on home project ? three servers with Gluster and >> NFS-Ganesha. My goal is to create HA NFS share with three copies of each >> file on each server. >> >> My systems are CentOS 7.3 Minimal install with the latest updates and >> the most current RPMs from "centos-gluster310" repository. >> >> I followed this tutorial: >> http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using- >> nfs-ganesha-and-glusterfs-one-step-at-a-time/ >> (second half that describes multi-node HA setup) >> >> with a few exceptions: >> >> 1. All RPMs are from "centos-gluster310" repo that is installed by "yum >> -y install centos-release-gluster" >> 2. I have three nodes (not four) with "replica 3" volume. >> 3. I created empty ganesha.conf and not empty ganesha-ha.conf in >> "/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog post is >> outdated, this is now requirement) >> 4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't needed >> anymore. >> >> > Please refer to http://gluster.readthedocs.io/ > en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/ > > It is being updated with latest changes happened wrt setup. > > When I finish configuration, all is good. nfs-ganesha.service is active >> and running and from client I can ping all three VIPs and I can mount >> NFS. Copied files are replicated to all nodes. >> >> But when I restart nodes (one by one, with 5 min. delay between) then I >> cannot ping or mount (I assume that all VIPs are down). So my setup >> definitely isn't HA. >> >> I found that: >> # pcs status >> Error: cluster is not currently running on this node >> > > This means pcsd service is not up. Did you enable (systemctl enable pcsd) > pcsd service so that is comes up post reboot automatically. If not please > start it manually. > > >> and nfs-ganesha.service is in inactive state. Btw. I didn't enable >> "systemctl enable nfs-ganesha" since I assume that this is something >> that Gluster does. >> > > Please check /var/log/ganesha.log for any errors/warnings. > > We recommend not to enable nfs-ganesha.service (by default), as the shared > storage (where the ganesha.conf file resides now) should be up and running > before nfs-ganesha gets started. > So if enabled by default it could happen that shared_storage mount point > is not yet up and it resulted in nfs-ganesha service failure. If you would > like to address this, you could have a cron job which keeps checking the > mount point health and then start nfs-ganesha service. > > Thanks, > Soumya > > >> I assume that my issue is that I followed instructions in blog post from >> 2015/10 that are outdated. Unfortunately I cannot find anything better ? >> I spent whole day by googling. >> >> Would you be so kind and check the instructions in blog post and let me >> know what steps are wrong / outdated? Or please do you have more current >> instructions for Gluster+Ganesha setup? >> >> Thank you. >> >> Kind regards, >> Adam >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> >>-- Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170503/aa25166b/attachment.html>
Mahdi Adnan
2017-May-03 09:32 UTC
[Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot
Hi, Same here, when i reboot the node i have to manually execute "pcs cluster start gluster01" and pcsd already enabled and started. Gluster 3.8.11 Centos 7.3 latest Installed using CentOS Storage SIG repository -- Respectfully Mahdi A. Mahdi ________________________________ From: gluster-users-bounces at gluster.org <gluster-users-bounces at gluster.org> on behalf of Adam Ru <ad.ruckel at gmail.com> Sent: Wednesday, May 3, 2017 12:09:58 PM To: Soumya Koduri Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot Hi Soumya, thank you very much for your reply. I enabled pcsd during setup and after reboot during troubleshooting I manually started it and checked resources (pcs status). They were not running. I didn?t find what was wrong but I?m going to try it again. I?ve thoroughly checked http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/ and I can confirm that I followed all steps with one exception. I installed following RPMs: glusterfs-server glusterfs-fuse glusterfs-cli glusterfs-ganesha nfs-ganesha-xfs and the guide referenced above specifies: glusterfs-server glusterfs-api glusterfs-ganesha glusterfs-api is a dependency of one of RPMs that I installed so this is not a problem. But I cannot find any mention to install nfs-ganesha-xfs. I?ll try to setup the whole environment again without installing nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries). Again, thank you for you time to answer my previous message. Kind regards, Adam On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri <skoduri at redhat.com<mailto:skoduri at redhat.com>> wrote: Hi, On 05/02/2017 01:34 AM, Rudolf wrote: Hi Gluster users, First, I'd like to thank you all for this amazing open-source! Thank you! I'm working on home project ? three servers with Gluster and NFS-Ganesha. My goal is to create HA NFS share with three copies of each file on each server. My systems are CentOS 7.3 Minimal install with the latest updates and the most current RPMs from "centos-gluster310" repository. I followed this tutorial: http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/ (second half that describes multi-node HA setup) with a few exceptions: 1. All RPMs are from "centos-gluster310" repo that is installed by "yum -y install centos-release-gluster" 2. I have three nodes (not four) with "replica 3" volume. 3. I created empty ganesha.conf and not empty ganesha-ha.conf in "/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog post is outdated, this is now requirement) 4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't needed anymore. Please refer to http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/ It is being updated with latest changes happened wrt setup. When I finish configuration, all is good. nfs-ganesha.service is active and running and from client I can ping all three VIPs and I can mount NFS. Copied files are replicated to all nodes. But when I restart nodes (one by one, with 5 min. delay between) then I cannot ping or mount (I assume that all VIPs are down). So my setup definitely isn't HA. I found that: # pcs status Error: cluster is not currently running on this node This means pcsd service is not up. Did you enable (systemctl enable pcsd) pcsd service so that is comes up post reboot automatically. If not please start it manually. and nfs-ganesha.service is in inactive state. Btw. I didn't enable "systemctl enable nfs-ganesha" since I assume that this is something that Gluster does. Please check /var/log/ganesha.log for any errors/warnings. We recommend not to enable nfs-ganesha.service (by default), as the shared storage (where the ganesha.conf file resides now) should be up and running before nfs-ganesha gets started. So if enabled by default it could happen that shared_storage mount point is not yet up and it resulted in nfs-ganesha service failure. If you would like to address this, you could have a cron job which keeps checking the mount point health and then start nfs-ganesha service. Thanks, Soumya I assume that my issue is that I followed instructions in blog post from 2015/10 that are outdated. Unfortunately I cannot find anything better ? I spent whole day by googling. Would you be so kind and check the instructions in blog post and let me know what steps are wrong / outdated? Or please do you have more current instructions for Gluster+Ganesha setup? Thank you. Kind regards, Adam _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users -- Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170503/38680e43/attachment.html>
Soumya Koduri
2017-May-03 12:54 UTC
[Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot
On 05/03/2017 02:39 PM, Adam Ru wrote:> Hi Soumya, > > thank you very much for your reply. > > I enabled pcsd during setup and after reboot during troubleshooting I > manually started it and checked resources (pcs status). They were not > running. I didn?t find what was wrong but I?m going to try it again.Oh..forgot to mention did you have 'pacemaker' service enabled too (if not needs to be started) Thanks, Soumya> > I?ve thoroughly checked > http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/ > and I can confirm that I followed all steps with one exception. I > installed following RPMs: > glusterfs-server > glusterfs-fuse > glusterfs-cli > glusterfs-ganesha > nfs-ganesha-xfs > > and the guide referenced above specifies: > glusterfs-server > glusterfs-api > glusterfs-ganesha > > glusterfs-api is a dependency of one of RPMs that I installed so this is > not a problem. But I cannot find any mention to install nfs-ganesha-xfs. > > I?ll try to setup the whole environment again without installing > nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries). > > Again, thank you for you time to answer my previous message. > > Kind regards, > Adam > > On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri <skoduri at redhat.com > <mailto:skoduri at redhat.com>> wrote: > > Hi, > > On 05/02/2017 01:34 AM, Rudolf wrote: > > Hi Gluster users, > > First, I'd like to thank you all for this amazing open-source! > Thank you! > > I'm working on home project ? three servers with Gluster and > NFS-Ganesha. My goal is to create HA NFS share with three copies > of each > file on each server. > > My systems are CentOS 7.3 Minimal install with the latest > updates and > the most current RPMs from "centos-gluster310" repository. > > I followed this tutorial: > http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/ > <http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/> > (second half that describes multi-node HA setup) > > with a few exceptions: > > 1. All RPMs are from "centos-gluster310" repo that is installed > by "yum > -y install centos-release-gluster" > 2. I have three nodes (not four) with "replica 3" volume. > 3. I created empty ganesha.conf and not empty ganesha-ha.conf in > "/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog > post is > outdated, this is now requirement) > 4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't > needed > anymore. > > > Please refer to > http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/ > <http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/> > > It is being updated with latest changes happened wrt setup. > > When I finish configuration, all is good. nfs-ganesha.service is > active > and running and from client I can ping all three VIPs and I can > mount > NFS. Copied files are replicated to all nodes. > > But when I restart nodes (one by one, with 5 min. delay between) > then I > cannot ping or mount (I assume that all VIPs are down). So my setup > definitely isn't HA. > > I found that: > # pcs status > Error: cluster is not currently running on this node > > > This means pcsd service is not up. Did you enable (systemctl enable > pcsd) pcsd service so that is comes up post reboot automatically. If > not please start it manually. > > > and nfs-ganesha.service is in inactive state. Btw. I didn't enable > "systemctl enable nfs-ganesha" since I assume that this is something > that Gluster does. > > > Please check /var/log/ganesha.log for any errors/warnings. > > We recommend not to enable nfs-ganesha.service (by default), as the > shared storage (where the ganesha.conf file resides now) should be > up and running before nfs-ganesha gets started. > So if enabled by default it could happen that shared_storage mount > point is not yet up and it resulted in nfs-ganesha service failure. > If you would like to address this, you could have a cron job which > keeps checking the mount point health and then start nfs-ganesha > service. > > Thanks, > Soumya > > > I assume that my issue is that I followed instructions in blog > post from > 2015/10 that are outdated. Unfortunately I cannot find anything > better ? > I spent whole day by googling. > > Would you be so kind and check the instructions in blog post and > let me > know what steps are wrong / outdated? Or please do you have more > current > instructions for Gluster+Ganesha setup? > > Thank you. > > Kind regards, > Adam > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://lists.gluster.org/mailman/listinfo/gluster-users > <http://lists.gluster.org/mailman/listinfo/gluster-users> > > > > > -- > Adam
Soumya Koduri
2017-May-03 12:57 UTC
[Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot
On 05/03/2017 03:02 PM, Mahdi Adnan wrote:> Hi, > > > Same here, when i reboot the node i have to manually execute "pcs > cluster start gluster01" and pcsd already enabled and started. > > Gluster 3.8.11 > > Centos 7.3 latest > > Installed using CentOS Storage SIG repository > >Could you please check too if the pacemaker service is also in started state post reboot of the node. Thanks, Soumya> > -- > > Respectfully* > **Mahdi A. Mahdi* > > ------------------------------------------------------------------------ > *From:* gluster-users-bounces at gluster.org > <gluster-users-bounces at gluster.org> on behalf of Adam Ru > <ad.ruckel at gmail.com> > *Sent:* Wednesday, May 3, 2017 12:09:58 PM > *To:* Soumya Koduri > *Cc:* gluster-users at gluster.org > *Subject:* Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down > after reboot > > Hi Soumya, > > thank you very much for your reply. > > I enabled pcsd during setup and after reboot during troubleshooting I > manually started it and checked resources (pcs status). They were not > running. I didn?t find what was wrong but I?m going to try it again. > > I?ve thoroughly checked > http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/ > and I can confirm that I followed all steps with one exception. I > installed following RPMs: > glusterfs-server > glusterfs-fuse > glusterfs-cli > glusterfs-ganesha > nfs-ganesha-xfs > > and the guide referenced above specifies: > glusterfs-server > glusterfs-api > glusterfs-ganesha > > glusterfs-api is a dependency of one of RPMs that I installed so this is > not a problem. But I cannot find any mention to install nfs-ganesha-xfs. > > I?ll try to setup the whole environment again without installing > nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries). > > Again, thank you for you time to answer my previous message. > > Kind regards, > Adam > > On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri <skoduri at redhat.com > <mailto:skoduri at redhat.com>> wrote: > > Hi, > > On 05/02/2017 01:34 AM, Rudolf wrote: > > Hi Gluster users, > > First, I'd like to thank you all for this amazing open-source! > Thank you! > > I'm working on home project ? three servers with Gluster and > NFS-Ganesha. My goal is to create HA NFS share with three copies > of each > file on each server. > > My systems are CentOS 7.3 Minimal install with the latest > updates and > the most current RPMs from "centos-gluster310" repository. > > I followed this tutorial: > http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/ > <http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/> > (second half that describes multi-node HA setup) > > with a few exceptions: > > 1. All RPMs are from "centos-gluster310" repo that is installed > by "yum > -y install centos-release-gluster" > 2. I have three nodes (not four) with "replica 3" volume. > 3. I created empty ganesha.conf and not empty ganesha-ha.conf in > "/var/run/gluster/shared_storage/nfs-ganesha/" (referenced blog > post is > outdated, this is now requirement) > 4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't > needed > anymore. > > > Please refer to > http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/ > <http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/> > > It is being updated with latest changes happened wrt setup. > > When I finish configuration, all is good. nfs-ganesha.service is > active > and running and from client I can ping all three VIPs and I can > mount > NFS. Copied files are replicated to all nodes. > > But when I restart nodes (one by one, with 5 min. delay between) > then I > cannot ping or mount (I assume that all VIPs are down). So my setup > definitely isn't HA. > > I found that: > # pcs status > Error: cluster is not currently running on this node > > > This means pcsd service is not up. Did you enable (systemctl enable > pcsd) pcsd service so that is comes up post reboot automatically. If > not please start it manually. > > > and nfs-ganesha.service is in inactive state. Btw. I didn't enable > "systemctl enable nfs-ganesha" since I assume that this is something > that Gluster does. > > > Please check /var/log/ganesha.log for any errors/warnings. > > We recommend not to enable nfs-ganesha.service (by default), as the > shared storage (where the ganesha.conf file resides now) should be > up and running before nfs-ganesha gets started. > So if enabled by default it could happen that shared_storage mount > point is not yet up and it resulted in nfs-ganesha service failure. > If you would like to address this, you could have a cron job which > keeps checking the mount point health and then start nfs-ganesha > service. > > Thanks, > Soumya > > > I assume that my issue is that I followed instructions in blog > post from > 2015/10 that are outdated. Unfortunately I cannot find anything > better ? > I spent whole day by googling. > > Would you be so kind and check the instructions in blog post and > let me > know what steps are wrong / outdated? Or please do you have more > current > instructions for Gluster+Ganesha setup? > > Thank you. > > Kind regards, > Adam > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://lists.gluster.org/mailman/listinfo/gluster-users > <http://lists.gluster.org/mailman/listinfo/gluster-users> > > > > > -- > Adam