Gandalf Corvotempesta
2017-Jun-29 17:41 UTC
[Gluster-users] How to shutdown a node properly ?
Init.d/system.d script doesn't kill gluster automatically on reboot/shutdown? Il 29 giu 2017 5:16 PM, "Ravishankar N" <ravishankar at redhat.com> ha scritto:> On 06/29/2017 08:31 PM, Renaud Fortier wrote: > > Hi, > > Everytime I shutdown a node, I lost access (from clients) to the volumes > for 42 seconds (network.ping-timeout). Is there a special way to shutdown a > node to keep the access to the volumes without interruption ? Currently, I > use the ?shutdown? or ?reboot? command. > > `killall glusterfs glusterfsd glusterd` before issuing shutdown or > reboot. If it is a replica or EC volume, ensure that there are no pending > heals before bringing down a node. i.e. `gluster volume heal volname info` > should show 0 entries. > > > > My setup is : > > -4 gluster 3.10.3 nodes on debian 8 (jessie) > > -3 volumes Distributed-Replicate 2 X 2 = 4 > > > > Thank you > > Renaud > > > _______________________________________________ > Gluster-users mailing listGluster-users at gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170629/e04baa00/attachment.html>
On Thu, Jun 29, 2017 at 12:41 PM, Gandalf Corvotempesta < gandalf.corvotempesta at gmail.com> wrote:> Init.d/system.d script doesn't kill gluster automatically on > reboot/shutdown? > > Sounds less like an issue with how it's shutdown but an issue with howit's mounted perhaps. My gluster fuse mounts seem to handle any one node being shutdown just fine as long as quorum is maintained. Il 29 giu 2017 5:16 PM, "Ravishankar N" <ravishankar at redhat.com> ha scritto:> >> On 06/29/2017 08:31 PM, Renaud Fortier wrote: >> >> Hi, >> >> Everytime I shutdown a node, I lost access (from clients) to the volumes >> for 42 seconds (network.ping-timeout). Is there a special way to shutdown a >> node to keep the access to the volumes without interruption ? Currently, I >> use the ?shutdown? or ?reboot? command. >> >> `killall glusterfs glusterfsd glusterd` before issuing shutdown or >> reboot. If it is a replica or EC volume, ensure that there are no pending >> heals before bringing down a node. i.e. `gluster volume heal volname info` >> should show 0 entries. >> >> >> >> My setup is : >> >> -4 gluster 3.10.3 nodes on debian 8 (jessie) >> >> -3 volumes Distributed-Replicate 2 X 2 = 4 >> >> >> >> Thank you >> >> Renaud >> >> >> _______________________________________________ >> Gluster-users mailing listGluster-users at gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170629/ac687511/attachment.html>
On my nodes, when i use the system.d script to kill gluster (service glusterfs-server stop) only glusterd is killed. Then I guess the shutdown doesn?t kill everything ! De : Gandalf Corvotempesta [mailto:gandalf.corvotempesta at gmail.com] Envoy? : 29 juin 2017 13:41 ? : Ravishankar N <ravishankar at redhat.com> Cc : gluster-users at gluster.org; Renaud Fortier <Renaud.Fortier at fsaa.ulaval.ca> Objet : Re: [Gluster-users] How to shutdown a node properly ? Init.d/system.d script doesn't kill gluster automatically on reboot/shutdown? Il 29 giu 2017 5:16 PM, "Ravishankar N" <ravishankar at redhat.com<mailto:ravishankar at redhat.com>> ha scritto: On 06/29/2017 08:31 PM, Renaud Fortier wrote: Hi, Everytime I shutdown a node, I lost access (from clients) to the volumes for 42 seconds (network.ping-timeout). Is there a special way to shutdown a node to keep the access to the volumes without interruption ? Currently, I use the ?shutdown? or ?reboot? command. `killall glusterfs glusterfsd glusterd` before issuing shutdown or reboot. If it is a replica or EC volume, ensure that there are no pending heals before bringing down a node. i.e. `gluster volume heal volname info` should show 0 entries. My setup is : -4 gluster 3.10.3 nodes on debian 8 (jessie) -3 volumes Distributed-Replicate 2 X 2 = 4 Thank you Renaud _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170629/3dd0349f/attachment.html>
This is my fstab : GLUSTER-NODE1:/volume-test /test glusterfs defaults,direct-io-mode=disable,_netdev,backup-volfile-servers=GLUSTER-NODE2:GLUSTER-NODE3:GLUSTER-NODE4 0 0 Do you see something wrong ? De : gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] De la part de David Gossage Envoy? : 29 juin 2017 14:26 ? : Gandalf Corvotempesta <gandalf.corvotempesta at gmail.com> Cc : gluster-users at gluster.org Objet : Re: [Gluster-users] How to shutdown a node properly ? On Thu, Jun 29, 2017 at 12:41 PM, Gandalf Corvotempesta <gandalf.corvotempesta at gmail.com<mailto:gandalf.corvotempesta at gmail.com>> wrote: Init.d/system.d script doesn't kill gluster automatically on reboot/shutdown? Sounds less like an issue with how it's shutdown but an issue with how it's mounted perhaps. My gluster fuse mounts seem to handle any one node being shutdown just fine as long as quorum is maintained. Il 29 giu 2017 5:16 PM, "Ravishankar N" <ravishankar at redhat.com<mailto:ravishankar at redhat.com>> ha scritto: On 06/29/2017 08:31 PM, Renaud Fortier wrote: Hi, Everytime I shutdown a node, I lost access (from clients) to the volumes for 42 seconds (network.ping-timeout). Is there a special way to shutdown a node to keep the access to the volumes without interruption ? Currently, I use the ?shutdown? or ?reboot? command. `killall glusterfs glusterfsd glusterd` before issuing shutdown or reboot. If it is a replica or EC volume, ensure that there are no pending heals before bringing down a node. i.e. `gluster volume heal volname info` should show 0 entries. My setup is : -4 gluster 3.10.3 nodes on debian 8 (jessie) -3 volumes Distributed-Replicate 2 X 2 = 4 Thank you Renaud _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170629/1c8ea081/attachment.html>
On 06/30/2017 12:40 AM, Renaud Fortier wrote:> > On my nodes, when i use the system.d script to kill gluster (service > glusterfs-server stop) only glusterd is killed. Then I guess the > shutdown doesn?t kill everything ! >Killing glusterd does not kill other gluster processes. When you shutdown a node, everything obviously gets killed but the client does not get notified immediately that the brick went down, leading for it to wait for the 42 second ping-timeout after which it assumes the brick is down. When you kill the brick manually before shutdown, the client immediate receives the notification and you don't see the hang. See Xavi's description in Bug 1054694. So if it is a planned shutdown or reboot, it is better to kill the gluster processes before shutting the node down. BTW, you can use https://github.com/gluster/glusterfs/blob/master/extras/stop-all-gluster-processes.sh which automatically checks for pending heals etc before killing the gluster processes. -Ravi> *De :*Gandalf Corvotempesta [mailto:gandalf.corvotempesta at gmail.com] > *Envoy? :* 29 juin 2017 13:41 > *? :* Ravishankar N <ravishankar at redhat.com> > *Cc :* gluster-users at gluster.org; Renaud Fortier > <Renaud.Fortier at fsaa.ulaval.ca> > *Objet :* Re: [Gluster-users] How to shutdown a node properly ? > > Init.d/system.d script doesn't kill gluster automatically on > reboot/shutdown? > > Il 29 giu 2017 5:16 PM, "Ravishankar N" <ravishankar at redhat.com > <mailto:ravishankar at redhat.com>> ha scritto: > > On 06/29/2017 08:31 PM, Renaud Fortier wrote: > > Hi, > > Everytime I shutdown a node, I lost access (from clients) to > the volumes for 42 seconds (network.ping-timeout). Is there a > special way to shutdown a node to keep the access to the > volumes without interruption ? Currently, I use the ?shutdown? > or ?reboot? command. > > `killall glusterfs glusterfsd glusterd` before issuing shutdown or > reboot. If it is a replica or EC volume, ensure that there are no > pending heals before bringing down a node. i.e. `gluster volume > heal volname info` should show 0 entries. > > > My setup is : > > -4 gluster 3.10.3 nodes on debian 8 (jessie) > > -3 volumes Distributed-Replicate 2 X 2 = 4 > > Thank you > > Renaud > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170630/8e9ea0c8/attachment.html>