Josh Boon
2015-Oct-26 16:41 UTC
[Gluster-users] VM fs becomes read only when one gluster node goes down
Andre, I've not explored using a DNS solution to publish the gluster cluster addressing space but things you'll want to check out are network.ping-timeout and whether or not your VM goes read-only on filesystem error. If your network is consistent and robust tuning network.ping-timeout to a very low value such as three seconds will instruct the client to drop that client on failure. The default value for this is 42 seconds which will cause your VM to go read-only as you've seen. You could also choose to have your VM's mount their partitions errors=continue as well depending on the filesystem they run. Our setup has timeout at seven seconds and errors=continue and has survived both testing and storage node segfaults. No data integrity issues have presented yet but our data is mostly temporal so integrity hasn't been tested thoroughly. Also we're qemu 2.0 running gluster 3.6 on ubuntu 14.04 for those curious. Best, Josh From: "Roman" <romeo.r at gmail.com> To: "Krutika Dhananjay" <kdhananj at redhat.com> Cc: "gluster-users" <gluster-users at gluster.org>, gluster-devel at gluster.org Sent: Monday, October 26, 2015 1:33:57 PM Subject: Re: [Gluster-users] VM fs becomes read only when one gluster node goes down Hi, got backupvolfile-server=NODE2NAMEHERE in fstab ? :) 2015-10-23 5:24 GMT+03:00 Krutika Dhananjay < kdhananj at redhat.com > : Could you share the output of 'gluster volume info', and also information as to which node went down on reboot? -Krutika BQ_BEGIN From: "Andr? Bauer" < abauer at magix.net > To: "gluster-users" < gluster-users at gluster.org > Cc: gluster-devel at gluster.org Sent: Friday, October 23, 2015 12:15:04 AM Subject: [Gluster-users] VM fs becomes read only when one gluster node goes down Hi, i have a 4 node Glusterfs 3.5.6 Cluster. My VM images are in an replicated distributed volume which is accessed from kvm/qemu via libgfapi. Mount is against storage.domain.local which has IPs for all 4 Gluster nodes set in DNS. When one of the Gluster nodes goes down (accidently reboot) a lot of the vms getting read only filesystem. Even when the node comes back up. How can i prevent this? I expect that the vm just uses the replicated file on the other node, without getting ro fs. Any hints? Thanks in advance. -- Regards Andr? Bauer _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users BQ_END -- Best regards, Roman. _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151026/5f7155a0/attachment.html>
André Bauer
2015-Oct-26 19:08 UTC
[Gluster-users] VM fs becomes read only when one gluster node goes down
Thanks guys! My volume info is attached at the bottom of this mail... @ Josh As you can see, i already have a 5 second ping timeout set. I will try it with 3 seconds. Not sure, if i want to have errors=continue on the fs level but i will give it a try, if its the only possibility to get automatic failover work. @ Roman I use qemu with libgfapi to access the images. So no glusterfs entries in fstab for my vm hosts. It also seems this is kind of deprecated: http://blog.gluster.org/category/mount-glusterfs/ "`backupvolfile-server` - This option did not really do much rather than provide a 'shell' script based failover which was highly racy and wouldn't work during many occasions. It was necessary to remove this to make room for better options (while it is still provided for backward compatibility in the code)" @ all Can anybody tell me how Glusterfs handles this internaly? Is the libgfapi client already aware of the server which replicates the image? Is there a way i can configure it manualy for a volume? Volume Name: vmimages Type: Distributed-Replicate Volume ID: 029285b2-dfad-4569-8060-3827c0f1d856 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: storage1.domain.local:/glusterfs/vmimages Brick2: storage2.domain.local:/glusterfs/vmimages Brick3: storage3.domain.local:/glusterfs/vmimages Brick4: storage4.domain.local:/glusterfs/vmimages Options Reconfigured: network.ping-timeout: 5 performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server auth.allow: 192.168.0.21,192.168.0.22,192.168.0.23,192.168.0.24,192.168.0.25,192.168.0.26 server.allow-insecure: on storage.owner-uid: 2000 storage.owner-gid: 2000 Regards Andr? Am 26.10.2015 um 17:41 schrieb Josh Boon:> Andre, > > I've not explored using a DNS solution to publish the gluster cluster > addressing space but things you'll want to check out > are network.ping-timeout and whether or not your VM goes read-only on > filesystem error. If your network is consistent and robust > tuning network.ping-timeout to a very low value such as three seconds > will instruct the client to drop that client on failure. The default > value for this is 42 seconds which will cause your VM to go read-only as > you've seen. You could also choose to have your VM's mount their > partitions errors=continue as well depending on the filesystem they run. > Our setup has timeout at seven seconds and errors=continue and has > survived both testing and storage node segfaults. No data integrity > issues have presented yet but our data is mostly temporal so integrity > hasn't been tested thoroughly. Also we're qemu 2.0 running gluster 3.6 > on ubuntu 14.04 for those curious. > > Best, > Josh > > ------------------------------------------------------------------------ > *From: *"Roman" <romeo.r at gmail.com> > *To: *"Krutika Dhananjay" <kdhananj at redhat.com> > *Cc: *"gluster-users" <gluster-users at gluster.org>, gluster-devel at gluster.org > *Sent: *Monday, October 26, 2015 1:33:57 PM > *Subject: *Re: [Gluster-users] VM fs becomes read only when one gluster > node goes down > > Hi, > got backupvolfile-server=NODE2NAMEHERE in fstab ? :) > > 2015-10-23 5:24 GMT+03:00 Krutika Dhananjay <kdhananj at redhat.com > <mailto:kdhananj at redhat.com>>: > > Could you share the output of 'gluster volume info', and also > information as to which node went down on reboot? > > -Krutika > ------------------------------------------------------------------------ > > *From: *"Andr? Bauer" <abauer at magix.net <mailto:abauer at magix.net>> > *To: *"gluster-users" <gluster-users at gluster.org > <mailto:gluster-users at gluster.org>> > *Cc: *gluster-devel at gluster.org <mailto:gluster-devel at gluster.org> > *Sent: *Friday, October 23, 2015 12:15:04 AM > *Subject: *[Gluster-users] VM fs becomes read only when one > gluster node goes down > > Hi, > > i have a 4 node Glusterfs 3.5.6 Cluster. > > My VM images are in an replicated distributed volume which is > accessed > from kvm/qemu via libgfapi. > > Mount is against storage.domain.local which has IPs for all 4 > Gluster > nodes set in DNS. > > When one of the Gluster nodes goes down (accidently reboot) a > lot of the > vms getting read only filesystem. Even when the node comes back up. > > How can i prevent this? > I expect that the vm just uses the replicated file on the other > node, > without getting ro fs. > > Any hints? > > Thanks in advance. > > -- > Regards > Andr? Bauer > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > -- > Best regards, > Roman. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-- Mit freundlichen Gr??en Andr? Bauer MAGIX Software GmbH Andr? Bauer Administrator August-Bebel-Stra?e 48 01219 Dresden GERMANY tel.: 0351 41884875 e-mail: abauer at magix.net abauer at magix.net <mailto:Email> www.magix.com <http://www.magix.com/> Gesch?ftsf?hrer | Managing Directors: Dr. Arnd Schr?der, Klaus Schmidt Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205 Find us on: <http://www.facebook.com/MAGIX> <http://www.twitter.com/magix_de> <http://www.youtube.com/wwwmagixcom> <http://www.magixmagazin.de> ---------------------------------------------------------------------- The information in this email is intended only for the addressee named above. Access to this email by anyone else is unauthorized. If you are not the intended recipient of this message any disclosure, copying, distribution or any action taken in reliance on it is prohibited and may be unlawful. MAGIX does not warrant that any attachments are free from viruses or other defects and accepts no liability for any losses resulting from infected email transmissions. Please note that any views expressed in this email may be those of the originator and do not necessarily represent the agenda of the company. ----------------------------------------------------------------------