I am running gluster as a storage backend for VM storage (KVM guests). If one of the bricks is taken offline (even for an instant), on bringing it back up it runs the metadata check. This causes the guest to both stop responding until the check finishes and also to ruin data that was in process (sql data for instance). I'm guessing the file is being locked while checked. Is there any way to fix for this? Without being able to fix for this, I'm not certain how viable gluster will be, or can be for VM storage. Justice London jlondon at lawinfo.com
Justice,>From posts from the community on this user list, I knowthat there are folks that run hundreds of VMs out of gluster. So it's probably more about the data usage than just a generic viability statement as you made in your post. Gluster does not support databases, though many people use them on gluster without much problem. Please let me know if you see some problem with unstructured file data on VMs. I would be happy to help debug that problem. Regards, Tejas. ----- Original Message ----- From: "Justice London" <jlondon at lawinfo.com> To: gluster-users at gluster.org Sent: Friday, April 16, 2010 12:52:19 AM Subject: [Gluster-users] Self heal with VM Storage I am running gluster as a storage backend for VM storage (KVM guests). If one of the bricks is taken offline (even for an instant), on bringing it back up it runs the metadata check. This causes the guest to both stop responding until the check finishes and also to ruin data that was in process (sql data for instance). I'm guessing the file is being locked while checked. Is there any way to fix for this? Without being able to fix for this, I'm not certain how viable gluster will be, or can be for VM storage. Justice London jlondon at lawinfo.com _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Thanks, that will help reproduce internally. Regards, Tejas. ----- Original Message ----- From: "Justice London" <jlondon at lawinfo.com> To: "Tejas N. Bhise" <tejas at gluster.com> Cc: gluster-users at gluster.org Sent: Friday, April 16, 2010 9:03:50 PM Subject: RE: [Gluster-users] Self heal with VM Storage After the self heal finishes it sort of works. Usually this destroys InnoDB if you're running a database. Most often, though, it also causes some libraries and similar to not be properly read in by the VM guest which means you have to reboot it to fix for this. It should be fairly easy to reproduce... just shut down a storage brick (any configuration... it doesn't seem to matter). Make sure of course that you have a running VM guest (KVM, etc) using the gluster mount. You'll then turn off(unplug, etc.) one of the storage bricks and wait a few minutes... then re-enable it. Justice London jlondon at lawinfo.com -----Original Message----- From: Tejas N. Bhise [mailto:tejas at gluster.com] Sent: Thursday, April 15, 2010 7:41 PM To: Justice London Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Self heal with VM Storage Justice, Thanks for the description. So, does this mean that after the self heal is over after some time, the guest starts to work fine ? We will reproduce this inhouse and get back. Regards, Tejas. ----- Original Message ----- From: "Justice London" <jlondon at lawinfo.com> To: "Tejas N. Bhise" <tejas at gluster.com> Cc: gluster-users at gluster.org Sent: Friday, April 16, 2010 1:18:36 AM Subject: RE: [Gluster-users] Self heal with VM Storage Okay, but what happens on a brick shutting down and being added back to the cluster? This would be after some live data has been written to the other bricks.>From what I was seeing access to the file is locked. Is this not the case?If file access is being locked it will obviously cause issues for anything trying to read/write to the guest at the time. Justice London jlondon at lawinfo.com -----Original Message----- From: Tejas N. Bhise [mailto:tejas at gluster.com] Sent: Thursday, April 15, 2010 12:33 PM To: Justice London Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Self heal with VM Storage Justice,>From posts from the community on this user list, I knowthat there are folks that run hundreds of VMs out of gluster. So it's probably more about the data usage than just a generic viability statement as you made in your post. Gluster does not support databases, though many people use them on gluster without much problem. Please let me know if you see some problem with unstructured file data on VMs. I would be happy to help debug that problem. Regards, Tejas. ----- Original Message ----- From: "Justice London" <jlondon at lawinfo.com> To: gluster-users at gluster.org Sent: Friday, April 16, 2010 12:52:19 AM Subject: [Gluster-users] Self heal with VM Storage I am running gluster as a storage backend for VM storage (KVM guests). If one of the bricks is taken offline (even for an instant), on bringing it back up it runs the metadata check. This causes the guest to both stop responding until the check finishes and also to ruin data that was in process (sql data for instance). I'm guessing the file is being locked while checked. Is there any way to fix for this? Without being able to fix for this, I'm not certain how viable gluster will be, or can be for VM storage. Justice London jlondon at lawinfo.com _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Hi, I've logged a bug to fix this issue and can be tracked at: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=831 The fix should be available for the upcoming 3.1 release. Pavan ----- Original Message ----- From: "Tejas N. Bhise" <tejas at gluster.com> To: "Justice London" <jlondon at lawinfo.com> Cc: gluster-users at gluster.org Sent: Saturday, April 17, 2010 8:57:13 AM Subject: Re: [Gluster-users] Self heal with VM Storage Thanks, that will help reproduce internally. Regards, Tejas. ----- Original Message ----- From: "Justice London" <jlondon at lawinfo.com> To: "Tejas N. Bhise" <tejas at gluster.com> Cc: gluster-users at gluster.org Sent: Friday, April 16, 2010 9:03:50 PM Subject: RE: [Gluster-users] Self heal with VM Storage After the self heal finishes it sort of works. Usually this destroys InnoDB if you're running a database. Most often, though, it also causes some libraries and similar to not be properly read in by the VM guest which means you have to reboot it to fix for this. It should be fairly easy to reproduce... just shut down a storage brick (any configuration... it doesn't seem to matter). Make sure of course that you have a running VM guest (KVM, etc) using the gluster mount. You'll then turn off(unplug, etc.) one of the storage bricks and wait a few minutes... then re-enable it. Justice London jlondon at lawinfo.com -----Original Message----- From: Tejas N. Bhise [mailto:tejas at gluster.com] Sent: Thursday, April 15, 2010 7:41 PM To: Justice London Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Self heal with VM Storage Justice, Thanks for the description. So, does this mean that after the self heal is over after some time, the guest starts to work fine ? We will reproduce this inhouse and get back. Regards, Tejas. ----- Original Message ----- From: "Justice London" <jlondon at lawinfo.com> To: "Tejas N. Bhise" <tejas at gluster.com> Cc: gluster-users at gluster.org Sent: Friday, April 16, 2010 1:18:36 AM Subject: RE: [Gluster-users] Self heal with VM Storage Okay, but what happens on a brick shutting down and being added back to the cluster? This would be after some live data has been written to the other bricks.>From what I was seeing access to the file is locked. Is this not the case?If file access is being locked it will obviously cause issues for anything trying to read/write to the guest at the time. Justice London jlondon at lawinfo.com -----Original Message----- From: Tejas N. Bhise [mailto:tejas at gluster.com] Sent: Thursday, April 15, 2010 12:33 PM To: Justice London Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] Self heal with VM Storage Justice,>From posts from the community on this user list, I knowthat there are folks that run hundreds of VMs out of gluster. So it's probably more about the data usage than just a generic viability statement as you made in your post. Gluster does not support databases, though many people use them on gluster without much problem. Please let me know if you see some problem with unstructured file data on VMs. I would be happy to help debug that problem. Regards, Tejas. ----- Original Message ----- From: "Justice London" <jlondon at lawinfo.com> To: gluster-users at gluster.org Sent: Friday, April 16, 2010 12:52:19 AM Subject: [Gluster-users] Self heal with VM Storage I am running gluster as a storage backend for VM storage (KVM guests). If one of the bricks is taken offline (even for an instant), on bringing it back up it runs the metadata check. This causes the guest to both stop responding until the check finishes and also to ruin data that was in process (sql data for instance). I'm guessing the file is being locked while checked. Is there any way to fix for this? Without being able to fix for this, I'm not certain how viable gluster will be, or can be for VM storage. Justice London jlondon at lawinfo.com _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users