Krutika Dhananjay
2016-Apr-14 10:02 UTC
[Gluster-users] "gluster volume heal datastore2 info" blocking, heal not completing
Would you happen to know what those 6 entries that need heal correspond to? Assuming heal-info reported the status at least once without hanging. Also, could you share the contents of glfsheal-datastore.log, specifically from the machine where you ran the `heal info` command? -Krutika On Wed, Apr 13, 2016 at 8:13 PM, Lindsay Mathieson < lindsay.mathieson at gmail.com> wrote:> On 14/04/2016 12:19 AM, Krutika Dhananjay wrote: > >> Hmm what version of gluster was the hang seen on? >> > > > Ah yes, sorry - 3.7.9 > > The heal was triggered by a "killall glusterfsd" on node vna as a test, I > then restarted the glusterd service to bring it back up. > > Eventually I shutdown the VM's and stopped all gluster servcies, then > brought them back up. Eventually the last 4 shards healed, but it took > around an hour. > > Prior to that I had done rolling updates and restarts across all three > node, heals between reboots completed reasonably quickly with no i/o > freezes. > > I off to bed now I'm afraid. Rather late here :) > > -- > Lindsay Mathieson > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160414/eba804b8/attachment.html>
Krutika Dhananjay
2016-Apr-14 10:52 UTC
[Gluster-users] "gluster volume heal datastore2 info" blocking, heal not completing
OK here is one thing you could do if you are able to recreate this hang easily on your setup. The next time you find heal-info hanging, could you do the following: # gluster volume statedump <VOL> and share the statedump logs? You will find them at /var/run/gluster/ -Krutika On Thu, Apr 14, 2016 at 3:32 PM, Krutika Dhananjay <kdhananj at redhat.com> wrote:> Would you happen to know what those 6 entries that need heal correspond > to? Assuming heal-info reported the status at least once without hanging. > Also, could you share the contents of glfsheal-datastore.log, specifically > from the machine where you ran the `heal info` command? > > -Krutika > > On Wed, Apr 13, 2016 at 8:13 PM, Lindsay Mathieson < > lindsay.mathieson at gmail.com> wrote: > >> On 14/04/2016 12:19 AM, Krutika Dhananjay wrote: >> >>> Hmm what version of gluster was the hang seen on? >>> >> >> >> Ah yes, sorry - 3.7.9 >> >> The heal was triggered by a "killall glusterfsd" on node vna as a test, I >> then restarted the glusterd service to bring it back up. >> >> Eventually I shutdown the VM's and stopped all gluster servcies, then >> brought them back up. Eventually the last 4 shards healed, but it took >> around an hour. >> >> Prior to that I had done rolling updates and restarts across all three >> node, heals between reboots completed reasonably quickly with no i/o >> freezes. >> >> I off to bed now I'm afraid. Rather late here :) >> >> -- >> Lindsay Mathieson >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160414/c0dee206/attachment.html>