Gluster users - Aug 2015 - how to reboot all bricks safely and seamlessly

On Wed, 2015-08-12 at 17:09 +0530, Ravishankar N wrote:
> 
> On 08/11/2015 10:06 PM, Kingsley wrote:
> > Hi,
> >
> > If you need to reboot all bricks in a volume, what's the best way
to do
> > this seamlessly?
> >
> > I did this a few days ago by rebooting one, then waiting for
"gluster
> > volume info" on another brick to show it back online before doing
the
> > next, and so on. However, it went a bit wrong and I ended up with a
> > corruption. It fixed itself after a while, but because the system then
> > had a backlog of stuff to catch up with, it didn't fix itself for
about
> > a day.
> >
> > I wonder that I didn't leave enough time for the freshly rebooted
brick
> > to apply all of the updates to itself that happened while it was
> > rebooting. So, what's the best way to find out whether a brick has
fully
> > synced itself with the other bricks, before rebooting the next one?
> 
> 'gluster volume heal <volname> info` must show zero entries. If
it
> doesn't, you can manually launch self heal by 'gluster volume heal 
> <volname>`
Thanks for that, Ravi, that's very useful to know. I did try the volume
heal, but I couldn't see how to figure out how far it had got in
percentage terms. Is there a command that I can issue that gives an
indication of how far through the heal process has got, eg "healed 370
or 1022 objects"?

-- 
Cheers,
Kingsley.

On 08/12/2015 05:20 PM, Kingsley wrote:
> On Wed, 2015-08-12 at 17:09 +0530, Ravishankar N wrote:
>> On 08/11/2015 10:06 PM, Kingsley wrote:
>>> Hi,
>>>
>>> If you need to reboot all bricks in a volume, what's the best
way to do
>>> this seamlessly?
>>>
>>> I did this a few days ago by rebooting one, then waiting for
"gluster
>>> volume info" on another brick to show it back online before
doing the
>>> next, and so on. However, it went a bit wrong and I ended up with a
>>> corruption. It fixed itself after a while, but because the system
then
>>> had a backlog of stuff to catch up with, it didn't fix itself
for about
>>> a day.
>>>
>>> I wonder that I didn't leave enough time for the freshly
rebooted brick
>>> to apply all of the updates to itself that happened while it was
>>> rebooting. So, what's the best way to find out whether a brick
has fully
>>> synced itself with the other bricks, before rebooting the next one?
>> 'gluster volume heal <volname> info` must show zero entries.
If it
>> doesn't, you can manually launch self heal by 'gluster volume
heal
>> <volname>`
> Thanks for that, Ravi, that's very useful to know. I did try the volume
> heal, but I couldn't see how to figure out how far it had got in
> percentage terms. Is there a command that I can issue that gives an
> indication of how far through the heal process has got, eg "healed 370
> or 1022 objects"?
The statistics option for the heal command should give you the number of 
files healed per crawl. You can refer to 
https://github.com/gluster/glusterfs/blob/release-3.7/doc/features/afr-statistics.md
for an explanation. We do not have an indication in terms of percentage, 
but 'gluster volume heal <volname> info` always lists the files that 
need healing. If that is non-zero, it means self-heal is not yet complete.

Hope this helps.
-Ravi