Gluster users - Aug 2015 - how to reboot all bricks safely and seamlessly

On 08/11/2015 10:06 PM, Kingsley wrote:
> Hi,
>
> If you need to reboot all bricks in a volume, what's the best way to do
> this seamlessly?
>
> I did this a few days ago by rebooting one, then waiting for "gluster
> volume info" on another brick to show it back online before doing the
> next, and so on. However, it went a bit wrong and I ended up with a
> corruption. It fixed itself after a while, but because the system then
> had a backlog of stuff to catch up with, it didn't fix itself for about
> a day.
>
> I wonder that I didn't leave enough time for the freshly rebooted brick
> to apply all of the updates to itself that happened while it was
> rebooting. So, what's the best way to find out whether a brick has
fully
> synced itself with the other bricks, before rebooting the next one?
'gluster volume heal <volname> info` must show zero entries. If it 
doesn't, you can manually launch self heal by 'gluster volume heal 
<volname>`

-Ravi
>
> I need to avoid having to shut the volume down as it would cause a
> service outage.
>
> Cheers,
> Kingsley.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

On Wed, 2015-08-12 at 17:09 +0530, Ravishankar N wrote:
> 
> On 08/11/2015 10:06 PM, Kingsley wrote:
> > Hi,
> >
> > If you need to reboot all bricks in a volume, what's the best way
to do
> > this seamlessly?
> >
> > I did this a few days ago by rebooting one, then waiting for
"gluster
> > volume info" on another brick to show it back online before doing
the
> > next, and so on. However, it went a bit wrong and I ended up with a
> > corruption. It fixed itself after a while, but because the system then
> > had a backlog of stuff to catch up with, it didn't fix itself for
about
> > a day.
> >
> > I wonder that I didn't leave enough time for the freshly rebooted
brick
> > to apply all of the updates to itself that happened while it was
> > rebooting. So, what's the best way to find out whether a brick has
fully
> > synced itself with the other bricks, before rebooting the next one?
> 
> 'gluster volume heal <volname> info` must show zero entries. If
it
> doesn't, you can manually launch self heal by 'gluster volume heal 
> <volname>`
Thanks for that, Ravi, that's very useful to know. I did try the volume
heal, but I couldn't see how to figure out how far it had got in
percentage terms. Is there a command that I can issue that gives an
indication of how far through the heal process has got, eg "healed 370
or 1022 objects"?

-- 
Cheers,
Kingsley.