Gluster users - Aug 2015 - how to reboot all bricks safely and seamlessly

Hi,

If you need to reboot all bricks in a volume, what's the best way to do
this seamlessly?

I did this a few days ago by rebooting one, then waiting for "gluster
volume info" on another brick to show it back online before doing the
next, and so on. However, it went a bit wrong and I ended up with a
corruption. It fixed itself after a while, but because the system then
had a backlog of stuff to catch up with, it didn't fix itself for about
a day.

I wonder that I didn't leave enough time for the freshly rebooted brick
to apply all of the updates to itself that happened while it was
rebooting. So, what's the best way to find out whether a brick has fully
synced itself with the other bricks, before rebooting the next one?

I need to avoid having to shut the volume down as it would cause a
service outage.

Cheers,
Kingsley.

Well as you mentioned you might have rebooted the other node of the replica
pair when the self heal was in progress. AFR team can help you with details
if there is a way to detect whether heal is in progress or not.

-Atin
Sent from one plus one
On Aug 11, 2015 10:06 PM, "Kingsley" <gluster at
gluster.dogwind.com> wrote:
> Hi,
>
> If you need to reboot all bricks in a volume, what's the best way to do
> this seamlessly?
>
> I did this a few days ago by rebooting one, then waiting for "gluster
> volume info" on another brick to show it back online before doing the
> next, and so on. However, it went a bit wrong and I ended up with a
> corruption. It fixed itself after a while, but because the system then
> had a backlog of stuff to catch up with, it didn't fix itself for about
> a day.
>
> I wonder that I didn't leave enough time for the freshly rebooted brick
> to apply all of the updates to itself that happened while it was
> rebooting. So, what's the best way to find out whether a brick has
fully
> synced itself with the other bricks, before rebooting the next one?
>
> I need to avoid having to shut the volume down as it would cause a
> service outage.
>
> Cheers,
> Kingsley.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150811/2d29c770/attachment.html>

On 08/11/2015 10:06 PM, Kingsley wrote:
> Hi,
>
> If you need to reboot all bricks in a volume, what's the best way to do
> this seamlessly?
>
> I did this a few days ago by rebooting one, then waiting for "gluster
> volume info" on another brick to show it back online before doing the
> next, and so on. However, it went a bit wrong and I ended up with a
> corruption. It fixed itself after a while, but because the system then
> had a backlog of stuff to catch up with, it didn't fix itself for about
> a day.
>
> I wonder that I didn't leave enough time for the freshly rebooted brick
> to apply all of the updates to itself that happened while it was
> rebooting. So, what's the best way to find out whether a brick has
fully
> synced itself with the other bricks, before rebooting the next one?
'gluster volume heal <volname> info` must show zero entries. If it 
doesn't, you can manually launch self heal by 'gluster volume heal 
<volname>`

-Ravi
>
> I need to avoid having to shut the volume down as it would cause a
> service outage.
>
> Cheers,
> Kingsley.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users