Gluster users - Jul 2017 - Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

Hi,

is there a recommended way to upgrade Gluster cluster when upgrading
to newer revision? I experienced filesystem corruption on several but
not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.

After upgrading one of two nodes, I checked peer status and volume
heal info, everything seemed fine so I upgraded second node and then
two VMs remounted root as read-only and dmesg contained I/O errors.

This did not happen in the past while following the same upgrade
procedure. Documentation mentions only upgrades to higher minor
version. Is there a recommended way to upgrade or did I do something
wrong that should be avoided?

Thanks for any suggestion
Pavel Szalbot

I upgraded from 3.8.12 to 3.8.13 without issues.

Two replicated volumes with online update, upgraded clients first and followed
by servers upgrade, "stop glusterd, pkill gluster*, update gluster*, start
glusterd, monitor healing process and logs, after completion proceed to the
other node"

check gluster logs for more information.


--

Respectfully
Mahdi A. Mahdi

________________________________
From: gluster-users-bounces at gluster.org <gluster-users-bounces at
gluster.org> on behalf of Pavel Szalbot <pavel.szalbot at gmail.com>
Sent: Monday, July 10, 2017 7:52:10 PM
To: gluster-users
Subject: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused
underlying VM fs corruption

Hi,

is there a recommended way to upgrade Gluster cluster when upgrading
to newer revision? I experienced filesystem corruption on several but
not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.

After upgrading one of two nodes, I checked peer status and volume
heal info, everything seemed fine so I upgraded second node and then
two VMs remounted root as read-only and dmesg contained I/O errors.

This did not happen in the past while following the same upgrade
procedure. Documentation mentions only upgrades to higher minor
version. Is there a recommended way to upgrade or did I do something
wrong that should be avoided?

Thanks for any suggestion
Pavel Szalbot
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170710/afc4c78f/attachment.html>

On Mon, Jul 10, 2017 at 10:33 PM, Mahdi Adnan <mahdi.adnan at outlook.com>
wrote:
> I upgraded from 3.8.12 to 3.8.13 without issues.
>
> Two replicated volumes with online update, upgraded clients first and
> followed by servers upgrade, "stop glusterd, pkill gluster*, update
> gluster*, start glusterd, monitor healing process and logs, after
> completion proceed to the other node"
>
You should first upgrade servers and then clients. New servers can
understand old clients, but it is not easy for old servers to understand
new clients in case it started doing something new.

> check gluster logs for more information.
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> ------------------------------
> *From:* gluster-users-bounces at gluster.org <gluster-users-bounces@
> gluster.org> on behalf of Pavel Szalbot <pavel.szalbot at
gmail.com>
> *Sent:* Monday, July 10, 2017 7:52:10 PM
> *To:* gluster-users
> *Subject:* [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13)
> caused underlying VM fs corruption
>
> Hi,
>
> is there a recommended way to upgrade Gluster cluster when upgrading
> to newer revision? I experienced filesystem corruption on several but
> not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.
>
> After upgrading one of two nodes, I checked peer status and volume
> heal info, everything seemed fine so I upgraded second node and then
> two VMs remounted root as read-only and dmesg contained I/O errors.
>
> This did not happen in the past while following the same upgrade
> procedure. Documentation mentions only upgrades to higher minor
> version. Is there a recommended way to upgrade or did I do something
> wrong that should be avoided?
>
> Thanks for any suggestion
> Pavel Szalbot
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170711/25bc8001/attachment.html>