Gluster users - Jul 2017 - Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

On Mon, Jul 10, 2017 at 10:33 PM, Mahdi Adnan <mahdi.adnan at outlook.com>
wrote:
> I upgraded from 3.8.12 to 3.8.13 without issues.
>
> Two replicated volumes with online update, upgraded clients first and
> followed by servers upgrade, "stop glusterd, pkill gluster*, update
> gluster*, start glusterd, monitor healing process and logs, after
> completion proceed to the other node"
>
You should first upgrade servers and then clients. New servers can
understand old clients, but it is not easy for old servers to understand
new clients in case it started doing something new.

> check gluster logs for more information.
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> ------------------------------
> *From:* gluster-users-bounces at gluster.org <gluster-users-bounces@
> gluster.org> on behalf of Pavel Szalbot <pavel.szalbot at
gmail.com>
> *Sent:* Monday, July 10, 2017 7:52:10 PM
> *To:* gluster-users
> *Subject:* [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13)
> caused underlying VM fs corruption
>
> Hi,
>
> is there a recommended way to upgrade Gluster cluster when upgrading
> to newer revision? I experienced filesystem corruption on several but
> not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.
>
> After upgrading one of two nodes, I checked peer status and volume
> heal info, everything seemed fine so I upgraded second node and then
> two VMs remounted root as read-only and dmesg contained I/O errors.
>
> This did not happen in the past while following the same upgrade
> procedure. Documentation mentions only upgrades to higher minor
> version. Is there a recommended way to upgrade or did I do something
> wrong that should be avoided?
>
> Thanks for any suggestion
> Pavel Szalbot
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170711/25bc8001/attachment.html>

Noted, many thanks


--

Respectfully
Mahdi A. Mahdi

________________________________
From: Pranith Kumar Karampuri <pkarampu at redhat.com>
Sent: Tuesday, July 11, 2017 6:41:28 AM
To: Mahdi Adnan
Cc: Pavel Szalbot; gluster-users
Subject: Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13)
caused underlying VM fs corruption



On Mon, Jul 10, 2017 at 10:33 PM, Mahdi Adnan <mahdi.adnan at
outlook.com<mailto:mahdi.adnan at outlook.com>> wrote:

I upgraded from 3.8.12 to 3.8.13 without issues.

Two replicated volumes with online update, upgraded clients first and followed
by servers upgrade, "stop glusterd, pkill gluster*, update gluster*, start
glusterd, monitor healing process and logs, after completion proceed to the
other node"

You should first upgrade servers and then clients. New servers can understand
old clients, but it is not easy for old servers to understand new clients in
case it started doing something new.


check gluster logs for more information.


--

Respectfully
Mahdi A. Mahdi

________________________________
From: gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at
gluster.org> <gluster-users-bounces at
gluster.org<mailto:gluster-users-bounces at gluster.org>> on behalf of
Pavel Szalbot <pavel.szalbot at gmail.com<mailto:pavel.szalbot at
gmail.com>>
Sent: Monday, July 10, 2017 7:52:10 PM
To: gluster-users
Subject: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused
underlying VM fs corruption

Hi,

is there a recommended way to upgrade Gluster cluster when upgrading
to newer revision? I experienced filesystem corruption on several but
not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.

After upgrading one of two nodes, I checked peer status and volume
heal info, everything seemed fine so I upgraded second node and then
two VMs remounted root as read-only and dmesg contained I/O errors.

This did not happen in the past while following the same upgrade
procedure. Documentation mentions only upgrades to higher minor
version. Is there a recommended way to upgrade or did I do something
wrong that should be avoided?

Thanks for any suggestion
Pavel Szalbot
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users



--
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170711/72b4b09d/attachment.html>

>
> You should first upgrade servers and then clients. New servers can
> understand old clients, but it is not easy for old servers to understand
new
> clients in case it started doing something new.
But isn't that the reason op-version exists? So that regardless of
client/server mix, nobody tries to do "new" things above the current
op-version?

He is not changing mayor versions, just a small step from 3.8.12 to
3.8.13. Corruption should not be happening.

Diego

On Tue, Jul 11, 2017 at 5:12 PM, Diego Remolina <dijuremo at gmail.com>
wrote:
> >
> > You should first upgrade servers and then clients. New servers can
> > understand old clients, but it is not easy for old servers to
understand
> new
> > clients in case it started doing something new.
>
> But isn't that the reason op-version exists? So that regardless of
> client/server mix, nobody tries to do "new" things above the
current
> op-version?
>
> He is not changing mayor versions, just a small step from 3.8.12 to
> 3.8.13. Corruption should not be happening.
>
For some reason 3.8 upgrade guide is not where it is supposed to be.
We highly recommend upgrading servers ahead of clients.
https://github.com/nixpanic/glusterdocs/commit/f6d48dc17f2cb6ee4680e372520ec3358641b2bc

I think at some point it is better to make this mandatory. Without a
predefined way of upgrading, it is very difficult to fix bugs in backward
compatible manner.

I am not sure why the corruption happened either :-(. Pavel, could you give
log files may be?

> Diego
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170711/7dceb292/attachment.html>