Gluster users - May 2017 - Rebalance + VM corruption - current status and request for feedback

Hi,

In the past couple of weeks, we've sent the following fixes concerning VM
corruption upon doing rebalance -
https://review.gluster.org/#/q/status:merged+project:glusterfs+branch:master+topic:bug-1440051

These fixes are very much part of the latest 3.10.2 release.

Satheesaran within Red Hat also verified that they work and he's not seeing
corruption issues anymore.

I'd like to hear feedback from the users themselves on these fixes (on your
test environments to begin with) before even changing the status of the bug
to CLOSED.

Although 3.10.2 has a patch that prevents rebalance sub-commands from being
executed on sharded volumes, you can override the check by using the
'force' option.

For example,

# gluster volume rebalance myvol start force

Very much looking forward to hearing from you all.

Thanks,
Krutika
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170517/d39f880d/attachment.html>

Thank you so much mate.

I'll finish the test tomorrow and let you know the results.

--

Respectfully
Mahdi A. Mahdi

________________________________
From: Krutika Dhananjay <kdhananj at redhat.com>
Sent: Wednesday, May 17, 2017 6:59:20 AM
To: gluster-user
Cc: Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier; Mahdi Adnan
Subject: Rebalance + VM corruption - current status and request for feedback

Hi,

In the past couple of weeks, we've sent the following fixes concerning VM
corruption upon doing rebalance -
https://review.gluster.org/#/q/status:merged+project:glusterfs+branch:master+topic:bug-1440051

These fixes are very much part of the latest 3.10.2 release.

Satheesaran within Red Hat also verified that they work and he's not seeing
corruption issues anymore.

I'd like to hear feedback from the users themselves on these fixes (on your
test environments to begin with) before even changing the status of the bug to
CLOSED.

Although 3.10.2 has a patch that prevents rebalance sub-commands from being
executed on sharded volumes, you can override the check by using the
'force' option.

For example,

# gluster volume rebalance myvol start force

Very much looking forward to hearing from you all.

Thanks,
Krutika
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170519/a7d690fa/attachment.html>

Good morning,


So i have tested the new Gluster 3.10.2, and after starting rebalance two VMs
were paused due to storage error and third one was not responding.

After rebalance completed i started the VMs and it did not boot, and throw an
XFS wrong inode error into the screen.


My setup:

4 nodes running CentOS7.3 with Gluster 3.10.2

4 bricks in distributed replica with group set to virt.

I added the volume to ovirt and created three VMs, i ran a loop to create 5GB
file inside the VMs.

Added new 4 bricks to the existing nodes.

Started rebalane "with force to bypass the warning message"

VMs started to fail after rebalancing.



--

Respectfully
Mahdi A. Mahdi

________________________________
From: Krutika Dhananjay <kdhananj at redhat.com>
Sent: Wednesday, May 17, 2017 6:59:20 AM
To: gluster-user
Cc: Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier; Mahdi Adnan
Subject: Rebalance + VM corruption - current status and request for feedback

Hi,

In the past couple of weeks, we've sent the following fixes concerning VM
corruption upon doing rebalance -
https://review.gluster.org/#/q/status:merged+project:glusterfs+branch:master+topic:bug-1440051

These fixes are very much part of the latest 3.10.2 release.

Satheesaran within Red Hat also verified that they work and he's not seeing
corruption issues anymore.

I'd like to hear feedback from the users themselves on these fixes (on your
test environments to begin with) before even changing the status of the bug to
CLOSED.

Although 3.10.2 has a patch that prevents rebalance sub-commands from being
executed on sharded volumes, you can override the check by using the
'force' option.

For example,

# gluster volume rebalance myvol start force

Very much looking forward to hearing from you all.

Thanks,
Krutika
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170526/6e2cc25c/attachment.html>

Could you provide the rebalance and mount logs?

-Krutika

On Fri, May 26, 2017 at 3:17 PM, Mahdi Adnan <mahdi.adnan at outlook.com>
wrote:
> Good morning,
>
>
> So i have tested the new Gluster 3.10.2, and after starting rebalance two
> VMs were paused due to storage error and third one was not responding.
>
> After rebalance completed i started the VMs and it did not boot, and throw
> an XFS wrong inode error into the screen.
>
>
> My setup:
>
> 4 nodes running CentOS7.3 with Gluster 3.10.2
>
> 4 bricks in distributed replica with group set to virt.
>
> I added the volume to ovirt and created three VMs, i ran a loop to create
> 5GB file inside the VMs.
>
> Added new 4 bricks to the existing nodes.
>
> Started rebalane "with force to bypass the warning message"
>
> VMs started to fail after rebalancing.
>
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> ------------------------------
> *From:* Krutika Dhananjay <kdhananj at redhat.com>
> *Sent:* Wednesday, May 17, 2017 6:59:20 AM
> *To:* gluster-user
> *Cc:* Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier; Mahdi
> Adnan
> *Subject:* Rebalance + VM corruption - current status and request for
> feedback
>
> Hi,
>
> In the past couple of weeks, we've sent the following fixes concerning
VM
> corruption upon doing rebalance - https://review.gluster.org/#/
> q/status:merged+project:glusterfs+branch:master+topic:bug-1440051
>
> These fixes are very much part of the latest 3.10.2 release.
>
> Satheesaran within Red Hat also verified that they work and he's not
> seeing corruption issues anymore.
>
> I'd like to hear feedback from the users themselves on these fixes (on
> your test environments to begin with) before even changing the status of
> the bug to CLOSED.
>
> Although 3.10.2 has a patch that prevents rebalance sub-commands from
> being executed on sharded volumes, you can override the check by using the
> 'force' option.
>
> For example,
>
> # gluster volume rebalance myvol start force
>
> Very much looking forward to hearing from you all.
>
> Thanks,
> Krutika
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170526/f4169738/attachment.html>