Gluster users - Oct 2016 - [URGENT] Add-bricks to a volume corrupted the files

On 19/10/2016 11:08 PM, Kevin Lemonnier wrote:
> Yes, to be honest I wasn't even planning on rebalancing just yet, I was
planning
> on letting it run a few days before, see if everything's fine, and then
maybe
> rebalance. I tried the rebalance when everything came crashing down, hoping
that
> might fix it, but it didn't.
I had a go at reproducing it last night Kevin with 3.8.4

- new volume

- Initial 2 bricks on two hosts

- copied a windows VM on to it

- Started some load (Crystal DiskMark in the VM)

- Added a 3rd brick and node



The 3rd bricks started healing right away and there were no issues. 
Eventually it was all healed and a normal rep 3 volume. Tonight I'lll 
compare the bricks with md5 but for now it seems ok.

- It was only one 40GB VM

- Not sure if the procedure is how it originally started for you.


If you'd like me to try a different process I can give it a goo, have 
the space and time and the cluster has been quite for weeks - time to 
break something :)


-- 
Lindsay Mathieson

> 
> I had a go at reproducing it last night Kevin with 3.8.4
> 
> - new volume
> 
> - Initial 2 bricks on two hosts
> 
> - copied a windows VM on to it
> 
> - Started some load (Crystal DiskMark in the VM)
> 
> - Added a 3rd brick and node
> 
For the size it's already more that what I have, most of our VMs are
20 Gb or less. They are almost all Linux webservers, the load on the disk
comes from MySQL almost exclusivly, but I guess load is load.
Still, it's not much, I think we currently have something like 15 Mb/s
average writing on the volume, so nothing crazy.

Did you add a third brick in replication, if I understand it correctly ?
That's not the problem I had, that does seem to be working fine.
What broke was adding 3 new bricks to a replica 3, bumping it from 1 x 3
to a 2 x 3. That doesn't start a heal, at least not until you start a
rebalance, but I didn't even get to that point :(

That's the only bug I've experienced so far in 3.7.12, everything else
(including increasing the replica count) seems to be working perfectly fine.
That's why I'm still installing that version, even though it's
outdated.

Thanks !
-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Digital signature
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20161019/4b628dd0/attachment.sig>