Gluster users - Nov 2016 - 3.7.16 with sharding corrupts VMDK files when adding and removing bricks

> 
>    Having to create multiple cluster is not a solution and is much more
>    expansive.
>    And if you corrupt data from a single cluster you still have issues
> 
Sure, but thinking about it later we realised that it might be for the better.
I believe when sharding is enabled the shards will be dispersed across all the
replica sets, making it that losing a replica set will kill all your VMs.

Imagine a 16x3 volume for example, losing 2 bricks could bring the whole thing
down if they happen to be in the same replica set. (I might be wrong about the
way gluster disperse shards, it's my understanding only, never had the
chance
to test it).
With multiple small clusters, we have the same disk space in the end but not
that problem, it's a bit more annoying to manage but for now that's
allright.
> 
>    I'm also subscribed to moosefs and lizardfs mailing list and I
don't
>    recall any single data corruption/data loss event
> 
Never used those, might be just because there are less users ? Really have no
idea,
maybe you are right.
>    If you change the shard size on a populated cluster,A  you break all
>    existing data.
Not really shocked there. Guess the cli should warn you when you try re-setting
the option though, that would be nice.

-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Digital signature
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20161112/b94ce3c9/attachment.sig>

Il 12 nov 2016 12:53, "Kevin Lemonnier" <lemonnierk at
ulrar.net> ha scritto:
> Sure, but thinking about it later we realised that it might be for the
better.
> I believe when sharding is enabled the shards will be dispersed across
all the
> replica sets, making it that losing a replica set will kill all your VMs.
>
> Imagine a 16x3 volume for example, losing 2 bricks could bring the whole
thing
> down if they happen to be in the same replica set. (I might be wrong
about the
> way gluster disperse shards, it's my understanding only, never had the
chance
> to test it).
> With multiple small clusters, we have the same disk space in the end but
not
> that problem, it's a bit more annoying to manage but for now that's
allright.

I don't use EC because i really love the "gluster feature" to have
plain
files stored and not encoded in any way.
> Not really shocked there. Guess the cli should warn you when you try
re-setting
> the option though, that would be nice.
Exactly. I've proposed a warning in the cli when changing the shard size
but this is still unfixed and this is scaring me
it's a critical bug, IMHO, and should be addressed asap or any user could
destroy the whole cluster with a simple command and no warning at all.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20161112/2d2880f1/attachment.html>

> Sure, but thinking about it later we realised that it might be for the
better.
> I believe when sharding is enabled the shards will be dispersed across all
the
> replica sets, making it that losing a replica set will kill all your VMs.
>
> Imagine a 16x3 volume for example, losing 2 bricks could bring the whole
thing
> down if they happen to be in the same replica set. (I might be wrong about
the
> way gluster disperse shards, it's my understanding only, never had the
chance
> to test it).
> With multiple small clusters, we have the same disk space in the end but
not
> that problem, it's a bit more annoying to manage but for now that's
allright.
>
>>    I'm also subscribed to moosefs and lizardfs mailing list and I
don't
>>    recall any single data corruption/data loss event
>>
> Never used those, might be just because there are less users ? Really have
no idea,
> maybe you are right.
I can add to this. I've been using MooseFS for general file storage with
Samba for over a year now for >25 million files shared to 350+ users.

I've *never* lost even a single file. We had some issues with
permissions but that needed a couple of lines added to our smb.conf
(CTDB cluster).

On the other hand at home, I tried to use GlusterFS for VM images in a
simple replica 2 setup with Pacemaker for HA. VMs were constantly
failing en masse even without making any changes. Very often the images
got corrupted and had to be restored from backups. This was over a year
ago but motivated me to try the VMs on MooseFS. Since then I've not had
a single problem with unexpected downtime or corruption.

It's not the fastest FS in the world but it's well balanced and has a
focus on consistency and reliability, Documentation clearly explains
where all the chunks of your files will be so you can clearly define
your resilience and recovery strategies. IMHO GlusterFS would be a great
product if it tried to:

a) Add less features per release, and/or slowing down the release cycle.
Maybe have a "Feature" branch like RozoFS, with a separate Stable and
Testing/Current. Stable is safe, Testing is risky, and "Feature" is
for
those that need to try new, well, features.
b) Concentrate on issues like split-brain, healing, and scaling online
without data loss. Seems to be a common theme on the list where healing
doesn't work without tinkering. It should really "just work".
c) Have a peek at BeeGFS. It's a very well-performing FS that has its
focus on HPC. You can't stand to lose many thousands of CPU-hours of
work if your FS goes down, and it has to be fast.

The biggest question for me is what is the target market for GlusterFS?
Is it:

HPC (performance, reliability on the large scale, ie loss of one file is
OK, all not, no funky features)
VM storage (much the same as HPC but large file performance required,no
loss or corruption of blocks within a file)
General File (medium performance OK, small file and random access
paramount, resilience and consistency need to be 99.999%, features such
as ACLs and XATTRs, snapshots required)

i think if the documentation/wiki addressed these questions it would
make it easier for newcomers to evaluate the product.


>
>>    If you change the shard size on a populated cluster,A  you break all
>>    existing data.
>
This needs to be a warning or clearly documented. If you lose a couple
of PB of data in a professional role, I'd not fancy your employment
prospects. I've always had the feeling that GlusterFS is a bit of a
playground for new features and the only way to really have a stable
storage system is to stump up the cash to RedHat (and we've purchased a
lot of RHEL/RHEV licences), but having so many problems in the community
version really even puts me off buying the full package!

Cheers

Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).