Gluster users - Feb 2012 - Exorbitant cost to achieve redundancy??

I'm trying to justify a GlusterFS storage system for my technology
development group and I want to get some clarification on
something that I can't seem to figure out architecture wise...

My storage system will be rather large. Significant fraction of a
petabyte and will require scaling in size for at least one decade.

from what I understand GlusterFS achieves redundancy through
replication. And from the documentation: Section 5.5 Creating
Distributed Replicated Volumes the note says "The number of bricks
should be a multiple of the replica count for a distributed replicated 
volume."

Is this telling me that if I want to be able to suffer 2 bricks failing
that I have to deploy three bricks at a time and the amount of space
I wind up with available is essentially equal to only that provided
by a single brick?

In other words... GlusterFS TRIPLES all my storage costs to provide
2 brick fault tolerance?

How do I get redundancy in GlusterFS while getting reasonable
storage costs where I am not wasting 50% of my investment or
more in providing copies to obtain redundancy?

Thank you.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120213/c7c9ac1c/attachment.html>

You don't have to leave all your redundancy to Gluster. You can put Gluster
on two (or more) systems which are each running RAID5, for instance. Then it
would take a minimum of 4 drives failing (2 on each array) before Gluster
should lose any data. Each system would require N+1 drives, so double your
drives plus two. (There are reasons to consider RAID other than 5, but I'll
leave that discussion out for now.)

As for "reasonable storage costs," have you priced the alternatives? 

Best,
Whit

On Mon, Feb 13, 2012 at 04:15:16PM -0800, Jeff Wiegley
wrote:
> I'm trying to justify a GlusterFS storage system for my technology
> development group and I want to get some clarification on
> something that I can't seem to figure out architecture wise...
> 
> My storage system will be rather large. Significant fraction of a
> petabyte and will require scaling in size for at least one decade.
> 
> from what I understand GlusterFS achieves redundancy through
> replication. And from the documentation: Section 5.5 Creating
> Distributed Replicated Volumes the note says "The number of bricks
> should be a multiple of the replica count for a distributed replicated
volume."
> 
> Is this telling me that if I want to be able to suffer 2 bricks failing
> that I have to deploy three bricks at a time and the amount of space
> I wind up with available is essentially equal to only that provided
> by a single brick?
> 
> In other words... GlusterFS TRIPLES all my storage costs to provide
> 2 brick fault tolerance?
> 
> How do I get redundancy in GlusterFS while getting reasonable
> storage costs where I am not wasting 50% of my investment or
> more in providing copies to obtain redundancy?
> 
> Thank you.
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

On Mon, 13 Feb 2012, Jeff Wiegley wrote:
> I'm trying to justify a GlusterFS storage system for my technology
> development group and I want to get some clarification on
> something that I can't seem to figure out architecture wise...
>
> My storage system will be rather large. Significant fraction of a
> petabyte and will require scaling in size for at least one decade.
>
> from what I understand GlusterFS achieves redundancy through
> replication. And from the documentation: Section 5.5 Creating
> Distributed Replicated Volumes the note says "The number of bricks
> should be a multiple of the replica count for a distributed replicated 
> volume."
>
> Is this telling me that if I want to be able to suffer 2 bricks failing
> that I have to deploy three bricks at a time and the amount of space
> I wind up with available is essentially equal to only that provided
> by a single brick?
>
> In other words... GlusterFS TRIPLES all my storage costs to provide
> 2 brick fault tolerance?
>
> How do I get redundancy in GlusterFS while getting reasonable
> storage costs where I am not wasting 50% of my investment or
> more in providing copies to obtain redundancy?
It is a real problem that I hope gets addressed soon, until then about the 
only think you can do is rely on underlying hardware redundancy. We have 
just over 100 TB on Gluster with every brick using hardware RAID 6 cards 
with a hot standby. I know its not the solution your looking for, but 
for now its all we have with Gluster.
><>
Nathan Stratton                                CTO, BlinkMind, Inc.
nathan at robotics.net                         nathan at blinkmind.com
http://www.robotics.net                        http://www.blinkmind.com

On Mon, Feb 13, 2012 at 09:18:34PM -0600, Nathan Stratton
wrote:
> 
> On Mon, 13 Feb 2012, Whit Blauvelt wrote:
> 
> >You don't have to leave all your redundancy to Gluster. You can put
Gluster
> >on two (or more) systems which are each running RAID5, for instance.
Then it
> >would take a minimum of 4 drives failing (2 on each array) before
Gluster
> >should lose any data. Each system would require N+1 drives, so double
your
> >drives plus two. (There are reasons to consider RAID other than 5, but
I'll
> >leave that discussion out for now.)
> 
> That's great with a few nodes, but the problem is with Gluster and
> many notes. We run all our notes with RAID6, but the more nodes you
> have the more likely you will have a node failure. This is what I
> think Jeff was worried about.
Sure, nodes can fail. But Jeff's subject was "Exorbitant cost to
achieve
redundancy??" That was a nice contrast to those who've found Gluster
far
cheaper than solutions they'd gone with before. Your systems could always be
hit by natural or man-made disaster at any location too. Seems to be more of
the former lately, and there's plenty of threat of the latter too. Remote DR
should go without saying if the data's valuable. If Gluster gets the geo-rep
thing working right, it'll be the low-cost solution there too.

If someone has a design to get redundancy more cost-effectively, that'd be
great. But as far as I can see - and I'm only in the trenches here, not on
the mountain top - as long as we're essentially on drives, we're going
to
need a RAIDed set locally, synchronously mirrored on anther RAIDed set by
whatever tech you like (e.g. Gluster), and a third set of possibly slower,
cheaper drives far away, with your data asynchronously mirrored for DR. No
equivalent solution is going to be cheaper than the collection of physical
drives. I'm sure my thinking's too conventional. Please suggest better.

Best,
Whit

Hi,

On Monday 13 February 2012 16:15:16 Jeff Wiegley wrote:
> In other words... GlusterFS TRIPLES all my storage costs to provide
> 2 brick fault tolerance?
> How do I get redundancy in GlusterFS while getting reasonable
> storage costs where I am not wasting 50% of my investment or
> more in providing copies to obtain redundancy?
Show me any kind of redundancy without multiplying the efforts!

Take a simple raid1 with two disks: How do you achieve fault-tolerance against 
one failing drive without storing the data on a second disk?
When you need tolerance against two failing disks (at the same time), you have 
to have at least three disks containing the data.

For bigger setups there are raid-levels that work with more then two disks and 
are tolerant against one or two failed drives, but then you "loose"
one or two
disks in your array for checksums. And these have a lot of disadvantages too.

As cheap as disk-space got the last years (save the last 4 months since the 
flood), most admins just use raid1 and be done with it. (Yes, I am an advocate 
of baarf, though not an "official member":)

Now the problem with raid inside one machine is that you still have the 
single-point-of-failure of motherboard, cpu, memory, psu(*), controller and 
network(to a point). With systems like glusterfs, moosefs, drbd and others you 
have your raid span multiple machines removing these spofs while preserving 
the advantage of local disk-reads. When you use the fs that way...

And on a side-note: I don't know what you get per hour but taking low
it-wages
in germany it takes probably less then one man-week of data-recovery to 
amortize the "investment" of doubling disks and machine for
redundancy.
And when the data is lost due to missing redundancy, its not only one persons 
work that is lost... The additional hardware pays off faster then your 
boss/client can think about the expenses.

Have fun,

Arnold

(*) Yes, you can have multiple psu in one machine. And thats nice too when you 
switch your machine from one ups to another. But power is still distributed by 
one power-distribution-board. Which is why I count the psu still as a spof.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120214/f27f703a/attachment.sig>