Gluster users - May 2011 - Storage Design Overview

Hi,

We are building a storage cluster with high availability and high
scalability in mind, and GlusterFS project seems very interesting for our
purposes.

We do have some basic questions before we get down and dirty, and would be
grateful if someone could shed some light on this.

1.  Can we mount a GlusterFS on a client and expect it to provide sustained
throughput near wirespeed?  Is there any subjective comparison between
reading from GlusterFS and reading from local drives?  Does it put extra
pressure on the client?

2.  Reliability + Scalability means Distributed Replicated volumes.
 Initially this might be enough for our needs, but as our read requirements
grow, the Striped option looks promising.  Is it possible to mix Distributed
+ Replicated + Striped?

3.  What happens to very large files.  Say 100 GB files.  Are they kept as a
single file in every node that has the file?  Or is it split up and
distributed in blocks?

That's it for now.  I hope these were not too elementary to ask in this
list.  And, thank you beforehand for your time and effort in replying to my
email.

Regards
HASSAN
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110511/4f71efcb/attachment.html>

Hi Hassan.

Answers inline below.

James Burnash, Unix Engineering
T. 201-239-2248
jburnash at knight.com | www.knight.com
[cid:image001.jpg at 01CC0FB7.A78A6B00]
545 Washington Ave. | Jersey City, NJ

From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at
gluster.org] On Behalf Of Nyamul Hassan
Sent: Wednesday, May 11, 2011 8:36 AM
To: gluster-users at gluster.org
Subject: [SPAM?] [Gluster-users] Storage Design Overview
Importance: Low

Hi,

We are building a storage cluster with high availability and high scalability in
mind, and GlusterFS project seems very interesting for our purposes.

We do have some basic questions before we get down and dirty, and would be
grateful if someone could shed some light on this.


1.      Can we mount a GlusterFS on a client and expect it to provide sustained
throughput near wirespeed?  <No>

2.      Is there any subjective comparison between reading from GlusterFS and
reading from local drives?  <A number of instances on the mailing list. Not
sure if these exist on a web page or the wiki at this point. Perhaps I?ll do
that when I get some free time.>

3.      Does it put extra pressure on the client? <What do you mean by
pressure? My clients (HP ProLiant DL360 G5 Quad Core with 32GB RAM) show up to
2GB of memory usage when the native Gluster client is used for mounts ? but that
is dependent on what you set the client cache max for ? in my case, 2GB. CPU
utilization is usually negligible in my systems, network bandwidth utilization
and I/O throughput ? depend on what the files sizes and access patterns look
like>

2.  Reliability + Scalability means Distributed Replicated volumes.  Initially
this might be enough for our needs, but as our read requirements grow, the
Striped option looks promising.  Is it possible to mix Distributed + Replicated
+ Striped? <I don?t believe that this configuration is currently
supported>

3.  What happens to very large files.  Say 100 GB files.  Are they kept as a
single file in every node that has the file?  Or is it split up and distributed
in blocks? <Storage is as single file on the appropriate nodes per your
volume configuration. The exception to this is Striped, which I believe still
stores the pieces of the striped file as actual files on the nodes.

That's it for now.  I hope these were not too elementary to ask in this
list.  And, thank you beforehand for your time and effort in replying to my
email.

Regards
HASSAN



DISCLAIMER:
This e-mail, and any attachments thereto, is intended only for use by the
addressee(s)named herein and
may contain legally privileged and/or confidential information. If you are not
the intended recipient of this
e-mail, you are hereby notified that any dissemination, distribution or copying
of this e-mail and any attachments
thereto, is strictly prohibited. If you have received this in error, please
immediately notify me and permanently
delete the original and any printout thereof. E-mail transmission cannot be
guaranteed to be secure or error-free.
The sender therefore does not accept liability for any errors or omissions in
the contents of this message which
arise as a result of e-mail transmission.
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY
Knight Capital Group may, at its discretion, monitor and review the content of
all e-mail communications.

http://www.knight.com<http://www.knight.com/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110511/e4558d42/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 5063 bytes
Desc: image001.jpg
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110511/e4558d42/attachment.jpg>

On 05/11/2011 08:35 AM, Nyamul Hassan wrote:
> 1.  Can we mount a GlusterFS on a client and expect it to provide 
> sustained throughput near wirespeed?  Is there any subjective
> comparison between reading from GlusterFS and reading from local
> drives?  Does it put extra pressure on the client?
As others have said, it depends on what you're doing.  I've measured
over
900MB/s per server across three servers from one client using 10GbE, but
only for very well-behaved workloads - large sequential I/O with many
concurrent threads.  Use fewer threads or smaller requests, throw in more
random or synchronous operations, do more stuff with metadata than data,
and your performance will drop from very good to quite poor.
> 2.  Reliability + Scalability means Distributed Replicated volumes. 
> Initially this might be enough for our needs, but as our read 
> requirements grow, the Striped option looks promising.  Is it
> possible to mix Distributed + Replicated + Striped?
It is possible in the code, but there's no configuration support for it.
In other words, you can't do it with "gluster volume create" and
any
"gluster volume set" is likely to undo any manual changes you had made
to the volume configuration files.

I've generally found the stripe translator to be of very limited use
anyway.  The overhead from splitting and reassembling requests, and even
from just having another translator in the stack, has overwhelmed any
advantage from splitting an individual I/O across connections in every
test I've done.  The only compelling reason I've heard for using the
stripe translator has nothing to do with performance.  Without striping,
the maximum size of a file is limited to the maximum available space on
any one brick.  You can use striping to distribute the space used by
that file across multiple bricks and thus get beyond that limit.
> 3.  What happens to very large files.  Say 100 GB files.  Are they
> kept as a single file in every node that has the file?  Or is it
> split up and distributed in blocks?
The "distribute" translator (a.k.a. DHT) will place the entire
contents
of a file onto one of its component subvolumes - a single brick, or a
replica set if you're doing replication as well.  Without striping,
that's the end of the story.  With striping, DHT will place the entire
file onto one stripe set, which will then store the contents on N bricks
or replica sets below that.  See above for an explanation of why this
might be useful in some very limited cases.