Gluster users - Feb 2014 - Understanding Gluster Replication/Distribute

I am new to Gluster and I am having a hard time grasping how Gluster 
functions in distribute mode vs. distribute+replication. I am planning 
on having 5 servers, with each server hosting a raid6-backed 36TB brick. 
For simplicity, lets just pretend this is a 40TB brick. Here are my 
questions:

1. If I do a distribute configuration only, usable capacity of the 
Gluster volume will be 5x40TB or 200TB?

2. In this configuration, what would clients see if one of the servers 
were to fail?

3. When the server comes back up, what steps would need to be taken to 
make the Gluster volume consistent again?

4. if I do a distributed replicated (2) volume, will my usable capacity 
become 160TB or 100TB, or perhaps something else entirely?

5. In this configuration, one server may be removed for maintenance and 
the file system stays consistent?

Thanks for your time.

Scott Dungan

Replies inline:

On 7 February 2014 10:11, Scott Dungan <scott at gps.caltech.edu>
wrote:
> I am new to Gluster and I am having a hard time grasping how Gluster
> functions in distribute mode vs. distribute+replication. I am planning on
> having 5 servers, with each server hosting a raid6-backed 36TB brick. For
> simplicity, lets just pretend this is a 40TB brick. Here are my questions:
>
> 1. If I do a distribute configuration only, usable capacity of the Gluster
> volume will be 5x40TB or 200TB?
Using "40TB" as a round number per brick:

distribute (no replicate) would be a single ~200TB GlusterFS volume.
> 2. In this configuration, what would clients see if one of the servers were
> to fail?
Lots of errors.  Typically, every fifth file or directory would be
missing, and you'd see lots of question marks in your "ls -l"
output.
> 3. When the server comes back up, what steps would need to be taken to make
> the Gluster volume consistent again?
In a distribute-only setup, there's no redundancy.  So there's no
"consistency" so to speak.  When the missing volume came online, the
files it holds would be available again.
> 4. if I do a distributed replicated (2) volume, will my usable capacity
> become 160TB or 100TB, or perhaps something else entirely?
5 servers is an uneven amount of bricks.  You'd end up with 120TB, but
40TB of that wouldn't be replicated.  A 6th brick would solve that
problem, and you'd have ~120TB in full distribute+replicate(2).
> 5. In this configuration, one server may be removed for maintenance and the
> file system stays consistent?
Theoretically yes.  I try to keep my replicated brick downtime to a
minimum though.  Similar to the ideas behind a RAID mirror, I don't
like running in production on only one copy of something for too long.

-Dan