Gluster users - Nov 2012 - New GlusterFS Config with 6 x Dell R720xd's and 12x3TB storage

Howdy,

We have the following hardware that we are going to use for a GlusterFS cluster.
6 x Dell R720xd's (16 cores, 96G)
PERC H710 Raid controller
12 x 3TB SAS drives
2 x 270G SAS OS drives configured in RAID 1

I've installed each of these with CentOS 6.3 x86_64 on the built in RAID 1
storage.

I now need to decide how to configure the 12 x 3TB disks in each server,
followed by partitioning / formatting them in the OS.

The PERC H710 supports RAID 0,1,5,6,10,50,60. Ideally we'd like to get good
performance, maximize storage capacity and still have parity :-)

Any suggestions on how to configure the backend storage?
* RAID level
* Number of virtual disks (supports 1 per RAID set)
* Stripe Element Size: 64, 128, 256, 512KB, 1MB
* Read Policy: Read Ahead, Adaptive Read Ahead, No Read Ahead
* Write Policy: Write Back, Write Through, Force Write Back
* Disk Cache Policy: Enabled, Disabled

According to the PERC manual, FastPath (provides IO performance for block sizes
smaller than the virtual disk's stripe size) is available when the virtual
disk is configured with Write Through and No Read Ahead cache policies. This is
only available for RAID 0,1,5,6

We plan to test multiple configurations, but figured I'd check to with the
expertise here before we started.

Thanks for any suggestions,

Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121130/e36b00ad/attachment.html>

On Fri, Nov 30, 2012 at 07:21:54PM +0000, Mike Hanby
wrote:
>    We have the following hardware that we are going to use for a GlusterFS
>    cluster.
> 
>    6 x Dell R720xd's (16 cores, 96G)
Heavily over-specified, especially the RAM. Having such large amounts of RAM
can even cause problems if you're not careful.  You probably want to use
sysctl and /etc/sysctl.conf to set

    vm.dirty_background_ratio=1
    vm.dirty_ratio=5   (or less)

so that dirty disk blocks are written to disk sooner, otherwise you may find
the system locks up for several minutes at a time as it flushes the enormous
disk cache.

I use 4 cores + 8GB for bricks with 24 disks (and they are never CPU-bound)
>    I now need to decide how to configure the 12 x 3TB disks in each
>    server, followed by partitioning / formatting them in the OS.
> 
>    The PERC H710 supports RAID 0,1,5,6,10,50,60. Ideally we'd like to
get
>    good performance, maximize storage capacity and still have parity :-)
For performance: RAID10
For maximum storage capacity: RAID5 or RAID6
>    * Stripe Element Size: 64, 128, 256, 512KB, 1MB
Depends on workload. With RAID10 and lots of concurrent clients, I'd tend to
use a 1MB stripe size. Then R/W by one client is likely to be on a different
disk to R/W by another client, and although throughput to individual clients
will be similar to a single disk, the total throughput is maximised.

If your accesses are mostly by a single client, then you may not get enough
readahead to saturate the disks with such a large stripe size; with RAID5/6
your writes may be slow if you can't write a stripe at a time (which may be
possible if you have a battery-backed card).  So for these scenarios
something like 256K may work better.  But you do need to test it.

Finally: you don't mention your network setup.

With 12 SATA disks, you can expect to get 25-100MB/sec *per disk* depending
on how sequential and how large the transfers are.  So your total disk
throughput is potentially 12 times that, i.e.  300-1200MB/sec.  The bottom
end of this range is easily achievable, and is already 2.5 times a 1G link.
At the top end you could saturate a 10G link.

So if you have only 1G networking it's very likely going to be the
bottleneck.

Regards,

Brian.