Gluster users - Sep 2012 - GlusterFS performance

GlusterFS newbie (less than a week) here. Running GlusterFS 3.2.6 servers 
on Dell PE2900 systems with four 3.16 GHz Xeon cores and 16 GB memory 
under CentOS 5.8.

For this test, I have a distributed volume of one brick only, so no 
replication. I have made performance measurements with both dd and 
Bonnie++, and they confirm each other; here I report only the dd numbers 
(using bs=1024k). File size is 1 TB. The brick is a RAID5 set of six 1 TB 
SATA drives with RAID done in the Perc controller; file system is ext4.

On the server:

* using dd to write to the GlusterFS volume (w/fuse): 581 MB/sec.
* using dd to read from the volume (of=/dev/null): 607 MB/sec.

On a fairly low-spec client system (CentOS 6.3, Pentium 4, 3.0 GHz), I 
get:

* dd write to gfs: 99 MB/sec.
* dd read from gfs: 15 MB/sec.

Note that the write performance is good and the read performance is very 
low.

Using NFS to read from the same server (using the Kernel NFS server) gives 
80 MB/sec, and iperf tells me 117 MB/sec, so I don't believe that there is 
anything fundamentally wrong with the network. Using MooseFS on the same 
hardware gives me a read and write performances very close to the NFS 
values.

Using a distributed volume of 2 bricks (no replication): write 45 MB/sec, 
read 13 MB/sec.

Using a replicated volume of 2 bricks: write 23 MB/sec, read 13 MB/sec.

I understand why writing to a replicated volume loses 50% of the 
performance, but I don't understand (1) why the read performance is always 
so low, even with a single brick, and (2) why writing to a 2-brick 
distributed non-replicated volume is only half the performance of a 
1-brick volume.

Someone give me a clue, please.

Steve

On Tue, Sep 25, 2012 at 5:35 PM, Steve Thompson <smt at cbe.cornell.edu>
wrote:
>
> On a fairly low-spec client system (CentOS 6.3, Pentium 4, 3.0 GHz), I get:
>
> * dd write to gfs: 99 MB/sec.
> * dd read from gfs: 15 MB/sec
>
This is different from any other benchmark I've seen. I haven't seen
that
much of a disparity before.

Have you tried something other than dd? Such as copying files to and from a
directory?

-JM
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120926/eb6f54f7/attachment.html>

Well, lookee what I found. Since I am not smart enough to know exactly 
when to give up, I installed GlusterFS 3.3.0 in place of 3.2.6 on both 
servers and clients. Using the tests (file foo.bar being on the GlusterFS 
volume):

 	dd if=/dev/zero of=foo.bar bs=1024k count=1000
 	dd if=foo.bar of=/dev/null bs=1024k

I find the following client performance, using a distributed volume of 2 
bricks and no replication, and with no further tuning:

GlusterFS 3.2.6:
 	read = 13 MB/sec
 	write = 45 MB/sec

GlusterFS 3.3.0:
 	read = 54 MB/sec
 	write = 97 MB/sec

Firefox, running from a GlusterFS-based home volume, is also nearly three 
times faster to open with 3.3.0 than with 3.2.6 (I have a bunch of tabs, 
and it now opens in 4 seconds vs 11 seconds).

Read performance is still a bit lower than I'd like, but what an 
improvement. And I have no idea why.

Steve