Gluster users - Aug 2009 - Performance

Hello all,
 
I have 2 servers both exporting 36 partitions. The client mirrors the 36
partitions on each server and puts them all into a DHT.
 
dd if=/dev/zero of=/home/webspace_glust/zeros bs=1024 count=1024000
Takes 8 minutes, compared to 30 seconds vi NFS, now granted I'm using
mostly default settings but a transfer rate of 2.1Mb/s vs about 40Mb/s
seems low.
 
What am I doing wrong? Where should I start with the performance tuning?
 
Josh.

----- "Hiren Joshi" <josh at moonfruit.com> wrote:
> Hello all,
>  
> I have 2 servers both exporting 36 partitions. The client mirrors the
> 36 partitions on each server and puts them all into a DHT.
>  
> dd if=/dev/zero of=/home/webspace_glust/zeros bs=1024 count=1024000
> Takes 8 minutes, compared to 30 seconds vi NFS, now granted I'm using
> mostly default settings but a transfer rate of 2.1Mb/s vs about
> 40Mb/s seems low.
  
> What am I doing wrong? Where should I start with the performance
> tuning?
a) Use a bigger block size. 1k is too small to get decent performance.
We suggest using atleast 128k.

b) Make sure you have the write-behind translator loaded on the client
side, with a cache-size set to atleast 1MB.

Vikas
-- 
Engineer - http://gluster.com/

On 08/12/2009 11:24 AM, Hiren Joshi wrote:
> Hello all,
>
> I have 2 servers both exporting 36 partitions. The client mirrors the 36
> partitions on each server and puts them all into a DHT.
>
> dd if=/dev/zero of=/home/webspace_glust/zeros bs=1024 count=1024000
> Takes 8 minutes, compared to 30 seconds vi NFS, now granted I'm using
> mostly default settings but a transfer rate of 2.1Mb/s vs about 40Mb/s
> seems low.
>
> What am I doing wrong? Where should I start with the performance tuning?
How about start with a 1:1 comparison? Unless is somehow doing 
mirroring, you should remove the mirroring from your solution before 
comparing.

Are you using striping? If you are not using striping, then the file 
"/home/webspace_glust" is going to be assigned to a single partition, 
and you would only be using a mirror of one partition.

36 partitions on each server - the word "partition" is ambiguous. Are 
they 36 separate drives? Or multiple partitions on the same drive. If 
multiple partitions on the same drive, this would be a bad idea, as it 
would require the disk head to move back and forth between the 
partitions, significantly increasing the latency, and therefore 
significantly reducing the performance. If each partition is on its own 
drive, you still won't see benefit unless you have many clients 
concurrently changing many different files. In your above case, it's 
touching a single file in sequence, and having a cluster is costing you 
rather than benefitting you.

As for the 2.1Mb/s vs 40Mb/s, I have no clue. I'm new to Gluster myself, 
and I have yet to install it on my node cluster (10 servers) and perform 
timing myself.

Cheers,
mark

-- 
Mark Mielke<mark at mielke.cc>