Gluster users - Sep 2009 - is glusterfs DHT really distributed?

Hi All,

I noticed a very weird phenomenon when I'm copying data (200KB image 
files) to our glusterfs storage.  When I run only run client, it copies 
roughly 20 files per second and as soon as I start a second client on 
another machine, the copy rate of the first client immediately degrade 
to 5 files per second.   When I stop the second client, the first client 
will immediately speed up to the original 20 files per second.  When I 
run 15 clients, the aggregate throughput is about 8 files per second, 
much worse than running only one client.  Neither CPU nor network is 
saturated.  My volume file is attached.  The servers are running on a 66 
node cluster and the clients are a 15-node cluster.

We have 33x2 servers and at most 15 separate machines, with each server 
serving < 0.5 clients on average.  I cannot think of a reason for a 
distributed system to behave like this.  There must be some kind of 
central access point.

- Wei





-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: g.vol
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090928/f90fe372/attachment.ksh>

On 09/28/2009 10:35 AM, Wei Dong wrote:
> Hi All,
>
> I noticed a very weird phenomenon when I'm copying data (200KB image 
> files) to our glusterfs storage.  When I run only run client, it 
> copies roughly 20 files per second and as soon as I start a second 
> client on another machine, the copy rate of the first client 
> immediately degrade to 5 files per second.   When I stop the second 
> client, the first client will immediately speed up to the original 20 
> files per second.  When I run 15 clients, the aggregate throughput is 
> about 8 files per second, much worse than running only one client.  
> Neither CPU nor network is saturated.  My volume file is attached.  
> The servers are running on a 66 node cluster and the clients are a 
> 15-node cluster.
>
> We have 33x2 servers and at most 15 separate machines, with each 
> server serving < 0.5 clients on average.  I cannot think of a reason 
> for a distributed system to behave like this.  There must be some kind 
> of central access point.
Although there is probably room for the GlusterFS folk to optimize...

You should consider directory write operations to involve the whole 
cluster. Creating a file is a directory write operation. Think of how it 
might have to do self-heal across the cluster, make sure the name is 
right and not already in use across the cluster, and such things.

Once you get to reads and writes for a particular file, it should be 
distributed.

Cheers,
mark

-- 
Mark Mielke<mark at mielke.cc>