Gluster users - Sep 2009 - Rsync

Hello all,
 
I'm getting what I think is bizarre behaviour.... I have about 400G to
rsync (rsync -av) onto a gluster share, the data is in a directory
structure which has about 1000 directories per parent and about 1000
directories in each of them.
 
When I try to rsync an end leaf directory (this has about 4 dirs and 100
files in each) the operation takes about 10 seconds. When I go one level
above (1000 dirs with about 4 dirs in each with about 100 files in each)
the operation takes about 10 minutes.
 
Now, if I then go one level above that (that's 1000 dirs with 1000 dirs
in each with about 4 dirs in each with about 100 files in each) the
operation takes days! Top shows glusterfsd takes 300-600% cpu usage
(2X4core), I have about 48G of memory (usage is 0% as expected).
 
Has anyone seen anything like this? How can I speed it up?
 
Thanks,
 
Josh.

I forgot to mention, the mount is mounted with direct-io, would this
make a difference? 
> -----Original Message-----
> From: gluster-users-bounces at gluster.org 
> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Hiren Joshi
> Sent: 22 September 2009 11:40
> To: gluster-users at gluster.org
> Subject: [Gluster-users] Rsync
> 
> Hello all,
>  
> I'm getting what I think is bizarre behaviour.... I have about 400G to
> rsync (rsync -av) onto a gluster share, the data is in a directory
> structure which has about 1000 directories per parent and about 1000
> directories in each of them.
>  
> When I try to rsync an end leaf directory (this has about 4 
> dirs and 100
> files in each) the operation takes about 10 seconds. When I 
> go one level
> above (1000 dirs with about 4 dirs in each with about 100 
> files in each)
> the operation takes about 10 minutes.
>  
> Now, if I then go one level above that (that's 1000 dirs with 
> 1000 dirs
> in each with about 4 dirs in each with about 100 files in each) the
> operation takes days! Top shows glusterfsd takes 300-600% cpu usage
> (2X4core), I have about 48G of memory (usage is 0% as expected).
>  
> Has anyone seen anything like this? How can I speed it up?
>  
> Thanks,
>  
> Josh.
>