rsync - Oct 2002 - hundreds of thousands files

I am rsyncing several hunderds of thousands of files in several
directories.  The way I got rsync to work for me was I wrote a script
which NFS mounts the directory I am rsyncing first before starting, and
then it goes through the directories and rsyncs them in bite sizeable
chunks by going a few directories deep and starting there..

Is there any way to have this as an option inside rsync directly?  It
would only have to be a change in the client.  Basicly when it realizes
there are thousands of files, instead of trying to do it in one big sweep
and bombing out, make it so it can do a directory at a time.

Like first rsync all the directories OMITTING individual files, and then
go through the directory tree and go all the way deep and rsync them
directory by directory instead of the whole tree at once.

Any one have any ideas on this?????


Ryan

On Wed, Oct 30, 2002 at 01:40:10PM -0700, Ryan wrote:
> I am rsyncing several hunderds of thousands of files in several
> directories.  The way I got rsync to work for me was I wrote a script
> which NFS mounts the directory I am rsyncing first before starting, and
> then it goes through the directories and rsyncs them in bite sizeable
> chunks by going a few directories deep and starting there..
> 
> Is there any way to have this as an option inside rsync directly?  It
> would only have to be a change in the client.  Basicly when it realizes
> there are thousands of files, instead of trying to do it in one big sweep
> and bombing out, make it so it can do a directory at a time.
> 
> Like first rsync all the directories OMITTING individual files, and then
> go through the directory tree and go all the way deep and rsync them
> directory by directory instead of the whole tree at once.
> 
> Any one have any ideas on this?????
Directory-at-a-time traversal and other ideas have been
discussed serveral times.  Probably the best recent
discussions have had subject lines refering to
"superlifter".

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt