Hi All I would like to use rsync to synchronize two file system tree over WAN periodically. The file system trees contain millions of small image files. Base on our understanding, rsync try to generate file name list before doing any operation, we easily run short of memory and have impact on our normal file system activities. I wonder if rsync can have an option that when it scan the file system tree and accumulate to N number of files, it process these files before scanning further. This might break the operation on files with multiple hard links or detecting rename. But we are ok on this. Thanks! -- Ming Zhang @#$%^ purging memory... (*!% http://blackmagic02881.wordpress.com/ http://www.linkedin.com/in/blackmagic02881 --------------------------------------------
Ming Zhang wrote...> I wonder if rsync can have an option that when it scan the file system > tree and accumulate to N number of files, it process these files before > scanning further.I'd suggest to run rsync in each directory, without the --recursive option. But this should happen within rsync else fork and parameter parsing would take too much ressorces.> This might break the operation on files with multiple hard links or > detecting rename. But we are ok on this.Indeed. Somehow I get the feeling such a feature is requested at least once per month. Christoph
On 6/22/07, Ming Zhang <blackmagic02881@gmail.com> wrote:> I wonder if rsync can have an option that when it scan the file system > tree and accumulate to N number of files, it process these files before > scanning further.The current CVS rsync works this way ("incremental recursion") by default. It will be released as rsync 3.0.0 at some point.> This might break the operation on files with multiple hard links or > detecting rename. But we are ok on this.Right; if you ask for any of those things, the CVS rsync doesn't use incremental recursion mode. Matt