How hard would it be to have rsync do the file recursion scan on both the source tree and the destination tree at the same time in parallel? Would that require a protocol change, or could just a program change be enough? I have some very large directories I'd like to syncronize. Total time to scan through these millions of files is a substantial portion of an hour or even exceeds it. It just goes slower if these large critical time blocks have to be done sequentially. It would also be nice if rsync didn't have to collect the entire tree in RAM all at once, but instead, both source and destination could recurse their respective trees in sync with each other and copy, create, delete, as things go through that parallel recursion. -- ----------------------------------------------------------------------------- | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ | | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ | -----------------------------------------------------------------------------
Phil Howard wrote...> I have some very large directories I'd like to syncronize. Total time to > scan through these millions of files is a substantial portion of an hour > or even exceeds it. It just goes slower if these large critical time blocks > have to be done sequentially.Yeah, same here.> It would also be nice if rsync didn't have to collect the entire tree in > RAM all at once, but instead, both source and destination could recurse > their respective trees in sync with each other and copy, create, delete, > as things go through that parallel recursion.I'm not sure, perhaps hard link preservation would cause some headache here. But as long as you're not using -H at all, there shouldn't be a principal problem. Christoph
On Fri, Dec 02, 2005 at 05:29:32PM -0600, Phil Howard wrote:> How hard would it be to have rsync do the file recursion scan on both the > source tree and the destination tree at the same time in parallel? Would > that require a protocol change, or could just a program change be enough?It might be possible to have rsync stat all the files as they arrive from the sender and mark the changed ones using flags, but it is unclear how much of a time savings that would be since rsync would still need to wait until all the file list arrived before starting to transfer files and make changes (it might be able to create directories, symlinks, and devices during this time, but that would require a protocol bump due to how rsync currently reports these events). It would be interesting to do some tests and see if it helps. What I'd like to do with rsync in the long run is to have it begin transferring each directory just after it is scanned, but that will require a protocol bump and a lot of changes to how rsync works. I have some ideas on how this might be implemented, and will hopefully be able to look into this soon. ..wayne..