rsync - Nov 2008 - Rsync & parallelizing it for files in same directory

Hi there,

 I haven't yet delved into the rsync source code, and thus would need
to ask a few questions first before going into the wrong direction.

 We have not, one but several cases[1] of doing Disaster Recovery type
backup/synchronization with typically 1 to 2 million of files.

Now yes, I know I can go and write a perl/python/<flavour_of_the_year>
script to recursively do the synchronizations in parallel, but it
still leaves issues with the directories with lots of medium sized
files, and not easy ways to split them up, unless....

Now the question I'm wondering about:

 - Ease of doing a threading type setup inside rsync.
 - Use threading architecture, to generate the checksums in parallel
(we have >24 cores per system around here....)
 - rsync then opens multiple connections to the remote site (Bandwidth
optimization with the latencies involved)
 - rsync then typically sends the files in a round robin fashion over
the "links"
 - or rsync does it directory per link for small directories.

We do have multicore systems coming out now as standard on laptops, so
a threading architecture is something to consider.


Note [1] and please don't ask about the design/architectures of these
systems, as I know, but it doesn't help preaching to rocks

-- 
Hendrik Visage

On Fri, 2008-11-28 at 12:40 +0100, Hendrik Visage wrote:
> Now the question I'm wondering about:
> 
>  - Ease of doing a threading type setup inside rsync.
>  - Use threading architecture, to generate the checksums in parallel
> (we have >24 cores per system around here....)
>  - rsync then opens multiple connections to the remote site (Bandwidth
> optimization with the latencies involved)
>  - rsync then typically sends the files in a round robin fashion over
> the "links"
>  - or rsync does it directory per link for small directories.
There is a similar enhancement request:

https://bugzilla.samba.org/show_bug.cgi?id=5124

This would take a major redesign of rsync which I don't think is going
to happen anytime soon.  It may make more sense to write a new, better
designed tool (possibly adopting some of Wayne and Martin Pool's ideas
for "superlifter") that is capable of parallelism.  I would be
interested in helping if I could find the time.

Matt