rsync - Jan 2016 - Why is my rsync transfer slow?

Just chiming in here,

I haven’t read all the previous posts so may be repeating…

Mike Bombich has a good piece on benchmarks for various source/destination
scenarios with rsync.

https://bombich.com/kb/ccc3/how-long-should-clone-or-backup-take

Note that copying to sparsebundle on local media is as fast as to local disk.
Sparsebundle is a mac filesystem whereas NAS is not so certain file metadata are
not covered like Hard links,Acls and groups and owners. So it is important to
turn off those rsync options to scan those file parts on NAS. But sparsebundle
on NAS is ok for HFS options.

Network backups are always slower but you can see that sparseimage on network
volume (AFP) is better according to Mike’s chart.

I also develop an rsync based backup app (backupList+) and some users have
reported very slow times even to sparse images on NAS but I am not the expert
Mike is on these matters. Network issues vary for sure….

Cheers,  Rob

Robert DuToit <rdutoit at comcast.net> wrote:
> Mike Bombich has a good piece on benchmarks for various source/destination
scenarios with rsync.
> 
> https://bombich.com/kb/ccc3/how-long-should-clone-or-backup-take
I hadn't seen that link, thanks.
There's an interesting anomaly in the first chart. Not unsurprisingly, for a
single source connection method - increasing the speed of the destination
connection increases throughput. And vice-versa, for a given destination
connection type, faster source connections give more throughput.
Almost !
The fact that internal SATA to internal SATA is slower than SATA-FW800 (either
way, but SATA to FW800 is most pronounced) suggests that there is an internal
bottleneck when using two internal SATA devices simultaneously on the machine
used for the tests.
Also, internal SATA <-> FW800 is the only combination where there's
significant asymmetry in rates, and it does look like there is a write speed
issue on the internal SATA. It would be interesting to see what speed something
like "dd if=/dev/zero of=/dev/${device} bs=1m" gets.

What I can say is his figures for SATA -> FW800 are in the same ballpark as I
get (with a different backup package) on big files.

> Network backups are always slower but you can see that sparseimage on
network volume (AFP) is better according to Mike’s chart.
That's not all that surprising really. If you think about it, it means the
source computer can cache filesystem metadata and dirty data locally and only
deals with the host for a relatively small number of relatively large files. So
it really just has to throw a list of blocks to write at the destination, and
the writes can be queued and (presumably if async options are set) overlap
requests to keep the pipes full.
Much reduced network overhead compared with dealing with a large number of
remote file operations - not so much the bandwidth required, but the latency of
all the round trip network conversations that need to take place.