David,
What OS is on each end?
This sounds like (maybe) classic "TCP buffers too small by
default",
which limits the amount of data in-flight for an individual
TCP flow to min(senders_TCP_send_buf, receivers_TCP_recv_buf),
assuming a clean network path.
(If you're running current Linux, it should autotune those buffers
up to a max-configured-value. But many out-of-box max's are too
small, anyway).
In the old days, rsync clients had no way to ask for a specific TCP
buffer size.
So you were stuck with whatever is default for the OS.
Thankfully, rsync developers (Wayne) added a CLI option awhile ago, like:
rsync -av --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 host::module
This requests ~2MB for send and receive buffer from the OS.
(If the OS has a smaller upper bound, it might silently not give you
what you ask for).
Need to do this sort of ask-for-large-buffers on both ends,
whether client or server/daemon.
If working properly, and if the OS is configured to allow what you
ask for, and if the network is reasonably clean (few losses),
and the on-path routers have reasonable (bw*delay) buffers
to help when there's an occasional loss,
you should be able to get very high single-flow rsync performance.
It's worked for me, anyway.
Not sure of the impact of in-line h/w-VPN's...
Let me know if I can help (perhaps unicast?).
Once it's resolved, it's be nice to post a note back to the
list on what the resolution is.
But it sounds suspiciously like TCP-buffer-limited, from what
you've said so far...
(although CA->germany 440/500 doesn't sound bad...)
Another diagnostic approach would be to verify that you can
get good single-flow performance with iperf (specifying reasonable buffers)
or similar.
If you can't get good performance with iperf, then you won't
be able to with single-flow rsync either...
If iperf is tripping over some loss-rate on the path, then you *could*
look at more aggressive TCP stacks - but let's save that for later.
Larry
--
At 5:01 PM -0700 8/10/06, David Halliday wrote:>Rsync speed seems to be effected more by latency than bandwidth, why is
>this?
>
>I have been happily rsyncing all over the world for some time, but today I
>had cause to sync 40GB of data to all our remote sites causing me to look
>closer at the statistics.
>
>We effectively pull data using.
>
>rsync -az --delete --force hqserver::filesystem /export/sync/filesystem
>
>Our network is a 20Meg+ T3 at HQ and a T1/E1 at each remote site. All
>running Hardware VPN over internet.
>
>When rsync is the only network user I am only getting a fraction of the
>available bandwith utilized. With the above command running at each site I
>used 'show int' on the VPN router to get a breakdown of bandwidth
use on a
>site to site basis.
>
>Total Outbound Usage 5354000 bits/sec (out of 20Meg) at California
>>From CA to Arizona 1071000bits/sec (Out of 1.5Meg)
>>From CA to Texas 838000bits/sec (Out of 1.5Meg)
>>From CA to Boston 672000bits/sec (Out of 1.5Meg)
>>From CA to France 429000bits/dec (Out of 2.0Meg)
>>From CA to UK 312000bits/sec (Out of 2.0Meg)
>>From CA to Germany 444000bits/dec (Out of 500K)
>>From CA to Korea 452000bits/dec (Out of 1.5Meg)
>>From CA to Taiwan 395000bits/dec (Out of 1.5Meg)
>>From CA to Singapore 340000bits/dec (Out of unknown)
>
>Finally, when I run several bandwidth using apps (such as ftp, scp etc)
>concurrently I can get close to line speed at any site all sites.
>
>>From this I conclude a single rsync at a site can not utilize all
available
>bandwidth. It is also interesting to note that the bandwidth the rsync is
>able to utilize is directly proportional to the distance (latency) to the
>remote site.
>
>Now the question, Is there any way to optimize these transfers so that a
>single rsync can use the entire bandwidth available?
>
>I considered trying different block sizes but most references I found
>suggested that I leave it to rsync to find the optimum size.
>
>Dave.
>
>--
>To unsubscribe or change options:
>https://lists.samba.org/mailman/listinfo/rsync
>Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html