rsync - Jan 2011 - Issue with bwlimit after resuming a partial transfer due to a communication break

I am using rsync version 3.0.7. My rsync server is running on the cloud.

I have set the bwlimit to 10 i.e. 10 KBps. When I try to sync files the
bwlimit is as what is
set. I have enabled storage of partial files. When a communication break
causes rsync to
time out during a transfer the partial file is stored in the partial
directory. I then try rsync
again when the link is up. At this point I see that the bwlimit of 10KBps is
not followed.
rsync transfers the data as fast as possible.

I tried the latest nightly build and the issue is seen there too. Has anyone
come across a
similar issue.

Thanks,
Alistair
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20110121/a1a905b5/attachment.html>

On Fri 21 Jan 2011, Alistair Dsouza wrote:
> 
> I have set the bwlimit to 10 i.e. 10 KBps. When I try to sync files the
> bwlimit is as what is
> set. I have enabled storage of partial files. When a communication break
> causes rsync to
> time out during a transfer the partial file is stored in the partial
> directory. I then try rsync
> again when the link is up. At this point I see that the bwlimit of 10KBps
is
> not followed.
> rsync transfers the data as fast as possible.
How did you measure this data rate?
Are you using the --progress output? Then you need to realize that what
is shown there is not the data rate over the network, but the effective
rate at which the file is transferred. As the first part of the file is
in the partial directory, the data that needs to be transmitted is not
much more than the block checksums plus a little overhead. So then it
looks like the file is being transmitted a lot faster.

The same thing can be seen when using -z while transferring a highly
compressable file: --progress will show a much higher transfer rate than
what you've configured with --bwlimit.

This phenomenon can be easily verified by doing it over a dialup line,
or other medium that physically can't go faster than a given rate:
--progress will show higher rates than would be possible.



Paul

Thanks Paul and Wayne. The behaviour is clear now.

Regards,
Alistair


On Sat, Jan 22, 2011 at 11:01 PM, Wayne Davison <wayned at samba.org>
wrote:
> On Sat, Jan 22, 2011 at 2:22 AM, Alistair Dsouza <alistaird at
gmail.com>wrote:
>
>> Literal data: 6,496 bytes
>>
>
> That is how much literal data rsync had to send to finish the file, so it
> is not surprising that it finished quickly.  See also the Total bytes
> received (about 40K), so it would not be surprising to have a 10K/sec
> limited connection finish in 4 seconds (in the example you cited it took
> longer than than, only averaging 5K/sec, presumably due to all the copying
> of basis-file data that went on).
>
> So, in your testing, be sure to not use a file that contains repetitive
> data.  When sending a new file, only the -z option helps rsync to send
> repetitive data with reduced bandwidth.  But when copying over a partial
> basis file, the normal checksum matching will copy local basis-file data
for
> both the partial data and for any matching chunks in the unsent part of the
> file that match chunks in the earlier part of the file.
>
> ..wayne..
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20110122/3bceddd8/attachment.html>