rsync - Jan 2007 - rsync with a single big file (100M)

Hi,

Anybody knows how rsync transfer one single file if there is difference
between the two files, will rsync re-transfer the whole file or only
transfer the difference? In my experiment, i found rsync would retranster
the whole file again if there's a difference.

What I did were:
I manually created a huge.txt (file size 100813288 ), run rsync command to
transfer this file from server 1 to server 2(server2 does not have this file
yet). The transfer time is 10 sec:
Start Time: Thu Jan 18 17:06:26 SGT 2007
building file list ... done
huge.txt
wrote 538885 bytes  read 101 bytes  51332.00 bytes/sec
total size is 100813288  speedup is 187.04
End Time: Thu Jan 18 17:06:36 SGT 2007

Append some text into huge.txt (now file size is 100819275) in server1, run
rsync command to transfer this file from server1 to server2 (server2 has
this file already at the moment). The transfer time is 17 sec:
 
Start Time: Thu Jan 18 17:08:33 SGT 2007
building file list ... done
huge.txt
wrote 255 bytes  read 70395 bytes  4037.14 bytes/sec
total size is 100819275  speedup is 1427.02
End Time: Thu Jan 18 17:08:50 SGT 2007

As you can see that the second time transfer is much longer than the first
time transfer, although there's only difference at end of the file. So i
feel rsync retransfer the whole file if there's difference.

The version of rsync I am using is  2.6.2
The command I used is 
/usr/local/bin/rsync --verbose --recursive --perms --links --owner --group
--compress --times --stats --delete --no-whole-file --progress
--password-file=/etc/rsyncd.pass /test/rsyncTest/huge.txt
elitesuser@haelis01::rsync

Is there anyways to force rsync to transfer only the difference?

Thanks
dejie
-- 
View this message in context:
http://www.nabble.com/rsync-with-a-single-big-file-%28100M%29-tf3079412.html#a8555649
Sent from the Samba - rsync mailing list archive at Nabble.com.

On Tue 23 Jan 2007, dejie wrote:
> 
> Anybody knows how rsync transfer one single file if there is difference
> between the two files, will rsync re-transfer the whole file or only
> transfer the difference? In my experiment, i found rsync would retranster
Only the difference.
> the whole file again if there's a difference.
No, your experiment does not show that...
> What I did were:
> I manually created a huge.txt (file size 100813288 ), run rsync command to
> transfer this file from server 1 to server 2(server2 does not have this
file
> yet). The transfer time is 10 sec:
> Start Time: Thu Jan 18 17:06:26 SGT 2007
> building file list ... done
> huge.txt
> wrote 538885 bytes  read 101 bytes  51332.00 bytes/sec
> total size is 100813288  speedup is 187.04
> End Time: Thu Jan 18 17:06:36 SGT 2007
> 
> Append some text into huge.txt (now file size is 100819275) in server1, run
> rsync command to transfer this file from server1 to server2 (server2 has
> this file already at the moment). The transfer time is 17 sec:
>  
> Start Time: Thu Jan 18 17:08:33 SGT 2007
> building file list ... done
> huge.txt
> wrote 255 bytes  read 70395 bytes  4037.14 bytes/sec
> total size is 100819275  speedup is 1427.02
> End Time: Thu Jan 18 17:08:50 SGT 2007
> 
> As you can see that the second time transfer is much longer than the first
> time transfer, although there's only difference at end of the file. So
i
> feel rsync retransfer the whole file if there's difference.
No, this only shows that the process took longer the second time. That's
because the file at the destination has to be read, checksums calculated
etc. to find the differences (and of course also the matches), and then
to build the new destination file and move it in place. However,
if you look at the byte counts:
> wrote 538885 bytes  read 101 bytes  51332.00 bytes/sec
vs.
> wrote 255 bytes  read 70395 bytes  4037.14 bytes/sec
you see that the second transfer did in fact transfer much less bytes
than the first time.


Rsync optimizes network traffic, not absolute speed. If your network is
fast enough, use --whole-file which will transfer the whole file which
will be faster in that case than finding and transferring the
differences, and merging those differences into the destination file.


Paul Slootman