rsync - Dec 2007 - problems with rsync 2.6.9 and large files (up to 20GB)

Hi,

I'm Jordi and I work at the university of barcelona. I'm trying to make
a
backup of our several clusters. In the past I worked with rsync with a very
good results.

When I try to backup some large directories (for example > 1.5TB with a lot
of large files >20GB) whit this command:

rsync -aulHI --delete --partial --modify-window=2 --log-format=" %t %o %l
%f
" -
-stats --exclude-from=/home/jingles/soft/sinc/sinc_conf/exclude_files.txt
--temp
-dir=/data/tmp --bwlimit=10000 -e "ssh -o ForwardX11=no -i
"/root/.ssh/id_rsa" -
l root" "/home" server_name:"/data/cerqt2/"


it seems that rsync is working and finishing but when appears the message
that summarizes the transferred files rsync starts the backup again (taking
a lot of time)

....
sent 640278263924 bytes  received 217256421 bytes  8610488.88 bytes/sec
total size is 783944943854  speedup is 1.22
rsync warning: some files vanished before they could be transferred (code
24) at
 main.c(977) [sender=2.6.9]
 2007/12/11 17:02:04 del. 0
home/jingles/soft/sinc/bin/home/g1benjamin/timel-fre
q.po15740
....

I must say that these clusters are working full day so there are several
changes in the data stored in our hard disks.

I googled for this problem and I found that there are a patch file to solve
this problem (you can see the thread in
http://www.mail-archive.com/rsync@lists.samba.org/msg17815.html
) but I can't found it.

Can you help me? The youngest 3.0 version solve this problem?


Thanks for any suggestion,

regards,

jordi
-------------- next part --------------
HTML attachment scrubbed and removed

On Wed, 2007-12-12 at 14:28 +0100, gorka barracuda
wrote:
> it seems that rsync is working and finishing but when appears the
> message that summarizes the transferred files rsync starts the backup
> again (taking a lot of time)
> 
> ....
> sent 640278263924 bytes  received 217256421 bytes  8610488.88
> bytes/sec
> total size is 783944943854  speedup is 1.22
> rsync warning: some files vanished before they could be transferred
> (code 24) at 
>  main.c(977) [sender=2.6.9]
>  2007/12/11 17:02:04 del. 0
> home/jingles/soft/sinc/bin/home/g1benjamin/timel-fre
> q.po15740 
> ....
I've never known rsync to start over like that.  Are you using a script
that runs rsync repeatedly until it exits with code 0?
> I googled for this problem and I found that there are a patch file to
> solve this problem (you can see the thread in
> http://www.mail-archive.com/rsync@lists.samba.org/msg17815.html ) but
> I can't found it.
That thread is about an optimization that makes the delta-transfer
algorithm much faster on large files.  The optimization is included in
current development versions of rsync 3.0.0.  This probably is not
related to the problem with rsync starting over.

Matt