rsync - Aug 2007 - Rsync stops randomly (using vanished files bash script)

Hey,

I've been trying to transfer about 90GB from one Server to another using 
rsync. But rsync stops randomly. I use the bash script provided in a bug 
report, to block errors about vanished files!

Here is how I start rsync:
./rsync-no-vanished -vvvvv -a -e ssh --delete --progress -z 
root@kilu3.de:/var/www/users /data/oldkilu3/ > /data/rsync.kilu3n &

When I log in into the server after a couple of hours, rsync is not 
running anymore and has transfered only like 10% of the files. I tried 
it 5 times over 3 days and both servers were definitely up and running 
the whole time. Here are some the tails of 3 of my output files:

# tail -f /data/rsync.kilu2n2
data recv 16378 at 78016976
data recv 16378 at 78033354
data recv 52 at 78049732
data recv 16378 at 78049784
data recv 16378 at 78066162
data recv 45 at 78082540
data recv 16378 at 78082585
data recv 16378 at 78098963
data recv 43 at 78115341
data rec

# tail -f /data/rsync.kilu3n
recv_file_name(users/heinicke)
[sender] make_file(users/zhuzhenhua77,*,2)
recv_file_name(users/zhuzhenhua77)
[sender] make_file(users/langhoff,*,2)
recv_file_name(users/langhoff)
[sender] make_file(users/neal,*,2)
recv_file_name(users/neal)
[sender] make_file(users/qingdu,*,2)
recv_file_name(users/qingdu)
[sender] make_file(users/fuch

# tail -f /data/rsync.kilu2n
[sender] make_file(users/jugendkreis/forum/images/smiles/icon_arrow.gif,*,2)
recv_file_name(users/jugendkreis/forum/images/smiles/icon_arrow.gif)
[sender] 
make_file(users/jugendkreis/forum/images/smiles/icon_biggrin.gif,*,2)
recv_file_name(users/jugendkreis/forum/images/smiles/icon_biggrin.gif)
[sender] 
make_file(users/jugendkreis/forum/images/smiles/icon_confused.gif,*,2)
recv_file_name(users/jugendkreis/forum/images/smiles/icon_confused.gif)
[sender] make_file(users/jugendkreis/forum/images/smiles/icon_cool.gif,*,2)
recv_file_name(users/jugendkreis/forum/images/smiles/icon_cool.gif)
[sender] make_file(users/jugendkreis/forum/images/smiles/icon_cry.gif,*,2)
recv_file_name(users


The files are just cut of, and I don't see why. Could anybody explain 
this to me?

Regards,
Samy

I don't think this is possible. Pretty much no services except ssh are 
running on the machine that I'm running the rsync client on. The 
partition I'm transferring the files to hast 400GB of space left and 3.7 
GB of my 4GB memory are still free...
Any other suggestions?

Do you think it could be possible that the machine which I'm fetching 
the data from run out of ram? I think the the client should have thrown 
some error message at least.

Regards,
Samy

Andre Majorel schrieb:
> On 2007-08-23 18:38 +0200, Samuel Vogel wrote:
>
>   
>> When I log in into the server after a couple of hours, rsync is
>> not running anymore and has transfered only like 10% of the
>> files. I tried it 5 times over 3 days and both servers were
>> definitely up and running the whole time.
>>     
>
> Ran out of memory ?
>
>

At 06:58 24.08.2007 +0200, Samuel Vogel wrote:
>I don't think this is possible. Pretty much no services except ssh are
running on the machine that I'm running the rsync client on. The partition
I'm transferring the files to hast 400GB of space left and 3.7 GB of my 4GB
memory are still free...
>Any other suggestions?
>
>Do you think it could be possible that the machine which I'm fetching
the data from run out of ram? I think the the client should have thrown some
error message at least.
What rsync version are you using? Up to he last official (2.6.9) there
seems to be an error that rsync can deadlock, see here:

http://marc.info/?l=rsync&m=116932447411768&w=2

I also had this problem, especially with big files over ssh. I now tried
the current cvs version and it works much better, it never stalled anymore
in the last few days.

bye  Fabi

Thanks for the reply. Do you mean on the rsync server or client side? 
The client has 4GB of Ram, the server only has 1GB. I try to download 
from the server to the client.
I'm sure that there is no file bigger than 500MB. And there can't be any
files 1GB, because we have 1GB quotas in place!

Like mentioned in the other reply to my question. I will try to use the 
latest CVS snapshot!

I'll report if I have success...


Andre Majorel schrieb:
> On 2007-08-24 06:58 +0200, Samuel Vogel wrote:
>
>  
>> I don't think this is possible. Pretty much no services except ssh 
>> are running on the machine that I'm running the rsync client on.
The
>> partition I'm transferring the files to hast 400GB of space left
and
>> 3.7 GB of my 4GB memory are still free...
>> Any other suggestions?
>>
>> Do you think it could be possible that the machine which I'm
fetching
>> the data from run out of ram? I think the the client should have 
>> thrown some error message at least.
>>     
>
> If you have directories with many files, rsync becomes a memory
> hog. I've had it bring down a 1-GB machine while copying a 10-GB
> news spool.
>
>

I installed the CVS version on both sides!

Fabian Cenedese schrieb:
> At 16:43 24.08.2007 +0200, Samuel Vogel wrote:
>   
>> I installed the latest CVS Version (today about 1pm). Again, the same
problem occured...
>>     
>
> Did you have it on both sides? Did rsync say "sending incremental
> file list" (probably only with -v)? It only does so if the rsync on
the other
> side understands it as well.
>
> bye  Fabi
>
>
>

On 8/23/07, Samuel Vogel <samy-delux@gmx.de>
wrote:
> I've been trying to transfer about 90GB from one Server to another
using
> rsync. But rsync stops randomly. I use the bash script provided in a bug
> report, to block errors about vanished files!
I wonder if this is the same problem that caused rsync to quit with
the message about exit code 20.  Is the disk almost full, or is there
a quota or file-size resource limit that might be preventing rsync
from writing to the log?  Does rsync work correctly if you call it
directly without the rsync-no-vanished script?

Matt