rsync - Dec 2009 - retransfer fail of large files with inplace and broken pipe

Hi,

i have to tranfer large files each 5-100 GB (mo-fri) over dsl line. 
unfortunately dsl lines are often not very stable and i got a broken pipe error.
(dsl lines are getting a new ip if they are broken or at least after a 
reconnect every 24 hours)

i had a script which detect the rsync error and restart the transmission.
this means that if a file has transfered e.g. 80 % i start again from beginning.

using partial and partial-dir was no solution to resync because rsync cut the 
original file (e.g. from 20 GB to 15 GB) which means that i have to transfer 
the whole rest of 5 GB.

so i had a look at --inplace which I thougt could do the trick, but inplace is 
updating the timestamp and if the script start a retransfer after a broken pipe
it fails because the --inplace file is newer than the original file of the 
sender.

using ignore-times could be a solution but slow down the whole process to much.

is there a option to tell rsync not to change the time of a --inplace 
transfered file, or maybe preserve the mtime and do a comparison of mtime 
instead of ctime.

Thx
Tom

tom raschel wrote:
> Hi,
> 
> i have to tranfer large files each 5-100 GB (mo-fri) over dsl line.
> unfortunately dsl lines are often not very stable and i got a broken
> pipe error.
> (dsl lines are getting a new ip if they are broken or at least after a
> reconnect every 24 hours)
> 
> i had a script which detect the rsync error and restart the
> transmission.
> this means that if a file has transfered e.g. 80 % i start again from
> beginning.
> 
> using partial and partial-dir was no solution to resync because rsync
> cut the
> original file (e.g. from 20 GB to 15 GB) which means that i have to
> transfer
> the whole rest of 5 GB.
> 
> so i had a look at --inplace which I thougt could do the trick, but
> inplace is
> updating the timestamp and if the script start a retransfer after a
> broken pipe
> it fails because the --inplace file is newer than the original file of
> the
> sender.
> 
> using ignore-times could be a solution but slow down the whole process
> to much.
> 
> is there a option to tell rsync not to change the time of a --inplace
> transfered file, or maybe preserve the mtime and do a comparison of
> mtime
> instead of ctime.
> 
> Thx
> Tom
> 
> 
> 
> --
> Please use reply-all for most replies to avoid omitting the mailing
> list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-
> questions.html

First: Transfer or re-transfer. I think, particularly with bad connections,
you need to treat those VERY differently.

For the initial transfer, --partial should help.

For retransfers, where stuff in the middle has changed, I would expect the 
necessary state information to exist ONLY in the two running processes, and
that information is lost if the connection goes down.
This includes the connection dying because both sides are going through the 
file and have nothing worthwhile to say to each other.

As usual, flames invited if I've got any of this wrong.

On Sat, Dec 12, 2009 at 11:21 PM, tom raschel <raschel at edvantice.de>
wrote:
> so i had a look at --inplace which I thougt could do the trick, but inplace
> is updating the timestamp and if the script start a retransfer after a
> broken pipe it fails because the --inplace file is newer than the original
> file of the sender.
>
Are you using --update (-u)?  If so, turn that off.  If not, rsync won't
skip a file that is newer, so something else is afoot.

..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20091213/36a21b8f/attachment.html>

Hi,

retransfer of large fail with inplace after a broken pipe is working now.
(thx again to wayne)

but it is much more slow as if a "normal" rsync job.

I have read that setting the --backup option could help. (have not tried it
yet)

But --backup option would halve the space, which is not desirable.

Is there a way to tell rsync to delete the --backup file after an successful
sync.

thx
Tom


"tom raschel" <raschel at edvantice.de> schrieb im Newsbeitrag 
news:loom.20091213T075221-530 at post.gmane.org...
> Hi,
>
> i have to tranfer large files each 5-100 GB (mo-fri) over dsl line.
> unfortunately dsl lines are often not very stable and i got a broken pipe 
> error.
> (dsl lines are getting a new ip if they are broken or at least after a
> reconnect every 24 hours)
>
> i had a script which detect the rsync error and restart the transmission.
> this means that if a file has transfered e.g. 80 % i start again from 
> beginning.
>
> using partial and partial-dir was no solution to resync because rsync cut 
> the
> original file (e.g. from 20 GB to 15 GB) which means that i have to 
> transfer
> the whole rest of 5 GB.
>
> so i had a look at --inplace which I thougt could do the trick, but 
> inplace is
> updating the timestamp and if the script start a retransfer after a broken 
> pipe
> it fails because the --inplace file is newer than the original file of the
> sender.
>
> using ignore-times could be a solution but slow down the whole process to 
> much.
>
> is there a option to tell rsync not to change the time of a --inplace
> transfered file, or maybe preserve the mtime and do a comparison of mtime
> instead of ctime.
>
> Thx
> Tom
>
>
>
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

OK,

have tried now --inplace with --backup option but syncing the files does 
consume much more time than a normal rsync process,
so this is not a reliable solution.

Thx
Tom


"Tom" <raschel at edvantice.de> schrieb im Newsbeitrag 
news:hg7dg6$emj$1 at ger.gmane.org...
> Hi,
>
> retransfer of large fail with inplace after a broken pipe is working now.
> (thx again to wayne)
>
> but it is much more slow as if a "normal" rsync job.
>
> I have read that setting the --backup option could help. (have not tried 
> it
> yet)
>
> But --backup option would halve the space, which is not desirable.
>
> Is there a way to tell rsync to delete the --backup file after an 
> successful
> sync.
>
> thx
> Tom
>
>
> "tom raschel" <raschel at edvantice.de> schrieb im
Newsbeitrag
> news:loom.20091213T075221-530 at post.gmane.org...
>> Hi,
>>
>> i have to tranfer large files each 5-100 GB (mo-fri) over dsl line.
>> unfortunately dsl lines are often not very stable and i got a broken
pipe
>> error.
>> (dsl lines are getting a new ip if they are broken or at least after a
>> reconnect every 24 hours)
>>
>> i had a script which detect the rsync error and restart the
transmission.
>> this means that if a file has transfered e.g. 80 % i start again from 
>> beginning.
>>
>> using partial and partial-dir was no solution to resync because rsync
cut
>> the
>> original file (e.g. from 20 GB to 15 GB) which means that i have to 
>> transfer
>> the whole rest of 5 GB.
>>
>> so i had a look at --inplace which I thougt could do the trick, but 
>> inplace is
>> updating the timestamp and if the script start a retransfer after a 
>> broken pipe
>> it fails because the --inplace file is newer than the original file of 
>> the
>> sender.
>>
>> using ignore-times could be a solution but slow down the whole process
to
>> much.
>>
>> is there a option to tell rsync not to change the time of a --inplace
>> transfered file, or maybe preserve the mtime and do a comparison of
mtime
>> instead of ctime.
>>
>> Thx
>> Tom
>>
>>
>>
>> -- 
>> Please use reply-all for most replies to avoid omitting the mailing
list.
>> To unsubscribe or change options: 
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
>>
>
>
>
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

On Sun, 2009-12-13 at 07:21 +0000, tom raschel wrote:
> i have to tranfer large files each 5-100 GB (mo-fri) over dsl line. 
> unfortunately dsl lines are often not very stable and i got a broken pipe
error.
> (dsl lines are getting a new ip if they are broken or at least after a 
> reconnect every 24 hours)
> 
> i had a script which detect the rsync error and restart the transmission.
> this means that if a file has transfered e.g. 80 % i start again from
beginning.
> 
> using partial and partial-dir was no solution to resync because rsync cut
the
> original file (e.g. from 20 GB to 15 GB) which means that i have to
transfer
> the whole rest of 5 GB.
Indeed.  I entered an enhancement request to handle this situation
better:

https://bugzilla.samba.org/show_bug.cgi?id=7123

-- 
Matt