rsync - Jul 2025 - Slow large file copy

Hi Danti,

away from my computer now, so I cannot check, but I suspect the --inplace to be
the culprit. If rsync has to overwrite the file while transferring, it may not
be able to check blocks in the original?

Regards
Hardy

-- 
I'm so stupid today, I could rule America.

On 29.07.25 12:55, Gionatan Danti via rsync wrote:
> Hi all,
> rsync seems slow to copy large files. The issue is not related to the
transfer itself, which is quite fast, but to the discovery of different blocks.
> 
> For example, transferring a big (multi-GB) file from src (remote) to dst
(local) with "--inplace" (to avoid a whole-file copy on dst) shows the
following:
> 
> - if dst does not have a local file, or if -W was given, the copy
immediately starts (but it clearly needs to transfer all blocks);
> 
> - if dst has a previous file version (and -W was not given), first dst
reads the entire file, then src does the same, next different blocks are copied
and finally dst re-read the entire file.
> 
> As dst and src file scans do not happen concurrently, and including the
last whole-file read by dst, copy time is vastly increased at about 3x compared
to an optimized process (with concurrent scans and without the final read).
> 
> Am I missing something? Can rsync be faster for large-file copies?
> 
> Regards.
>

Il 2025-07-30 08:49 Hardy via rsync ha scritto:
> Hi Danti,
> 
> away from my computer now, so I cannot check, but I suspect the 
> --inplace to be the culprit. If rsync has to overwrite the file while 
> transferring, it may not be able to check blocks in the original?
> 
> Regards
> Hardy
Hi, I tried with and without --inplace, but results are similar. If 
anything, omitting --inplace is worse, as it means rewriting the whole 
file on the receiving side.

Thanks.

-- 
Danti Gionatan

Yepp, found it in the man page:

--inplace
[ ... ]
o The efficiency of rsync's delta-transfer algorithm may be reduced if some
data in the destination file is overwritten  before  it
   can  be  copied to a position later in the file.  This does not apply if you
use --backup, since rsync is smart enough to use the
   backup file as the basis file for the transfer.

So NOT using --inplace will save a lot of network transmission at the cost of
local disk copying. Can you not tolerate the temporary disk space?

Hardy

On 30.07.25 08:49, Hardy wrote:
> Hi Danti,
> 
> away from my computer now, so I cannot check, but I suspect the --inplace
to be the culprit. If rsync has to overwrite the file while transferring, it may
not be able to check blocks in the original?
> 
> Regards
> Hardy
> 
> -- 
> I'm so stupid today, I could rule America.
> 
> On 29.07.25 12:55, Gionatan Danti via rsync wrote:
>> Hi all,
>> rsync seems slow to copy large files. The issue is not related to the
transfer itself, which is quite fast, but to the discovery of different blocks.
>>
>> For example, transferring a big (multi-GB) file from src (remote) to
dst (local) with "--inplace" (to avoid a whole-file copy on dst) shows
the following:
>>
>> - if dst does not have a local file, or if -W was given, the copy
immediately starts (but it clearly needs to transfer all blocks);
>>
>> - if dst has a previous file version (and -W was not given), first dst
reads the entire file, then src does the same, next different blocks are copied
and finally dst re-read the entire file.
>>
>> As dst and src file scans do not happen concurrently, and including the
last whole-file read by dst, copy time is vastly increased at about 3x compared
to an optimized process (with concurrent scans and without the final read).
>>
>> Am I missing something? Can rsync be faster for large-file copies?
>>
>> Regards.
>>
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html