rsync - Feb 2019 - rsync rewrites all blocks of large files although it uses delta transfer

On Wednesday, February 13, 2019 11:29:44 AM EET Kevin Korb via rsync 
<rsync at lists.samba.org> wrote:
> With --backup in order to end up with 2 files it has to write out a
> whole new file.
> Sure, it only sent the differences (normally that means
> over the network but there is no network here) but the writing end was
> told to duplicate the file being updated before updating it.
The copy is needed for the comparison of the blocks as "--inplace"
overwrites
the destination file. I've tried without "--backup" but then the
delta
transfers too much data - close to the size of the backed-up files.

The copy is in a temp file system which is discarded after the backup (by
"rm
-rf"). This temp filesystem is not log structured or copy-on-write so
having a
copy there is not a big problem. Although I don't want a backup of all files
which are modified but rather a TMPDIR.

The ideal workflow would be to compare SRC and DST and write changed blocks to 
the TMPDIR, then read them from TMPDIR and apply it to DST.



 
Cheers
--
Delian

It can't do what you want.  The closest thing would be --compare-dest.

On 2/13/19 5:26 PM, Delian Krustev wrote:
> On Wednesday, February 13, 2019 11:29:44 AM EET Kevin Korb via rsync 
> <rsync at lists.samba.org> wrote:
>> With --backup in order to end up with 2 files it has to write out a
>> whole new file.
>> Sure, it only sent the differences (normally that means
>> over the network but there is no network here) but the writing end was
>> told to duplicate the file being updated before updating it.
> 
> The copy is needed for the comparison of the blocks as
"--inplace" overwrites
> the destination file. I've tried without "--backup" but then
the delta
> transfers too much data - close to the size of the backed-up files.
> 
> The copy is in a temp file system which is discarded after the backup (by
"rm
> -rf"). This temp filesystem is not log structured or copy-on-write so
having a
> copy there is not a big problem. Although I don't want a backup of all
files
> which are modified but rather a TMPDIR.
> 
> The ideal workflow would be to compare SRC and DST and write changed blocks
to
> the TMPDIR, then read them from TMPDIR and apply it to DST.
> 
> 
> 
>  
> Cheers
> --
> Delian
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			https://sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20190213/da0a7610/signature.sig>

On 2019-02-13 5:26 p.m., Delian Krustev via rsync wrote:
> 
> The copy is needed for the comparison of the blocks as
"--inplace" overwrites
> the destination file. I've tried without "--backup" but then
the delta
> transfers too much data - close to the size of the backed-up files.
> 
It's cool that --backup can be used as source data that way, a feature
was unaware of.. but I think you found the cause of your problem right
here as well.

If the --inplace delta is as large as the filesize, then the
structure/location of the data has changed enough that the whole file
would have to be written out in any case.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: remi.vcf
Type: text/x-vcard
Size: 193 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20190213/1cf5807a/remi.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20190213/1cf5807a/signature.sig>

On Wednesday, February 13, 2019 6:25:59 PM EET Remi Gauvin 
<remi at georgianit.com> wrote:
> If the --inplace delta is as large as the filesize, then the
> structure/location of the data has changed enough that the whole file
> would have to be written out in any case.
This is not the case.
If you see my original post you would have noticed that the delta transfer 
finds only about 20 MB of differences within the almost 2G datafile.

The problem with --inplace without --backupdir is that delta transfers can no 
longer work efficiently.


Cheers
--
Delian