rsync - May 2018 - [Bug 13423] New: Checksum option does not work as expected when append-verify is used

https://bugzilla.samba.org/show_bug.cgi?id=13423

            Bug ID: 13423
           Summary: Checksum option does not work as expected when
                    append-verify is used
           Product: rsync
           Version: 3.1.3
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: dariuszb at me.com
        QA Contact: rsync-qa at samba.org

Checksum option does not work as expected when append-verify is used. I can see
that checksums are created for all files but they are not synced.




rsync sample files to remote site:

# rsync -azhcv --append-verify  --progress --stats . 
rsyncclient at 192.168.142.2::DBTest/DBtest/
sending incremental file list
./
Firefox 60.0.dmg
         55.37M 100%    8.40MB/s    0:00:06 (xfr#1, to-chk=3/6)
Skype-8.19.0.1.dmg
         92.68M 100%    8.35MB/s    0:00:10 (xfr#2, to-chk=2/6)
SuperDuper!-2.dmg
          6.09M 100%    5.16MB/s    0:00:01 (xfr#3, to-chk=1/6)
SuperDuper!.dmg
          6.09M 100%    7.20MB/s    0:00:00 (xfr#4, to-chk=0/6)

Number of files: 6 (reg: 5, dir: 1)
Number of created files: 4 (reg: 4)
Number of deleted files: 0
Number of regular files transferred: 4
Total file size: 160.23M bytes
Total transferred file size: 160.23M bytes
Literal data: 160.23M bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.388 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 159.83M
Total bytes received: 95

sent 159.83M bytes  received 95 bytes  8.20M bytes/sec
total size is 160.23M  speedup is 1.00


Now i use hexcurse and change one of the source files - just flip some bytes.

As -c option is present I would expect that rsync detects that one file has
changed and updates it accordingly. I run rsync again - no files are synced - I
can confirm it by manually checking files' hashes. 

# rsync -azhcv --append-verify  --progress --stats . 
rsyncclient at 192.168.142.2::DBTest/DBtest/
sending incremental file list

Number of files: 6 (reg: 5, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 160.23M bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.402 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 272
Total bytes received: 12

sent 272 bytes  received 12 bytes  81.14 bytes/sec
total size is 160.23M  speedup is 564,202.91



I remove --append-verify option and run rsync again

# rsync -azhcv  --progress --stats .  rsyncclient at
192.168.142.2::DBTest/DBtest/
sending incremental file list
Firefox 60.0.dmg
         55.37M 100%  101.92MB/s    0:00:00 (xfr#1, to-chk=3/6)

Number of files: 6 (reg: 5, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 160.23M bytes
Total transferred file size: 55.37M bytes
Literal data: 7.44K bytes
Matched data: 55.36M bytes
File list size: 0
File list generation time: 0.390 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 7.44K
Total bytes received: 52.14K

sent 7.44K bytes  received 52.14K bytes  7.94K bytes/sec
total size is 160.23M  speedup is 2,689.70


This time files were synced as expected

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13423

--- Comment #1 from Kevin Korb <rsync at sanitarium.net> ---
If the file has not grown then there is nothing for --append[-verify] to do. 
Also, without --itemize-changes you have no reporting from --checksum.  You
probably shouldn't have either option.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13423

--- Comment #2 from dariuszb at me.com ---
Yes it is clear that --append-verify will not update the same size files.

But --checksum should check hashes of all files and trigger update if
different.

What is happening here looks like append-verify overwrites checksum
functionality.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13423

--- Comment #3 from Kevin Korb <rsync at sanitarium.net>
---
>From man rsync --append:
> If a file needs to be transferred and its size on the receiver is the same
or longer than the size on the sender, the file is skipped.
--append-verify does the verify AFTER an append happens so if --append won't
copy the file neither will --append-verify.  Since --append essentially means
only care if the source file is bigger the file is ignored.

The combination is why I said you probably don't want either option and
especially not both options.  With both, all files on both ends are checksummed
first (even files that are only on one side!)  Then files that are bigger on
the source are appended to.  Then the files are verified.  Then if the
verification fails rsync throws a warning and redoes the whole file --inplace
style.

Normally I recommend against --checksum but I have no Mac experience and I
don't really know how your disk image files work.  In many cases it is
actually
faster to use --ignore-times (delta-xfer everything regardless of timestamp)
and maybe --inplace instead of --checksum.  Usually --checksum is used when you
can't afford to write out a new file because of limited/slow writes or
because
you are retaining the old version and don't want a new version unless it is
actually new.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13423

--- Comment #4 from dariuszb at me.com ---
Thank you for clarification.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.