rsync - Jun 2012 - [Bug 8990] New: --append-verify doesn't spot file differences, even with --checksum

https://bugzilla.samba.org/show_bug.cgi?id=8990

           Summary: --append-verify doesn't spot file differences, even
                    with --checksum
           Product: rsync
           Version: 3.0.9
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: samba-bugzilla at ookypooky.com
         QAContact: rsync-qa at samba.org


Example:

$ echo "hello" > test1.txt

$ echo "world" > test2.txt

$ rsync --append test1.txt test2.txt

$ diff test1.txt test2.txt
--- test1.txt   2012-06-08 10:23:31.059215000 +0100
+++ test2.txt   2012-06-08 10:23:36.891225000 +0100
@@ -1 +1 @@
-hello
+world

$ rsync --append-verify test1.txt test2.txt

$ diff test1.txt test2.txt
--- test1.txt   2012-06-08 10:23:31.059215000 +0100
+++ test2.txt   2012-06-08 10:23:36.891225000 +0100
@@ -1 +1 @@
-hello
+world

$ rsync --append-verify --checksum test1.txt test2.txt

$ diff test1.txt test2.txt
--- test1.txt   2012-06-08 10:23:31.059215000 +0100
+++ test2.txt   2012-06-08 10:23:36.891225000 +0100
@@ -1 +1 @@
-hello
+world

$ rsync --append-verify --checksum --ignore-times test1.txt test2.txt

$ diff test1.txt test2.txt
--- test1.txt   2012-06-08 10:23:31.059215000 +0100
+++ test2.txt   2012-06-08 10:23:36.891225000 +0100
@@ -1 +1 @@
-hello
+world

Bug confirmed in rsync-3.0.6 and 3.0.9. This syncs correctly with rsync-2.6.9
(using only --append).

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8990

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
            Summary|--append-verify doesn't     |It might be nice to make
                   |spot file differences, even |--append-verify also
                   |with --checksum             |transfer non-appending
                   |                            |files
           Severity|normal                      |enhancement

--- Comment #1 from Wayne Davison <wayned at samba.org> 2012-06-16
17:03:30 UTC ---
The way append currently works is that it only transfers files that have data
to append.  From the manpage:

"If a file needs to be transferred and its size on the receiver is
the same or longer than the size on the sender, the file is skipped."

Thus, it is assumed that if you are using --append (or --append-verify) that
you have narrowed down your transfer list to just files that need to be
appended and that you know have unchanged data prior to the append bits.

It might be nice to allow rsync to do a non-appending transfer for any files it
finds that have a non-longer length (possibly only with --append-verify).  Such
a feature would require some way to alert the sender what kind of file is
currently being processed (possibly using the sum data fields), and that is not
currently possible.  I'm thinking that we don't want to do that with
--append
because it is already dangerous if your transfer list is not limited to just
the files that are known good besides needing some appending.  The
--append-verify option is already telling rsync that you aren't sure about
the
files, so it might be nice if that option also mode did more updating work. 
However, such a change would be incompatible with older rsyncs and is not
currently planned.

I'll change this into an enhancement request for future consideration.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8990

--- Comment #2 from Matt A <samba-bugzilla at ookypooky.com> 2012-06-26
08:07:39 UTC ---
(In reply to comment #1)
> Thus, it is assumed that if you are using --append (or --append-verify)
that
> you have narrowed down your transfer list to just files that need to be
> appended and that you know have unchanged data prior to the append bits.
I can see that that is the intended behaviour for --append, but then what does
--append-verify do differently? Prior to version 3.0.0, --append did check that
the files matched after attempting an append, and the way the documentation is
written gives the impression that this behaviour is now implemented by
--append-verify. To quote:

  --append-verify
              This  works  just  like the --append option, but the existing
data on the receiving side is included in the full-file checksum verification
step, which will cause a file to be resent if the final verification step fails
(rsync uses a normal, non-appending --inplace transfer for the resend).

              Note: prior to rsync 3.0.0, the --append option worked like
--append-verify, so if you are interacting with an older rsync (or the transfer
is using a protocol prior  to  30), specifying either append option will
initiate an --append-verify transfer.

I'm pretty sure the current behaviour is a bug.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8990

--- Comment #3 from Andrey Bond <plboy at mail.ru> ---
Hello,
I make backup users data. There is the problem with file that are reduced.

For example there is the file outlook.pst with size more 2GB. I use the
--inplace and --append-verify options. When this file become larger the sync is
working properly. But if this file become smaller, because the user erase some
messages, the file will not be syncroniced.

Now I start rsync twice for backup of snapshot with --append-verify after
whithout --append-verify. If I start rsync without --append-verify first the
rsync copy the whole file (instead of copying the changes) if size is larger
then distenation file.

I think the need to check all files with modified date by checksum. May be with
aditional option.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.