rsync - Nov 2015 - [Bug 11609] New: Incorrect (or at least dangerous) behaviour of --append-verify

https://bugzilla.samba.org/show_bug.cgi?id=11609

            Bug ID: 11609
           Summary: Incorrect (or at least dangerous) behaviour of
                    --append-verify
           Product: rsync
           Version: 3.1.1
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: swordangel at gmail.com
        QA Contact: rsync-qa at samba.org

This has been reported at the MacPorts bug tracker already
(https://trac.macports.org/ticket/49657), as I first discovered this using
rsync 3.1.1 from MacPorts, on OS X 10.11.1; but I'm told that this needs to
be
reported upstream.

As far as I can understand, one use of --append-verify is that in case of
interruption, rsync will leave the destination file in its partial state so
that it can be resumed at a later point.

Consider, for example:

source volume: "/Volume/DyingHDD"
destination volume: "/Volume/NewHDD"
Suppose I have a 2 GB file "/Volume/DyingHDD/00027.MTS" and my
DyingHDD is
really dying: its read speed has degraded to around 5 MB/s and it's so
unstable
that it can't read 10 GB without suddenly going offline (OS X would think
that
it has not been ejected cleanly).

So I try to salvage whatever is on DyingHDD by doing "rsync -av
--append-verify
--timeout=60 /Volume/DyingHDD/ /Volume/NewHDD/". (You may point out that
ddrescue should be a better tool for the job, but that's beside the point.)
Now, a few hundred MBs into 00027.MTS, DyingHDD suddenly disappears. I would
expect rsync to leave "/Volume/NewHDD/00027.MTS" in its partial state,
at the
few hundred MB mark, and quit.

But that's not what happened. Instead, rsync proceeded to fill
"/Volume/NewHDD/00027.MTS" all the way to 2 GB at the full speed of
NewHDD
(about 150 MB/s), and leave its timestamp at the current system time. Whatever
rsync filled the end of the file with must have been garbage (there is no way
the OS could have cached the whole source file in RAM).

So the next time I call "rsync -av --append-verify --timeout=60
/Volume/DyingHDD/ /Volume/NewHDD/", rsync just skips over 00027.MTS.

--partial has a similar behaviour

I have been able to reproduce the same behaviour in a Debian 8 VM in
VirtualBox:
1. I put a read-only SD card on my Mac;
2. I shared it across to the Debian 8 VM using VirtualBox's shared folder
function;
3. I then started rsync in the Debian 8 VM to copy some 2GB file from the SD
card;
4. A few hundred MBs in, I yanked the SD card from my Mac;
5. rsync proceeded to fill the destination file with garbage;

Output from rsync:
root at debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# rsync -av --progress
--append-verify --timeout=60 00013.MTS
/media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/sending incremental file list
00013.MTS
  2,125,037,568 100%  144.59MB/s    0:00:14 (xfr#1, to-chk=0/1)
rsync: read errors mapping
"/media/sf_PRIVATE/AVCHD/BDMV/STREAM/00013.MTS":
Protocol error (71)
WARNING: 00013.MTS failed verification -- update retained (will try again).
file has vanished: "/media/sf_PRIVATE/AVCHD/BDMV/STREAM/00013.MTS"

sent 2,125,556,484 bytes  received 922,216 bytes  94,510,164.44 bytes/sec
total size is 2,125,037,568  speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1183) [sender=3.1.1]

But:
root at debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# ls -l
/media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/00013.MTS 
-rwxrwx--- 1 root vboxsf 2125037568 Nov 18 22:51
/media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/00013.MTS

And the next time I call rsync again:
root at debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# rsync -av --progress
--append-verify --timeout=60 00013.MTS
/media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/
sending incremental file list

sent 53 bytes  received 16 bytes  138.00 bytes/sec
total size is 2,125,037,568  speedup is 30,797,645.91

rsync just skipped over the file.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.