rsync - Sep 2013 - [Bug 10170] New: rsync should support reflink similar to cp --reflink

https://bugzilla.samba.org/show_bug.cgi?id=10170

           Summary: rsync should support reflink similar to cp --reflink
           Product: rsync
           Version: 3.1.0
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: samba at shubin.ca
         QAContact: rsync-qa at samba.org


new filesystems support the reflink system call...
the standard GNU cp utility now supports this too.
this functionality is invaluable for future rsync.
@BasketCase recommended I write this as a feature.

Here is some background reference:
http://thread.gmane.org/gmane.linux.file-systems/31535

Thanks in advance, happy to answer any questions I can!

James

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10170

--- Comment #1 from Kevin Korb <rsync at sanitarium.net> 2013-09-27
00:31:00 UTC ---
*** Bug 9041 has been marked as a duplicate of this bug. ***

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10170

David Taylor <davidt-samba-bugzilla at yadt.co.uk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |davidt-samba-bugzilla at yadt.
                   |                            |co.uk

--- Comment #2 from David Taylor <davidt-samba-bugzilla at yadt.co.uk> ---
Created attachment 10585
  --> https://bugzilla.samba.org/attachment.cgi?id=10585&action=edit
prototype patch for --reflink support

I have written a rough draft of a patch to enable support for (Btrfs) reflinks
in rsync.  It is very lightly tested.

Only Btrfs is supported (using BTRFS_IOC_CLONE), and this is intended to
provide a better alternative to --inplace for updating a backup which will be
snapshotted repeatedly.  It avoids the problems of --inplace with
partial updates and stale hardlinks.

It adds two new options:

--reflink

When updating an existing file, rather than creating an entirely new temporary
file, rsync will create a 'reflink' copy of the original file and update
it in
place.  When creating a backup copy of a file, rsync will also create a
'reflink' copy rather than rewriting the file.  If a 'reflink'
copy cannot be
performed it will fall back to the normal, non-reflink behaviour.

--reflink-always behaves like --reflink, but disables the fallback if a reflink
fails.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10170

--- Comment #3 from roland <devzero at web.de> ---
fantastic !

did not know about that feature in btrfs, but as i already did inplace backups
with rsync on btrfs, i think i will give that a try.

i just wondered about reflink support in zfs and putting a link here for
reference: https://github.com/zfsonlinux/zfs/issues/405

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10170

James <purpleidea at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |purpleidea at gmail.com

--- Comment #4 from James <purpleidea at gmail.com> ---
(In reply to David Taylor from comment #2)

Sweet... Glad to hear someone is hacking on this.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10170

--- Comment #5 from kdave <dave at jikos.cz> ---
(In reply to David Taylor from comment #2)

A few things:

* thanks for working on reflink support in rsync :)

* btrfs is not the only filesystem to support reflink, ocfs2 does as well, so
you might make the formulations more generic

* for btrfs, there are 2 types of the cloning ioctl:
 1) that does file-to-file clone (same for ocfs2), IOC_CLONE
 2) range cloning, IOC_CLONE_RANGE

I believe for rsync option 2 should be implemented in the similar way --inplace
works.

* the proposed patch does not work in all scenarios:

$ btrfs subvol create subv1
$ btrfs subvol create subv2
$ <create big file subv1/testfile>
$ rsync --reflink subv1/testfile subv2

according to strace, rsync does not call the clone ioctl at all (none of the
forked processes)

$ rsync --reflink-always subv1/testfile subv2
testfile
    240,475,844 100%   43.54MB/s    0:00:05 (xfr#1, to-chk=0/1)
rsync: open "/subv2/testfile": No such file or directory (2)
rsync: reflink of "/subv2/.testfile.8bxNk0" failed: No such file or
directory
(2)
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1165) [sender=3.1.2dev]

and the target file is not created, the same command without --reflink works.

* for speed comparison, I did 'cp --reflink subv1/testfile subv2' and it
takes
near to no time, compared to 5 seconds for bare copy, I'm expecting that
rsync
--reflink would take comparable time to cp+reflink

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10170

--- Comment #6 from Brian J. Murrell <brian at interlinx.bc.ca> ---
Does this --reflink feature have any parity/functionality with --link-dest,
which is often used in "snapshot" style [i.e. daily] backup scripts on
non-snapshottable  filesystems such as XFS?

XFS supports COW reflinks but not snapshots and as such rsync --link-dest is
more appropriate for creating new backups.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.