rsync - May 2012 - [Bug 8918] New: Use fiemap to quickly detect zero ranges of source file

https://bugzilla.samba.org/show_bug.cgi?id=8918

           Summary: Use fiemap to quickly detect zero ranges of source
                    file
           Product: rsync
           Version: 3.1.0
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: matt at mattmccutchen.net
         QAContact: rsync-qa at samba.org


Rsync could use the fiemap ioctl
(https://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/filesystems/fiemap.txt;hb=HEAD)
on the source file to detect any ranges that do not have data on the filesystem
and thus are guaranteed to read as zero, without actually reading them.  This
would be most useful in combination with writing the destination file sparsely
(--sparse), but it would be safe to use in any case as a sender-side
optimization.

Originally proposed at https://bugzilla.redhat.com/show_bug.cgi?id=525545.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8918

--- Comment #1 from Arne Jansen <jansen at webgods.de> 2012-05-08 05:59:34
UTC ---
you could also use the SEEK_HOLE/SEEK_DATA interface which linux added
recently.
This would have the advantage of being Solaris-compatible.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8918

--- Comment #2 from Theodore Ts'o <tytso at mit.edu> 2012-05-08
10:01:13 UTC ---
Using FIEMAP has some real potential problems if the file was just recently
written (and has blocks which were recently written, and where their final
location on disk has not been determined yet).   You can work around this using
the explicitly or implicitly forcing an fsync if this case is found, but
supporting SEEK_HOLE/SEEK_DATA avoids this problem.  The tradeoff is that only
the very latest kernels support SEEK_HOLE/SEEK_DATA.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8918

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #3 from Wayne Davison <wayned at samba.org> 2012-06-16
18:01:35 UTC ---
I would like to include support for SEEK_HOLE and SEEK_DATA in rsync's
--sparse
code, but the rsync protocol doesn't yet support indicating holes in the
data
between the sender and the receiver (the receiver just scans for zeros and
leaves holes when they are found).

We'd also need configure support (if it can be compiled) and run-time
support
(if the current kernel supports the feature and detection if the source
filesystem supports it too).

I'll look into this at some point.  Patches welcomed.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8918

Matt McCutchen <matt at mattmccutchen.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Use fiemap to quickly       |Use OS support to detect
                   |detect zero ranges of       |zero ranges of source file
                   |source file                 |

--- Comment #4 from Matt McCutchen <matt at mattmccutchen.net> 2012-06-17
00:51:29 UTC ---
(In reply to comment #3)
> the rsync protocol doesn't yet support indicating holes in the data
> between the sender and the receiver (the receiver just scans for zeros and
> leaves holes when they are found).
Strictly speaking, the protocol issue is orthogonal and is covered by bug 5801.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.