rsync - Sep 2008 - DO NOT REPLY [Bug 5801] New: Sparse (-S) option doesn't work with new files

https://bugzilla.samba.org/show_bug.cgi?id=5801

           Summary: Sparse (-S) option doesn't work with new files
           Product: rsync
           Version: 3.1.0
          Platform: Other
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: martin@scharrer-online.de
         QAContact: rsync-qa@samba.org


Excuse me, this seems to be a known issue, but I couldn't find any related
existing bug for it.

The 'handle sparse files efficiently' option (--sparse/-S) doesn't
work with
new files, i.e. files with doesn't exist on the destination. In this case
all
sparse blocks are transmitted as zero blocks which slows down the sync process
heavily in some application.

This issue was discussed already:
http://lists.samba.org/archive/rsync/2003-August/006998.html
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=337827



OT:
I detected this today when I was scripting something to sync some large video
files in mkv format (around 1.1GB) with the last MB transfered first in order
to watch the incomplete video while it was transfered. (BTW: An option in rsync
for this would be awesome!!). To do this I created a sparse copy of the video
with only the last MB holding real data using 'dd if=... of=.sparse/...
count=0
bs=1M seek=$SIZE_IN_MB_MINUS_ONE' and then rsyncing the .sparse dir first
using
-S. To my surprise this was as slow as the rsyncing of the real file.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5801


matt@mattmccutchen.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement
            Summary|Sparse (-S) option doesn't  |Speed up delta-transfer of
                   |work with new files         |zeros in new files with --
                   |                            |sparse




------- Comment #1 from matt@mattmccutchen.net  2008-10-01 20:17 CST -------
As currently designed, the --sparse option only affects writing of the
destination file, not the transfer of the data, so this is an enhancement
request.  -z may help.  It may be worth adding a special case for zero blocks
to the delta-transfer algorithm, as suggested in the Debian bug report, to
improve the behavior for sparse files without the full CPU-time penalty of
compression.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5801





------- Comment #2 from martin at scharrer-online.de  2009-10-05 09:16 CST
-------
I would like to repeat my interest in such a feature. I frequently copying
large sparse files over a not-so-fast network connections. At the moment I have
to pre-create the destination files as large sparse files using an extra
script.
In my opinion this should be done by rsync itself.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5801





------- Comment #3 from matt at mattmccutchen.net  2009-11-21 00:00 CST -------
Fedora has a similar RFE:

https://bugzilla.redhat.com/show_bug.cgi?id=525545

There, the further suggestion is made that the receiver should be able to skip
long runs of zeros without reconstructing them in memory.  A simple approach
would be to skip one delta-transfer block at a time.  Or, if the protocol is
changed (rather than just having the generator fabricate an all-zero block), it
would be easy to add a token representing a run of zeros of arbitrary length.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5801

--- Comment #4 from grarpamp at gmail.com 2012-03-10 22:53:06 UTC ---
Maybe related:
https://bugzilla.samba.org/show_bug.cgi?id=7854

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=5801

Björn Jacke <bj at sernet.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |grarpamp at gmail.com

--- Comment #5 from Björn Jacke <bj at sernet.de> ---
*** Bug 7854 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are the QA Contact for the bug.