rsync - May 2013 - [Bug 9864] New: Allow permanent compression of destination files

https://bugzilla.samba.org/show_bug.cgi?id=9864

           Summary: Allow permanent compression of destination files
           Product: rsync
           Version: 3.1.0
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: me at haravikk.com
         QAContact: rsync-qa at samba.org


Rsync already supports compression for improving transfer speed, which is
great, but it would also be nice to see the option for permanent compression to
reduce space on the target file-system.

Here is an example of what options might look like:

--dest-compress - enables compression of files on the destination.
--dest-compress-level - sets the compression level for files on the
destination. Any value from 1-9 implies --dest-compress, while 0 disables it.
If this parameter is not supplied then --compress-level will be used if set,
otherwise default compression is used.
--dest-compress-ext - an optional extension to append to compressed files, e.g
- 'gz'. This extension is checked when checking for files that already
exist on
the destination, so that compressed files are not deleted.
--source-compress-ext - specifies an extension to be checked for on the source,
targeting files that need to be decompressed before sending. This is useful in
cases where a destination structure created with compression needs to be synced
to another target in a decompressed form. Where possible decompression should
be skipped if transfer compression or destination compression are also enabled
(i.e - a delta update is not being performed).

In keeping with ticket #8915:
--dest-compress-minsize - the minimum size of uncompressed file that
compression will be attempted upon, e.g - if a file is under 4mb then
compression may be skipped.
--dest-compress-maxsize - the maximum size of uncompressed file that
compression will be attempted upon, e.g - skip compression on files larger than
1gb to save processing.


The implication of this feature of course is that a sync/backup destination can
be a lot smaller than the source destination, allowing you to get more out of
the space; especially handy on common file-systems such as ext4 that do not
natively support compression.

However, this saving is at the cost of processing, as a file on the destination
may need to be decompressed on-the-fly in order to compare checksums, and
changes cannot be applied via --inplace or --append. If a file is new to the
destination however then if --compress-level and --dest-compress-level are the
same then it should be possible to simply write the file straight to disk in
the compressed form; however if any of the compression details conflict then a
gzip -d optionally followed by a gzip may be required to remove and
(optionally) re-apply compression.

Of course the savings won't be as good as a single compressed archive would
be,
but it retains the freedom of a synced structure but with some of the space
requirement reduced.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=9864

--- Comment #1 from roland <devzero at web.de> 2013-05-07 20:10:20 UTC ---
http://lists.samba.org/archive/rsync/2006-November/016822.html

http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=source-filter_dest-filter.diff;h=7de358b9c4e567191aa11759822f6fd0c54d307b;hb=HEAD

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=9864

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX

--- Comment #2 from Wayne Davison <wayned at samba.org> 2014-01-19
23:27:18 UTC ---
I think this is better solved by a compressing filesystem, such a btrfs or a
fuse overlay.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=9864

Haravikk <me at haravikk.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WONTFIX                     |

--- Comment #3 from Haravikk <me at haravikk.com> 2014-01-20 12:51:56 UTC
---
(In reply to comment #2)
> I think this is better solved by a compressing filesystem, such a btrfs or
a
> fuse overlay.
Normally I'd agree with you, but the availability is still surprisingly
poor;
for example most of my rsync use is onto a Synology NAS, but all it seems to
support is ext4 with no real ability to customise it (trying to change mount
options almost always causes it to assume the disk(s) have gone bad =/).
Granted that's poor design on Synology's part, but I've worked with
plenty of
other limited environments where compression would have been nice.

Since rsync already has compression built-in when transferring data, it seems
like a good idea to take fuller advantage of it.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=9864

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |WONTFIX

--- Comment #4 from Wayne Davison <wayned at samba.org> 2014-01-20
20:05:53 UTC ---
This is not something I'm going to support.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.