rsync - Feb 2007 - DO NOT REPLY [Bug 4383] New: Idea: Store files in compressed format and visa versa

https://bugzilla.samba.org/show_bug.cgi?id=4383

           Summary: Idea: Store files in compressed format and visa versa
           Product: rsync
           Version: 3.0.0
          Platform: Other
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: kilburna@iservnetworks.com
         QAContact: rsync-qa@samba.org


The -z compresses files or deltas in transist, but on the receiver side
(assuming push) files are stored in uncompressed form. What would be useful is
to store the file in compressed format much like gzip or bzip2 does. 

A switch to store compressed or uncompressed could be used in push or pull.

This might not be so easy because sync uses "context-compression" by
"priming"
the compressor with hits and discarding the compressed output, but I put it up
as an idea because we are storing >3TB of data.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=4383


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement




------- Comment #1 from wayned@samba.org  2007-02-10 23:43 MST -------
I think that it would probably be better to implement this outside of rsync,
such as creating a fuse filesystem that compresses the files that it receives. 
I know that at least one project exists to implement a compressing fuse
filesystem.

I'll leave this open for now to see if anyone wants to discuss this.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=4383


devzero at web.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |devzero at web.de




------- Comment #2 from devzero at web.de  2009-08-07 05:24 CST -------
maybe the patches/source-filter_dest-filter.diff is interesting for you !?

i did not test it, but it looks related to what you want.

+dit(bf(--source-filter=COMMAND)) This option allows the user to specify a
+filter program that will be applied to the contents of all transferred
+regular files before the data is sent to destination.  COMMAND will receive
+the data on its standard input and it should write the filtered data to
+standard output.  COMMAND should exit non-zero if it cannot process the
+data or if it encounters an error when writing the data to stdout.
+
+Example: --source-filter="gzip -9" will cause remote files to be
+compressed.
+Use of --source-filter automatically enables --whole-file.
+If your filter does not output the same number of bytes that it received on
+input, you should use --times-only to disable size and content checks on
+subsequent rsync runs.
+
+dit(bf(--dest-filter=COMMAND)) This option allows you to specify a filter
+program that will be applied to the contents of all transferred regular
+files before the data is written to disk.  COMMAND will receive the data on
+its standard input and it should write the filtered data to standard
+output.  COMMAND should exit non-zero if it cannot process the data or if
+it encounters an error when writing the data to stdout.
+
+Example: --dest-filter="gzip -9" will cause remote files to be
compressed.
+Use of --dest-filter automatically enables --whole-file.
+If your filter does not output the same number of bytes that it
+received on input, you should use --times-only to disable size and
+content checks on subsequent rsync runs.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.