rsync - Sep 2007 - compression of source and target files

Hi - there's a flag for rsync to compress the files in transit - is it
possible
to compress one side (target) with gzip and have rsync still work
correctly?

Any help would be appreciated.

-- Ken

Chuck Wolber wrote:
> On Fri, 21 Sep 2007, Kenneth Simpson wrote:
>
>   
>> Hi - there's a flag for rsync to compress the files in transit - is
it
>> possible to compress one side (target) with gzip and have rsync still 
>> work correctly?
>>     
>
> It'll still work correctly, but compressing a compressed file can
actually
> make it slightly bigger and wastes CPU cycles in the process.
>
> ..Chuck..
>
>   
 Sorry, I neglected to mention the source is uncompressed but
we need to compress the target file because we're running out
of disk space and the files are highly compressible.


We can't compress the source since the files are large and
compressing the source would create other problems.
 
The original thought was to use a file system with compression (I
think Linux has such a beast) but this would at least require a
kernel rebuild which we won't be able to do for awhile.

The second thought was that we might be able to gzip on the fly
and have rsync work correctly (since it's compressing them in
transit.)

On Fri, 21 Sep 2007, Kenneth Simpson wrote:
> Hi - there's a flag for rsync to compress the files in transit - is it 
> possible to compress one side (target) with gzip and have rsync still 
> work correctly?
It'll still work correctly, but compressing a compressed file can actually 
make it slightly bigger and wastes CPU cycles in the process.

..Chuck..


-- 
http://www.quantumlinux.com
 Quantum Linux Laboratories, LLC.
 ACCELERATING Business with Open Technology

"The idea that any one of us [presidential candidates] can bring about
this change is a fantasy, it is not the truth! We need you to bring about
the change on all these issues, we need you involved, we need you taking
responsibility!"
		--John Edwards

Hi Ken,

no, that's nothing rsync supports.

indeed, it would be nice and great for backups - but for now
you need a filesystem which supports transparent compression
to have your rsync`ed data compressed.

on linux, watch out for reiser4 or zfs-fuse

regards
roland


----- Original Message ----- 
From: "Kenneth Simpson" <ken@clearshape.com>
To: <rsync@lists.samba.org>
Sent: Friday, September 21, 2007 9:24 PM
Subject: compression of source and target files

> Hi - there's a flag for rsync to compress the files in transit - is it
> possible
> to compress one side (target) with gzip and have rsync still work
> correctly?
>
> Any help would be appreciated.
>
> -- Ken
>
> -- 
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

oh, this is interesting patch - thanks for giving pointer.

i have tried it and it and it looks interesting, but somewhat incomplete.

i can transfer remote files to a local dir and they are being compressed on the
local side, but (quite logical) this breaks size/content-checking. 

being also mentioned in the patch:

+Use of --dest-filter automatically enables --whole-file.
+If your filter does not output the same number of bytes that it
+received on input, you should use --times-only to disable size and
+content checks on subsequent rsync runs.

so, just deciding transfer based on timestamp is a little bit unsafe for me.

i tried "--checksum" but it didn`t work as expected.

seems this option needs to be touched/enhanced to work with 
--dest-filter/--source-filter or maybe even size/content-checking 
can be implemented by just making sure rsync takes the uncompressed
size for comparison. this will use much cpu, though....

regards
roland

> -----Urspr?ngliche Nachricht-----
> Von: "Matt McCutchen" <hashproduct+rsync@gmail.com>
> Gesendet: 23.09.07 04:11:43
> An: "Kenneth Simpson" <ken@clearshape.com>
> CC: rsync@lists.samba.org
> Betreff: Re: compression of source and target files
> 
> On 9/21/07, Kenneth Simpson <ken@clearshape.com> wrote:
> >  Sorry, I neglected to mention the source is uncompressed but
> > we need to compress the target file because we're running out
> > of disk space and the files are highly compressible.
> 
> You might try the experimental patch
"source-filter_dest-filter.diff"
> that comes in patches/ of the rsync source distribution.  It adds
> options that claim to do what you want, but I have never tested it;
> your mileage may vary.
> 
> Matt
> -- 
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> 

_______________________________________________________________________
Jetzt neu! Sch?tzen Sie Ihren PC mit McAfee und WEB.DE. 3 Monate
kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220