rsync - Jan 2005 - Transforming file names & contents

Hi. I work at an online content provider. We have magazine articles in 
thousands of files stored on our servers. Files are added to the 
repository on a regular basis. I use rsync extensively (thank you to 
the authors) to periodically copy newly added files to my local copy of 
the repository for development purposes. rsync wonderfully reduces the 
amount of file transfer necessary.

The size of the repository has grown dramatically, which is fine on the 
servers but putting a strain on my system (a PowerBook). So I added 
support to our app for compressed versions of the files.

However, now all the files locally are suffixed with ".gz", whereas
all
the files on the server are not. I would like to continue to use rsync, 
but tell it that a file locally ending in ".gz" matches a file of the 
same name on the server without. It can't use checksums, obviously, 
because the targets are compressed.

Is there any way to do this in rsync? If not, may I request it as a 
feature? I'd be happy to implement it myself, but there's just no time 
to familiarize myself with the code and do it.

Related to that, it would be cool to tell rsync to run the files 
through gzip (or any other program) as it writes them. Both of these 
features should be generalized, of course.

TIA!

-- 
Rick

On Thu, Jan 20, 2005 at 09:57:59PM -0800, Rick Mann
wrote:
> Is there any way to [have locally-compressed files] in rsync?
No, not at this time.  To implement it would require some new kind of a
"quick check" algorithm, which would probably require a database of
some
kind (the existing algorithm compares the file's modified-time and its
size, so that won't work for having a shorter file on one side of the
transfer).  Nothing like this is currently planned, but others have
written tools that talk the rsync protocol and interface to a database
instead of a filesystem.  Another option would be to use some kind of a
compressed filesystem.

..wayne..

--- Wayne Davison wrote:
>  Another option would be to use some kind of a compressed filesystem.
Do you know of one that works on Linux? I searched for this a few months back
but came up empty.

Thanks,
Joe

===="An eye for an eye soon makes the whole world blind."
   --Mahatma Gandhi

Hi,

The following problem has appeared in the past but I have not seen
satisfactory solution. 
It is common when synchronizing large amount of files over slow link. 

I would like to synchronize 1TB of data between local and remote sites
over WAN.
For the initial synchronization, I would like to transfer the files from
the local site to the remote site using tapes,
and use Rsync only for the incremental synchronization.

After copying the files from the tapes at the remote site, it would have
been nice if Rsync had an option that would scan
the files and generate the checksums/hash values necessary for the
incremental synchronization step.
(so that I would not need to use Rsync for the initial transfer).
If this was an option, do I need to generate the checksums/hash values
on the local site prior to incremental synchronization ?

Is it complicated to add to Rsync an option that would just generate
checksums/hash values of files so that 
the solution above becomes feasible ?.
--
IsraeL Gold, WiGo Technologies, rhglaw1@barak.net.il
-- 
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html