Hi. I work at an online content provider. We have magazine articles in thousands of files stored on our servers. Files are added to the repository on a regular basis. I use rsync extensively (thank you to the authors) to periodically copy newly added files to my local copy of the repository for development purposes. rsync wonderfully reduces the amount of file transfer necessary. The size of the repository has grown dramatically, which is fine on the servers but putting a strain on my system (a PowerBook). So I added support to our app for compressed versions of the files. However, now all the files locally are suffixed with ".gz", whereas all the files on the server are not. I would like to continue to use rsync, but tell it that a file locally ending in ".gz" matches a file of the same name on the server without. It can't use checksums, obviously, because the targets are compressed. Is there any way to do this in rsync? If not, may I request it as a feature? I'd be happy to implement it myself, but there's just no time to familiarize myself with the code and do it. Related to that, it would be cool to tell rsync to run the files through gzip (or any other program) as it writes them. Both of these features should be generalized, of course. TIA! -- Rick
On Thu, Jan 20, 2005 at 09:57:59PM -0800, Rick Mann wrote:> Is there any way to [have locally-compressed files] in rsync?No, not at this time. To implement it would require some new kind of a "quick check" algorithm, which would probably require a database of some kind (the existing algorithm compares the file's modified-time and its size, so that won't work for having a shorter file on one side of the transfer). Nothing like this is currently planned, but others have written tools that talk the rsync protocol and interface to a database instead of a filesystem. Another option would be to use some kind of a compressed filesystem. ..wayne..
--- Wayne Davison wrote:> Another option would be to use some kind of a compressed filesystem.Do you know of one that works on Linux? I searched for this a few months back but came up empty. Thanks, Joe ===="An eye for an eye soon makes the whole world blind." --Mahatma Gandhi
israel Gold
2005-Jan-23 10:44 UTC
Transferring large amount of files w/ Rsync - Initial transfer
Hi, The following problem has appeared in the past but I have not seen satisfactory solution. It is common when synchronizing large amount of files over slow link. I would like to synchronize 1TB of data between local and remote sites over WAN. For the initial synchronization, I would like to transfer the files from the local site to the remote site using tapes, and use Rsync only for the incremental synchronization. After copying the files from the tapes at the remote site, it would have been nice if Rsync had an option that would scan the files and generate the checksums/hash values necessary for the incremental synchronization step. (so that I would not need to use Rsync for the initial transfer). If this was an option, do I need to generate the checksums/hash values on the local site prior to incremental synchronization ? Is it complicated to add to Rsync an option that would just generate checksums/hash values of files so that the solution above becomes feasible ?. -- IsraeL Gold, WiGo Technologies, rhglaw1@barak.net.il -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html