similar to: Rsync: Re: patch to enable faster mirroring of large filesyst ems

I was at first, but then removed it. The results were still insufficiently fast. > Were you using the -c option of rsync? It sounds like you > were and it's > extremely slow. I knew somebody who once went to > extraordinary lengths to > avoid the overhead of -c, making a big patch to rsync to > cache checksums, > when all he had to do was not use -c.

In my particular case, it is reasonable to assume that the size and timestamp will change when the file is updated. (We are looking at it as a patching mechanism.) Right now it's actually using update time only, I should modify it to check the file size as well. Is there a way you could query your database to tell you which extents have data that has been modified within a certain timeframe?

Keating, Tim [TKeating@origin.ea.com] writes: > - If there's a mismatch, the client sends over the entire .checksum > file. The server does the compare and sends back a list of files to > delete and a list of files to update. (And now I think of it, it > would probably be better if the server just sent the client back the > list of files and let the client figure out what it

Keating, Tim [TKeating@origin.ea.com] writes: > Is there a way you could query your database to tell you which > extents have data that has been modified within a certain timeframe? Not in any practical way that I know of. It's not normally a major hassle for us since rsync is used for a central backup that occurs on a large enough time scale that the timestamp does normally change

Not sure, I am re-running this to ensure that I was not smoking crack at the time :) > -----Original Message----- > From: Dave Dykstra [mailto:dwd@bell-labs.com] > Sent: Thursday, November 29, 2001 1:12 PM > To: Keating, Tim > Cc: rsync@samba.org > Subject: Re: Rsync: Re: patch to enable faster mirroring of large > filesyst ems > > > On Thu, Nov 29, 2001 at

I just took a look at the 2.5.5 codebase to see how easy it would be to write a little driver script that downloads a big directory tree from an rsync daemon the chunky way (get a list of a module's subdirectories and do the transfer by subdirectory). The reason for doing this is obvious when you have large directory trees, as is the case for many of us. Unfortunately the way list_only is

Rsync list: Alberto and I have done a couple more exchanges by private email, and we found that he wasn't turning on my include/exclude optimization in his test because he had an "exclude" directive in rsyncd.conf. He has now removed that and run the test again. His very interesting results are below with my comments. Note that his case is rather pathological because he's got

Dear all, I've been trying to track down a problem with timeouts when pulling data from an rsync daemon and I have now run out of any useful ideas. The problem manifests itself when I try to transfer a large directory tree on a slow client machine. What happens then is that the client rsync process successfully receives the list of files from the server, then begins checking the local

Chris, to put things in the right prespective, you should read (if you haven't done so already) the original paper describing the design behind batch mode. The design and implementation of this functionality goes back to a project called the Internet2 Distributed Storage Infrastructure (I2-DSI). As part of that project, the authors created a modified version of rsync (called rsync+)

Hi, I am thinking about a non-standard usage of rsync (at least not mentioned in the man file) I want to synchronized my collegues home directories(trees) each night AND store rsync's internal updating commands (reversed) to be able to restore the state of a directory the day before. This would require - saving the internal updating commands in a separate directory - reversing these commands

I am puzzled as to why I can't get the option --link-dest to work properly. When I use this option when both source and destinations are on a local filesystem the hard-linking of the target against the link-dest directory does work, but when the source is a remote directory (via ssh or rsync server) hard links are not created. I suspect it has something to do with setting the correct

Wayne, I have been using the rsync snapshot from 1/25 for the last few days and everything seems quite solid so far. I include below a few nit-picks in case you're looking for thing to tidy up. I'm also looking forward to hearing news about the --filter option. I'm thinking it is going to be very useful for backup purposes. What I would want to be able to do is, on a

Hi Everybody, I'm new to this list, but I have been using rsync for quite some time. First, congratulations to the rsync team for a very fine piece of software! I'm wondering whether rsync could help me to perform the following task: I have 5 million files on one side of the ocean, 100000 of which must be copied to the other side. Both numbers grow with time, and occasionally, some

Our project is considering supporting a mirror site which is going to be off the network (essentially a stand-alone mirror for a local LAN in a place without internet connectivity). So I am in the (unfortunate) position of having to decide how to do this (or if this can be done at all). The current plan is to set up a PC running linux with a 120GB drive and a DVD reader on the remote site and

At the moment, if you start an rsyncd that's not running as root using default settings it will have some trouble. rsyncd tries to use chroot by default, but this will always fail if it's not started by root. It does emit an error message in this case, but I wonder if some people find this a bit confusing until they discover the setting. I have in the past. It might be better that if

Is there a good place to get information about the list of all possible output and error messages rsync generates? Or should I just muck around the source code (which I haven't looked at yet) and find them? I am doing something where I would like to parse rsync's output using Perl into a set of data structures. I already have something that works under normal conditions. Eventually

I've just started using rsync, and it's way cool. I checked the FAQ, the broken mailing list archive search function, and scanned the archives, but didn't see this mentioned. I presume it's a fairly common need: I'm mirroring files automatically and daily from one site to another, and anticipate the day when the size of the files may exceed the size of the destination disk.

Hello! As I was found rsync do not detect file renaming. If I just copy my backup.0.tgz (many Mbytes in size having it's md5) to backup.1.tgz (which will be equial in size and md5) it will be the same file in fact... Rsync will delete old file (backup.0.tgz) on the receiving side and download new one (backup.1.tgz). I do not think there are any difficulties to detect this situation and

Is there any chance this can be added into the distribution as it sounds really nifty. Another suggestion unless I have read the following - would it be useful to have a command option in rsync to generate the file list by doing the "find" and outputting into a standard format? (As this would make it less OS specific or kludgy?) Cheers, Lachlan. At 16:06 19/11/01 -0500, you wrote:

I have attached a patch that adds 4 options to rsync that have helped me to speed up my mirroring. I hope this is useful to someone else, but I fear that my relative inexperience with rsync has caused me to miss a way to do what I want without having to patch the code. So please let me know if I'm all wet. Here's my story: I have a large filesystem (around 20 gigabytes of data) that