I've decided to proceed down another path to solve this problem, at least until rsync is able. I've begun to code a perl script that keeps a lookup list of file and directory names to inodes. What will occur is the pointer list check will be run just before the rsync job and perform name changes and moves before rsync has it's chance. I've come to realize how complex this task is. Using inodes is a help, but linux likes to re-use deleted file/folder inodes. I've considered this problem, but feel it won't really be a problem as rsync runs right after and will replace files that were improperly moved instead of just being deleted and a new file sync'ed. I did have another question, that is an rsync issue. Is there any way to change the dry-run option to display a probable transfer size if the command was run. I would like to be able to stop a possible rsync job if there is more then a particular threshold of files to be synchronized. Maybe there is another way to do this? I would like the system to say, for example, 'looks like there's 400MB's to transfer so cancel the transfer and email the admin instead'. Thanks. Frank Thomas -------------- next part -------------- HTML attachment scrubbed and removed
On 10/9/07, Frank Thomas <fthomas@radiology.ca> wrote:> I've decided to proceed down another path to solve this problem, at least > until rsync is able.You aren't happy with --detect-renamed ?> Is there any way to > change the dry-run option to display a probable transfer size if the command > was run.If you pass --stats, rsync will print a "Total transferred file size", which is the total size of all regular files that would be transferred. The amount of data that rsync would actually send over the network may be much less due to delta transfers, but there's no way to predict this amount in advance. As an alternative, you could go ahead and attempt a real run but limit the total amount of data sent by interposing a "head -c <nbytes>" into the connection. If the limit is hit before the run finishes, "head" will exit and the connection will be dropped, allowing you to take corrective action; be sure to use --delete-after to avoid premature deletion of destination files whose source counterparts have been moved. Reply if you would like more help setting this up. Matt
On 10/10/07, Frank Thomas <fthomas@radiology.ca> wrote:> First, I'm not sure how to safely compile this into rsyncI don't know what you mean by "safely". To get an rsync executable with the patch, download an rsync source package from http://rsync.samba.org/ (your choice of the stable rsync 2.6.9 or a snapshot or prelease of rsync 3.0.0), extract it, apply the patch "patches/detect-renamed.diff", ./configure, and make.> second, has this patch been widely tested?I imagine that it has been lightly tested. You can help by testing it some more!> Secondarily, I would love more help on setting up to limit transfer > sizes. I tried the --stats option, but I didn't seem to get the results > that I was looking for. Maybe I made a mistake, as I used the -n option > to perform a dry run and it showed me a zero amount.Was the "Total transferred file size" zero? That just means that no regular files would be transferred, presumably because none of the source files had changed since the last run. If the "Total transferred file size" was zero but rsync itemized transfers, then something is probably wrong. Matt
On 10/11/07, Frank Thomas <fthomas@radiology.ca> wrote:> Thanks Matt for your reply. What my problem is, I didn't install rsync > from source, but through a rpm package. I wonder if it is possible to > apply the patch and re-create the rpm, to minimize the work involved.Yes. Obtain the rsync source RPM from your distribution, add "patch -p1 <patches/detect-renamed.diff" to the end of the %prep section in the spec file, and rebuild the RPM. There's general information about how to rebuild RPMs at: http://bradthemad.org/tech/notes/patching_rpms.php Matt
On 10/11/07, Frank Thomas <fthomas@radiology.ca> wrote:> Thanks Matt again for the info. So now I'm feeling foolish. I tried to just pull the latest version of this from the cvs and I keep getting the following errors when I try to click onto 'download'.Yes, the cvsweb seems to be broken; I have reported the problem to the Samba webmaster. However, I don't understand why you would want to download just detect-renamed.diff from the CVS. For best results, you should use the same version of detect-renamed.diff as the rest of the source files; you can either check out the entire CVS tree with a CVS client or download a recent release or snapshot tarball from http://rsync.samba.org/ftp/rsync/ . And you only need to do any of this if you want to replace the source tarball included in the source RPM with something newer. Matt
BTW, the detect-renamed.diff in the current CVS rsync seems to be very badly broken, so don't use it yet. I have been working on a fixed version. Matt On 10/12/07, Frank Thomas <fthomas@radiology.ca> wrote:> To be honest, I didn't even think of doing that. I was specifically looking for the patch. I didn't even know that it was available in that tarball. I have really no experience in making or modifying rpm's. New stuff to learn! > > Frank. > > ________________________________ > > From: hashproduct@gmail.com on behalf of Matt McCutchen > Sent: Thu 10/11/2007 5:58 PM > To: Frank Thomas > Cc: rsync@lists.samba.org > Subject: Re: Recognizing directory name changes > > > > On 10/11/07, Frank Thomas <fthomas@radiology.ca> wrote: > > Thanks Matt again for the info. So now I'm feeling foolish. I tried to just pull the latest version of this from the cvs and I keep getting the following errors when I try to click onto 'download'. > > Yes, the cvsweb seems to be broken; I have reported the problem to the > Samba webmaster. > > However, I don't understand why you would want to download just > detect-renamed.diff from the CVS. For best results, you should use > the same version of detect-renamed.diff as the rest of the source > files; you can either check out the entire CVS tree with a CVS client > or download a recent release or snapshot tarball from > http://rsync.samba.org/ftp/rsync/ . And you only need to do any of > this if you want to replace the source tarball included in the source > RPM with something newer. > > Matt > ************************************************************************************************************************************ > Warning: This message and any attachments are intended only for the use of the individual or entity to which they are addressed and may contain confidential and personal information which may be subject to the "Freedom of Information and Protection of Privacy (FOIPP) Act" and other legislations. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited and may violate privacy and copyright laws. If you are not the intended recipient, please notify the sender immediately by return phone or fax, and delete any electronic or hard copy in your possession. Thank-you. > ************************************************************************************************************************************ > > >
On 10/12/07, Frank Thomas <fthomas@radiology.ca> wrote:> Thank you for telling me of it's issues. When the patch approaches testing, please contact me and I can setup a test environment to test transfers in the Gig's.I have fixed the patch and Wayne has committed my fixes to the CVS. You may test it now. Matt