rsync - Aug 2008 - File Locking

First let me say that rsync is a wonderful program and thank you all for the
hard work that went into it's creation.  On to my question.  I have a
situation that requires the files that rsync is uploading/downloading to be
locked.  The reason is because it is possible to have more than one copy of
rsync running and without file locking, the additional copies simply retry to
upload/download a file that a previous copy is already working on.  What I am
hoping for is a way for rsync to notice that the file is already being copied
and therefore move on to the next file.

I found a previous post from 2005 where someone made a patch for rsync that used
flock to lock files but I can't find any other mention of the patch?  Is
this possible with rsync perhaps via a patch?  If not, any other ideas?  By the
way, I am using rsync on Windows under cygwin.

Thanks to all.

-Kyle

A different Kyle wrote:
> I have a situation that requires the files that rsync is
> uploading/downloading to be locked.  The reason is because it is possible
> to have more than one copy of rsync running and without file locking,
> the additional copies simply retry to upload/download a file that a
> previous copy is already working on.
The way I solved this problem for a data-mirroring system was to use a
small wrapper script that ensures only one invocation of rsync is ever
running at one time. This proved to be a robust solution for our situation.

--Kyle

> A different Kyle wrote:
> I have a situation that requires the files that rsync is
> uploading/downloading to be locked.  The reason is because it is possible
> to have more than one copy of rsync running and without file locking,
> the additional copies simply retry to upload/download a file that a
> previous copy is already working on.
>> The way I solved this problem for a data-mirroring system was to use a
>> small wrapper script that ensures only one invocation of rsync is ever
>> running at one time. This proved to be a robust solution for our
situation.
>> --Kyle
Thanks for the idea but I sometimes need more than one copy to run at the same
time so that won't work.  File locking is about the only option I can see.

Thank you.

On Mon 18 Aug 2008, [EMAIL PROTECTED] wrote:
> 
> >> The way I solved this problem for a data-mirroring system was to
use a
> >> small wrapper script that ensures only one invocation of rsync is
ever
> >> running at one time. This proved to be a robust solution for our
situation.
> 
> >> --Kyle
> 
> Thanks for the idea but I sometimes need more than one copy to run at the 
> same time so that won't work.  File locking is about the only option I
can
> see.
>> If you need to run more than one rsync over the same tree at the same
>> time, you may need to rethink your tree layout and/or your approach to
>> whatever problem you're trying to fix.  Running two rsyncs
sequentially
>> will usually be faster than two concurrent ones, as that might cause
>> disk thrashing (the heads are continually seeking to and from where the
>> respective rsync processes are working).
>>  Paul Slootman
Thanks again, but that is still not what I am looking for.  Perhaps more info is
needed.

I have an identical set of directories at two locations.  When a file is added
to one location, I'll call it the source side, I want to run a script that
picks up that file and copies it to the other location, say the destination
side.  Simple enough.

However, I want to schedule the script to run, say every 15 minutes.  That way
if a file is put on the source side, the cript will pick it up and begin copying
it.  However, if the file is a few hundred MB, it might take longer than 15
minutes to copy it.  So the next time the script runs, I need rsync to skip that
file since it is still being copied from the first run and move to the next
file.  That same thing might be repeated during the next run.

In other words, I can't wait until the first run has completed the large
copy to begin copying additional files.  I want to start a second, third,
fourth, etc copy that begins working on any additional files that may have been
placed on the source side.


Thank you all again for the input, but it still looks to me like I need some
type of file locking.  Again, all other input is welcome because there might be
a better way.

- Kyle