rsync - Dec 2009 - preventing multiple rsync processes backing up same file with cron backup

If I have an rsync cron job that runs say every 15 minutes, what happens if
there is a file that process 1 has started to copy across, but hasn't
finished before the next process starts?

Does rsync ignore this partially copied file, or does it try to copy it
across again in the second cron job?

Many thanks in advance,

Chris 
-- 
View this message in context:
http://old.nabble.com/preventing-multiple-rsync-processes-backing-up-same-file-with-cron-backup-tp26897620p26897620.html
Sent from the Samba - rsync mailing list archive at Nabble.com.

On Tue, Dec 22, 2009 at 10:56:21PM -0800, snowc wrote:
> 
> If I have an rsync cron job that runs say every 15 minutes, what happens if
> there is a file that process 1 has started to copy across, but hasn't
> finished before the next process starts?
> 
> Does rsync ignore this partially copied file, or does it try to copy it
> across again in the second cron job?
Probably it is not an rsync job to prevent such cases. You can put to crontab
simple shell
script which will do proper locking and execute rsync only if there is no other
instances.

For example: http://sysadvent.blogspot.com/2009/12/cron-practices.html

On Wed, Dec 23, 2009 at 09:54:05AM +0000, chris snow
wrote:
> Say I want to backup a directory every 15 minutes from cron.  I wrap cron
in
> a script that does locking to ensure only one instance of the script runs
at
> any one time.
> 
> At some stage during the day, I have a file that is 1Gb that gets added to
> the directory that I am backing up, and then a lot of smaller files.
> 
> The next rsync job starts copying the big file, but it may take an hour to
> transfer the big file across the network and whilst that is happening, the
> other smaller files will have to wait.  I no longer have my directory
> synchronised every 15 minutes.
It is physically can not be done, because your network throughput does not allow
to sync
everything during 15 minute interval. Even if you sync small files, the bigger
ones still
not synchronized.

But you can try to play with --max-size and --min-size parameters and start two
jobs in parallel
(one for big files, other for small). Independend locking for each job type will
be required.

On Tue, Dec 22, 2009 at 10:56 PM, snowc <chsnow123 at gmail.com> wrote:
> Does rsync ignore this partially copied file, or does it try to copy it
> across again in the second cron job?
>
If the one job catches up with the other, they will both copy the same file,
each to its own temp file.  If you have delete processing going on, one
rsync may even delete a temp file for the other (if the delete scan happens
at the right time).

..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20091223/e1c5d027/attachment.html>