rsync - Oct 2003 - Patch to revive tmpfiles

This is a patch to fix one annoyance of having rsync processes race:

I usually keep our servers synced with the following script, run by cron.
#!/bin/sh
lockfile -r 2 -l 1000 /tmp/synchome.lock || exit 1
rsync -e ssh -avHP --delete zorro01:/home/\* /home >/dev/null
rm -f /tmp/synchome.lock
--

Sometimes my users (including myself) are in a hurry and syncronise files and
directories in their homes themselves. This bypasses the lock above and two or
more rsync processes are in a race.

What happens is that the competing rsync processes delete each others temporary
files a lot of network traffic is wasted since rsyncing fails when the temp file
is deleted.

The included patch tries to remedy this situation by reviving the temp file.

Patch (Also to be found at: http://xinit.se/~hes/rsync-2.5.6-revive.patch):
--- receiver.c.orig     2003-01-21 00:32:17.000000000 +0100
+++ receiver.c          2003-10-06 14:30:35.000000000 +0200
@@ -457,13 +457,21 @@
                if (fd1 != -1) {
                        close(fd1);
                }
-               close(fd2);
+
+               /* Revive temp file if someone removed it during receive */
+               fd1 = do_open(fnametmp, O_RDONLY, 0);
+               if (fd1 == -1) {
+                       sprintf(fnametmp, "/proc/self/fd/%d", fd2);
+               } else {
+                       close(fd1);
+               }

                if (verbose > 2)
                        rprintf(FINFO,"renaming %s to
%s\n",fnametmp,fname);

                finish_transfer(fname, fnametmp, file);

+               close(fd2);
                cleanup_disable();

                if (!recv_ok) {
--

Hans Eric Sandstr?m
MailCORE AB

ps. A --lock option to mutually exclude updates to the same file from competing
rsyncs would be a nice feature. That would sort of obsolete the lockfile trick
above.

On Mon, Oct 06, 2003 at 03:22:04PM +0200, Hans Eric Sandstr?m
wrote:
> This is a patch to fix one annoyance of having rsync processes race:
> 
> I usually keep our servers synced with the following script, run by cron.
> #!/bin/sh
> lockfile -r 2 -l 1000 /tmp/synchome.lock || exit 1
> rsync -e ssh -avHP --delete zorro01:/home/\* /home >/dev/null
> rm -f /tmp/synchome.lock
> --
> 
> Sometimes my users (including myself) are in a hurry and syncronise files
and directories in their homes themselves. This bypasses the lock above and two
or more rsync processes are in a race.
> 
> What happens is that the competing rsync processes delete each others
temporary files a lot of network traffic is wasted since rsyncing fails when the
temp file is deleted.
> 
> The included patch tries to remedy this situation by reviving the temp
file.
Interesting idea.  Very platform dependant though.
> 
-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

Do stay on list.

On Wed, Oct 08, 2003 at 11:54:44PM +0200, Hans Eric Sandstr?m
wrote:
> Do you think there should be something like:
> #ifdef HAVEPROCFS
> <patch>
> #endif
> Around the patch.
It would have to be discussed.  But something like that
could be considered.

It isn't just a matter of whether they have procfs, it also
matters whether /proc is organised the same way and had the
file descriptors.  Some /proc filesystems don't have
/proc/self.  Others don't have /proc/*/fd but do have
/dev/fd*.  Further, there have been discussions on
linux-kernel regarding changing /proc/fd and even getting
rid of or modifying how /proc/self is done.

What would probably be most suitable would be to have
autoconf generate one of two printf format strings so
you could generate the path something vaguely like this:

| #if defined(PROC_SELF_FD) || defined(PROC_PID_FD)
| /* check for missing tempfile */
| #if defined(PROC_SELF_FD) 
| 	sprintf(fnametmp, PROC_SELF_FD, fd);
| #elif defined(PROC_PID_FD)
| 	sprintf(fnametmp, PROC_PID_FD, getpid(), fd);
| #endif
| #endif

> A fdlink( fd, filename) syscall would also have been nice to have. But no
such
> beast seems to exists in any OS i use.
And you won't see it happen.  This was proposed a few months
ago on lkml and it was rightly shot down on the basis of
security issues.
> On the other subject (locking):
> What do you think about taking an exclusive lock on the destination file.
To
> avoid two different rsyncs overrunning each other. Actually there is no
> locking att all in the code and the copying should preferrably be atomic.
> 
> The exclusive lock would block one of the rsync processes. A nonblocking
lock
> would have to handle a write error to the locked file. I looked at the code
> but could'nt really follow how errors where handled so I did'nt try
that.
> 
> Then there is a catch 22 in the code when the new file does'nt exist.
Is it
> safe to create an empty desitnation file to take the lock on.
I'm not too keen on adding locking.  It gets into issues of
making sure locks are released and in some cases filesystem
limitations.  We are talking about a corner case that
as far as i can tell doesn't really cause more problems than
that of wasting resources with multiple syncs.

The disconnected tempfile reconnection is only interesting
to me because of the oddness i imagine must be happening
when the tempfile disappears out from under rsync.  I'm not
sure if i wouldn't rather just have rsync bomb out at that
point with a good error message such as "Tempfile for %d
missing -- probable rsync collision".
> 
> Hans Eric
> 
> ----- Original Message ----- 
> From: "jw schultz" <jw@pegasys.ws>
> To: <rsync@lists.samba.org>
> Sent: Wednesday, October 08, 2003 10:56 PM
> Subject: Re: Patch to revive tmpfiles
> 
> 
> On Mon, Oct 06, 2003 at 03:22:04PM +0200, Hans Eric Sandstr?m wrote:
> > This is a patch to fix one annoyance of having rsync processes race:
> >
> > I usually keep our servers synced with the following script, run by
cron.
> > #!/bin/sh
> > lockfile -r 2 -l 1000 /tmp/synchome.lock || exit 1
> > rsync -e ssh -avHP --delete zorro01:/home/\* /home >/dev/null
> > rm -f /tmp/synchome.lock
> > --
> >
> > Sometimes my users (including myself) are in a hurry and syncronise
files
> and directories in their homes themselves. This bypasses the lock above and
> two or more rsync processes are in a race.
> >
> > What happens is that the competing rsync processes delete each others
> temporary files a lot of network traffic is wasted since rsyncing fails
when
> the temp file is deleted.
> >
> > The included patch tries to remedy this situation by reviving the temp
file.
> 
> Interesting idea.  Very platform dependant though.
> 
> >
> 
> -- 
> ________________________________________________________________
> J.W. Schultz            Pegasystems Technologies
> email address: jw@pegasys.ws
> 
> Remember Cernan and Schmitt
> -- 
> To unsubscribe or change options:
> http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> 
-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt