On Thu, Jan 06, 2005 at 02:59:44AM +0800, Jeff Pitman
wrote:> The attached patch (extremely ugly) attempts to resolve this by
> foregoing the rename step until the end.
Thanks for the patch. It looks like it implements the basics of the
idea, but I think that it will be better to use deterministic temp-file
names like the --partial-dir logic gives us. I also chose to use an
option name of --delay-renames. So, I've created my own version of this
post-transfer renaming heuristic, which is available from here:
http://rsync.samba.org/ftp/unpacked/rsync/patches/delay-renames.diff
The patch is relative to the very latest CVS or "nightly" tar file.
One unresolved issue I thought of is how rsync should deal with an
absolute --partial-dir pathname: the above implementation woul put
all the partial files into that one dir (a relative pathname, like
the default, ensures that the files get sprinkled around).
> The first big [problem] in running the testsuite is that the parent
> dirs mtime is not getting preserved. [...] Next, hardlinks need some
> work.
These problems stem from a bug in the synchronization between the
receiver and the generator that I just discovered: the generator does
not wait until all the redo & delete-after work is done before it begins
the final hard-link pass and the final directory-time-tweaking pass.
Because of this, the extra renaming loop added at the end of the
receiver's work didn't get done before the generator got going on the
hard-linking and directory-time tweaking. I have checked-in a fix for
this, so the above delay-renames.diff works fine with a test-suite that
has been modified to use --delay-renames on all the possible commands.
Here's some of the design items that the patch deals with:
- Complain if the user combines --inplace with --delay-renames.
- A client sender needs to pass the new option to a server receiver.
- We use the --partial-dir logic to make all the temp-file names
deterministic, using the default name of ".~tmp~" if the user did
not specify the --partial-dir option. Note that these dirs get
auto-excluded by the normal --partial-dir handling, so they will not
be transferred by default, and they will not be deleted by --delete
or --delete-after if the prior transfer did not complete and some
temp files got left around (they will also be used to speed up the
transfer when it is restarted).
The patch also fixes/changes a few things from your patch:
- The rename loop must happen prior to the delete-after logic in the
receiver.
- It is good to avoid twiddling the flist->flags bits in the forked
receiver because it will cause all the shared file-list memory
(between the generator and the receiver) to become unshared (on
systems that use copy-on-write memory when forking).
- It is good to avoid adding large amounts of memory to the flist
object (adding MAXPATHLEN chars of extra memory for each file
transferred would bloat rsync's memory by a huge amount for large
transfers).
I've attached a 3-hunk patch that kluges in the use of the new
--delay-renames option into all the (relevant) test-suite rsync
commands (for temporary testing). Apply it after the main patch,
if desired.
..wayne..
-------------- next part --------------
--- options.c 2005-01-10 10:16:54 +0000
+++ options.c 2005-01-10 10:30:07 +0000
@@ -456,6 +456,7 @@
{"protocol", 0, POPT_ARG_INT, &protocol_version, 0,
0, 0 },
{"server", 0, POPT_ARG_NONE, &am_server, 0, 0, 0
},
{"help", 'h', POPT_ARG_NONE, 0, 'h',
0, 0 },
+ {"delay-renames", 0, POPT_ARG_NONE, 0, 0, 0, 0 },
{0,0,0,0, 0, 0, 0}
};
@@ -947,6 +948,8 @@
bwlimit_writemax = 512;
}
+ if (inplace)
+ delay_renames = 0;
if (delay_renames && !partial_dir)
partial_dir = ".~tmp~";
--- runtests.sh 2004-02-04 05:34:29 +0000
+++ runtests.sh 2005-01-10 10:27:33 +0000
@@ -168,7 +168,7 @@
exit 2
fi
-RSYNC="$rsync_bin"
+RSYNC="$rsync_bin --delay-renames"
export rsync_bin RSYNC