rsync - Jan 2006 - propagate atimes with rsync-2.6.6 (fwd)

Dear Martin Pool.

We regularly use rsync for making backups of our file systems but we have
noticed that the atimes are not transferred with the files and are also
always updated on the sender's side. Therefore, we have created a modified
version of rsync based on rsync-2.6.6 protocol version 29 which transfers
the access times with the transferred files and also allows to preserve
the access times on the sending side. The transfer of the atimes is done
by default while the preservation of the source atimes is switched on by
an additional option (-X --atime-preserve). To guarantee compatibility
with previous protocol versions, we have introduced a new protocol version
(30). If the remote protocol version is older than 30, rsync behaves as
before. The changes made to the reference version are small and kept in
accordance with the original style. The affected files are:
	flist.c
	generator.c
	options.c
	proto.h
	rsync.1
	rsync.c
	rsync.h
	rsync3.txt
	sender.c
and	util.c
Please find the corresponding patch attached to this email.
We have thouroughly tested our new protocol under Linux and also compiled
it under Unix (Aix). It works as expected under Linux while some of the
changes were redundant under Unix.
We consider the preservation of the source access times as useful since
this information might be worth being backed-up as well.
We hope our approach will find your attention and would be glad if it were
included in the next update of rsync.
Please let us know about your suggestions for improvements.
Yours truly,

Christian Konz
-------------- next part --------------
diff -uNr rsync-2.6.6/flist.c rsync-2.6.6_patch/flist.c
--- rsync-2.6.6/flist.c	2005-07-07 21:49:14.000000000 +0200
+++ rsync-2.6.6_patch/flist.c	2006-01-07 14:04:38.000000000 +0100
@@ -54,6 +54,7 @@
 extern int implied_dirs;
 extern int copy_links;
 extern int copy_unsafe_links;
+extern int remote_protocol;
 extern int protocol_version;
 extern int sanitize_paths;
 extern int orig_umask;
@@ -314,6 +315,7 @@
 {
 	unsigned short flags;
 	static time_t modtime;
+	static time_t acctime;
 	static mode_t mode;
 	static int64 dev;
 	static dev_t rdev;
@@ -329,7 +331,7 @@
 
 	if (!file) {
 		write_byte(f, 0);
-		modtime = 0, mode = 0;
+		modtime = 0, acctime = 0, mode = 0;
 		dev = 0, rdev = makedev(0, 0);
 		rdev_major = 0;
 		uid = 0, gid = 0;
@@ -379,6 +381,8 @@
 	else
 		modtime = file->modtime;
 
+        acctime = file->acctime;
+
 #ifdef SUPPORT_HARD_LINKS
 	if (file->link_u.idev) {
 		if (file->F_DEV == dev) {
@@ -431,6 +435,8 @@
 	write_longint(f, file->length);
 	if (!(flags & XMIT_SAME_TIME))
 		write_int(f, modtime);
+	if (remote_protocol >= 30)
+		write_int(f, acctime);
 	if (!(flags & XMIT_SAME_MODE))
 		write_int(f, to_wire_mode(mode));
 	if (preserve_uid && !(flags & XMIT_SAME_UID)) {
@@ -506,6 +512,7 @@
 					      unsigned short flags, int f)
 {
 	static time_t modtime;
+	static time_t acctime;
 	static mode_t mode;
 	static int64 dev;
 	static dev_t rdev;
@@ -524,7 +531,7 @@
 	struct file_struct *file;
 
 	if (!flist) {
-		modtime = 0, mode = 0;
+    	        modtime = 0, acctime = 0, mode = 0;
 		dev = 0, rdev = makedev(0, 0);
 		rdev_major = 0;
 		uid = 0, gid = 0;
@@ -578,6 +585,10 @@
 	file_length = read_longint(f);
 	if (!(flags & XMIT_SAME_TIME))
 		modtime = (time_t)read_int(f);
+	if (remote_protocol >= 30)
+		acctime = (time_t)read_int(f);
+	else
+		acctime = time(NULL);
 	if (!(flags & XMIT_SAME_MODE))
 		mode = from_wire_mode(read_int(f));
 
@@ -630,6 +641,7 @@
 
 	file->flags = 0;
 	file->modtime = modtime;
+	file->acctime = acctime;
 	file->length = file_length;
 	file->mode = mode;
 	file->uid = uid;
@@ -882,6 +894,7 @@
 
 	file->flags = flags;
 	file->modtime = st.st_mtime;
+	file->acctime = st.st_atime;
 	file->length = st.st_size;
 	file->mode = st.st_mode;
 	file->uid = st.st_uid;
diff -uNr rsync-2.6.6/generator.c rsync-2.6.6_patch/generator.c
--- rsync-2.6.6/generator.c	2005-07-28 21:06:03.000000000 +0200
+++ rsync-2.6.6_patch/generator.c	2005-11-25 11:33:33.000000000 +0100
@@ -917,6 +917,13 @@
 						  fnamecmpbuf, 1,
 						  itemizing && verbose > 1,
 						  code) == 0) {
+					if (set_modtime(fname,file->modtime,
+							file->acctime) != 0) {
+						rsyserr(FERROR, errno, 
+							"failed to set times on %s",
+							full_fname(fname));
+						return;
+					}
 					if (preserve_hard_links
 					    && file->link_u.links) {
 						hard_link_cluster(file, ndx,
diff -uNr rsync-2.6.6/options.c rsync-2.6.6_patch/options.c
--- rsync-2.6.6/options.c	2005-05-19 10:52:17.000000000 +0200
+++ rsync-2.6.6_patch/options.c	2006-01-11 15:56:00.000000000 +0100
@@ -48,6 +48,7 @@
 int preserve_uid = 0;
 int preserve_gid = 0;
 int preserve_times = 0;
+int preserve_atime = 0;
 int omit_dir_times = 0;
 int update_only = 0;
 int cvs_exclude = 0;
@@ -286,6 +287,7 @@
   rprintf(F," -g, --group                 preserve group\n");
   rprintf(F," -D, --devices               preserve devices (root
only)\n");
   rprintf(F," -t, --times                 preserve times\n");
+  rprintf(F," -X, --atime-preserve        preserve atimes on SRC\n");
   rprintf(F," -O, --omit-dir-times        omit directories when preserving
times\n");
   rprintf(F," -S, --sparse                handle sparse files
efficiently\n");
   rprintf(F," -n, --dry-run               show what would have been
transferred\n");
@@ -412,6 +414,7 @@
   {"group",           'g', POPT_ARG_NONE,  
&preserve_gid, 0, 0, 0 },
   {"devices",         'D', POPT_ARG_NONE,  
&preserve_devices, 0, 0, 0 },
   {"times",           't', POPT_ARG_NONE,  
&preserve_times, 0, 0, 0 },
+  {"atime-preserve",  'X', POPT_ARG_NONE,  
&preserve_atime, 0, 0, 0 },
   {"omit-dir-times",  'O', POPT_ARG_VAL,   
&omit_dir_times, 2, 0, 0 },
   {"checksum",        'c', POPT_ARG_NONE,  
&always_checksum, 0, 0, 0 },
   {"verbose",         'v', POPT_ARG_NONE,   0, 'v',
0, 0 },
diff -uNr rsync-2.6.6/proto.h rsync-2.6.6_patch/proto.h
--- rsync-2.6.6/proto.h	2005-07-07 21:49:14.000000000 +0200
+++ rsync-2.6.6_patch/proto.h	2005-11-23 23:51:54.000000000 +0100
@@ -266,7 +266,7 @@
 void print_child_argv(char **cmd);
 void out_of_memory(char *str);
 void overflow_exit(char *str);
-int set_modtime(char *fname, time_t modtime);
+int set_modtime(char *fname, time_t modtime, time_t acctime);
 int create_directory_path(char *fname, int base_umask);
 int full_write(int desc, char *ptr, size_t len);
 int copy_file(char *source, char *dest, mode_t mode);
diff -uNr rsync-2.6.6/rsync.1 rsync-2.6.6_patch/rsync.1
--- rsync-2.6.6/rsync.1	2005-07-28 21:31:08.000000000 +0200
+++ rsync-2.6.6_patch/rsync.1	2006-01-11 15:56:09.000000000 +0100
@@ -374,6 +374,7 @@
  -g, --group                 preserve group
  -D, --devices               preserve devices (root only)
  -t, --times                 preserve times
+ -X, --atime-preserve        don't change atimes on source
  -O, --omit-dir-times        omit directories when preserving times
  -S, --sparse                handle sparse files efficiently
  -n, --dry-run               show what would have been transferred
@@ -751,7 +752,11 @@
 modified cannot be effective; in other words, a missing \fB-t\fP or \fB-a\fP
will
 cause the next transfer to behave as if it used \fB-I\fP, causing all files to
be
 updated (though the rsync algorithm will make the update fairly efficient
-if the files haven\&'t actually changed, you\&'re much better
off using \fB-t\fP)\&.
+if the files haven\&'t actually changed, you\&'re much better
off using
+\fB-t\fP)\&. The access times are transferred along with the files by
default\&.
+.IP 
+.IP "\fB-X, --atime-preserve\fP" 
+This tells rsync to not change the access times of the source files\&.
 .IP 
 .IP "\fB-O, --omit-dir-times\fP" 
 This tells rsync to omit directories when
diff -uNr rsync-2.6.6/rsync3.txt rsync-2.6.6_patch/rsync3.txt
--- rsync-2.6.6/rsync3.txt	2001-09-12 16:35:39.000000000 +0200
+++ rsync-2.6.6_patch/rsync3.txt	2006-01-19 18:20:33.957510912 +0100
@@ -124,10 +124,6 @@
 
   - Scripting support.  
 
-  - Propagate atimes and do not modify them.  This is very ugly on
-    Unix.  It might be better to try to add O_NOATIME to kernels, and
-    call that.
-
   - Unicode.  Probably just use UTF-8 for everything.
 
   - Open authentication system.  Can we use PAM?  Is SASL an adequate
diff -uNr rsync-2.6.6/rsync.c rsync-2.6.6_patch/rsync.c
--- rsync-2.6.6/rsync.c	2005-03-14 18:06:08.000000000 +0100
+++ rsync-2.6.6_patch/rsync.c	2005-11-23 23:53:43.000000000 +0100
@@ -73,7 +73,7 @@
 		flags |= PERMS_SKIP_MTIME;
 	if (!(flags & PERMS_SKIP_MTIME)
 	    && cmp_modtime(st->st_mtime, file->modtime) != 0) {
-		if (set_modtime(fname,file->modtime) != 0) {
+		if (set_modtime(fname,file->modtime,file->acctime) != 0) {
 			rsyserr(FERROR, errno, "failed to set times on %s",
 				full_fname(fname));
 			return 0;
diff -uNr rsync-2.6.6/rsync.h rsync-2.6.6_patch/rsync.h
--- rsync-2.6.6/rsync.h	2005-05-03 19:00:47.000000000 +0200
+++ rsync-2.6.6_patch/rsync.h	2005-11-28 17:59:19.000000000 +0100
@@ -66,7 +66,7 @@
 #define FLAG_HLINK_TOL (1<<4)	/* receiver/generator */
 
 /* update this if you make incompatible changes */
-#define PROTOCOL_VERSION 29
+#define PROTOCOL_VERSION 30
 
 /* We refuse to interoperate with versions that are not in this range.
  * Note that we assume we'll work with later versions: the onus is on
@@ -509,6 +509,7 @@
 		struct hlink *links;
 	} link_u;
 	time_t modtime;
+	time_t acctime;
 	uid_t uid;
 	gid_t gid;
 	mode_t mode;
diff -uNr rsync-2.6.6/sender.c rsync-2.6.6_patch/sender.c
--- rsync-2.6.6/sender.c	2005-05-19 10:52:24.000000000 +0200
+++ rsync-2.6.6_patch/sender.c	2006-01-07 14:28:06.000000000 +0100
@@ -37,6 +37,7 @@
 extern int inplace;
 extern int batch_fd;
 extern int write_batch;
+extern int preserve_atime;
 extern struct stats stats;
 extern struct file_list *the_file_list;
 extern char *log_format;
@@ -357,6 +358,14 @@
 					full_fname(fname));
 			}
 		}
+/* reset original access and modtime if preserve_atime=1 */
+		if (preserve_atime) {
+			if (set_modtime(fname,st.st_mtime,st.st_atime) != 0) {
+				rsyserr(FERROR, errno,
+					"failed to set times on %s",
+					full_fname(fname));
+			}
+		}
 		close(fd);
 
 		free_sums(s);
diff -uNr rsync-2.6.6/util.c rsync-2.6.6_patch/util.c
--- rsync-2.6.6/util.c	2005-07-07 21:49:14.000000000 +0200
+++ rsync-2.6.6_patch/util.c	2005-11-23 23:59:42.000000000 +0100
@@ -128,7 +128,7 @@
 
 
 
-int set_modtime(char *fname, time_t modtime)
+int set_modtime(char *fname, time_t modtime, time_t acctime)
 {
 	if (verbose > 2) {
 		rprintf(FINFO, "set modtime of %s to (%ld) %s",
@@ -142,17 +142,17 @@
 	{
 #ifdef HAVE_UTIMBUF
 		struct utimbuf tbuf;
-		tbuf.actime = time(NULL);
+		tbuf.actime = acctime;
 		tbuf.modtime = modtime;
 		return utime(fname,&tbuf);
 #elif defined HAVE_UTIME
 		time_t t[2];
-		t[0] = time(NULL);
+		t[0] = acctime;
 		t[1] = modtime;
 		return utime(fname,t);
 #else
 		struct timeval t[2];
-		t[0].tv_sec = time(NULL);
+		t[0].tv_sec = acctime;
 		t[0].tv_usec = 0;
 		t[1].tv_sec = modtime;
 		t[1].tv_usec = 0;

On Tue, Jan 24, 2006 at 05:46:36PM +0100, Christian Konz T3
wrote:
> Therefore, we have created a modified version of rsync based on
> rsync-2.6.6 protocol version 29 which transfers the access times with
> the transferred files and also allows to preserve the access times on
> the sending side.
First -- thanks for the patch.  You've unfortunately duplicated some
work that was already done (see the file patches/atimes.diff in the
rsync source), but your version does do things a little differently:

- The atimes.diff patch does not require a protocol bump because the
  new behavior is not enabled by default; it is turned on via the
  --atimes (-U) option.  (I personally don't think that preserving
  the atime should happen by default, not even with -a.)

- The atimes.diff patch omits directories from its access-time
  preservation (since it is very likely that the atime on directories
  will change outside of rsync's control and some systems don't like
  the atime to be forced on a directory).

- The atimes.diff patch adds a new XMIT_SAME_ATIME flag that avoids
  sending the same atime value twice in a row (just as the regular
  code uses XMIT_SAME_TIME for the mtime value).

- The atimes.diff patch changes the output of --itemize-changes when
  --atimes is specified to indicate which files needed an access-time
  change.

- Your patch has an option for leaving the access time unchanged on
  the source where the atimes.diff patch does not (though see below
  for my comments on this).

The main reason that the --atimes option has not made it into the main
source is that I was not willing to increase the amount of per-file
memory needed to make room for a value that most people didn't need
(since the preservation of access times is not commonly required).
However, I recently modified the atimes.diff to make it only use extra
memory when the --atimes option was actually specified, so I am
considering adding the option to the next version (its inclusion now
mainly depends on if I consider this memory-saving design too much of
a kluge or not).

Here are some comments on your patch:

- Your code should not be checking the remote_protocol variable, but
  the protocol_version variable.  This is because the negotiated
  protocol version can differ from the maximum remote value (mainly
  by the use of the --protocol option).

- You have code that sets the atime on a file that was hard-linked
  via --link-dest.  This is probably the right choice (and one that
  the atimes.diff patch fails to consider):  it can mean that older
  version(s) of identical files can get the access time updated, but
  it's probably not an important enough of a difference to force a
  new version to be copied into the destination (rather than
  hard-linked to a --link-dest version).
  
- Your change in set_perms() only sets the atime if the mtime differs.

- Your patch's optional preservation of the access times on the
  sending side is done by forcing both the access time and the modify
  time to the values that were acquired at the start of a file's
  transfer.  This can be unsafe if someone managed to change the file
  during the time that the file was being sent.  You should at least
  re-stat the file at the end of the transfer to make sure that the
  modify time hasn't changed (and skip the update if it has).  Even
  then, I'd be uneasy about this method of trying to leave the access
  time unchanged on the source because there would still be a chance
  that it would obliterate an updated mtime.

Overall you did a very nice job of changing the code.  Thanks again
for helping out.

..wayne..