rsync - Jan 2015 - wrong transfer of app packages using --backup

Dear All,

I have been seeing some strange behavior using rsync 3.1.1 on OSX with the 3
standard patches applied when using the --backup and --backup-Dir options.

This works as expected on 3.0.9, moving any files on dest that are not on source
to the backup folder. If no changes then the backup folder is not even created.

-aHAXN --fileflags --force-change --protect-decmpfs ?delete --backup
--backup-dir=BL_Archive --stats ?progress -vvv

on 3.1.1 , where dest and source are already  in sync, any apps and rtfd files
(both are ?packages? of other files) on the dest are replicated in the backup
folder. They aren?t removed from the dest. They have a cross through them
(corrupted) and I noticed that not all the files within these corrupted apps are
present, mostly nib files, frameworks etc..

Anyone else experience this?

I ran with verbose but the only references to the backup folder are setting uids
on that backup folder. It is created early on in the output.

Created backup_dir BL_Archive/
set uid of BL_Archive/testfolder from 0 to 501
set uid of BL_Archive/testfolder/AppKiDo.app from 0 to 501
set uid of BL_Archive/testfolder/AppKiDo.app/Contents from 0 to 501
set uid of BL_Archive/testfolder/AppKiDo.app/Contents/MacOS from 0 to 501

Thanks,  Rob


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1387 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20150116/47218e49/attachment.bin>

Hi Again,

Update : I have spent some hours trying to track down where this happens, mostly
in backup.c.

This just seems to concern any files which are .app or .nib or .rtfd as they are
packages and act like directories. As below, 3.0.9 does not try to copy these at
all.

This only happens when there is no source file present on destination. It then
sends the app package folder to the backup_Dir which is corrupted since it
contains nothing inside.

It seems that it is making the backup directories for the .app package and also
the .nib files inside. This means it is corrupted on first pass and if copying
many files there can be many of these corrupts in backup_Dir which is not great.

So I am trying to find where in backup.c (or maybe rsync.c) this decision is
being made. Probably somewhere between  validate_backup_dir and copy_valid_path.
My C skills aren?t that great so following this path is difficult.

Thanks,  Rob




On Jan 16, 2015, at 9:41 AM, Robert DuToit <rdutoit at comcast.net> wrote:
> Dear All,
> 
> I have been seeing some strange behavior using rsync 3.1.1 on OSX with the
3 standard patches applied when using the --backup and --backup-Dir options.
> 
> This works as expected on 3.0.9, moving any files on dest that are not on
source to the backup folder. If no changes then the backup folder is not even
created.
> 
> -aHAXN --fileflags --force-change --protect-decmpfs ?delete --backup
--backup-dir=BL_Archive --stats ?progress -vvv
> 
> on 3.1.1 , where dest and source are already  in sync, any apps and rtfd
files (both are ?packages? of other files) on the dest are replicated in the
backup folder. They aren?t removed from the dest. They have a cross through them
(corrupted) and I noticed that not all the files within these corrupted apps are
present, mostly nib files, frameworks etc..
> 
> Anyone else experience this?
> 
> I ran with verbose but the only references to the backup folder are setting
uids on that backup folder. It is created early on in the output.
> 
> Created backup_dir BL_Archive/
> set uid of BL_Archive/testfolder from 0 to 501
> set uid of BL_Archive/testfolder/AppKiDo.app from 0 to 501
> set uid of BL_Archive/testfolder/AppKiDo.app/Contents from 0 to 501
> set uid of BL_Archive/testfolder/AppKiDo.app/Contents/MacOS from 0 to 501
> 
> Thanks,  Rob
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1387 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20150119/e9cc0706/attachment.bin>

Dear All,

I found a workaround for the corrupted backup app package files though there is
probably a more intelligent way.

When the destination is empty or missing (rather than an update) 

rsync.c calls make_backup and returns 1 when it encounters 

if (x_lstat(fname, &sx.st, NULL) < 0)
 		return 1;

the calling code in rsync.c:

if ( make_backups > 0 && overwriting_basis) {
		int ok = make_backup(fname, False);
		if (!ok)
			return 1;
		if (ok == 1 && fnamecmp == fname)
			fnamecmp = get_backup_name(fname);
	}

in 3.0.9  get_backup_name just returns the backup_Dir path but in 3.1.1 it
proceeds to copy_valid_path which then creates the folder structure for the DIR,
which in this case is an APP package or rtfd file which should not be copied at
all. I don?t think any dirs should be created if there are no backups to be made
so probably the older simple get_backup_name is what is needed here. I don?t
know if it is possible to differentiate app packages from regular directories
and it doesn?t matter if the app file is really being copied to the backup_Dir.
But the backup Dirs are being created in this case though there are no backups
to be made.

As a workaround I moved

if (x_lstat(fname, &sx.st, NULL) < 0)
 		return 1;

to the begiining in make_backup and added it also to copy_valid_path so it would
return the call from rsync instead of making screwy directories.

This is my diff, assuming one runs fileflags, crtimes, and hfs_compression diffs
first for OSX and rsync 3.1.1.

Rob


diff --git a/backup.c b/backup.c
--- a/backup.c
+++ b/--- a/backup.c
@@ -65,6 +65,9 @@
 	BOOL ret = True;
 	stat_x sx;
 	char *b, *rel = backup_dir_buf + backup_dir_len, *name = rel;
+    
+    if (x_lstat(fname, &sx.st, NULL) < 0)
+		return 1;
 
 	for (f = fname, b = rel; *f && *f == *b; f++, b++) {
 		if (*b == '/')
@@ -215,17 +218,18 @@
 	stat_x sx;
 	struct file_struct *file;
 	int save_preserve_xattrs;
-	char *buf = get_backup_name(fname);
-	int ret = 0;
-
-	if (!buf)
-		return 0;
-
-	init_stat_x(&sx);
+    int ret = 0;
+	
+    init_stat_x(&sx);
 	/* Return success if no file to keep. */
 	if (x_lstat(fname, &sx.st, NULL) < 0)
 		return 1;
-
+    
+    char *buf = get_backup_name(fname);
+	
+	if (!buf)
+		return 0;
+	
 	/* Try a hard-link or a rename first.  Using rename is not atomic, but
 	 * is more efficient than forcing a copy for larger files when no hard-
 	 * linking is possible. */


 
On Jan 16, 2015, at 9:41 AM, Robert DuToit <rdutoit at comcast.net> wrote:
> Dear All,
> 
> I have been seeing some strange behavior using rsync 3.1.1 on OSX with the
3 standard patches applied when using the --backup and --backup-Dir options.
> 
> This works as expected on 3.0.9, moving any files on dest that are not on
source to the backup folder. If no changes then the backup folder is not even
created.
> 
> -aHAXN --fileflags --force-change --protect-decmpfs ?delete --backup
--backup-dir=BL_Archive --stats ?progress -vvv
> 
> on 3.1.1 , where dest and source are already  in sync, any apps and rtfd
files (both are ?packages? of other files) on the dest are replicated in the
backup folder. They aren?t removed from the dest. They have a cross through them
(corrupted) and I noticed that not all the files within these corrupted apps are
present, mostly nib files, frameworks etc..
> 
> Anyone else experience this?
> 
> I ran with verbose but the only references to the backup folder are setting
uids on that backup folder. It is created early on in the output.
> 
> Created backup_dir BL_Archive/
> set uid of BL_Archive/testfolder from 0 to 501
> set uid of BL_Archive/testfolder/AppKiDo.app from 0 to 501
> set uid of BL_Archive/testfolder/AppKiDo.app/Contents from 0 to 501
> set uid of BL_Archive/testfolder/AppKiDo.app/Contents/MacOS from 0 to 501
> 
> Thanks,  Rob
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1387 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20150122/73844343/attachment.bin>

I have noticed another glitch,

On OSX with the 3 usual patches fileflags, crtimes, hfs_compression  and using
?backup and ?backup-dir=PATH

Whenever source has a trailing slash, the backup dir is doubled in depth. 

So when backup_Dir =  ?BL_Archive" and source is ?some/folder/path? the
usual correct result  at root of backup folder is

?BL_Archive/some/file/path?

but when trailing slash = ?some/folder/path/?

result = ?BL_Archive/BL_Archive/some/file/path?

I tried this copying to different locations, root, nested etc and all the same
result. ON 3.0.9 the slash does not do this - you get the normal backup_Dir.

 backup_dir_buf  before stringjoin(?) in copy_valid_path is normal and after
stringjoin(?) it becomes the ?doubleDir.? which goes to do_mkdir.

Again, have tried for days to sort this out but can?t follow the trail right.

Thanks,  Rob






On Jan 22, 2015, at 7:16 PM, Robert DuToit <rdutoit at comcast.net> wrote:
> Dear All,
> 
> I found a workaround for the corrupted backup app package files though
there is probably a more intelligent way.
> 
> When the destination is empty or missing (rather than an update) 
> 
> rsync.c calls make_backup and returns 1 when it encounters 
> 
> if (x_lstat(fname, &sx.st, NULL) < 0)
> 		return 1;
> 
> the calling code in rsync.c:
> 
> if ( make_backups > 0 && overwriting_basis) {
> 		int ok = make_backup(fname, False);
> 		if (!ok)
> 			return 1;
> 		if (ok == 1 && fnamecmp == fname)
> 			fnamecmp = get_backup_name(fname);
> 	}
> 
> in 3.0.9  get_backup_name just returns the backup_Dir path but in 3.1.1 it
proceeds to copy_valid_path which then creates the folder structure for the DIR,
which in this case is an APP package or rtfd file which should not be copied at
all. I don?t think any dirs should be created if there are no backups to be made
so probably the older simple get_backup_name is what is needed here. I don?t
know if it is possible to differentiate app packages from regular directories
and it doesn?t matter if the app file is really being copied to the backup_Dir.
But the backup Dirs are being created in this case though there are no backups
to be made.
> 
> As a workaround I moved
> 
> if (x_lstat(fname, &sx.st, NULL) < 0)
> 		return 1;
> 
> to the begiining in make_backup and added it also to copy_valid_path so it
would return the call from rsync instead of making screwy directories.
> 
> This is my diff, assuming one runs fileflags, crtimes, and hfs_compression
diffs first for OSX and rsync 3.1.1.
> 
> Rob
> 
> 
> diff --git a/backup.c b/backup.c
> --- a/backup.c
> +++ b/--- a/backup.c
> @@ -65,6 +65,9 @@
> 	BOOL ret = True;
> 	stat_x sx;
> 	char *b, *rel = backup_dir_buf + backup_dir_len, *name = rel;
> +    
> +    if (x_lstat(fname, &sx.st, NULL) < 0)
> +		return 1;
> 
> 	for (f = fname, b = rel; *f && *f == *b; f++, b++) {
> 		if (*b == '/')
> @@ -215,17 +218,18 @@
> 	stat_x sx;
> 	struct file_struct *file;
> 	int save_preserve_xattrs;
> -	char *buf = get_backup_name(fname);
> -	int ret = 0;
> -
> -	if (!buf)
> -		return 0;
> -
> -	init_stat_x(&sx);
> +    int ret = 0;
> +	
> +    init_stat_x(&sx);
> 	/* Return success if no file to keep. */
> 	if (x_lstat(fname, &sx.st, NULL) < 0)
> 		return 1;
> -
> +    
> +    char *buf = get_backup_name(fname);
> +	
> +	if (!buf)
> +		return 0;
> +	
> 	/* Try a hard-link or a rename first.  Using rename is not atomic, but
> 	 * is more efficient than forcing a copy for larger files when no hard-
> 	 * linking is possible. */
> 
> 
> 
> On Jan 16, 2015, at 9:41 AM, Robert DuToit <rdutoit at comcast.net>
wrote:
> 
>> Dear All,
>> 
>> I have been seeing some strange behavior using rsync 3.1.1 on OSX with
the 3 standard patches applied when using the --backup and --backup-Dir options.
>> 
>> This works as expected on 3.0.9, moving any files on dest that are not
on source to the backup folder. If no changes then the backup folder is not even
created.
>> 
>> -aHAXN --fileflags --force-change --protect-decmpfs ?delete --backup
--backup-dir=BL_Archive --stats ?progress -vvv
>> 
>> on 3.1.1 , where dest and source are already  in sync, any apps and
rtfd files (both are ?packages? of other files) on the dest are replicated in
the backup folder. They aren?t removed from the dest. They have a cross through
them (corrupted) and I noticed that not all the files within these corrupted
apps are present, mostly nib files, frameworks etc..
>> 
>> Anyone else experience this?
>> 
>> I ran with verbose but the only references to the backup folder are
setting uids on that backup folder. It is created early on in the output.
>> 
>> Created backup_dir BL_Archive/
>> set uid of BL_Archive/testfolder from 0 to 501
>> set uid of BL_Archive/testfolder/AppKiDo.app from 0 to 501
>> set uid of BL_Archive/testfolder/AppKiDo.app/Contents from 0 to 501
>> set uid of BL_Archive/testfolder/AppKiDo.app/Contents/MacOS from 0 to
501
>> 
>> Thanks,  Rob
>> 
>> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1387 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20150125/ef72e2c5/attachment.bin>