On Thu, May 03, 2018 at 01:09:30AM -0400, Kevin Korb via rsync wrote:> --delete only works when you sync a dir not individual files. So... > rsync -vai --delete --include='g*' --include='G*' --exclude='*' > /share/Public/videos/ /share/Backup-HDD-03That kinda didn't work: rsync -vai --delete --include='S[A-Z]*' --include='S[a-h]*' --exclude='*' \ /share/Public/videos/ /share/Backup-HDD-04 I deleted, manually, a directory on the backup HDD to test this out and got the following result: NASD30354.ZADMIN <795> ./Backup-HDD-04.list.sh sending incremental file list .d..t...... ./ cd+++++++++ Sense8/ cd+++++++++ Sense8/Season-1/ cd+++++++++ Sense8/Season-2/ sent 25,683 bytes received 82 bytes 17,176.67 bytes/sec total size is 1,666,354,406,916 speedup is 64,675,117.68 None of the files in those two directories were copied to the backup HDD after the directories were created. I then modified the script so that I could include directories that start with "s": rsync -vain --delete --include='S[A-Z]*' --include='S[a-h]*' \ --include='s*' --exclude='*' \ /share/Public/videos/ /share/Backup-HDD-04 I then got the following result (yes, in dry-run mode): NASD30354.ZADMIN <802> ./Backup-HDD-04.list.sh sending incremental file list>f+++++++++ Sense8/Season-1/sense8.s01e01.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e02.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e03.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e04.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e05.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e06.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e07.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e08.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e09.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e10.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e11.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-1/sense8.s01e12.1080p.webrip.x264-2hd.mkv >f+++++++++ Sense8/Season-2/sense8.s02e01.1080p.webrip.x264-skgtv.mkv >f+++++++++ Sense8/Season-2/sense8.s02e02.1080p.webrip.x264-skgtv.mkv >f+++++++++ Sense8/Season-2/sense8.s02e03.1080p.webrip.x264-skgtv.mkv >f+++++++++ Sense8/Season-2/sense8.s02e04.1080p.webrip.x264-skgtv.mkv >f+++++++++ Sense8/Season-2/sense8.s02e05.1080p.webrip.x264-skgtv.mkv >f+++++++++ Sense8/Season-2/sense8.s02e06.1080p.webrip.x264-morose.mkv >f+++++++++ Sense8/Season-2/sense8.s02e07.1080p.webrip.x264-morose.mkv >f+++++++++ Sense8/Season-2/sense8.s02e08.1080p.webrip.x264-morose.mkv >f+++++++++ Sense8/Season-2/sense8.s02e09.1080p.webrip.x264-morose.mkv >f+++++++++ Sense8/Season-2/sense8.s02e10.1080p.webrip.x264-morose.mkvsent 26,729 bytes received 132 bytes 53,722.00 bytes/sec total size is 1,749,617,318,885 speedup is 65,135,971.07 (DRY RUN) I kinda see what is going on, based upon the man page and filter rules. I was worried about getting stuff that starts with "s", so I renamed a file so that the frst letter was now a "s" and it was listed for copying. Since a majority of the files start with all-caps, I don't think I'll run into an issue of copying more files that I really wanted. Thanks for the tip, as I now have something to work with. MB -- e-mail: vidiot at vidiot.com | vidiot at vidiot.net /~\ The ASCII 6082066843 at email.uscc.net (140 char limit) \ / Ribbon Campaign Visit - URL: http://vidiot.com/ X Against http://vidiot.net/ / \ HTML Email "What do you say Beckett. Wanna have a baby?" - Castle to Det. Beckett "How long have I been gone?" Alexis after seeing Castle and Beckett w/ baby - Castle - 11/25/13
Note that these are working because S* and s* are being applied to all the dirs and files within dirs. So if you had x/Something it wouldn't get copied and if you had S/else it wouldn't get copied. Maybe you want to add --prune-empty-dirs and an --include='*/' so that rsync will look inside of all dirs for matching files. On 05/03/2018 02:50 PM, Mike Brown via rsync wrote:> On Thu, May 03, 2018 at 01:09:30AM -0400, Kevin Korb via rsync wrote: >> --delete only works when you sync a dir not individual files. So... >> rsync -vai --delete --include='g*' --include='G*' --exclude='*' >> /share/Public/videos/ /share/Backup-HDD-03 > > That kinda didn't work: > > rsync -vai --delete --include='S[A-Z]*' --include='S[a-h]*' --exclude='*' \ > /share/Public/videos/ /share/Backup-HDD-04 > > I deleted, manually, a directory on the backup HDD to test this out and > got the following result: > > NASD30354.ZADMIN <795> ./Backup-HDD-04.list.sh > sending incremental file list > .d..t...... ./ > cd+++++++++ Sense8/ > cd+++++++++ Sense8/Season-1/ > cd+++++++++ Sense8/Season-2/ > > sent 25,683 bytes received 82 bytes 17,176.67 bytes/sec > total size is 1,666,354,406,916 speedup is 64,675,117.68 > > None of the files in those two directories were copied to the backup HDD > after the directories were created. > > I then modified the script so that I could include directories that start > with "s": > > rsync -vain --delete --include='S[A-Z]*' --include='S[a-h]*' \ > --include='s*' --exclude='*' \ > /share/Public/videos/ /share/Backup-HDD-04 > > I then got the following result (yes, in dry-run mode): > > NASD30354.ZADMIN <802> ./Backup-HDD-04.list.sh > sending incremental file list >> f+++++++++ Sense8/Season-1/sense8.s01e01.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e02.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e03.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e04.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e05.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e06.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e07.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e08.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e09.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e10.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e11.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-1/sense8.s01e12.1080p.webrip.x264-2hd.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e01.1080p.webrip.x264-skgtv.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e02.1080p.webrip.x264-skgtv.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e03.1080p.webrip.x264-skgtv.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e04.1080p.webrip.x264-skgtv.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e05.1080p.webrip.x264-skgtv.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e06.1080p.webrip.x264-morose.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e07.1080p.webrip.x264-morose.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e08.1080p.webrip.x264-morose.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e09.1080p.webrip.x264-morose.mkv >> f+++++++++ Sense8/Season-2/sense8.s02e10.1080p.webrip.x264-morose.mkv > > sent 26,729 bytes received 132 bytes 53,722.00 bytes/sec > total size is 1,749,617,318,885 speedup is 65,135,971.07 (DRY RUN) > > I kinda see what is going on, based upon the man page and filter rules. > I was worried about getting stuff that starts with "s", so I renamed a file > so that the frst letter was now a "s" and it was listed for copying. > > Since a majority of the files start with all-caps, I don't think I'll > run into an issue of copying more files that I really wanted. > > Thanks for the tip, as I now have something to work with. > > MB >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20180503/02e25f96/signature.sig>
On Thu, May 03, 2018 at 02:59:11PM -0400, Kevin Korb via rsync wrote:> Note that these are working because S* and s* are being applied to all > the dirs and files within dirs. So if you had x/Something it wouldn't > get copied and if you had S/else it wouldn't get copied. Maybe you want > to add --prune-empty-dirs and an --include='*/' so that rsync will look > inside of all dirs for matching files.All not working so when when working with the next HDD backup: rsync -vain --prune-empty-dirs --delete \ --include='[T-Z]*' --include='*/' --exclude='*' \ /share/Public/videos/ /share/Backup-HDD-05 When run without the --prune-empty-dirs it picked up a directory starting with iZ, but didn't list the sub-directory and its files. With the --prune-empty-dirs, it was no longer listed. It should never have been listed in the first place as I never included "i". But, worse than that, the --include='*/' is causing ALL directories in /share/Public/videos/ to be queued up for transfer to the backup HDD. The filter rule is not working out in practice. For example, the following directories do not get copied: Victoria/Season-1 Victoria/Season-2 Why? Because the "S" in Season is getting excluded because it is not in the initial include rule. How do you get the filter rule to only look at the first part of the path and ignore the remaining part of the path? The man page for Filter RULES has the following: As the list of files/directories to transfer is built, rsync checks each name to be transferred against the list of include/exclude pat‐ terns in turn, and the first matching pattern is acted on: if it is an exclude pattern, then that file is skipped; if it is an include pattern then that filename is not skipped; if no matching pattern is found, then the filename is not skipped. That tells me that the first rule I have "--include='[T-Z]*'" says that Victoria/Season-1 and all its contents should be copied. But, it is checking again after the "/", thereby excluding the directory. :-( Looking further at INCLUDE/EXCLUDE PATTERN RULES, I found the following: o a trailing "dir_name/***" will match both the directory (as if "dir_name/" had been specified) and everything in the directory (as if "dir_name/**" had been specified). This behavior was added in version 2.6.7. So, changing "--include='[T-Z]*'" to "--include='[T-Z]*/***'" seems to have done the trick (of course removing the "--include='*/'"). This has been a real brain teaser. MB -- e-mail: vidiot at vidiot.com | vidiot at vidiot.net /~\ The ASCII 6082066843 at email.uscc.net (140 char limit) \ / Ribbon Campaign Visit - URL: http://vidiot.com/ X Against http://vidiot.net/ / \ HTML Email "What do you say Beckett. Wanna have a baby?" - Castle to Det. Beckett "How long have I been gone?" Alexis after seeing Castle and Beckett w/ baby - Castle - 11/25/13