On Mon, Mar 28, 2016 at 08:11:07AM -0400, Kevin Korb wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > What does --itemize-changes say about these files? > > Also, what is --atimes? Is this the Apple modified rsync for OSX? > > On 03/28/2016 07:51 AM, Albert Berger wrote: > > Greetings, > > > > when the daily system backup with rsync is performed, thousands of > > files (e.g. from /usr directory), which are not modified are > > uselessly transferrred to the backup storage every day. The rsync > > command is the following: > > > > rsync --update -DHAErlptgo --relative --atimes --delete-during > > ${SRCPATH} ${BACKUPDIR} > > > > Quick investigation shown, that these files are hard links, and the > > command option related to this behaviour is -H: without it the > > unmodified files are not copied. Seemingly this situation is > > described in the rsync manual: > > > > "If incremental recursion is active (see --recursive), rsync may > > transfer a missing hard-linked file before it finds that another > > link for that contents exists else‐ where in the hierarchy.... > > One way to avoid this inefficiency is to disable incremental > > recursion using the --no-inc-recursive option." > > > > Adding --no-inc-recursive to the aforementioned command line > > doesn't help (are --recursive and --no-inc-recursive supposed to be > > used in one option set?). And I use --recursive to copy entire > > directory tree, including subdirectories. Could someone advise > > please how one can prevent copying unmodified hard-linked files? > > > > Thanks, A. Berger. > >As an example of copying hard-linked files, the /usr/bin contains the following two hard-linked sets for two inodes: inode 1: ld ld.bfd inode 2: g++ i686-pc-linux-gnu-c++ i686-pc-linux-gnu-g++ c++ Below is the --itemized-changes output for those two sets which is produced when /usr/bin doesn't have any modifications and is synced with the aforementioned command to its backup copy: hf......u.. /usr/bin/ld.bfd hf......u.. /usr/bin/i686-pc-linux-gnu-c++ hf......u.. /usr/bin/i686-pc-linux-gnu-g++ hf......u.. /usr/bin/g++ As can bee seen, one file in each set is not listed at all, and all other hard-linked files are marked for updating. Is this behaviour normal? --atimes is an option for preserving access times. This option is added by patching rsync source with a patch from rsync-patches-3.1.1.tar.gz file from rsync download page. The system is ArchLinux, kernel 4.4.5. Regards, A.Berger.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 My guess is that it is the atimes patch causing the problem. Normally rsync would never say "u" in the itemized output as it is "reserved for future use". Plus copying the file would change the atime. On 03/28/2016 08:44 AM, Albert Berger wrote:> On Mon, Mar 28, 2016 at 08:11:07AM -0400, Kevin Korb wrote: >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> >> What does --itemize-changes say about these files? >> >> Also, what is --atimes? Is this the Apple modified rsync for >> OSX? >> >> On 03/28/2016 07:51 AM, Albert Berger wrote: >>> Greetings, >>> >>> when the daily system backup with rsync is performed, thousands >>> of files (e.g. from /usr directory), which are not modified >>> are uselessly transferrred to the backup storage every day. The >>> rsync command is the following: >>> >>> rsync --update -DHAErlptgo --relative --atimes --delete-during >>> ${SRCPATH} ${BACKUPDIR} >>> >>> Quick investigation shown, that these files are hard links, and >>> the command option related to this behaviour is -H: without it >>> the unmodified files are not copied. Seemingly this situation >>> is described in the rsync manual: >>> >>> "If incremental recursion is active (see --recursive), rsync >>> may transfer a missing hard-linked file before it finds that >>> another link for that contents exists else‐ where in the >>> hierarchy.... One way to avoid this inefficiency is to >>> disable incremental recursion using the --no-inc-recursive >>> option." >>> >>> Adding --no-inc-recursive to the aforementioned command line >>> doesn't help (are --recursive and --no-inc-recursive supposed >>> to be used in one option set?). And I use --recursive to copy >>> entire directory tree, including subdirectories. Could someone >>> advise please how one can prevent copying unmodified >>> hard-linked files? >>> >>> Thanks, A. Berger. >>> > > As an example of copying hard-linked files, the /usr/bin contains > the following two hard-linked sets for two inodes: > > inode 1: ld ld.bfd > > inode 2: g++ i686-pc-linux-gnu-c++ i686-pc-linux-gnu-g++ c++ > > Below is the --itemized-changes output for those two sets which is > produced when /usr/bin doesn't have any modifications and is synced > with the aforementioned command to its backup copy: > > hf......u.. /usr/bin/ld.bfd hf......u.. > /usr/bin/i686-pc-linux-gnu-c++ hf......u.. > /usr/bin/i686-pc-linux-gnu-g++ hf......u.. /usr/bin/g++ > > As can bee seen, one file in each set is not listed at all, and all > other hard-linked files are marked for updating. Is this behaviour > normal? > > --atimes is an option for preserving access times. This option is > added by patching rsync source with a patch from > rsync-patches-3.1.1.tar.gz file from rsync download page. > > The system is ArchLinux, kernel 4.4.5. > > > Regards, A.Berger. >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlb5KEUACgkQVKC1jlbQAQeWQACg2dJZ/rSs7z1UnsLMZ1ZZa98B 4yMAnR8zOH80RpyWUMB/OWecC2sCE/2W =XQ6i -----END PGP SIGNATURE-----
Thanks! On Mon, Mar 28, 2016 at 08:49:09AM -0400, Kevin Korb wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > My guess is that it is the atimes patch causing the problem. Normally > rsync would never say "u" in the itemized output as it is "reserved > for future use". Plus copying the file would change the atime. > > On 03/28/2016 08:44 AM, Albert Berger wrote: > > On Mon, Mar 28, 2016 at 08:11:07AM -0400, Kevin Korb wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > >> > >> What does --itemize-changes say about these files? > >> > >> Also, what is --atimes? Is this the Apple modified rsync for > >> OSX? > >> > >> On 03/28/2016 07:51 AM, Albert Berger wrote: > >>> Greetings, > >>> > >>> when the daily system backup with rsync is performed, thousands > >>> of files (e.g. from /usr directory), which are not modified > >>> are uselessly transferrred to the backup storage every day. The > >>> rsync command is the following: > >>> > >>> rsync --update -DHAErlptgo --relative --atimes --delete-during > >>> ${SRCPATH} ${BACKUPDIR} > >>> > >>> Quick investigation shown, that these files are hard links, and > >>> the command option related to this behaviour is -H: without it > >>> the unmodified files are not copied. Seemingly this situation > >>> is described in the rsync manual: > >>> > >>> "If incremental recursion is active (see --recursive), rsync > >>> may transfer a missing hard-linked file before it finds that > >>> another link for that contents exists else‐ where in the > >>> hierarchy.... One way to avoid this inefficiency is to > >>> disable incremental recursion using the --no-inc-recursive > >>> option." > >>> > >>> Adding --no-inc-recursive to the aforementioned command line > >>> doesn't help (are --recursive and --no-inc-recursive supposed > >>> to be used in one option set?). And I use --recursive to copy > >>> entire directory tree, including subdirectories. Could someone > >>> advise please how one can prevent copying unmodified > >>> hard-linked files? > >>> > >>> Thanks, A. Berger. > >>> > > > > As an example of copying hard-linked files, the /usr/bin contains > > the following two hard-linked sets for two inodes: > > > > inode 1: ld ld.bfd > > > > inode 2: g++ i686-pc-linux-gnu-c++ i686-pc-linux-gnu-g++ c++ > > > > Below is the --itemized-changes output for those two sets which is > > produced when /usr/bin doesn't have any modifications and is synced > > with the aforementioned command to its backup copy: > > > > hf......u.. /usr/bin/ld.bfd hf......u.. > > /usr/bin/i686-pc-linux-gnu-c++ hf......u.. > > /usr/bin/i686-pc-linux-gnu-g++ hf......u.. /usr/bin/g++ > > > > As can bee seen, one file in each set is not listed at all, and all > > other hard-linked files are marked for updating. Is this behaviour > > normal? > > > > --atimes is an option for preserving access times. This option is > > added by patching rsync source with a patch from > > rsync-patches-3.1.1.tar.gz file from rsync download page. > > > > The system is ArchLinux, kernel 4.4.5. > > > > > > Regards, A.Berger. > > > > - -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. Kevin at FutureQuest.net (work) > Orlando, Florida kmk at sanitarium.net (personal) > Web page: http://www.sanitarium.net/ > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iEYEARECAAYFAlb5KEUACgkQVKC1jlbQAQeWQACg2dJZ/rSs7z1UnsLMZ1ZZa98B > 4yMAnR8zOH80RpyWUMB/OWecC2sCE/2W > =XQ6i > -----END PGP SIGNATURE----- > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html