Kevin Korb <kmk at sanitarium.net>, I thank you greatly for your attempts to educate me, however as we get deeper into discussing more and more different rsync options, I feel that I am actually just getting more confused and frustrated. I've been sitting here, trying all sorts of different combinations and permutations of the various options we've discussed, and that you've told me about, but so far none of the combinations is producing the (seemingly simple) effect that I wanted. So, at the risk of trying your patience, may I please request that we begin again. Let's start from scratch, and perhaps you can guide me to the simplest possible solution to the problem I first came here with... hopefully a solution which is so simple that even *I* can't screw it up. So, let's consider the following simple hypothetical example... We have two directories within the current directory. They are named ./one/ and ./two/ Within each of these two directories, there is a set of ordinary files. The set of filenames present within each of the two directories is identical. Let's just say that each one contains the following three ordinary files: 001.jpg 002.jpg 003.jpg How may I use rsync to obtain a list of _just_ those files within these two directories that the cmp command would describe as being different between the two directories? What is the exact command line I should use? (And within the output from the given command, what must I look for, exactly, that designates files that are different, as opposed to those whose content is the same?) Regards, rfg P.S. I really do hope that I can get this to work with rsync. I do prefer not reinventing the wheel, but it is starting to seem simpler to me if I were to just write a Perl script that would walk two directory hierarchies, in parallel, and just repeatedly invoke the cmp command on all of the regular files found therein.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just add --itemize-changes and --checksum to what you were doing before and know that it will take a long time. On 09/30/2015 06:42 PM, Ronald F. Guilmette wrote:> > Kevin Korb <kmk at sanitarium.net>, > > I thank you greatly for your attempts to educate me, however as we > get deeper into discussing more and more different rsync options, I > feel that I am actually just getting more confused and frustrated. > I've been sitting here, trying all sorts of different combinations > and permutations of the various options we've discussed, and that > you've told me about, but so far none of the combinations is > producing the (seemingly simple) effect that I wanted. > > So, at the risk of trying your patience, may I please request that > we begin again. Let's start from scratch, and perhaps you can > guide me to the simplest possible solution to the problem I first > came here with... hopefully a solution which is so simple that even > *I* can't screw it up. > > So, let's consider the following simple hypothetical example... > > We have two directories within the current directory. They are > named ./one/ and ./two/ > > Within each of these two directories, there is a set of ordinary > files. The set of filenames present within each of the two > directories is identical. Let's just say that each one contains > the following three ordinary files: > > 001.jpg 002.jpg 003.jpg > > How may I use rsync to obtain a list of _just_ those files within > these two directories that the cmp command would describe as being > different between the two directories? What is the exact command > line I should use? (And within the output from the given command, > what must I look for, exactly, that designates files that are > different, as opposed to those whose content is the same?) > > > Regards, rfg > > > P.S. I really do hope that I can get this to work with rsync. I > do prefer not reinventing the wheel, but it is starting to seem > simpler to me if I were to just write a Perl script that would walk > two directory hierarchies, in parallel, and just repeatedly invoke > the cmp command on all of the regular files found therein. >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., - -*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., - -*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlYMZg8ACgkQVKC1jlbQAQfjgQCdGmGrIm3eol++rnB+fPGXApin jUEAnAsNTycPDsM4k+Kox5J2RSH4zldP =QfjT -----END PGP SIGNATURE-----
In message <560C660F.5000202 at sanitarium.net>, Kevin Korb <kmk at sanitarium.net> wrote:>Just add --itemize-changes and --checksum to what you were doing >before and know that it will take a long time.I'm still not getting to where I need to be. Maybe you can explain what has gone wrong in this very simple example: % mkdir one two % echo hello > one/hello % ln one/hello two/hello % echo different0 > one/foo % echo different1 > two/foo % rsync -n -v --itemize-changes -checksum -a one two Here is the output generated by that last command: sending incremental file list cd+++++++++ one/>f+++++++++ one/foo >f+++++++++ one/helloI fail to see how this helps me to know that in this case the files one/hello and two/hello are byte-for-byte identical AND also that the contents of the files one/foo and two/foo are in fact different. Where is the clear sign of a difference between the two "foo" files?
"Ronald F. Guilmette" <rfg at tristatelogic.com> wrote:> P.S. I really do hope that I can get this to work with rsync. > I do prefer not reinventing the wheel, but it is starting to seem > simpler to me if I were to just write a Perl script that would > walk two directory hierarchies, in parallel, and just repeatedly > invoke the cmp command on all of the regular files found therein.Just because rsync is an awesome hammer, it does not necessarily follow that every problem involving backups closely resembles a nail :) Since your source and backup are both local, I suspect using rsync as a comparison tool is overkill. (It may even be overkill for making the backups in the first place.) Had you considered "diff -q -r"? $ mkdir one two $ echo hello > one/hello $ ln one/hello two/hello $ echo different0 > one/foo $ echo different1 > two/foo $ diff -q -r one two Files one/foo and two/foo differ or, if you prefer $ diff -q -r -s one two Files one/foo and two/foo differ Files one/hello and two/hello are identical (I'm using gnu diff; other variants might have different command options.)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Yes, when it comes to local copies cp is significantly faster than rsync. Without --link-dest there isn't much advantage to using rsync for backups. The only thing you get beyond cp -au is --delete. Also, when it comes to static data like media files I like to keep an md5 file around with checksums for all the files. That way I can easily verify the files later on either the primary or backup storage and more importantly if they differ I can tell which one is correct. The cfv utility is great for file verifying. The -n option on it tells it to rename incorrect files. Then you can just do a simple rsync pass to delete the incorrectly named files and put back the now missing files. On 10/01/2015 01:53 AM, Perry Hutchison wrote:> "Ronald F. Guilmette" <rfg at tristatelogic.com> wrote: > >> P.S. I really do hope that I can get this to work with rsync. I >> do prefer not reinventing the wheel, but it is starting to seem >> simpler to me if I were to just write a Perl script that would >> walk two directory hierarchies, in parallel, and just repeatedly >> invoke the cmp command on all of the regular files found >> therein. > > Just because rsync is an awesome hammer, it does not necessarily > follow that every problem involving backups closely resembles a > nail :) > > Since your source and backup are both local, I suspect using rsync > as a comparison tool is overkill. (It may even be overkill for > making the backups in the first place.) Had you considered "diff > -q -r"? > > $ mkdir one two $ echo hello > one/hello $ ln one/hello two/hello $ > echo different0 > one/foo $ echo different1 > two/foo $ diff -q -r > one two Files one/foo and two/foo differ > > or, if you prefer > > $ diff -q -r -s one two Files one/foo and two/foo differ Files > one/hello and two/hello are identical > > (I'm using gnu diff; other variants might have different command > options.) >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., - -*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., - -*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlYM5wYACgkQVKC1jlbQAQdfRgCgkcyjw09g677+T+BUsdDWFb3c DwAAoL2q8+P/c8oolfNq9WEbhlIL2SvW =8K4f -----END PGP SIGNATURE-----
In message <560cca51.5cWVPTQcE1NFlU+U%perryh at pluto.rain.com>, perryh at pluto.rain.com (Perry Hutchison) wrote:>Just because rsync is an awesome hammer, it does not necessarily follow >that every problem involving backups closely resembles a nail :)An excellent and very apropos point.>Since your source and backup are both local, I suspect using rsync as >a comparison tool is overkill. (It may even be overkill for making >the backups in the first place.)In theory, I might be able to use cpio -p rather than rsync to make my backups, however for some reason that I can't remember anymore, when I originally set up my system for making backups, I looked at that possibility and concluded that rsync was preferable to cpio -p. I wish that I could remember why. (It might have had to do with better or more complete handling of ``weird stuff'' like extended file attributes, ACLs, device nodes, or other such oddities)>Had you considered "diff -q -r"?No, actually. thanks for the suggestion!>(I'm using gnu diff; other variants might have different command options.)The diff I have here on FreeBSD (9.1) seem to be GNU, so this will work. Regards, rfg