For some time now I've been using rsync on FreeBSD to make my system backups. Recently, I accidentally rm'd some files from one directory and I had to go and fetch copies off of my backup drive. After I had done so, I found that about 1/5 of them were corrupted. (They were all .jpg files, by the way.) To be clear, I most certainly *do not* attribute the apparent corruption to rsync. My backup drive is itself slightly dubious... a retired (so-called ``refurbished'') WD Black 1TB drive which the S.M.A.R.T. info showed already had well more than 30,000 hours on it before I ever laid hands on it. Also, and separately, I've been trying to use it... and others like it... in one of these open-on-the-top slot loading external/powered SATA to USB3 adapters. As I have learned recently, 2.5" drives are generally no problem to use with such adapters, but unless you have a utility fan pointing right at it, 3.5" drives that are placed into one of those adapters can pretty quickly get REALLY hot... a fact which, taken alone, may actually be the root cause of the file corruption on my backup drive. So anyway, I have a 1TB backup drive now that is chock full of files that I placed on it previously (using rsync)... the majority of which, by volume, are binary media files (i.e. mp3 songs, JPEGs, movies, etc.) and now I'd like to carefully check them all to see which ones may have gotten corrupted. So, my question: How best to do this? Looking at the rsync man page, I see a couple of options that may perhaps be relevant, specifically: -c, --checksum skip based on checksum, not mod-time & size -n, --dry-run perform a trial run with no changes made Can I use these to, in effect, check all of my backup files for integrity? Please note that the options I have been using to make my backups are as follows: -v -t -axHAXS --delete --fileflags --force-change I don't imagine that either the -n or -c options will interact badly with any of those, correct? Regards, rfg
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 First off, --fileflags --force-change are not in my man rsync so I don't know what those are. Second, you should look into using either ZFS subvolume snapshots or rsync --link-dest to maintain multiple backups. Now, for your actual question... Add --itemize-changes to your standard command line. -v is almost entirely useless without it anyway. To figure out and fix what is corrupt you have 2 paths: 1. Add --checksum for a single pass. This will take forever as rsync checksums everything even things it shouldn't expect to match (even things that are only on one end!). Anything that checksum finds that rsync wouldn't have otherwise found would have a 'c' but not a 't'. 2. Add --ignore-times for a single pass. Normally this doesn't take as long as --checksum. However, since you are using an external USB device which means you are also using --whole-file this will probably be even slower than --checksum. Either way, you need to get your system writing files correctly. On 09/30/2015 04:19 PM, Ronald F. Guilmette wrote:> > For some time now I've been using rsync on FreeBSD to make my > system backups. Recently, I accidentally rm'd some files from one > directory and I had to go and fetch copies off of my backup drive. > After I had done so, I found that about 1/5 of them were corrupted. > (They were all .jpg files, by the way.) > > To be clear, I most certainly *do not* attribute the apparent > corruption to rsync. My backup drive is itself slightly dubious... > a retired (so-called ``refurbished'') WD Black 1TB drive which the > S.M.A.R.T. info showed already had well more than 30,000 hours on > it before I ever laid hands on it. Also, and separately, I've been > trying to use it... and others like it... in one of these > open-on-the-top slot loading external/powered SATA to USB3 > adapters. As I have learned recently, 2.5" drives are generally no > problem to use with such adapters, but unless you have a utility > fan pointing right at it, 3.5" drives that are placed into one of > those adapters can pretty quickly get REALLY hot... a fact which, > taken alone, may actually be the root cause of the file corruption > on my backup drive. > > So anyway, I have a 1TB backup drive now that is chock full of > files that I placed on it previously (using rsync)... the majority > of which, by volume, are binary media files (i.e. mp3 songs, > JPEGs, movies, etc.) and now I'd like to carefully check them all > to see which ones may have gotten corrupted. > > So, my question: How best to do this? > > Looking at the rsync man page, I see a couple of options that may > perhaps be relevant, specifically: > > -c, --checksum skip based on checksum, not mod-time & > size -n, --dry-run perform a trial run with no > changes made > > Can I use these to, in effect, check all of my backup files for > integrity? > > Please note that the options I have been using to make my backups > are as follows: > > -v -t -axHAXS --delete --fileflags --force-change > > I don't imagine that either the -n or -c options will interact > badly with any of those, correct? > > > Regards, rfg >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., - -*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., - -*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlYMSlEACgkQVKC1jlbQAQeNCQCff5zt9XGv94L0uNCtZn6Yf1L5 WqEAoOit1g8CazYDIwp9YDjMDPMhU4Tr =ekEj -----END PGP SIGNATURE-----
In message <560C4A51.4040702 at sanitarium.net>, Kevin Korb <kmk at sanitarium.net> wrote:>First off, --fileflags --force-change are not in my man rsync so I >don't know what those are.These are probably (Free)BSD specific. Here's what the man page says: --fileflags preserve file-flags (aka chflags) --force-change affect user/system immutable files/dirs>Second, you should look into using either ZFS subvolume snapshots or >rsync --link-dest to maintain multiple backups.Thank you, but I have no real interest in switching to ZFS just now.>Now, for your actual question... >Add --itemize-changes to your standard command line. -v is almost >entirely useless without it anyway.Thanks! I certainly did not know about that option!>To figure out and fix what is corrupt you have 2 paths: > >1. Add --checksum for a single pass. This will take forever as rsync >checksums everything even things it shouldn't expect to match (even >things that are only on one end!). Anything that checksum finds that >rsync wouldn't have otherwise found would have a 'c' but not a 't'.I don't understand what you mean about 'c' and 't'. Are you talking about what will appear in the (itemized) change log? I am guessing so.>2. Add --ignore-times for a single pass.NOW I am REALLY confused! I don't understand at all what the functional difference is between these two options: -c, --checksum skip based on checksum, not mod-time & size -I, --ignore-times don't skip files that match size and time Could you please explain? Both of these options would seem to me to have exactly the same/identical effect.>... Normally this doesn't take >as long as --checksum. However, since you are using an external USB >device which means you are also using --whole-file...No, I am *not* using the --whole-file option. Indeed, up until this moment, I didn't even know that option existed! I thank you for bringing it to my attention. Now I plan to be using --whole-file in future when making all of my backups. (Because most of the files I back up are binary media files, I think that the delta algorithm is not really saving me that much in terms of run-time.) Having said that however, I need also to say that your comment (just above) is, for me, puzzling and downright cryptic. Yes, I am doing my backups to an external USB-connected device. But what has that fact got to do with the --whole-file option? I see no obvious connection at all.>... this will probably be even slower than --checksum.Why would using the --whole-file option during my attempts to verify my backup files cause things to run even slower? (This is not at all obvious.)>Either way, you need to get your system writing files correctly.Oh yes, clearly. I believe (for now) that simply having proper cooling applied to the external drive in question may be sufficient to prevent corruption of files put on the drive in futire. (I did run one of the LONG built-in drive self-test passes on this drive after I found that some files on it had been corrupted, but results from that were 100% OK. So I think the drive perhaps just got a bit flaky during a time when I *did not* have an small utility fan right next to it and pointing at it. I _do_ have that now.) Thank you for all of your detailed responses. Regards, rfg