Cool Thanks! Specifically, the timestamps on both <src> and <dest> match for "ls -l" but do not match for "ls -lu" or "ls -lc” The storage is just an regular HDD in a mac pro tower. I can’t imagine why it wouldn’t handle timestamps. Also of note - this problem doesn’t exist for every file, just the vast majority. So, that just makes it more confusing. Yes, I imagine rsync is not the best for linear tape but give the choice between cp (which is faster and causes less problems but offers almost zero verbosity) and rsync, I’ll choose rsync. If people know of other options, I’d be very happy to know of them. On 6/2/16, 6:33 PM, "Kevin Korb" <kmk at sanitarium.net> wrote:>The man page has a section on what all the itemize-changes flags do. > >There is a --ignore-times but the result is what you have now, re-copy >everything even if the timestamp matches. > >The best you can really do with storage that can't handle timestamps is >to use --update. But if you do that you need to get rid of --partial >(part of -P) or else rsync will never complete a file that was partially >copied since the partial copy is newer that the source file. > >OTOH, are you using rsync to copy files to a tape drive? If so then I >would think rsync is not the right tool for dealing with linear storage. > >On 06/02/2016 06:25 PM, McDowell, Blake wrote: >> OK. Thanks. Where can I find information regarding how to interpret >> —itemize-changes? >> >> The timestamps aren’t changing, so the target must not be storing them, >> which I have no idea why. The directory I’m writing to is 777. >> >> What is the flag to tell rsync to ignore the timestamps? >> >> Thanks, >> Blake >> >> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb" >> <rsync-bounces at lists.samba.org on behalf of kmk at sanitarium.net> wrote: >> >>> It is saying the timestamp is wrong and that it is copying the file and >>> changing the timestamp. If it does that every time then either the >>> timestamps are changing on the source or the target isn't storing them. >>> >>> On 06/02/2016 06:13 PM, McDowell, Blake wrote: >>>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can >>>> you >>>> provide some insight? >>>> >>>> This is a local transfer from an external drive to an internal drive >>>>all >>>> attached to one computer. >>>> >>>> >>>> rsync -aPh --itemize-changes -n >>>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint >>>> /Volumes/3TB_LTO/LT003A/ >>>> sending incremental file list >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat >>>>>iv >>>>> es >>>>> /2012_79_1_14_1_DER_01.mov >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat >>>>>iv >>>>> es >>>>> /2012_79_1_14_1_DER_02.mov >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>/1 >>>>> 19 >>>>> 9WP_00086400.dpx >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>/1 >>>>> 19 >>>>> 9WP_00086401.dpx >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>/1 >>>>> 19 >>>>> 9WP_00086402.dpx >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>/1 >>>>> 19 >>>>> 9WP_00086403.dpx >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>/1 >>>>> 19 >>>>> 9WP_00086404.dpx >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>/1 >>>>> 19 >>>>> 9WP_00086405.dpx >>>>> f..t....... >>>>> >>>>> >>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>/1 >>>>> 19 >>>>> 9WP_00086406.dpx >>>> >>>> >>>> ~Blake >>>> >>>> >>>> >>>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb" >>>> <rsync-bounces at lists.samba.org on behalf of kmk at sanitarium.net> wrote: >>>> >>>>> Instead of the second -v (or even the first) add --itemize-changes. >>>>>It >>>>> will tell you why each file is being copied. If the file timestamps >>>>> are >>>>> not correct then perhaps the underlying storage doesn't allow >>>>>arbitrary >>>>> mtime changes. >>>>> >>>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote: >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> At my work we use rsync to move files between drives and to LTO >>>>>>among >>>>>> other things. >>>>>> >>>>>> >>>>>> >>>>>> I¹m having an issue using rsync to move material between and >>>>>>external >>>>>> drive and an internal drive. >>>>>> >>>>>> >>>>>> >>>>>> We run ³rsync avvPh <src> <dest>² and most of the files keep >>>>>>writing >>>>>> every time I run this. It appears that the modification times are >>>>>>not >>>>>> being carried through to the destination resulting in the files >>>>>> continually wanting to re-write. >>>>>> >>>>>> >>>>>> >>>>>> I¹m hoping someone can help me figure this out. Any information you >>>>>> need >>>>>> from me (logs, etc) I¹m happy to try and provide. >>>>>> >>>>>> >>>>>> >>>>>> Many thanks, >>>>>> >>>>>> Blake >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._ >>>>>., >>>>> Kevin Korb Phone: (407) 252-6853 >>>>> Systems Administrator Internet: >>>>> FutureQuest, Inc. Kevin at FutureQuest.net (work) >>>>> Orlando, Florida kmk at sanitarium.net (personal) >>>>> Web page: http://www.sanitarium.net/ >>>>> PGP public key available on web site. >>>>> >>>>> >>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._ >>>>>., >>>>> >>> >>> -- >>> >>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >>> Kevin Korb Phone: (407) 252-6853 >>> Systems Administrator Internet: >>> FutureQuest, Inc. Kevin at FutureQuest.net (work) >>> Orlando, Florida kmk at sanitarium.net (personal) >>> Web page: http://www.sanitarium.net/ >>> PGP public key available on web site. >>> >>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >>> >> > >-- >~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. Kevin at FutureQuest.net (work) > Orlando, Florida kmk at sanitarium.net (personal) > Web page: http://www.sanitarium.net/ > PGP public key available on web site. >~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >
Rsync only cares about the modification time. The ls command usually abbreviates the timestamp so it is better to use the stat command on one of the problem files to see the full thing. On 06/02/2016 06:42 PM, McDowell, Blake wrote:> Cool Thanks! > > Specifically, the timestamps on both <src> and <dest> match for "ls -l" > but do not match for "ls -lu" or "ls -lc” > > The storage is just an regular HDD in a mac pro tower. I can’t imagine why > it wouldn’t handle timestamps. Also of note - this problem doesn’t exist > for every file, just the vast majority. So, that just makes it more > confusing. > > Yes, I imagine rsync is not the best for linear tape but give the choice > between cp (which is faster and causes less problems but offers almost > zero verbosity) and rsync, I’ll choose rsync. If people know of other > options, I’d be very happy to know of them. > > On 6/2/16, 6:33 PM, "Kevin Korb" <kmk at sanitarium.net> wrote: > >> The man page has a section on what all the itemize-changes flags do. >> >> There is a --ignore-times but the result is what you have now, re-copy >> everything even if the timestamp matches. >> >> The best you can really do with storage that can't handle timestamps is >> to use --update. But if you do that you need to get rid of --partial >> (part of -P) or else rsync will never complete a file that was partially >> copied since the partial copy is newer that the source file. >> >> OTOH, are you using rsync to copy files to a tape drive? If so then I >> would think rsync is not the right tool for dealing with linear storage. >> >> On 06/02/2016 06:25 PM, McDowell, Blake wrote: >>> OK. Thanks. Where can I find information regarding how to interpret >>> —itemize-changes? >>> >>> The timestamps aren’t changing, so the target must not be storing them, >>> which I have no idea why. The directory I’m writing to is 777. >>> >>> What is the flag to tell rsync to ignore the timestamps? >>> >>> Thanks, >>> Blake >>> >>> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb" >>> <rsync-bounces at lists.samba.org on behalf of kmk at sanitarium.net> wrote: >>> >>>> It is saying the timestamp is wrong and that it is copying the file and >>>> changing the timestamp. If it does that every time then either the >>>> timestamps are changing on the source or the target isn't storing them. >>>> >>>> On 06/02/2016 06:13 PM, McDowell, Blake wrote: >>>>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can >>>>> you >>>>> provide some insight? >>>>> >>>>> This is a local transfer from an external drive to an internal drive >>>>> all >>>>> attached to one computer. >>>>> >>>>> >>>>> rsync -aPh --itemize-changes -n >>>>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint >>>>> /Volumes/3TB_LTO/LT003A/ >>>>> sending incremental file list >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat >>>>>> iv >>>>>> es >>>>>> /2012_79_1_14_1_DER_01.mov >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat >>>>>> iv >>>>>> es >>>>>> /2012_79_1_14_1_DER_02.mov >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>> /1 >>>>>> 19 >>>>>> 9WP_00086400.dpx >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>> /1 >>>>>> 19 >>>>>> 9WP_00086401.dpx >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>> /1 >>>>>> 19 >>>>>> 9WP_00086402.dpx >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>> /1 >>>>>> 19 >>>>>> 9WP_00086403.dpx >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>> /1 >>>>>> 19 >>>>>> 9WP_00086404.dpx >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>> /1 >>>>>> 19 >>>>>> 9WP_00086405.dpx >>>>>> f..t....... >>>>>> >>>>>> >>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX >>>>>> /1 >>>>>> 19 >>>>>> 9WP_00086406.dpx >>>>> >>>>> >>>>> ~Blake >>>>> >>>>> >>>>> >>>>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb" >>>>> <rsync-bounces at lists.samba.org on behalf of kmk at sanitarium.net> wrote: >>>>> >>>>>> Instead of the second -v (or even the first) add --itemize-changes. >>>>>> It >>>>>> will tell you why each file is being copied. If the file timestamps >>>>>> are >>>>>> not correct then perhaps the underlying storage doesn't allow >>>>>> arbitrary >>>>>> mtime changes. >>>>>> >>>>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote: >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> At my work we use rsync to move files between drives and to LTO >>>>>>> among >>>>>>> other things. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I¹m having an issue using rsync to move material between and >>>>>>> external >>>>>>> drive and an internal drive. >>>>>>> >>>>>>> >>>>>>> >>>>>>> We run ³rsync avvPh <src> <dest>² and most of the files keep >>>>>>> writing >>>>>>> every time I run this. It appears that the modification times are >>>>>>> not >>>>>>> being carried through to the destination resulting in the files >>>>>>> continually wanting to re-write. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I¹m hoping someone can help me figure this out. Any information you >>>>>>> need >>>>>>> from me (logs, etc) I¹m happy to try and provide. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Many thanks, >>>>>>> >>>>>>> Blake >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._ >>>>>> ., >>>>>> Kevin Korb Phone: (407) 252-6853 >>>>>> Systems Administrator Internet: >>>>>> FutureQuest, Inc. Kevin at FutureQuest.net (work) >>>>>> Orlando, Florida kmk at sanitarium.net (personal) >>>>>> Web page: http://www.sanitarium.net/ >>>>>> PGP public key available on web site. >>>>>> >>>>>> >>>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._ >>>>>> ., >>>>>> >>>> >>>> -- >>>> >>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >>>> Kevin Korb Phone: (407) 252-6853 >>>> Systems Administrator Internet: >>>> FutureQuest, Inc. Kevin at FutureQuest.net (work) >>>> Orlando, Florida kmk at sanitarium.net (personal) >>>> Web page: http://www.sanitarium.net/ >>>> PGP public key available on web site. >>>> >>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >>>> >>> >> >> -- >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >> Kevin Korb Phone: (407) 252-6853 >> Systems Administrator Internet: >> FutureQuest, Inc. Kevin at FutureQuest.net (work) >> Orlando, Florida kmk at sanitarium.net (personal) >> Web page: http://www.sanitarium.net/ >> PGP public key available on web site. >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >> >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20160602/f86bb5c8/signature.sig>
In <D3762D63.17A7%mcdowellh at si.edu>, on 06/02/16 at 10:42 PM, "McDowell, Blake" <McDowellH at si.edu> said: Hi Blake,>The storage is just an regular HDD in a mac pro tower. I can t imagine >why it wouldn t handle timestamps. Also of note - this problem doesn t >exist for every file, just the vast majority. So, that just makes it more >confusing.Are the file systems the same on the source and the destination partitions? Check out --modify-window in the help. If the source and destination file systems have different timestamp precision, this is the usual solution. You can also try rsync /path-to-foo where path-to-foo is a directory or file. This will list the file timestamps as rsync interprets them. BTW, what version of rsync are you running? It might matter. Steven -- ---------------------------------------------------------------------- "Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. www.scoug.com www.arcanoae.com www.warpcave.com ----------------------------------------------------------------------
"McDowell, Blake" <McDowellH at si.edu> wrote:> The storage is just an regular HDD in a mac pro tower. I can't > imagine why it wouldn't handle timestamps. Also of note - this > problem doesn't exist for every file, just the vast majority. > So, that just makes it more confusing.The filesystem format (MacOS native? FAT?) might be a factor.> Yes, I imagine rsync is not the best for linear tape but give the > choice between cp (which is faster and causes less problems but > offers almost zero verbosity) and rsync, I???ll choose rsync. If > people know of other options, I???d be very happy to know of them.Best choice for magtape is probably something like tar, cpio, or pax (for a file-oriented backup), or the appropriate variant of dump(8) (to back up an entire filesystem -- but not all FS formats have a dump/restore suite available).
"McDowell, Blake" <McDowellH at si.edu> wrote:> The storage is just an regular HDD in a mac pro tower.Ah, is this the version of rsync that comes with OS X ? Are these HFS+ filesystems ? I vaguely recall that the OS X version is "hacked" to handle the file semantics of HFS+ filesystems. Hopefully someone else actually knows the details, I could be a bit wrong here, but IIRC it's something like : On a "*nix" filsystem, a file is a file - a chunk of data and some filesystem metadata. On HFS+, a file is comprised of up to 3 parts - the data fork, the resource fork (I don't believe this is widely used these days), and a chunk more metadata. "Regular" rsync only copies the data fork and that part of the metadata that maps to *nix filesystem semantics, the OS X version of rsync copies the whole file by way of quite a kludge - can't remember if it needs an extra cmd line switch to do this. The kludge is to treat the data fork as one file, and the resource fork plus metadata as another file. I vaguely recall that this means it does something like : 1) copy/sync data fork as one file 2) copy/sync resource fork as another file - put the bits together at the destination From memory (it's a while since I last used rsync for doing backups), in step 1 the files don't match because the destination file was modified during step 2 of the last copy - thus the file gets synced again. Again from memory, I think I used to see that every file with a resource fork would be copied in it's entirety every time. This could be a complete red herring of course, but it's something I've come across in the past/
perryh at pluto.rain.com (Perry Hutchison) wrote:> Best choice for magtape is probably something like tar, cpio, or pax > (for a file-oriented backup), or the appropriate variant of dump(8) > (to back up an entire filesystem -- but not all FS formats have a > dump/restore suite available).I wouldn't have thought rsync a good choice either. I've used both tar and cpio in the past - I don't think there's much to choose between them, "personal comfort" with the tool is probably as big a factor as anything. Used them with QIC, DAT, DLT, and SLR over the years. If you want "ease of use" and aren't averse to paid for/proprietary programs, then on a Mac I think Retrospect has a lot going for it. I've been using it for home and business use for something like 25 years - oh, that makes me feel old now ! It's exceedingly good for incremental/differential backups - I've tried a few others but always managed to find situations where they would miss copying files (not a good attribute for a backup system), an area Dantz has some patents on.
Thanks for that info Simon. The files system is Journaled HFS+ and I¹m running rsync version 3.1.2 protocol version 31. I run rsync exclusively through the CLI/Terminal, so I¹m not sure what the version that comes with OSX is, but it is not the GUI version. On 6/3/16, 5:17 AM, "rsync on behalf of Simon Hobson" <rsync-bounces at lists.samba.org on behalf of linux at thehobsons.co.uk> wrote:>"McDowell, Blake" <McDowellH at si.edu> wrote: > >> The storage is just an regular HDD in a mac pro tower. > >Ah, is this the version of rsync that comes with OS X ? Are these HFS+ >filesystems ? > >I vaguely recall that the OS X version is "hacked" to handle the file >semantics of HFS+ filesystems. Hopefully someone else actually knows the >details, I could be a bit wrong here, but IIRC it's something like : >On a "*nix" filsystem, a file is a file - a chunk of data and some >filesystem metadata. On HFS+, a file is comprised of up to 3 parts - the >data fork, the resource fork (I don't believe this is widely used these >days), and a chunk more metadata. "Regular" rsync only copies the data >fork and that part of the metadata that maps to *nix filesystem >semantics, the OS X version of rsync copies the whole file by way of >quite a kludge - can't remember if it needs an extra cmd line switch to >do this. >The kludge is to treat the data fork as one file, and the resource fork >plus metadata as another file. I vaguely recall that this means it does >something like : > >1) copy/sync data fork as one file >2) copy/sync resource fork as another file - put the bits together at the >destination > >From memory (it's a while since I last used rsync for doing backups), in >step 1 the files don't match because the destination file was modified >during step 2 of the last copy - thus the file gets synced again. Again >from memory, I think I used to see that every file with a resource fork >would be copied in it's entirety every time. > >This could be a complete red herring of course, but it's something I've >come across in the past/ > > > >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: >https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Hi Perry, The issue I¹m having is not writing to the LTO tape - rsync does that ok, but we have just switch over to gcp followed by rsync as a double-check. It has certainly helped with our LTO writing. The problem I¹m having with the files continually re-writing is for a local transfer from external HDD to internal HDD, both HFS+ Thanks, Blake On 6/3/16, 4:14 AM, "Perry Hutchison" <perryh at pluto.rain.com> wrote:>"McDowell, Blake" <McDowellH at si.edu> wrote: > >> The storage is just an regular HDD in a mac pro tower. I can't >> imagine why it wouldn't handle timestamps. Also of note - this >> problem doesn't exist for every file, just the vast majority. >> So, that just makes it more confusing. > >The filesystem format (MacOS native? FAT?) might be a factor. > >> Yes, I imagine rsync is not the best for linear tape but give the >> choice between cp (which is faster and causes less problems but >> offers almost zero verbosity) and rsync, I???ll choose rsync. If >> people know of other options, I???d be very happy to know of them. > >Best choice for magtape is probably something like tar, cpio, or pax >(for a file-oriented backup), or the appropriate variant of dump(8) >(to back up an entire filesystem -- but not all FS formats have a >dump/restore suite available).
Hi Steven, Yes, both file systems are the same. rsync -nri --modify-window=1 <src> <dest> Gives me the following for most files >f..T....... 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/ BWAG_R2_00138428.dpx Although a few have >f..T......n 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/ BWAG_R2_00135909.dpx Strangely, no matter what HDD I¹m transferring from it is always the same sequence of files that have the n - "A n means the create time (newness) is different and is being updated to the sender's value (requires --crtimes). (I¹m not quite sure I completely understand ‹-modify-window) Here is a <dest> file example of timestamps as rsync interprets them: -rwxrwxrwx 24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx Here is a <src> file example of timestamps as rsync interprets them: -rwxrwxrwx 24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx So, yeah, that is the problem, apparently. The timestamps are transferring. But, for the files that have the ³n² the timestamps appear to be fine and those ones don¹t continually transfer. A mystery to me as to why the timestamps work on these particular files. <src> -rwxrwxrwx 24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx <dest> -rwxrwxrwx 24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx I¹m running version 3.1.2 protocol 31. Thanks, Blake On 6/2/16, 7:26 PM, "rsync on behalf of Steven Levine" <rsync-bounces at lists.samba.org on behalf of steve53 at earthlink.net> wrote:>In <D3762D63.17A7%mcdowellh at si.edu>, on 06/02/16 > at 10:42 PM, "McDowell, Blake" <McDowellH at si.edu> said: > >Hi Blake, > >>The storage is just an regular HDD in a mac pro tower. I can t imagine >>why it wouldn t handle timestamps. Also of note - this problem doesn t >>exist for every file, just the vast majority. So, that just makes it more >>confusing. > >Are the file systems the same on the source and the destination >partitions? > >Check out > > --modify-window > >in the help. If the source and destination file systems have different >timestamp precision, this is the usual solution. > >You can also try > > rsync /path-to-foo > >where path-to-foo is a directory or file. This will list the file >timestamps as rsync interprets them. > >BTW, what version of rsync are you running? It might matter. > >Steven > >-- >---------------------------------------------------------------------- >"Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. >www.scoug.com www.arcanoae.com www.warpcave.com >---------------------------------------------------------------------- > > >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: >https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Hi Kevin, Stat is pretty cool! But, I can’t quite figure out some of the stuff it is telling me right now. I’ll try using —update but I’m hoping to figure out why the timestamps are not transferring on the majority of my files. Thanks, Blake On 6/2/16, 6:58 PM, "rsync on behalf of Kevin Korb" <rsync-bounces at lists.samba.org on behalf of kmk at sanitarium.net> wrote:>Rsync only cares about the modification time. The ls command usually >abbreviates the timestamp so it is better to use the stat command on one >of the problem files to see the full thing. > >On 06/02/2016 06:42 PM, McDowell, Blake wrote: >> Cool Thanks! >> >> Specifically, the timestamps on both <src> and <dest> match for "ls -l" >> but do not match for "ls -lu" or "ls -lc” >> >> The storage is just an regular HDD in a mac pro tower. I can’t imagine >>why >> it wouldn’t handle timestamps. Also of note - this problem doesn’t exist >> for every file, just the vast majority. So, that just makes it more >> confusing. >> >> Yes, I imagine rsync is not the best for linear tape but give the choice >> between cp (which is faster and causes less problems but offers almost >> zero verbosity) and rsync, I’ll choose rsync. If people know of other >> options, I’d be very happy to know of them. >> >> On 6/2/16, 6:33 PM, "Kevin Korb" <kmk at sanitarium.net> wrote: >> >>> The man page has a section on what all the itemize-changes flags do. >>> >>> There is a --ignore-times but the result is what you have now, re-copy >>> everything even if the timestamp matches. >>> >>> The best you can really do with storage that can't handle timestamps is >>> to use --update. But if you do that you need to get rid of --partial >>> (part of -P) or else rsync will never complete a file that was >>>partially >>> copied since the partial copy is newer that the source file. >>> >>> OTOH, are you using rsync to copy files to a tape drive? If so then I >>> would think rsync is not the right tool for dealing with linear >>>storage. >>> >>> On 06/02/2016 06:25 PM, McDowell, Blake wrote: >>>> OK. Thanks. Where can I find information regarding how to interpret >>>> —itemize-changes? >>>> >>>> The timestamps aren’t changing, so the target must not be storing >>>>them, >>>> which I have no idea why. The directory I’m writing to is 777. >>>> >>>> What is the flag to tell rsync to ignore the timestamps? >>>> >>>> Thanks, >>>> Blake >>>> >>>> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb" >>>> <rsync-bounces at lists.samba.org on behalf of kmk at sanitarium.net> wrote: >>>> >>>>> It is saying the timestamp is wrong and that it is copying the file >>>>>and >>>>> changing the timestamp. If it does that every time then either the >>>>> timestamps are changing on the source or the target isn't storing >>>>>them. >>>>> >>>>> On 06/02/2016 06:13 PM, McDowell, Blake wrote: >>>>>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. >>>>>>Can >>>>>> you >>>>>> provide some insight? >>>>>> >>>>>> This is a local transfer from an external drive to an internal drive >>>>>> all >>>>>> attached to one computer. >>>>>> >>>>>> >>>>>> rsync -aPh --itemize-changes -n >>>>>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint >>>>>> /Volumes/3TB_LTO/LT003A/ >>>>>> sending incremental file list >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Deriv >>>>>>>at >>>>>>> iv >>>>>>> es >>>>>>> /2012_79_1_14_1_DER_01.mov >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Deriv >>>>>>>at >>>>>>> iv >>>>>>> es >>>>>>> /2012_79_1_14_1_DER_02.mov >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D >>>>>>>PX >>>>>>> /1 >>>>>>> 19 >>>>>>> 9WP_00086400.dpx >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D >>>>>>>PX >>>>>>> /1 >>>>>>> 19 >>>>>>> 9WP_00086401.dpx >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D >>>>>>>PX >>>>>>> /1 >>>>>>> 19 >>>>>>> 9WP_00086402.dpx >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D >>>>>>>PX >>>>>>> /1 >>>>>>> 19 >>>>>>> 9WP_00086403.dpx >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D >>>>>>>PX >>>>>>> /1 >>>>>>> 19 >>>>>>> 9WP_00086404.dpx >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D >>>>>>>PX >>>>>>> /1 >>>>>>> 19 >>>>>>> 9WP_00086405.dpx >>>>>>> f..t....... >>>>>>> >>>>>>> >>>>>>> >>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D >>>>>>>PX >>>>>>> /1 >>>>>>> 19 >>>>>>> 9WP_00086406.dpx >>>>>> >>>>>> >>>>>> ~Blake >>>>>> >>>>>> >>>>>> >>>>>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb" >>>>>> <rsync-bounces at lists.samba.org on behalf of kmk at sanitarium.net> >>>>>>wrote: >>>>>> >>>>>>> Instead of the second -v (or even the first) add --itemize-changes. >>>>>>> It >>>>>>> will tell you why each file is being copied. If the file >>>>>>>timestamps >>>>>>> are >>>>>>> not correct then perhaps the underlying storage doesn't allow >>>>>>> arbitrary >>>>>>> mtime changes. >>>>>>> >>>>>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> At my work we use rsync to move files between drives and to LTO >>>>>>>> among >>>>>>>> other things. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I¹m having an issue using rsync to move material between and >>>>>>>> external >>>>>>>> drive and an internal drive. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We run ³rsync avvPh <src> <dest>² and most of the files keep >>>>>>>> writing >>>>>>>> every time I run this. It appears that the modification times are >>>>>>>> not >>>>>>>> being carried through to the destination resulting in the files >>>>>>>> continually wanting to re-write. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I¹m hoping someone can help me figure this out. Any information >>>>>>>>you >>>>>>>> need >>>>>>>> from me (logs, etc) I¹m happy to try and provide. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Many thanks, >>>>>>>> >>>>>>>> Blake >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> >>>>>>> >>>>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-, >>>>>>>._ >>>>>>> ., >>>>>>> Kevin Korb Phone: (407) 252-6853 >>>>>>> Systems Administrator Internet: >>>>>>> FutureQuest, Inc. Kevin at FutureQuest.net (work) >>>>>>> Orlando, Florida kmk at sanitarium.net (personal) >>>>>>> Web page: http://www.sanitarium.net/ >>>>>>> PGP public key available on web site. >>>>>>> >>>>>>> >>>>>>> >>>>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-, >>>>>>>._ >>>>>>> ., >>>>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._ >>>>>., >>>>> Kevin Korb Phone: (407) 252-6853 >>>>> Systems Administrator Internet: >>>>> FutureQuest, Inc. Kevin at FutureQuest.net (work) >>>>> Orlando, Florida kmk at sanitarium.net (personal) >>>>> Web page: http://www.sanitarium.net/ >>>>> PGP public key available on web site. >>>>> >>>>> >>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._ >>>>>., >>>>> >>>> >>> >>> -- >>> >>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >>> Kevin Korb Phone: (407) 252-6853 >>> Systems Administrator Internet: >>> FutureQuest, Inc. Kevin at FutureQuest.net (work) >>> Orlando, Florida kmk at sanitarium.net (personal) >>> Web page: http://www.sanitarium.net/ >>> PGP public key available on web site. >>> >>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >>> >> > >-- >~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. Kevin at FutureQuest.net (work) > Orlando, Florida kmk at sanitarium.net (personal) > Web page: http://www.sanitarium.net/ > PGP public key available on web site. >~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., >