Hi Steven, Yes, both file systems are the same. rsync -nri --modify-window=1 <src> <dest> Gives me the following for most files >f..T....... 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/ BWAG_R2_00138428.dpx Although a few have >f..T......n 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/ BWAG_R2_00135909.dpx Strangely, no matter what HDD I¹m transferring from it is always the same sequence of files that have the n - "A n means the create time (newness) is different and is being updated to the sender's value (requires --crtimes). (I¹m not quite sure I completely understand ‹-modify-window) Here is a <dest> file example of timestamps as rsync interprets them: -rwxrwxrwx 24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx Here is a <src> file example of timestamps as rsync interprets them: -rwxrwxrwx 24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx So, yeah, that is the problem, apparently. The timestamps are transferring. But, for the files that have the ³n² the timestamps appear to be fine and those ones don¹t continually transfer. A mystery to me as to why the timestamps work on these particular files. <src> -rwxrwxrwx 24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx <dest> -rwxrwxrwx 24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx I¹m running version 3.1.2 protocol 31. Thanks, Blake On 6/2/16, 7:26 PM, "rsync on behalf of Steven Levine" <rsync-bounces at lists.samba.org on behalf of steve53 at earthlink.net> wrote:>In <D3762D63.17A7%mcdowellh at si.edu>, on 06/02/16 > at 10:42 PM, "McDowell, Blake" <McDowellH at si.edu> said: > >Hi Blake, > >>The storage is just an regular HDD in a mac pro tower. I can t imagine >>why it wouldn t handle timestamps. Also of note - this problem doesn t >>exist for every file, just the vast majority. So, that just makes it more >>confusing. > >Are the file systems the same on the source and the destination >partitions? > >Check out > > --modify-window > >in the help. If the source and destination file systems have different >timestamp precision, this is the usual solution. > >You can also try > > rsync /path-to-foo > >where path-to-foo is a directory or file. This will list the file >timestamps as rsync interprets them. > >BTW, what version of rsync are you running? It might matter. > >Steven > >-- >---------------------------------------------------------------------- >"Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. >www.scoug.com www.arcanoae.com www.warpcave.com >---------------------------------------------------------------------- > > >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: >https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
the T means that the timestamp is wrong and rsync is not fixing it because you don't have --times or --archive in your command line. On 06/08/2016 08:17 PM, McDowell, Blake wrote:> Hi Steven, > > Yes, both file systems are the same. > > rsync -nri --modify-window=1 <src> <dest> > > Gives me the following for most files >f..T....... > 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/ > BWAG_R2_00138428.dpx > Although a few have >f..T......n > 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/ > BWAG_R2_00135909.dpx > > Strangely, no matter what HDD I¹m transferring from it is always the same > sequence of files that have the n - "A n means the create time (newness) > is different and is being updated to the sender's value (requires > --crtimes). > (I¹m not quite sure I completely understand ‹-modify-window) > > Here is a <dest> file example of timestamps as rsync interprets them: > -rwxrwxrwx 24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx > > Here is a <src> file example of timestamps as rsync interprets them: > -rwxrwxrwx 24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx > > > > So, yeah, that is the problem, apparently. The timestamps are > transferring. > > But, for the files that have the ³n² the timestamps appear to be fine and > those ones don¹t continually transfer. A mystery to me as to why the > timestamps work on these particular files. > > <src> > -rwxrwxrwx 24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx > <dest> > > > -rwxrwxrwx 24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx > > I¹m running version 3.1.2 protocol 31. > > > Thanks, > Blake > > > On 6/2/16, 7:26 PM, "rsync on behalf of Steven Levine" > <rsync-bounces at lists.samba.org on behalf of steve53 at earthlink.net> wrote: > >> In <D3762D63.17A7%mcdowellh at si.edu>, on 06/02/16 >> at 10:42 PM, "McDowell, Blake" <McDowellH at si.edu> said: >> >> Hi Blake, >> >>> The storage is just an regular HDD in a mac pro tower. I can t imagine >>> why it wouldn t handle timestamps. Also of note - this problem doesn t >>> exist for every file, just the vast majority. So, that just makes it more >>> confusing. >> >> Are the file systems the same on the source and the destination >> partitions? >> >> Check out >> >> --modify-window >> >> in the help. If the source and destination file systems have different >> timestamp precision, this is the usual solution. >> >> You can also try >> >> rsync /path-to-foo >> >> where path-to-foo is a directory or file. This will list the file >> timestamps as rsync interprets them. >> >> BTW, what version of rsync are you running? It might matter. >> >> Steven >> >> -- >> ---------------------------------------------------------------------- >> "Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. >> www.scoug.com www.arcanoae.com www.warpcave.com >> ---------------------------------------------------------------------- >> >> >> -- >> Please use reply-all for most replies to avoid omitting the mailing list. >> To unsubscribe or change options: >> https://lists.samba.org/mailman/listinfo/rsync >> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20160608/2ea21f07/signature.sig>
I ran the original transfer with the -a flag. It doesn't fix the problem. -----Original Message----- From: rsync [mailto:rsync-bounces at lists.samba.org] On Behalf Of Kevin Korb Sent: Wednesday, June 08, 2016 8:26 PM To: rsync at lists.samba.org Subject: Re: rsync keeps writing files over the T means that the timestamp is wrong and rsync is not fixing it because you don't have --times or --archive in your command line. On 06/08/2016 08:17 PM, McDowell, Blake wrote:> Hi Steven, > > Yes, both file systems are the same. > > rsync -nri --modify-window=1 <src> <dest> > > Gives me the following for most files >f..T....... > 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DP > X_R2/ > BWAG_R2_00138428.dpx > Although a few have >f..T......n > 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DP > X_R2/ > BWAG_R2_00135909.dpx > > Strangely, no matter what HDD I¹m transferring from it is always the > same sequence of files that have the n - "A n means the create time > (newness) is different and is being updated to the sender's value > (requires --crtimes). > (I¹m not quite sure I completely understand ‹-modify-window) > > Here is a <dest> file example of timestamps as rsync interprets them: > -rwxrwxrwx 24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx > > Here is a <src> file example of timestamps as rsync interprets them: > -rwxrwxrwx 24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx > > > > So, yeah, that is the problem, apparently. The timestamps are > transferring. > > But, for the files that have the ³n² the timestamps appear to be fine > and those ones don¹t continually transfer. A mystery to me as to why > the timestamps work on these particular files. > > <src> > -rwxrwxrwx 24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx > <dest> > > > -rwxrwxrwx 24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx > > I¹m running version 3.1.2 protocol 31. > > > Thanks, > Blake > > > On 6/2/16, 7:26 PM, "rsync on behalf of Steven Levine" > <rsync-bounces at lists.samba.org on behalf of steve53 at earthlink.net> wrote: > >> In <D3762D63.17A7%mcdowellh at si.edu>, on 06/02/16 >> at 10:42 PM, "McDowell, Blake" <McDowellH at si.edu> said: >> >> Hi Blake, >> >>> The storage is just an regular HDD in a mac pro tower. I can t >>> imagine why it wouldn t handle timestamps. Also of note - this >>> problem doesn t exist for every file, just the vast majority. So, >>> that just makes it more confusing. >> >> Are the file systems the same on the source and the destination >> partitions? >> >> Check out >> >> --modify-window >> >> in the help. If the source and destination file systems have >> different timestamp precision, this is the usual solution. >> >> You can also try >> >> rsync /path-to-foo >> >> where path-to-foo is a directory or file. This will list the file >> timestamps as rsync interprets them. >> >> BTW, what version of rsync are you running? It might matter. >> >> Steven >> >> -- >> --------------------------------------------------------------------- >> - "Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. >> www.scoug.com www.arcanoae.com www.warpcave.com >> --------------------------------------------------------------------- >> - >> >> >> -- >> Please use reply-all for most replies to avoid omitting the mailing list. >> To unsubscribe or change options: >> https://lists.samba.org/mailman/listinfo/rsync >> Before posting, read: >> http://www.catb.org/~esr/faqs/smart-questions.html > >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
In <D37E24FA.120D%mcdowellh at si.edu>, on 06/09/16 at 12:17 AM, "McDowell, Blake" <McDowellH at si.edu> said: Hi Blake, Please reply to the list.>rsync -nri --modify-window=1 <src> <dest>As others mentioned, you need need to use --times. This is needed so that we can see use output from --itemize-changes.>Gives me the following for most files >f..T....... >2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/ >BWAG_R2_00138428.dpx >Although a few have >f..T......n >2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/ >BWAG_R2_00135909.dpxAt a certain level this makes sense. Without --times, the timestamps are not used to determine whether or not a file needs to be transferred and, in addition, the receiver will set the timestamp to the current time on the receiver. This is what the docs mean when they say "Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective;">(I¹m not quite sure I completely understand -modify-window)You should not need --modify-window. Modify window is needed when the source and destination file systems have differing timestamp resolutions. For example if transferring to a Window's filesystem that had 2-second timestamp resolution from a *ix system with 1-second resolution, you would need to use --modify-window=2 to avoid spurious transfers.>Here is a <dest> file example of timestamps as rsync interprets them: >-rwxrwxrwx 24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx>Here is a <src> file example of timestamps as rsync interprets them: >-rwxrwxrwx 24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpxWithout --times, this is the expected behavior. The timestamps differ, so rsync will transfer the file. Because the file content is the same, the transfer will be quick, but a tranfer will happen. If you cannot use --times, you many need to use some combination of --update and --ignore-times an possibly --size-only to avoid selecting these files for transfer. Exactly which options will be approriate will depend on the content of your files and how the content changes. FWIW, if --times cannot set the timestamp correctly on the receiver, I would suspect an issue with the filesystem or your rsync build. Rsync uses the standard platform APIs for setting the timestamps so this should just work.>But, for the files that have the ³n²What "n" do you mean? Another FWIW, when testing, --dry-run (i.e. -n) is useful to understand which files will be transferred, especially when working with complex filters, but you need to run without -n to see that true results of the transfer. Steven -- ---------------------------------------------------------------------- "Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. www.scoug.com www.arcanoae.com www.warpcave.com ----------------------------------------------------------------------
Hi Steve, I run my first transfer always as rsync -avvPh. I now add in -i based on what I’ve learned here. My understanding from the rsync man page is that -a or ―archive implicitly includes -t or ―times. Am I mistaken there? Do I need to run ―times separately? The output of my first transfer is:>f++++++++++ 2012_79_1_79_1__New_Ark__Derivatives/2012_79_1_79_1_DER_01.mov12.09G 100% 132.21MB/s 0:01:27 (xfr#4, to-chk=13489/13497) When I immediately re-run the same rsync command in dry-run mode I get:>f..t....... 2012_79_1_79_1__New_Ark__Derivatives/2012_79_1_79_1_DER_01.movI am running about 13,000 files on this transfer and that is what is case for every file except for a very small amount that output: .f 2012_79_1_79_1__New_Ark__UHD_DPX/NewArk_00086558.dpx It is a mystery to me as to why these particular files are transferring their timestamps correctly. Re-running the rysnc command without dry run does not make only the timestamps transfer; the whole file retransfers and it takes the same length of time to transfer as the first time. This is continually the case, the timestamps never update. I can’t imagine what would be wrong with my rsync build - I pull it straight from homebrew/github /Users/medialab ?:? brew install rsync Warning: homebrew/dupes/rsync-3.1.2 already installed Maybe the file system… Any suggested troubleshooting for that or rsync? Thanks, Blake On 6/10/16, 11:43 PM, "rsync on behalf of Steven Levine" <rsync-bounces at lists.samba.org on behalf of steve53 at earthlink.net> wrote:>In <D37E24FA.120D%mcdowellh at si.edu>, on 06/09/16 > at 12:17 AM, "McDowell, Blake" <McDowellH at si.edu> said: > >Hi Blake, > >Please reply to the list. > >>rsync -nri --modify-window=1 <src> <dest> > >As others mentioned, you need need to use --times. This is needed so that >we can see use output from --itemize-changes. > >>Gives me the following for most files >f..T....... >>2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R >>2/ >>BWAG_R2_00138428.dpx >>Although a few have >f..T......n >>2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R >>2/ >>BWAG_R2_00135909.dpx > >At a certain level this makes sense. Without --times, the timestamps are >not used to determine whether or not a file needs to be transferred and, >in addition, the receiver will set the timestamp to the current time on >the receiver. > >This is what the docs mean when they say > >"Note that if this option is not used, the optimization that excludes >files that have not been modified cannot be effective;" > >>(I¹m not quite sure I completely understand -modify-window) > >You should not need --modify-window. Modify window is needed when the >source and destination file systems have differing timestamp resolutions. >For example if transferring to a Window's filesystem that had 2-second >timestamp resolution from a *ix system with 1-second resolution, you would >need to use --modify-window=2 to avoid spurious transfers. > >>Here is a <dest> file example of timestamps as rsync interprets them: >>-rwxrwxrwx 24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx > >>Here is a <src> file example of timestamps as rsync interprets them: >>-rwxrwxrwx 24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx > >Without --times, this is the expected behavior. The timestamps differ, so >rsync will transfer the file. Because the file content is the same, the >transfer will be quick, but a tranfer will happen. > >If you cannot use --times, you many need to use some combination of >--update and --ignore-times an possibly --size-only to avoid selecting >these files for transfer. Exactly which options will be approriate will >depend on the content of your files and how the content changes. > >FWIW, if --times cannot set the timestamp correctly on the receiver, I >would suspect an issue with the filesystem or your rsync build. Rsync >uses the standard platform APIs for setting the timestamps so this should >just work. > >>But, for the files that have the ³n² > >What "n" do you mean? > >Another FWIW, when testing, --dry-run (i.e. -n) is useful to understand >which files will be transferred, especially when working with complex >filters, but you need to run without -n to see that true results of the >transfer. > >Steven > >-- >---------------------------------------------------------------------- >"Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. >www.scoug.com www.arcanoae.com www.warpcave.com >---------------------------------------------------------------------- > > >-- >Please use reply-all for most replies to avoid omitting the mailing list. >To unsubscribe or change options: >https://lists.samba.org/mailman/listinfo/rsync >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html