samba-bugs@samba.org
2009-Apr-18 21:02 UTC
DO NOT REPLY [Bug 6276] New: crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 Summary: crtimes.patch does not preserve creation dates on Mac x86_64 only Product: rsync Version: 3.0.6 Platform: x86 OS/Version: Mac OS X Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: wayned@samba.org ReportedBy: bugzilla.samba.org@maclemon.at QAContact: rsync-qa@samba.org Summary: rsync with fileflags and crtimes patches does not preserve creation date/time on Mac OS X 10.5.6 under x86_64 when compiled as x86_64. I'm compiling rsync 3.0.6-pre1 (and 3.0.5) on Mac OS X Leopard 10.5.6 (9G2141) with gcc version 4.0.1 (Apple Inc. build 5490). I'm creating a 4 way universal binary for ppc, ppc64, i386 and x86_64. crtimes patch works fine for 3.0.5 and 3.0.6-pre1 on ppc, ppc64 and i386. However rsync does not preserve creation dates on Mac OS X when run as x86_64 binary. Running the same rsync with the same patch created as i386 (32 bit binary) DOES correctly preserve creation dates. This is how I build rsync 3.0.6-pre1 as a 4 way universal binary: patch -p1 <../patches/fileflags.diff ./prepare-source patch -p1 <../patches/crtimes.diff ./configure --prefix=/usr/local/maclemon-beta --disable-debug CC="gcc -std=gnu99 -mmacosx-version-min=10.4 -isysroot /Developer/SDKs/MacOSX10.4u.sdk" CFLAGS="-g -fprefetch-loop-arrays -funroll-loops -O3 -DHAVE_CONFIG_H -Wall -W -I./popt -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -arch ppc64 -arch x86_64" LDFLAGS="-mmacosx-version-min=10.4 -arch ppc -arch i386 -arch ppc64 -arch x86_64" make Result: MD5 (rsync) = 9cc18a386a256f6aecd9c46815088594 SHA1(rsync)= d4c9b32e088208e12821f96117b632f20750e08f rsync: Mach-O universal binary with 4 architectures rsync (for architecture ppc): Mach-O executable ppc rsync (for architecture i386): Mach-O executable i386 rsync (for architecture ppc64): Mach-O 64-bit executable ppc64 rsync (for architecture x86_64): Mach-O 64-bit executable x86_64 -rwxr-xr-x 1 pepi staff 2535144 18 Apr 22:49 rsync When created this way, rsync running on an intel Core2Duo based Mac will not preserve creation date/time. I've tested this with backupbouncer 0.1.3: ------------------ rsync3-maclemon-beta ------------------ This copier produced log output in: /Volumes/Dst/81-rsync3-maclemon-beta/log Verifying: basic-permissions ... ok (Critical) Verifying: timestamps ... ok (Critical) Verifying: symlinks ... ok (Critical) Verifying: symlink-ownership ... ok Verifying: hardlinks ... ok (Important) Verifying: resource-forks ... Sub-test: on files ... ok (Critical) Sub-test: on hardlinked files ... ok (Important) Verifying: finder-flags ... ok (Critical) Verifying: finder-locks ... ok Verifying: creation-date ... FAIL <-----------------FAILS here Verifying: bsd-flags ... ok Verifying: extended-attrs ... Sub-test: on files ... ok (Important) Sub-test: on directories ... ok (Important) Sub-test: on symlinks ... ok Verifying: access-control-lists ... Sub-test: on files ... ok (Important) Sub-test: on dirs ... ok (Important) Verifying: fifo ... ok Verifying: devices ... ok Verifying: combo-tests ... Sub-test: xattrs + rsrc forks ... ok Sub-test: lots of metadata ... ok When compiled as i386 (32bit) only (doesn't matter if I also compile in the ppc and ppc64 versions) The ONLY difference to the above build is the omission of -arch x86_64 in configure. patch -p1 <../patches/fileflags.diff ./prepare-source patch -p1 <../patches/crtimes.diff ./configure --prefix=/usr/local/maclemon-beta --disable-debug CC="gcc -std=gnu99 -mmacosx-version-min=10.4 -isysroot /Developer/SDKs/MacOSX10.4u.sdk" CFLAGS="-g -fprefetch-loop-arrays -funroll-loops -O3 -DHAVE_CONFIG_H -Wall -W -I./popt -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -arch ppc64" LDFLAGS="-mmacosx-version-min=10.4 -arch ppc -arch i386 -arch ppc64" make MD5 (rsync) = 85e347ea3cf8567b1b2f980c673d856f SHA1(rsync)= 23c2c188a80db12ca268202fa28ce32c1a02e804 rsync: Mach-O universal binary with 3 architectures rsync (for architecture ppc): Mach-O executable ppc rsync (for architecture i386): Mach-O executable i386 rsync (for architecture ppc64): Mach-O 64-bit executable ppc64 -rwxr-xr-x 1 pepi staff 1888720 18 Apr 22:56 rsync Now backup bouncer reports this: ------------------ rsync3-maclemon-beta ------------------ This copier produced log output in: /Volumes/Dst/81-rsync3-maclemon-beta/log Verifying: basic-permissions ... ok (Critical) Verifying: timestamps ... ok (Critical) Verifying: symlinks ... ok (Critical) Verifying: symlink-ownership ... ok Verifying: hardlinks ... ok (Important) Verifying: resource-forks ... Sub-test: on files ... ok (Critical) Sub-test: on hardlinked files ... ok (Important) Verifying: finder-flags ... ok (Critical) Verifying: finder-locks ... ok Verifying: creation-date ... ok <-------------- WORKS! Verifying: bsd-flags ... ok Verifying: extended-attrs ... Sub-test: on files ... ok (Important) Sub-test: on directories ... ok (Important) Sub-test: on symlinks ... ok Verifying: access-control-lists ... Sub-test: on files ... ok (Important) Sub-test: on dirs ... ok (Important) Verifying: fifo ... ok Verifying: devices ... ok Verifying: combo-tests ... Sub-test: xattrs + rsrc forks ... ok Sub-test: lots of metadata ... ok Conclusion: crtimes patch does not work when rsync 3.0.6-pre1 (and 3.0.5) is compiled for x86_64 and run on Core2Duo in 64bit mode on Mac OS X 10.5.6. crtimes patch does only work when rsync is compiled in 32bit mode (i386) or for any PPC build. Best regards Pepi -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs@samba.org
2009-Apr-25 21:59 UTC
DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 wayned@samba.org changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |WORKSFORME ------- Comment #1 from wayned@samba.org 2009-04-25 17:00 CST ------- I compiled an x86_64 binary of rsync on OS X, and it worked fine to preserve create times. I've seen other reports about combination binaries having issues, so I assume that this is an issue with the tools creating a combination binary, not in rsync. If you discover otherwise, please feel free to let me know. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs@samba.org
2009-Apr-25 22:09 UTC
DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 ------- Comment #2 from matt@mattmccutchen.net 2009-04-25 17:10 CST ------- The recommended way to make a universal binary of rsync is to compile a separate binary for each architecture and then combine them using "lipo". See: http://lists.samba.org/archive/rsync/2008-March/020271.html -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs@samba.org
2009-May-10 19:57 UTC
DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 ------- Comment #3 from bugzilla.samba.org@maclemon.at 2009-05-10 14:58 CST ------- Created an attachment (id=4136) --> (https://bugzilla.samba.org/attachment.cgi?id=4136&action=view) Output of rsync log when testing with backup bouncer 0.1.3 rsync 3.0.6 for crtimes -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs@samba.org
2009-May-10 19:58 UTC
DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 ------- Comment #4 from bugzilla.samba.org@maclemon.at 2009-05-10 14:59 CST ------- Created an attachment (id=4137) --> (https://bugzilla.samba.org/attachment.cgi?id=4137&action=view) Output of backup bouncer 0.1.3 when testing rsync 3.0.6 for crtimes problem -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs@samba.org
2009-May-10 20:02 UTC
DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 bugzilla.samba.org@maclemon.at changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|WORKSFORME | ------- Comment #5 from bugzilla.samba.org@maclemon.at 2009-05-10 15:02 CST ------- Retested with rsync 3.0.6 release version, and x86_64 on Mac OS X still does NOT preserve creation dates with the crtimes.patch. This time I've created a single (non-fat) binary for testing. Apply patches: patch -p1 <../patches/fileflags.diff ./prepare-source patch -p1 <../patches/crtimes.diff This is how I configure: ./configure --prefix=/usr/local/maclemon-beta --disable-debug CC="gcc -std=gnu99 -mmacosx-version-min=10.4 -isysroot /Developer/SDKs/MacOSX10.4u.sdk" CFLAGS="-g -fprefetch-loop-arrays -funroll-loops -O3 -DHAVE_CONFIG_H -Wall -W -I./popt -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch x86_64" LDFLAGS="-mmacosx-version-min=10.4 -arch x86_64" MD5 (rsync) = 6aab05fca8e521e8a6afa2d2099e7f85 SHA1(rsync)= 89f0f988ee01cf96ce5da22a249142545d426008 rsync: Mach-O 64-bit executable x86_64 -rwxr-xr-x 1 pepi staff 650544 10 Mai 21:44 rsync When testing crtimes with "make test" the test does pass. Yet when copying a file with rsync x86_64 on 10.5.6 the creation time of the copied file is set to the modification date of the file. Output of "make test" $ sudo make test pepi > root@marvelousmonolith.maclemon.lan Password: rsync_bin=`pwd`/rsync ./runtests.sh ===========================================================./runtests.sh running in /Users/pepi/unix/rsync/rsync-3.0.6/rsync-3.0.6 rsync_bin=/Users/pepi/unix/rsync/rsync-3.0.6/rsync-3.0.6/rsync srcdir=/Users/pepi/unix/rsync/rsync-3.0.6/rsync-3.0.6 TLS_ARGS= -L testuser=root os=Darwin marvelousmonolith.maclemon.lan 9.6.2 Darwin Kernel Version 9.6.2: Tue Jan 13 20:42:22 PST 2009; root:xnu-1228.9.80~1/RELEASE_I386 i386 preserve_scratch=no scratchbase=/Users/pepi/unix/rsync/rsync-3.0.6/rsync-3.0.6/testtmp PASS 00-hello PASS acls PASS backup PASS batch-mode PASS chgrp PASS chmod-option SKIP chmod-temp-dir (Can't find a tmp dir on a different file system) PASS chmod SKIP chown-fake (Can't chown (probably need root)) PASS chown PASS compare-dest PASS crtimes PASS daemon-gzip-download PASS daemon-gzip-upload PASS daemon SKIP default-acls (I don't know how to use your setfacl command) PASS delete SKIP devices-fake (Can't create char device node) PASS devices SKIP dir-sgid (Your filesystem doesn't use directory setgid; maybe it's BSD.) PASS duplicates PASS exclude PASS executability PASS files-from PASS fuzzy PASS hands PASS hardlinks PASS itemize PASS longdir PASS merge PASS missing PASS relative PASS ssh-basic PASS symlink-ignore PASS trimslash PASS unsafe-byname PASS unsafe-links PASS wildmatch SKIP xattrs (Unable to set an xattr) ------------------------------------------------------------ ----- overall results: 33 passed 6 skipped ------------------------------------------------------------ overall result is 0 Testing with backoup bouncer 0.1.3: This copier produced log output in: /Volumes/Dst/81-rsync3-maclemon-beta/log Verifying: basic-permissions ... ok (Critical) Verifying: timestamps ... ok (Critical) Verifying: symlinks ... ok (Critical) Verifying: symlink-ownership ... ok Verifying: hardlinks ... ok (Important) Verifying: resource-forks ... Sub-test: on files ... ok (Critical) Sub-test: on hardlinked files ... ok (Important) Verifying: finder-flags ... ok (Critical) Verifying: finder-locks ... ok Verifying: creation-date ... FAIL Verifying: bsd-flags ... ok Verifying: extended-attrs ... Sub-test: on files ... ok (Important) Sub-test: on directories ... ok (Important) Sub-test: on symlinks ... ok Verifying: access-control-lists ... Sub-test: on files ... ok (Important) Sub-test: on dirs ... ok (Important) Verifying: fifo ... ok Verifying: devices ... ok Verifying: combo-tests ... Sub-test: xattrs + rsrc forks ... ok Sub-test: lots of metadata ... ok Logs from the backup bouncer and rsync output are attached. The identical way to configure and build rsync as i386 (32bit) works fine for all creation timestamp tests. Be that with make test or by manually checking in the filesystem. x86_64 fails. All tests were done as root. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs at samba.org
2010-Feb-06 07:14 UTC
DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 ------- Comment #6 from steve at ortizaggies.com 2010-02-06 01:14 CST ------- Created an attachment (id=5288) --> (https://bugzilla.samba.org/attachment.cgi?id=5288&action=view) patch to fix creation dates bug on Mac x86_64 I ran into this same bug (6276) today, using the same tool (backup bouncer) to check how rsync would work on Mac OS X (Snow Leopard, 10.6.2). rsync passed every test except for the "creation-date" test... this was surprising because I had installed the crtimes (and fileflags) patch. I expected rsync to work flawlessly. I looked into it and figured out the problem was in syscall.c, which effects both rsync and the testsuite. Following is a summary of the problem, the fix, and steps to verify the problem and the fix. Problem: The get_create_time function calls the getattrlist system call, but the assumptions that worked on a 32-bit version don't hold for 64-bit. Here's an excerpt from the man page for getattrlist... ============================ The data returned in the buffer described by attrBuf and attrBufSize is formatted as follows. 1. The first element of the buffer is a u_int32_t that contains the overall length, in bytes, of the attributes returned. This size includes the length field itself. 2. Following the length field is a list of attributes. Each attribute is represented by a field of its type, where the type is given as part of the attribute description (below). 3. The attributes are placed into the attribute buffer in the order that they are described below. 4. Each attribute is aligned to a 4-byte boundary (including 64-bit data types). ============================ The key thing to note is that the data start 4 bytes into the buffer. The current implementation uses the following struct as the attrBuf... struct create_time { unsigned long length; struct timespec crtime; }; Assuming the unsigned long is 4 bytes, and the crtime field continues immediately afterwards, this would be valid. Unfortunately, what's happening is the crtime field starts 8 bytes into the struct, so it is 4 bytes offset, and the time returned is 0, which is also the value returned by get_create_time to indicate an error. Since this is not really a problem with the system call, it's a problem with how the result is being extracted, the errno is 0 (Unknown error). Unfortunately, the crtimes test in the test suite passes. crtimes.test calls on checkit in rsync.fns, which calls on rsync_ls_lR in the same file, which calls on the tls tool, which uses the same get_create_time function that's causing rsync to fail. tls produces the following stderr message with the --crtimes (or -N) option, but it results in a zero length output, which matches the other file listing it compares to "verify" that crtimes.test is working. That's why the test appears to pass. tls: get_create_time show-bug/from/foo: Unknown error: 0 Fix: So, the fix is fairly straightforward... the get_create_time function needs to grab the right set of bytes. My patch is attached... you can see I got rid of the struct create_time, and I use a char array to load results from the system call. It works, but if you want to do it differently, I won't mind one bit (or 4 bytes). Verify: Here's what you can do to verify the problem and the fix. Note, I think this problem is limited to Mac OS X (64-bit), so you'll need to test it accordingly. tar xzf rsync-3.0.7.tar.gz tar xzf rsync-patches-3.0.7.tar.gz cd rsync-3.0.7 patch -p1 <patches/fileflags.diff patch -p1 <patches/crtimes.diff ./configure make make test # it should pass all the tests, so here's how you can manually see it is really broken... mkdir -p show-bug/from mkdir show-bug/to touch -t 200111111111.11 show-bug/from/foo # created date/time touch -t 200212122222.22 show-bug/from/foo # modified date/time ./rsync -rtg --crtimes show-bug/from/ show-bug/to ./tls -N show-bug/from/foo # this is how the testsuite tries to verify it... broken test GetFileInfo -d show-bug/from/foo # shows created date GetFileInfo -d show-bug/to/foo # shows the modified date, but should show created... broken rsync Now, you should be able to run my patch and go through the same steps... you'll see rsync applies the correct creation date now. patch -p1 <../fix_get_create_time.patch # apply my patch (wherever you saved it) make make test # tests still pass, but tls works this time so it's more meaningful rm show-bug/to/foo ./rsync -rtg --crtimes show-bug/from/ show-bug/to ./tls -N show-bug/from/foo # shows correct output now, no errors ./tls -N show-bug/to/foo # output matches above, good GetFileInfo -d show-bug/from/foo # shows created date GetFileInfo -d show-bug/to/foo # output matches above, good More comments: To be thorough, the test for crtime should probably be updated too, so it fails if the files are empty. There are a lot of ways to do this... You may want to consider adding some output to stdout when tls fails... you could then grep for that text in the checkit function to see if any tls commands failed, and then you'd probably want to fail that test. Otherwise, similar problems to this could silently pass. Philosophically, maybe it would be better not to use tls in the testing, but I guess you've got to trust something for your tests. This was probably a rare case, where the same bug was affecting both rsync and the testsuite. Thanks to everyone who maintains rsync! It's a great tool, and I wouldn't have bothered to write up this fix if I didn't find it useful. Thanks to n8gray for backup bouncers, and Mike Bombich who has some nice rsync for Mac OS X help on his website. Here are some odds and ends... the rsync version info and my backup bouncer 0.2.0 test results for my modified rsync (the untouched version 3.0.7 fails the creation-date test, but this fix will produces a version of rsync that passes everything). If you need anything else to close out this bug, please let me know. I am glad to help. $ rsync --version rsync version 3.0.7 protocol version 30 Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, no iconv, symtimes, file-flags $ sudo ./bbouncer copy -c 15-rsync-3.0.7 -d /Volumes/Src /Volumes/Dst src = /Volumes/Src dst = /Volumes/Dst Enabling owners on src/dst disks /dev/disk1s1 on /Volumes/Src (hfs, local, journaled) /dev/disk2s1 on /Volumes/Dst (hfs, local, journaled) Cleaning. Copying with: rsync-3.0.7 ... ok ------------------ rsync-3.0.7 ------------------ Verifying: basic-permissions ... ok (Critical) Verifying: timestamps ... ok (Critical) Verifying: symlinks ... ok (Critical) Verifying: symlink-ownership ... ok Verifying: hardlinks ... ok (Important) Verifying: resource-forks ... Sub-test: on files ... ok (Critical) Sub-test: on hardlinked files ... ok (Important) Verifying: finder-flags ... ok (Critical) Verifying: finder-locks ... ok Verifying: creation-date ... ok Verifying: bsd-flags ... ok Verifying: extended-attrs ... Sub-test: on files ... ok (Important) Sub-test: on directories ... ok (Important) Sub-test: on symlinks ... ok Verifying: access-control-lists ... Sub-test: on files ... ok (Important) Sub-test: on dirs ... ok (Important) Verifying: fifo ... ok Verifying: devices ... ok Verifying: combo-tests ... Sub-test: xattrs + rsrc forks ... ok Sub-test: lots of metadata ... ok -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs at samba.org
2010-Feb-06 21:57 UTC
DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 ------- Comment #7 from wayned at samba.org 2010-02-06 15:57 CST ------- Thanks for the detailed analysis! I'm thinking that all we need to do is to change "unsigned long" to "uint32" in the create_time structure. I've checked in that change into the patches repo (both branches). Would you try that and see if it works? +struct create_time { -+ unsigned long length; ++ uint32 length; + struct timespec crtime; +}; -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs at samba.org
2010-Feb-11 18:19 UTC
DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
https://bugzilla.samba.org/show_bug.cgi?id=6276 wayned at samba.org changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution| |FIXED ------- Comment #8 from wayned at samba.org 2010-02-11 12:19 CST ------- I have used Steve's suggested fix of wrapping the structure with an alignment pragma: #pragma pack(push) #pragma pack(4) ...create_time struct here... #pragma pack(pop) Since this is specific to OS X, I think that should be OK. If that eventually turns out to be problematic, Steve's suggested fix to use a byte array will be used instead. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
Seemingly Similar Threads
- DO NOT REPLY [Bug 6276] crtimes.patch does not preserve creation dates on Mac x86_64 only
- fileflags.diff patch
- Passed all tests with flying colors on Mac OS X 10.4.11 - synopsis of installation and testing
- Bug with crtimes and hard links?
- [Bug 13522] New: Patch fileflags.diff and crtimes.diff