rsync - Nov 2011 - [Bug 8566] New: Spotlight comments (extended attributes) are not synced

https://bugzilla.samba.org/show_bug.cgi?id=8566

           Summary: Spotlight comments (extended attributes) are not
                    synced
           Product: rsync
           Version: 3.0.9
          Platform: All
        OS/Version: Mac OS X
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: p.org at gmx.at
         QAContact: rsync-qa at samba.org


rsync 3.0.9 on Mac OS X 10.6.8

As an example for working extended attributes: File labels are always properly
synced: Both during their first sync when the file + attributes is copied and
also after you change their states and sync again, the file remains untouched,
but the attributes are properly updated!

But Spotlight comments are not transferred, neither when a file is synced the
first time nor when the comment is changed, and another sync is then run.


For completeness, this is the command line I ran:

rsync \
--protect-args \
\
--archive \
--crtimes \
--hard-links \
--acls \
--xattrs \
--fileflags \
--one-file-system \
--force-change \
\
--exclude=".*" \
--exclude="Icon*" \
\
--verbose \
--stats \
--progress \
--itemize-changes \
--human-readable \
\
--dry-run \
\
~/Temporary/ \
/Volumes/ExternalDisk/All/

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #1 from Stefan Nowak <p.org at gmx.at> 2011-12-01 21:31:18 UTC
---
1) Could someone please answer how to successfully sync Spotlight comments?
Thanks!

2) As of 3.0.9 this functionality seems not to be integrated in the main
distribution, as my testing in my previous comment shows, but through patching
it seems to be possible

User Gnarlodious reports at:
https://discussions.apple.com/message/12142363#12142363
Downloaded 3.0.7 and applied the patch "fileflags.diff" (among others)
and then
comments seem to get properly synced.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #2 from Stefan Nowak <p.org at gmx.at> 2011-12-01 21:58:28 UTC
---
I downloaded 3.0.9 source + patches, patched in the 2 patches:
patch -p1 <patches/fileflags.diff
patch -p1 <patches/crtimes.diff
And although running with the appropriate arguments (as in the original
report), the comments still do not get synced!

Help please!

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

> I downloaded 3.0.9 source + patches, patched in the 2 patches:
> patch -p1 <patches/fileflags.diff
> patch -p1 <patches/crtimes.diff
> And although running with the appropriate arguments (as in the original
> report), the comments still do not get synced!
> 
> Help please!

I believe this issue has been previously mentioned. I do not remember the
outcome or if it has been resolved. There may be a duplicate bug in the tracking
system or something mentioned on the mailing list relating to this bug. I would
suggest having a hunt in the mailing list archives and the bug tracking system.

A quick search brings up the following (unfortunately no answers).
http://www.mail-archive.com/rsync at lists.samba.org/msg21846.html

If you use the latest development branch is the problem still present?

Hope this helps.


---------------------------------------------------------------------
This message is protected by LBackup, an open source backup solution.
http://www.lbackup.org

https://bugzilla.samba.org/show_bug.cgi?id=8566

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #3 from Wayne Davison <wayned at samba.org> 2011-12-24
18:28:18 UTC ---
Hmm, stock 3.0.7 with official fileflags & crtimes patches was working and
3.0.9 with those patches is not?  That's weird.  I don't have access to
an OS X
box at the moment, but should be able to look into this in the future.  If
anyone wants to dive in and debug, feel free.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #4 from Stefan Nowak <p.org at gmx.at> 2012-03-23 13:21:24 UTC
---
I'd appreciate some help!
I'm still unable to correctly rsync Finder / Spotlight comments.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #5 from Stefan Nowak <p.org at gmx.at> 2012-03-23 13:47:44 UTC
---
(In reply to comment #3)
> Hmm, stock 3.0.7 with official fileflags & crtimes patches was working
and 3.0.9 with those patches is not?  That's weird.
I now also downloaded source 3.0.7, patched it with 3.0.7, and built it, and it
doesn't work as well.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #6 from Mike Bombich <mike at bombich.com> 2012-03-23 15:25:19
UTC ---
My own build of rsync preserves Spotlight comments just fine. Those comments
are stored as extended attributes, so as long as you preserve xattrs, your
Spotlight comments should be preserved. Before you make the conclusion that
your Spotlight comments have *not* been preserved, verify whether the
"com.apple.metadata:kMDItemFinderComment" exists on the destination
file. Also,
you should verify that Spotlight is enabled on the destination.

mdutil -s /Volumes/Target

[bombich:/Volumes/Source] touch test
[bombich:/Volumes/Source] xattr --list test

<add a Spotlight comment in Finder>

[bombich:/Volumes/Source] xattr --list test
com.apple.metadata:kMDItemFinderComment

[bombich:/Volumes/Source] xattr --get com.apple.metadata:kMDItemFinderComment
test
com.apple.metadata:kMDItemFinderComment    bplist00^This is a tes   

[bombich:/Volumes/Source] ls -la /Volumes/Target/test
ls: /Volumes/Target/test: No such file or directory

<rsync Source to Target, preserving xattrs -- I used 3.0.6 + some various bug
fixes from 3.0.7-3.0.9>

[bombich:/Volumes/Source] ls -la /Volumes/Target/test
-rw-r--r--@ 1 bombich  wheel  0 Mar 23 11:18 /Volumes/Target/test

[bombich:/Volumes/Source] xattr --list /Volumes/Target/test 
com.apple.metadata:kMDItemFinderComment

[bombich:/Volumes/Source] xattr --get com.apple.metadata:kMDItemFinderComment
/Volumes/Target/test 
com.apple.metadata:kMDItemFinderComment    bplist00^This is a tes   

[Note: I use a custom build of the "xattr" tool so the arguments may
be
different on your system]

Lastly, you should note that Finder has some display problems with Spotlight
comments. I don't know if the problem resides within Finder or Spotlight,
but
I've seen odd delays associated with the proper information appearing the
the
Spotlight comments field. For example, the Spotlight comments appear in the
xattr on the destination file, but they do not appear immediately in the
Finder. Additionally, if you delete the source file, then re-touch it and Get
Info, the old Spotlight comments are still there, even though the xattr is
empty. So don't use the Finder as a measure of whether these comments are
preserved, use the xattr contents instead.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #7 from Stefan Nowak <p.org at gmx.at> 2012-03-24 14:02:49 UTC
---
Dear Mike, thanks for your effort!
Some questions remain, maybe you can help?

I performed the tests as advised, and can now confirm:
What I experience(d) all the time, is indeed a heavy Finder bug, rather than an
rsync bug!

xattr proofed, that after every rsync, all comments were properly synced!

It is the buggy Finder, which hides the proper comment info:
* Finder does not only conceal Finder comments of freshly synced files (0-10
seconds ago), but also from files, which were synced many months (!) and
sessions ago!
* Even if I select "Show Info" (CMD-I) for a destination file, which
was
recently or long ago rsynced, I get no info, even though xattr prooves, that it
is there!
* "Odd delay" is an understatement! This is a large scale bug.

Is there any way to force Finder to update its comment display?
(Or a Spotlight Update/Re-Indexing of the target volume?)

The methods below did not accomplish comment display update:
* Refresh a Finder window to force it to update:
http://hints.macworld.com/article.php?story=2009091413423819
* Restart Finder
* Log out & in again
* Restart Mac

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #8 from Stefan Nowak <p.org at gmx.at> 2012-03-24 14:15:36 UTC
---
Interesting: I now performed a .DS_Store removal, according to:
http://www.finetunedmac.com/forums/ubbthreads.php?ubb=showflat&Number=11266#Post11266

cd /Volumes/Destination/Folder
osascript -e 'tell application "Finder" to quit'
rm .DS_Store
rm ../.DS_Store
osascript -e 'tell application "Finder" to activate'

After that ALL comments are gone (in the Finder view).
Before that, at least those which I added at the destination were shown.
xattr -rl   prooves that all comments are still there, only now NONE shown in
Finder.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #9 from Stefan Nowak <p.org at gmx.at> 2012-03-24 14:45:18 UTC
---
mdutil -E /Volumes/Destination

Re-indexing the destination volume did not help either.
xattr -rl showed that all comment data is still there.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #10 from Stefan Nowak <p.org at gmx.at> 2013-04-05 06:56:45
UTC ---
User V.K. posted a script here:
https://discussions.apple.com/message/10549961#10549961

I ran V.K.'s script over the rsync backup'ed files and then the comments
were
restored! Also after disconnecting, un-powering, re-powering, re-connecting the
external harddisk, the restored comments persisted.

My post: https://discussions.apple.com/message/21704242#21704242

On my rsync backup destination the files had shown NO comments in the Finder
but
mdls -name kMDItemFinderComment <YourFile> had revealed that they were
stored
(at least somewhere).

For test purposes I created the folder "_1" with comment "1"
on the source with
Mac OS X Finder 10.6.8, ran rsync 3.0.7 (compiled with xattrs and others), and
on the destination again no comment was shown for folder "_1". After
running
the comment-restore-applescript over "_1" the comment was there! I
created
folder "_2" with comment "22" on the source, and copied it
via Finder. The
comment appeared on the destination as well.

CONCLUSION:
It seems rsync syncs the Finder comments to a deprecated storage place?!
Or Finder has a bug, ignoring info on valid storage positions.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8566

--- Comment #11 from Stefan Nowak <p.org at gmx.at> 2013-05-02 10:27:05
UTC ---
Apple Developer Bug Reporting Team replied today:

Engineering has determined that this issue behaves as intended based on the
following information:
Comments are stored primarily in the .DS_Store files and mirrored to Spotlight
for searching. xattrs are not the primary storage.
This is the effect of attributes being stored externally, and command line
tools not being aware of them.

---

Now I realize why my rsync failed:
I had used --exclude=".*" which excluded .DS_Store from being synced!


Finder's metadata handling being only .DS_Store-aware and mostly ignoring
xattrs poses the following problems:

PROBLEM 1: It is neither fish nor fowl

a) The labels ARE considered from xattrs, more precisely at least ALSO
considered from xattrs.
Proof: rsync xattr sync results immediately visible in Finder.

b) Whereas the comments are considered SOLELY from .DS_store.
Proof: Rsync results not visible in Finder.
Workaround: See comment 10

PROBLEM 2: Partial file+metadata syncs are IMPOSSIBLE by design without
.DS_Store merging capabilities

Even if I would include .DS_Store files in my sync, synching a folder partially
MUST fail by design. Why?

If metadata would be per file, it is transported correctly on whatever
selection of files I sync over.
If metadata is per one .DS_Store per folder for multiple files, and you sync
only a few files plus the .DS_Store of that folder, you OVERWRITE the .DS_Store
on the destination.
Then the metadata for the files being both in the source and destination are
correct, but those only at the destination LOOSE their metadata. So a sync
process must be .DS_Store aware and MERGE .DS_Store from source and
destination, which practically EXCLUDES MOST 3rd PARTY apps from properly
syncing metadata!!!

What is my application: My entire projects are too large for my internal disk.
Therefore I eventually keep them on an external disk. The actual projects are
always on my internal disk, and I rsync them over additively (without the
option "delete on destination") to my external disk every here and
then. If
eventually done, I sync a last time, and then erase on my internal disk.
Therefore my source (internal HD) is a subset of my external disk. And syncing
only parts over with a .DS_Store unaware tool would copy over only the synced
files' metadata but ERASE the metadata of all other files on the destination
(external HD).

EXAMPLE:
Source: 3, 4 -- all with metadata
Destination: 1, 2, 3, 7, 8 -- all with metadata
Outcome: If I sync over 3+4 with a DS_Store unaware sync tool, then 3 gets its
metadata updated, 4 gets copied freshly, and all others (1, 2, 7, 8) loose
their metadata!

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.