rsync - Mar 2008 - another iconv question

Hi All,
  I am trying to help my friend set up his rsync with iconv. Presently  
it works fine but re-copies every file with an umlaute in the  
filename. I saw a recent post about this and the fix but...

he ran "locale"  ( both source and dest are on same machine) and is  
running on a German-Swiss locale

xserve-backup-02:/Volumes/Backup RAID 8TB teleclub$ locale
LANGLC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL="C"

we tried adding option iconv=C and iconv=C,C  but no luck - it still  
re-copies very file with an umlaute.

I checked "iconv --list" on my Mac and see no "C" listed. I
am not
sure if "C" is correct either.


Example:

The correct file name would be "Action des Monats f?r vertonung.mov"  
and not "Action des Monats f\#303\#274r vertonung.mov"

but the log shows it not translated:

/Volumes/SAN_Video/Final Cut Pro Documents/Capture Scratch/Action des  
Monats/! Render/Action des Monats f\#303\#274r vertonung.mov
        32768   0%  344.09kB/s    0:54:11
     42205184   3%   40.25MB/s    0:00:26

Though the actual file name gets copied correctly to dest, obviously  
the mapping (if that is what it is called) is different causing rsync  
to update the file every time.

Any thoughts or direction appreciated. Thanks, Rob D

On Mon, Mar 31, 2008 at 09:54:16AM -0400, Robert DuToit
wrote:
> we tried adding option iconv=C and iconv=C,C  but no luck
That would appear to be a request for rsync to not do any character
conversion, in which case you shouldn't be using --iconv.

When using the --iconv option, you should describe the character set
that the files are in on the disk on both the sender and the receiver.
> [rsync] re-copies [e]very file with an umlaute.
Are you using (or implying) the -t (--times) option?  (Use the -i option
to see if the copy is due to time changes or something else.)  What do
you mean by "re-copies"?  Are you talking about copying the file away
and back again, which creates an extra copy of the file on the source
machine under a slightly different name?  If so, perhaps that character
cannot be converted to UTF-8 and back without changing in the process.

If you're talking about a single direction, make sure you've specified
the right character set for each machine.  Also make sure that the file
wasn't copied before with the wrong character set, since the new file
might be the right one.

..wayne..

On Mon, 2008-03-31 at 09:54 -0400, Robert DuToit wrote: 
> I am trying to help my friend set up his rsync with iconv. Presently  
> it works fine but re-copies every file with an umlaute in the  
> filename. I saw a recent post about this and the fix but...
> 
> he ran "locale"  ( both source and dest are on same machine) and
is
> running on a German-Swiss locale
Please be more specific about what your friend is trying to accomplish!
If he's trying to perform a nontrivial conversion, he definitely won't
accomplish it with "--iconv=.".  That option tells rsync to convert
from
the sending machine's default charset to the receiving machine's
default ?charset, but in his case, source and destination are on the
same machine, so these charsets are one and the same!  To convert from a
charset A to a charset B on a local run, he should pass "--iconv=A,B".
> xserve-backup-02:/Volumes/Backup RAID 8TB teleclub$ locale
> LANG> LC_COLLATE="C"
> LC_CTYPE="C"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL="C"
> 
> we tried adding option iconv=C and iconv=C,C  but no luck - it still  
> re-copies very file with an umlaute.
> 
> I checked "iconv --list" on my Mac and see no "C"
listed. I am not
> sure if "C" is correct either.
"C" is a "standard" locale whose associated charset is
ASCII.  Based on
the log output (below), your friend should probably be using
de_CH.UTF-8 .
> Example:
> 
> The correct file name would be "Action des Monats f?r
vertonung.mov"
> and not "Action des Monats f\#303\#274r vertonung.mov"
> 
> but the log shows it not translated:
> 
> /Volumes/SAN_Video/Final Cut Pro Documents/Capture Scratch/Action des  
> Monats/! Render/Action des Monats f\#303\#274r vertonung.mov
>         32768   0%  344.09kB/s    0:54:11
>      42205184   3%   40.25MB/s    0:00:26
All that's happening here is that the source filename is in UTF-8, but
rsync is escaping the two high bytes in its log output because they are
invalid in ASCII, the charset implied by the specified locale C.  If
your friend switches to a *.UTF-8 locale, rsync will show him the umlaut
as-is.
> Though the actual file name gets copied correctly to dest, obviously  
> the mapping (if that is what it is called) is different causing rsync  
> to update the file every time.
The output escaping issue won't cause recopying, but from what you say,
I can guess what the real problem is.  I notice that the source filename
is in composed UTF-8, and the Mac OS X HFS+ filesystem has an annoying
behavior of silently decomposing UTF-8 characters in filenames.  Suppose
the destination is on HFS+ and your friend is using --delete.  Rsync
will copy the file, but the destination filesystem will store its name
with a decomposed umlaut-u (three bytes 0x75, 0xcc, 0x88).  Rsync
compares binary filenames without regard for charset-specific
conventions, so on the next run, it will fail to recognize the
decomposed destination file as corresponding to the source file, delete
the destination file, and transfer the file again.  Essentially, rsync
tries and tries again to create a destination file with the same
(binary) name as the source file, but the filesystem keeps foiling it.

You can avoid this problem by passing --iconv=UTF-8,UTF8-MAC .  UTF8-MAC
is a pseudo-charset recognized by Mac OS X iconv in which all characters
are decomposed.  This way, rsync will decompose the source filename and
recognize it as matching the destination filename.

Wayne, please consider adding this material to the "copies every file"
entry on http://rsync.samba.org/FAQ.html .

Matt