[Rd] R string comparisons may vary with platform (plain text)

Prof Brian Ripley
2014-11-23T11:44:10Z

On 23/11/2014 09:39, peter dalgaard wrote:
>
>> On 23 Nov 2014, at 01:05 , Henrik Bengtsson <hb at
biostat.ucsf.edu> wrote:
>>
>> On Sat, Nov 22, 2014 at 12:42 PM, Duncan Murdoch
>> <murdoch.duncan at gmail.com> wrote:
>>> On 22/11/2014, 2:59 PM, Stuart Ambler wrote:
>>>> A colleague?s R program behaved differently when I ran it, and
we thought
>>>> we traced it probably to different results from string
comparisons as
>>>> below, with different R versions.  However the platforms also
differed.  A
>>>> friend ran it on a few machines and found that the comparison
behavior
>>>> didn?t correlate with R version, but rather with platform.
>>>>
>>>> I wonder if you?ve seen this.  If it?s not some setting I?m
unaware of,
>>>> maybe someone should look into it.  Sorry I haven?t taken the
time to read
>>>> the source code myself.
>>>
>>> Looks like a collation order issue.  See ?Comparison.
>>
>> With the oddity that both platforms use what look like similar locales:
>>
>> LC_COLLATE=en_US.UTF-8
>> LC_COLLATE=en_US.utf8
>
> It's the sort of thing thay I've tried to wrap my mind around
multiple times and failed, but have a look at
>
>
http://stackoverflow.com/questions/19967555/postgres-collation-differences-osx-v-ubuntu
>
> which seems to be essentially the same issue, just for Postgres. If you
have the stamina, also look into the python question that it links to.
>
> As I understand it, there are two potential reasons: Either the two
platforms are not using the same collation table for en_US, or at least one of
them is not fully implementing the Unicode Collation Algorithm.

And I have seen both with R.  At the very least, check if ICU is being
used (capabilities("ICU") in current R, maybe not in some of the
obsolete versions seen in this thread).

As a further possibility, there are choices in the UCA (in R, see
?icuSetCollate) and ICU can be compiled with different default choices.
  It is not clear to me what (if any) difference ICU versions make, but
in R-devel extSoftVersion() reports that.


> In general, collation is a minefield: Some languages have the same letters
in different order (e.g. Estonian with Z between S and T); accented characters
sort with the unaccented counterpart in some languages but as separate
characters in others; some locales sort ABab, others AaBb, yet others aAbB;
sometimes punctuation is ignored, sometimes not; sometimes multiple characters
count as one, etc.
>
As ?Comparison has long said.


--
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford
1 South Parks Road, Oxford OX1 3TG, UK