R devel - Jul 2010 - bug in identical()? [Was: [R-sig-ME] Failure to load lme4 on Mac]

Daniel,

thanks for the test case. I did run it in valgrind but nothing showed up,
however ...

I'm starting to have a suspicion that this has something to do with
identical() - look at this:
> identical(M1,M2)
[1] FALSE
> all(serialize(M1,NULL)==serialize(M2,NULL))
[1] TRUE
> identical(unserialize(serialize(M1,NULL)),unserialize(serialize(M2,NULL)))
[1] FALSE
> identical(unserialize(serialize(M1,NULL)),unserialize(serialize(M1,NULL)))
[1] FALSE

So I think this may be a bug in identical() mainly because of the last one.
I'll need to take identical() apart to see where it fails ... I'm CCing
this to R-devel as the current issue seems more like an R issue so more eyes can
have a look ...

Cheers,
Simon


[FWIW this is tested in today's R-devel (with valgrind level 2) on x86_64 OS
X 10.6.4 with lme4 from CRAN and Matrix form R-devel Recommended]


On Jul 17, 2010, at 4:50 AM, Daniel Myall wrote:
> I've done some further testing (R 2.11.1) and the issue is not limited
to Leopard.
> 
> Using the test:
> 
> library(lme4)
> y <- (1:20)*pi
> x <- (1:20)^2
> group <- gl(2,10)
> for (i in 1:10) {
>  M1 <- lmer (y ~     x + (    x | group))
>  M2 <- lmer (y ~     x + (    x | group))
>  print(identical(M1,M2))
> }
> 
> For CRAN lme4 and Matrix:
> 
> 32 bit on Leopard: R CMD check fails; different results (on most runs)
> 32 bit on Snow Leopard: R CMD check passes; different results (on some
runs).
> 64 bit on Snow Leopard: R CMD check passes; identical results
> 
> For SVN version of Matrix with CRAN lme4:
> 
> 32 bit on Snow Leopard: different results (on all runs).
> 64 bit on Snow Leopard: different results (on all runs)
> 
> For SVN version of Matrix with SVN lme4a:
> 
> 32 bit on Snow Leopard: different results (on all runs).
> 64 bit on Snow Leopard: identical results
> 
> I couldn't reproduce on Linux 32/64bit. Is it time to jump into
valgrind to try and find the cause?
> 
> Cheers,
> Daniel
> 
> 
> 
> On 17/07/10 5:51 PM, John Maindonald wrote:
>> In principle, maybe a Snow Leopard version might be posted
>> as an alternative, if someone can provide one.  But I take it
>> that the issue is now a bit wider than tests that fail on Leopard
>> vs passing on Snow Leopard?
>>   
> 
>

Ok, I think I found the issue. I'm not sure why this varies by platform but
the mismatch is due to the @env slot. Two environments are only identical if it
is *the* same environment (i.e. the same pointer). However, M1 and M2 have
different environments. The content of those environments is identical, but that
is irrelevant as it's not the same pointer. Hence identical(M1, M2) fails
(and serialize comparison succeeds as it cares only about the content).

So the short story is don't use identical() to compare the models (unless
you remove @env first). The long story raises the question whether identical()
should really return FALSE for environments like
> identical(new.env(),new.env())
[1] FALSE
I can see arguments both ways but for the purposes of comparing values there
should be an option that the above is TRUE.

To be honest I don't see why this has not shown up on other platforms as
that is a global issue... (I hope this is the full story - I didn't try all
the combinations to see if setting @env to the same environment will appease
identical() for all the models)

Cheers,
Simon


On Jul 17, 2010, at 3:49 PM, Simon Urbanek wrote:
> Daniel,
> 
> thanks for the test case. I did run it in valgrind but nothing showed up,
however ...
> 
> I'm starting to have a suspicion that this has something to do with
identical() - look at this:
> 
>> identical(M1,M2)
> [1] FALSE
>> all(serialize(M1,NULL)==serialize(M2,NULL))
> [1] TRUE
>>
identical(unserialize(serialize(M1,NULL)),unserialize(serialize(M2,NULL)))
> [1] FALSE
>>
identical(unserialize(serialize(M1,NULL)),unserialize(serialize(M1,NULL)))
> [1] FALSE
> 
> So I think this may be a bug in identical() mainly because of the last one.
I'll need to take identical() apart to see where it fails ... I'm CCing
this to R-devel as the current issue seems more like an R issue so more eyes can
have a look ...
> 
> Cheers,
> Simon
> 
> 
> [FWIW this is tested in today's R-devel (with valgrind level 2) on
x86_64 OS X 10.6.4 with lme4 from CRAN and Matrix form R-devel Recommended]
> 
> 
> On Jul 17, 2010, at 4:50 AM, Daniel Myall wrote:
> 
>> I've done some further testing (R 2.11.1) and the issue is not
limited to Leopard.
>> 
>> Using the test:
>> 
>> library(lme4)
>> y <- (1:20)*pi
>> x <- (1:20)^2
>> group <- gl(2,10)
>> for (i in 1:10) {
>> M1 <- lmer (y ~     x + (    x | group))
>> M2 <- lmer (y ~     x + (    x | group))
>> print(identical(M1,M2))
>> }
>> 
>> For CRAN lme4 and Matrix:
>> 
>> 32 bit on Leopard: R CMD check fails; different results (on most runs)
>> 32 bit on Snow Leopard: R CMD check passes; different results (on some
runs).
>> 64 bit on Snow Leopard: R CMD check passes; identical results
>> 
>> For SVN version of Matrix with CRAN lme4:
>> 
>> 32 bit on Snow Leopard: different results (on all runs).
>> 64 bit on Snow Leopard: different results (on all runs)
>> 
>> For SVN version of Matrix with SVN lme4a:
>> 
>> 32 bit on Snow Leopard: different results (on all runs).
>> 64 bit on Snow Leopard: identical results
>> 
>> I couldn't reproduce on Linux 32/64bit. Is it time to jump into
valgrind to try and find the cause?
>> 
>> Cheers,
>> Daniel
>> 
>> 
>> 
>> On 17/07/10 5:51 PM, John Maindonald wrote:
>>> In principle, maybe a Snow Leopard version might be posted
>>> as an alternative, if someone can provide one.  But I take it
>>> that the issue is now a bit wider than tests that fail on Leopard
>>> vs passing on Snow Leopard?
>>> 
>> 
>> 
>