thr3ads.net - R help - [R] S4 / operator "[" : Compatibility issue between lme4 and kml [Jun 2015]

If this information is useful, please help other people find it:
Share via:

cgenolin

2015-Jun-05 07:36 UTC

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

Hi all,

There is a compatibility issue between the package 'lme4' and my package
'kml'. I define the "[" operator. It works just fine in my
package (1). If I
try to use the lme4 package, then it does no longer work (2). Moreover, it
has some kind of strange behavior (3). Do you know what is wrong? Any idea
of how I can correct that?

Here is a reproductible example, and the same code with the result follows.

Thanks for your help
Christophe

--- 8< ----------------- Code for reproductible example -------------------
library(kml)

### Creating some data
dn <- gald(1)
dn["traj"]

### using lme4
library(lme4)
dn["traj"]

### Re-defining the "["
setMethod(   ### Simplified version ###
  "[",
  signature=signature(x="ClusterLongData", i="character",
j="ANY",drop="ANY"),
  definition=function (x, i, j="missing", ..., drop = TRUE){
      x <- as(x, "LongData")
      return(x[i, j])
    }
)

dn["traj"]
dn["traj"]

--- 8< ----------------- Execution of the previous code -------------------
> library(kml)Le chargement a n?cessit? le package : clv
Le chargement a n?cessit? le package : cluster
Le chargement a n?cessit? le package : class
Le chargement a n?cessit? le package : longitudinalData
Le chargement a n?cessit? le package : rgl
Le chargement a n?cessit? le package : misc3d> dn <- gald(1)
 ###########
### (1) the "[" operator works just fine
> dn["traj"]      t0   t1    t2    t3    t4   t5   t6    t7    t8    t9   t10
i1 -3.11 4.32  2.17  1.82  4.90 7.34 0.83 -2.70  5.36  4.96  3.16
i2 -7.11 1.40 -2.40 -2.96  4.31 0.50 1.25  0.52 -0.04  7.55  5.50
i3  2.80 6.23  6.08  2.87  2.58 2.88 6.58 -2.38  2.30 -1.74 -3.23
i4  2.24 0.91  6.50 10.92 11.32 7.79 7.78 10.69  9.15  1.07 -0.51

 ###########
### (2) using 'lme4', it does no longer work
> library(lme4)Le chargement a n?cessit? le package : Matrix
Le chargement a n?cessit? le package : Rcpp> dn["traj"]Error in x[i, j] :
  erreur d'?valuation de l'argument 'j' lors de la s?lection
d'une m?thode
pour la fonction '[' : Erreur : l'argument "j" est
manquant, avec aucune
valeur par d?faut

 ###########
### (3) If I define again the "[", it does not work the first time I
call
it, but it work the second time!> setMethod(+   "[",
+   signature=signature(x="ClusterLongData", i="character",
j="ANY",drop="ANY"),
+   definition=function (x, i, j="missing", ..., drop = TRUE){
+       x <- as(x, "LongData")
+       return(x[i, j])
+     }
+ )
[1] "["

### No working the first time I use it> dn["traj"]Error in dn["traj"] :
  l'argument "j" est manquant, avec aucune valeur par d?faut

### But working the second time> dn["traj"]      t0   t1    t2    t3    t4   t5   t6    t7    t8    t9   t10
i1 -3.11 4.32  2.17  1.82  4.90 7.34 0.83 -2.70  5.36  4.96  3.16
i2 -7.11 1.40 -2.40 -2.96  4.31 0.50 1.25  0.52 -0.04  7.55  5.50
i3  2.80 6.23  6.08  2.87  2.58 2.88 6.58 -2.38  2.30 -1.74 -3.23
i4  2.24 0.91  6.50 10.92 11.32 7.79 7.78 10.69  9.15  1.07 -0.51 



--
View this message in context:
http://r.789695.n4.nabble.com/S4-operator-Compatibility-issue-between-lme4-and-kml-tp4708236.html
Sent from the R help mailing list archive at Nabble.com.

Martin Maechler

2015-Jun-05 09:33 UTC

head link

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

>>>>> Christophe Genolini <cgenolin at u-paris10.fr>
>>>>>     on Fri, 5 Jun 2015 00:36:42 -0700 writes:
    > Hi all,
    > There is a compatibility issue between the package 'lme4' and
my package
    > 'kml'. I define the "[" operator. It works just fine
in my package (1). If I
    > try to use the lme4 package, then it does no longer work (2). Moreover,
it
    > has some kind of strange behavior (3). Do you know what is wrong? Any
idea
    > of how I can correct that?

    > Here is a reproductible example, and the same code with the result
follows.

Dear Christophe,
can you please specify the exact sessionInfo() ?
(after loading both kml and lme4).

'Matrix' has been updated on CRAN very recently, and 
'lme4' is about to be update very soon,
so this *is* a somewhat interesting and important problem,
but the exact versions of the packages *do* matter.

Bonnes salutations!

Martin Maechler
ETH Zurich

   [........................]

Martin Maechler

2015-Jun-05 10:21 UTC

head link

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

>>>>> Martin Maechler <maechler at stat.math.ethz.ch>
>>>>>     on Fri, 5 Jun 2015 11:33:46 +0200 writes:
>>>>> Christophe Genolini <cgenolin at u-paris10.fr>
>>>>>     on Fri, 5 Jun 2015 00:36:42 -0700 writes:
    >> Hi all,
    >> There is a compatibility issue between the package 'lme4'
and my package
    >> 'kml'. I define the "[" operator. It works just
fine in my package (1). If I
    >> try to use the lme4 package, then it does no longer work (2).
Moreover, it
    >> has some kind of strange behavior (3). Do you know what is wrong?
Any idea
    >> of how I can correct that?

    >> Here is a reproductible example, and the same code with the result
follows.

    > Dear Christophe,
    > can you please specify the exact sessionInfo() ?
    > (after loading both kml and lme4).

    > 'Matrix' has been updated on CRAN very recently, and 
    > 'lme4' is about to be update very soon,
    > so this *is* a somewhat interesting and important problem,
    > but the exact versions of the packages *do* matter.

As a matter of fact,  

1) the package versions don't seem to matter much.

2) lme4 is *not* involved directly:
   Much worse, it is the 'Matrix' package.

If you replace 'lme4' by 'Matrix' in your code, you get the same
symptoms.

... I'm investigating ..

Martin

Martin Maechler

2015-Jun-05 17:52 UTC

head link

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

>>>>> Christophe Genolini <cgenolin at u-paris10.fr>
>>>>>     on Fri, 5 Jun 2015 00:36:42 -0700 writes:
    > Hi all,
    > There is a compatibility issue between the package 'lme4' and
my package
    > 'kml'. I define the "[" operator. It works just fine
in my package (1). If I
    > try to use the lme4 package, then it does no longer work (2). Moreover,
it
    > has some kind of strange behavior (3). Do you know what is wrong? Any
idea
    > of how I can correct that?

    > Here is a reproductible example, and the same code with the result
follows.

    > Thanks for your help
    > Christophe

  [ ... I'm providing slightly different code below .... ]
> --- 8< ----------------- Execution of the previous code
-------------------
> > library(kml)
> Le chargement a n?cessit? le package : clv
> Le chargement a n?cessit? le package : cluster
> Le chargement a n?cessit? le package : class
> Le chargement a n?cessit? le package : longitudinalData
> Le chargement a n?cessit? le package : rgl
> Le chargement a n?cessit? le package : misc3d
> > dn <- gald(1)
>  ###########
> ### (1) the "[" operator works just fine
> > dn["traj"]
>       t0   t1    t2    t3    t4   t5   t6    t7    t8    t9   t10
> i1 -3.11 4.32  2.17  1.82  4.90 7.34 0.83 -2.70  5.36  4.96  3.16
> i2 -7.11 1.40 -2.40 -2.96  4.31 0.50 1.25  0.52 -0.04  7.55  5.50
> i3  2.80 6.23  6.08  2.87  2.58 2.88 6.58 -2.38  2.30 -1.74 -3.23
> i4  2.24 0.91  6.50 10.92 11.32 7.79 7.78 10.69  9.15  1.07 -0.51
>  ###########
> ### (2) using 'lme4', it does no longer work
> > library(lme4)
> Le chargement a n?cessit? le package : Matrix
> Le chargement a n?cessit? le package : Rcpp
> > dn["traj"]
> Error in x[i, j] :
>   erreur d'?valuation de l'argument 'j' lors de la
s?lection d'une m?thode
> pour la fonction '[' : Erreur : l'argument "j" est
manquant, avec aucune
> valeur par d?faut
>  ###########
> ### (3) If I define again the "[", it does not work the first
time I call
> it, but it work the second time!
> > setMethod("[",
> +   signature=signature(x="ClusterLongData",
i="character", j="ANY",drop="ANY"),
> +   definition=function (x, i, j="missing", ..., drop = TRUE){
> +       x <- as(x, "LongData")
> +       return(x[i, j])
> +     }
> + )
> [1] "["
> ### No working the first time I use it
> > dn["traj"]
> Error in dn["traj"] :
>   l'argument "j" est manquant, avec aucune valeur par d?faut
> ### But working the second time
> > dn["traj"]
>       t0   t1    t2    t3    t4   t5   t6    t7    t8    t9   t10
> i1 -3.11 4.32  2.17  1.82  4.90 7.34 0.83 -2.70  5.36  4.96  3.16
> i2 -7.11 1.40 -2.40 -2.96  4.31 0.50 1.25  0.52 -0.04  7.55  5.50
> i3  2.80 6.23  6.08  2.87  2.58 2.88 6.58 -2.38  2.30 -1.74 -3.23
> i4  2.24 0.91  6.50 10.92 11.32 7.79 7.78 10.69  9.15  1.07 -0.51 
I have made some investigations, but have to stop for now, and
leave this hopefully to others knowledgable about S4 method
dispatch, etc :

1) I am confident to say that you have uncovered an "unfelicity if
  not a bug" in R.

2) I am also pretty confident that the "[" methods that you
  define in 'kml' and in the package '

3) Diagnosing is not easy: As you have shown yourself above,
  in some situations the bug "bites" and if you repeat the *same*
  code, things work.

  This is related to the fact that S4 methods are __cached__
  (so next time they are found more quickly) under some
  circumstances, and the cache is cleared under other such circumstances.

3b) Actually, I am sure that we have seen +/- the same problem many
    months ago, in other contexts but did not get "down to it";
    and at the moment, I cannot quickly find where to look for
    the problem there...


##--- 8< ------------Commented (incl output) reproducible code--------------
library(kml)

### Creating some data
dn <- gald(1)
(dnt <- dn["traj"])

showMethods("[")
## Function: [ (package base)
## x="ClusterLongData", i="character"
## x="ListPartition", i="ANY"
## x="LongData", i="ANY"
## x="LongData", i="character"
##     (inherited from: x="LongData", i="ANY")
## x="LongData3d", i="ANY"
## x="nonStructure", i="ANY"
## x="ParChoice", i="ANY"
## x="ParKml", i="ANY"
## x="ParLongData", i="ANY"
## x="Partition", i="ANY"
## x="ParWindows", i="ANY"

### using Matrix  (or lme4, which 'Depends' on Matrix; hence same
effect)
library(Matrix)
dn["traj"]
## Error in x[i, j] :
##   error in evaluating the argument 'j' in selecting a method for
function '[': Error: argument "j" is missing, with no default
traceback()
## 3: x[i, j]
## 2: dn["traj"]
## 1: dn["traj"]
(ms <- methods(`[`)) ## 81 methods

##---- MM: debugging :
trace("[", browser, signature=c("ClusterLongData",
"character", "missing",   "missing"))
trace("[", browser, signature=c("LongData",       
"character", "character", "missing"))
dn["traj"]
## -> you get into the browser, just press   "c"   twice (once for
each "trace")
## ==> it works !!

## Remove the tracing :
untrace("[", signature=c("ClusterLongData",
"character", "missing",   "missing"))
untrace("[", signature=c("LongData",       
"character", "character", "missing"))
dn["traj"]
## Error in dn["traj"] : argument "j" is missing, with no
default

## Debugging only the *inner* function:
trace("[", browser, signature=c("LongData",       
"character", "character", "missing"))
dn["traj"]
## Error ....
untrace("[", signature=c("LongData",       
"character", "character", "missing"))

## Debugging only the *outer* function:
trace("[", browser, signature=c("ClusterLongData",
"character", "missing",   "missing"))
dn["traj"] ## -> debugger, press 'c'
## it works!
##       t0    t1    t2   t3   t4   t5   t6    t7    t8    t9   t10
## i1  7.38  4.80  4.80 0.73 0.58 2.22 0.55 -1.05  0.79  5.20  4.43
## i2  1.55  2.01 -0.29 2.12 4.44 7.33 9.09  4.76 12.18  5.92  8.06
## i3 13.60 14.72 10.15 9.25 8.70 8.34 6.71  5.84  3.44  2.10  2.17
## i4 -7.49 -1.80  0.08 2.51 6.61 4.56 8.96  3.05  3.41 -2.62 -4.09
untrace("[", signature=c("ClusterLongData",
"character", "missing",   "missing"))

##--- {end of reproducible code} --------------

cgenolin

2015-Jun-06 14:19 UTC

head link

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

Thanks a lot for your time. Two questions:

1/ Shall I submit a bug report?

2/ In your point 2), I cannot find the verb... Is my english not good enough
to understand it, or is it a "not-ended" sentence?

:-)



--
View this message in context:
http://r.789695.n4.nabble.com/S4-operator-Compatibility-issue-between-lme4-and-kml-tp4708236p4708282.html
Sent from the R help mailing list archive at Nabble.com.

Martin Morgan

2015-Jun-06 16:42 UTC

head link

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

On 06/05/2015 10:52 AM, Martin Maechler wrote:>>>>>> Christophe Genolini <cgenolin at u-paris10.fr>
>>>>>>      on Fri, 5 Jun 2015 00:36:42 -0700 writes:
>
>      > Hi all,
>      > There is a compatibility issue between the package 'lme4'
and my package
>      > 'kml'. I define the "[" operator. It works just
fine in my package (1). If I
>      > try to use the lme4 package, then it does no longer work (2).
Moreover, it
>      > has some kind of strange behavior (3). Do you know what is wrong?
Any idea
>      > of how I can correct that?
>
>      > Here is a reproductible example, and the same code with the
result follows.
>
>      > Thanks for your help
>      > Christophe
>
>    [ ... I'm providing slightly different code below .... ]
>
>> --- 8< ----------------- Execution of the previous code
-------------------
>
>>> library(kml)
>> Le chargement a n?cessit? le package : clv
>> Le chargement a n?cessit? le package : cluster
>> Le chargement a n?cessit? le package : class
>> Le chargement a n?cessit? le package : longitudinalData
>> Le chargement a n?cessit? le package : rgl
>> Le chargement a n?cessit? le package : misc3d
>>> dn <- gald(1)
>
>>   ###########
>> ### (1) the "[" operator works just fine
>
>>> dn["traj"]
>>        t0   t1    t2    t3    t4   t5   t6    t7    t8    t9   t10
>> i1 -3.11 4.32  2.17  1.82  4.90 7.34 0.83 -2.70  5.36  4.96  3.16
>> i2 -7.11 1.40 -2.40 -2.96  4.31 0.50 1.25  0.52 -0.04  7.55  5.50
>> i3  2.80 6.23  6.08  2.87  2.58 2.88 6.58 -2.38  2.30 -1.74 -3.23
>> i4  2.24 0.91  6.50 10.92 11.32 7.79 7.78 10.69  9.15  1.07 -0.51
>
>>   ###########
>> ### (2) using 'lme4', it does no longer work
>
>>> library(lme4)
>> Le chargement a n?cessit? le package : Matrix
>> Le chargement a n?cessit? le package : Rcpp
>>> dn["traj"]
>> Error in x[i, j] :
>>    erreur d'?valuation de l'argument 'j' lors de la
s?lection d'une m?thode
>> pour la fonction '[' : Erreur : l'argument "j"
est manquant, avec aucune
>> valeur par d?faut
>
>>   ###########
>> ### (3) If I define again the "[", it does not work the first
time I call
>> it, but it work the second time!
>>> setMethod("[",
>> +   signature=signature(x="ClusterLongData",
i="character", j="ANY",drop="ANY"),
>> +   definition=function (x, i, j="missing", ..., drop =
TRUE){
Your file has two definitions of

   setMethod("[", c("ClusterLongData", ...

I deleted the first one.

The second definition had

     signature=signature(x="ClusterLongData", i="character",
j="ANY",drop="ANY"),

whereas probably you mean to say that you'll handle

     signature=signature(x="ClusterLongData", i="character",
                         j="missing", drop="ANY")

The next line says

     definition=function (x, i, j="missing", ..., drop = TRUE){

which provides a default value for 'j' when j is not provided by the
user. Thus
later when you say

    x[i, j]

you are performing dn["traj", "missing"] when probably you
meant

   x[i, , drop=drop]

Making these changes, so the definition is

setMethod(
     "[",
     signature=signature(x="ClusterLongData", i="character",
j="missing",
       drop="ANY"),
     definition=function (x, i, j, ..., drop = TRUE){
         if (is.numeric(i)) {
             stop("[ClusterLongData:getteur]: to get a clusters list, use
['ci']")
         }else{}
         if (i %in% c("criterionValues",
"criterionValuesAsMatrix")){
             j <- x['criterionActif']
         }else{}
         if (i %in% c(CRITERION_NAMES, "criterionActif",
CLUSTER_NAMES,
                      "criterionValues",
"criterionValuesAsMatrix", "sorted",
                      "initializationMethod")) {
             x <- as(x, "ListPartition")
         }else{
             x <- as(x, "LongData")
         }
         x[i, , drop=drop]
     })

Allows operations to work correctly.

 > library(kml)
Loading required package: clv
Loading required package: cluster
Loading required package: class
Loading required package: longitudinalData
Loading required package: rgl
Loading required package: misc3d
 > library(Matrix)
 > x = gald(1)["traj"]
 > x
       t0    t1    t2    t3    t4    t5    t6    t7    t8    t9   t10
i1 -3.18 -1.19 -1.17  1.56 -0.70  1.78 -0.95 -2.00 -5.05  1.05  2.84
i2  3.51  1.72  6.97  6.09  7.81  8.33  9.54 14.38 16.14 12.82 13.86
i3  9.60 11.59  9.09  6.31  9.24  7.69  4.26 -0.80  2.70  1.63  1.21
i4 -0.54  3.80  6.05 10.41 12.60 12.32 10.33 11.05  7.89  5.21  0.67

It's hard to tell whether is an issue with the methods package, or just that
Matrix offered a better nearest 'method' than those provided by kml / 
longitudinalData.


>> +       x <- as(x, "LongData")
>> +       return(x[i, j])
>> +     }
>> + )
>> [1] "["
>
>> ### No working the first time I use it
>>> dn["traj"]
>> Error in dn["traj"] :
>>    l'argument "j" est manquant, avec aucune valeur par
d?faut
>
>> ### But working the second time
>>> dn["traj"]
>>        t0   t1    t2    t3    t4   t5   t6    t7    t8    t9   t10
>> i1 -3.11 4.32  2.17  1.82  4.90 7.34 0.83 -2.70  5.36  4.96  3.16
>> i2 -7.11 1.40 -2.40 -2.96  4.31 0.50 1.25  0.52 -0.04  7.55  5.50
>> i3  2.80 6.23  6.08  2.87  2.58 2.88 6.58 -2.38  2.30 -1.74 -3.23
>> i4  2.24 0.91  6.50 10.92 11.32 7.79 7.78 10.69  9.15  1.07 -0.51
>
> I have made some investigations, but have to stop for now, and
> leave this hopefully to others knowledgable about S4 method
> dispatch, etc :
>
> 1) I am confident to say that you have uncovered an "unfelicity if
>    not a bug" in R.
>
> 2) I am also pretty confident that the "[" methods that you
>    define in 'kml' and in the package '
>
> 3) Diagnosing is not easy: As you have shown yourself above,
>    in some situations the bug "bites" and if you repeat the
*same*
>    code, things work.
>
>    This is related to the fact that S4 methods are __cached__
>    (so next time they are found more quickly) under some
>    circumstances, and the cache is cleared under other such circumstances.
>
> 3b) Actually, I am sure that we have seen +/- the same problem many
>      months ago, in other contexts but did not get "down to it";
>      and at the moment, I cannot quickly find where to look for
>      the problem there...
>
>
> ##--- 8< ------------Commented (incl output) reproducible
code--------------
> library(kml)
>
> ### Creating some data
> dn <- gald(1)
> (dnt <- dn["traj"])
>
> showMethods("[")
> ## Function: [ (package base)
> ## x="ClusterLongData", i="character"
> ## x="ListPartition", i="ANY"
> ## x="LongData", i="ANY"
> ## x="LongData", i="character"
> ##     (inherited from: x="LongData", i="ANY")
> ## x="LongData3d", i="ANY"
> ## x="nonStructure", i="ANY"
> ## x="ParChoice", i="ANY"
> ## x="ParKml", i="ANY"
> ## x="ParLongData", i="ANY"
> ## x="Partition", i="ANY"
> ## x="ParWindows", i="ANY"
>
> ### using Matrix  (or lme4, which 'Depends' on Matrix; hence same
effect)
> library(Matrix)
> dn["traj"]
> ## Error in x[i, j] :
> ##   error in evaluating the argument 'j' in selecting a method for
function '[': Error: argument "j" is missing, with no default
> traceback()
> ## 3: x[i, j]
> ## 2: dn["traj"]
> ## 1: dn["traj"]
> (ms <- methods(`[`)) ## 81 methods
>
> ##---- MM: debugging :
> trace("[", browser, signature=c("ClusterLongData",
"character", "missing",   "missing"))
> trace("[", browser, signature=c("LongData",       
"character", "character", "missing"))
> dn["traj"]
> ## -> you get into the browser, just press   "c"   twice (once
for each "trace")
> ## ==> it works !!
>
> ## Remove the tracing :
> untrace("[", signature=c("ClusterLongData",
"character", "missing",   "missing"))
> untrace("[", signature=c("LongData",       
"character", "character", "missing"))
> dn["traj"]
> ## Error in dn["traj"] : argument "j" is missing, with
no default
>
> ## Debugging only the *inner* function:
> trace("[", browser, signature=c("LongData",       
"character", "character", "missing"))
> dn["traj"]
> ## Error ....
> untrace("[", signature=c("LongData",       
"character", "character", "missing"))
>
> ## Debugging only the *outer* function:
> trace("[", browser, signature=c("ClusterLongData",
"character", "missing",   "missing"))
> dn["traj"] ## -> debugger, press 'c'
> ## it works!
> ##       t0    t1    t2   t3   t4   t5   t6    t7    t8    t9   t10
> ## i1  7.38  4.80  4.80 0.73 0.58 2.22 0.55 -1.05  0.79  5.20  4.43
> ## i2  1.55  2.01 -0.29 2.12 4.44 7.33 9.09  4.76 12.18  5.92  8.06
> ## i3 13.60 14.72 10.15 9.25 8.70 8.34 6.71  5.84  3.44  2.10  2.17
> ## i4 -7.49 -1.80  0.08 2.51 6.61 4.56 8.96  3.05  3.41 -2.62 -4.09
> untrace("[", signature=c("ClusterLongData",
"character", "missing",   "missing"))
>
> ##--- {end of reproducible code} --------------
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

R help - Jun 2015 - S4 / operator "[" : Compatibility issue between lme4 and kml

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

[R] S4 / operator "[" : Compatibility issue between lme4 and kml

[R] S4 / operator "[" : Compatibility issue between lme4 and kml