R devel - Apr 2017 - Crash after (wrongly) applying product operator on S4 object that derives from list

Dear Hilmar

Perhaps this gives an indication of why the infinite recursion happens:

## after calling `*` on ma and a matrix:
> showMethods(classes=class(ma), includeDefs=TRUE, inherited = TRUE)
Function: * (package base)
e1="FOOCLASS", e2="matrix"
     (inherited from: e1="vector", e2="structure")
     (definition from function "Ops")
function (e1, e2)
{
     value <- callGeneric(e1, e2 at .Data)
     if (length(value) == length(e2)) {
         e2 at .Data <- value
         e2
     }
     else value
}


> is(ma, "vector")
[1] TRUE

I got that in a fresh session of
> sessionInfo()
R Under development (unstable) (2017-04-18 r72542)
Platform: x86_64-apple-darwin16.5.0 (64-bit)
Running under: macOS Sierra 10.12.4

Best wishes
Wolfgang

19.4.17 10:01, Hilmar Berger scripsit:
> Hi,
>
> following up on my own question, I found smaller example that does not
> require LIMMA:
>
> setClass("FOOCLASS",
>          representation("list")
> )
> ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>
>> ma * ma$M
> Error: C stack usage  7970512 is too close to the limit
>
>> library(xlsx)
> Loading required package: rJava
> Loading required package: xlsxjars
>> ma * ma$M
> ---> Crash
>
> xlsx seems to act like a catalyst here, with the product operator
> running in a deep nested iteration, exhausting the stack. Valgrind shows
> thousands of invalid stack accesses when loading xslx, which might
> contribute to the problem. Package xlsx has not been updated since 2014,
> so it might fail with more current versions of R or Java (I'm using
> Oracle Java 8).
>
> Still, even if xlsx was the package to be blamed for the crash, I fail
> to understand what exactly the product operator is trying to do in the
> multiplication of the matrix with the object.
>
> Best regards,
> Hilmar
>
> On 18/04/17 18:57, Hilmar Berger wrote:
>> Hi,
>>
>> this is a problem that occurs in the presence of two libraries (limma,
>> xlsx) and leads to a crash of R. The problematic code is the wrong
>> application of sweep or the product ("*") function on an
LIMMA MAList
>> object. To my knowledge, limma does not define a "*" method
for MAList
>> objects.
>>
>> If only LIMMA is loaded but not package xlsx, the code does not crash
>> but rather produces an error ("Error: C stack usage  7970512 is
too
>> close to the limit"). Loading only package rJava instead of xlsx
does
>> also not produce the crash but the error message instead. Note that
>> xlsx functions are not explicitly used.
>>
>> It could be reproduced on two different Linux machines running
>> R-3.2.5, R-3.3.0 and R-3.3.2.
>>
>> Code to reproduce the problem:
>> ---------------------------------
>> library(limma)
>> library(xlsx)
>>
>> # a MAList
>> ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>> M=matrix(rnorm(300), 30,10)))
>>
>> # This should actually be sweep(ma$M, ...) for functional code, but I
>> omitted the $M...
>> #sweep(ma, 2, c(1:10), "*")
>> # sweep will crash when doing the final operation of applying the
>> function over the input matrix, which in this case is function
"*"
>>
>> f = match.fun("*")
>> # This is not exactly the same as in sweep but it also tries to
>> multiply the MAList object with a matrix of same size and leads to the
>> crash
>> f(ma, ma$M)
>> # ma * ma$M has the same effect
>> ---------------------------------
>>
>> My output:
>>
>> R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
>> Copyright (C) 2016 The R Foundation for Statistical Computing
>> Platform: x86_64-pc-linux-gnu (64-bit)
>>
>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>> You are welcome to redistribute it under certain conditions.
>> Type 'license()' or 'licence()' for distribution
details.
>>
>>   Natural language support but running in an English locale
>>
>> R is a collaborative project with many contributors.
>> Type 'contributors()' for more information and
>> 'citation()' on how to cite R or R packages in publications.
>>
>> Type 'demo()' for some demos, 'help()' for on-line
help, or
>> 'help.start()' for an HTML browser interface to help.
>> Type 'q()' to quit R.
>>
>> > library(limma)
>> > library(xlsx)
>> Loading required package: rJava
>> Loading required package: xlsxjars
>> >
>> > sessionInfo()
>> R version 3.3.0 (2016-05-03)
>> Platform: x86_64-pc-linux-gnu (64-bit)
>> Running under: Ubuntu 14.04.5 LTS
>>
>> locale:
>>  [1] LC_CTYPE=en_US.UTF-8          LC_NUMERIC=C LC_TIME=en_US.UTF-8
>>  [4] LC_COLLATE=en_US.UTF-8        LC_MONETARY=en_US.UTF-8
>> LC_MESSAGES=en_US.UTF-8
>>  [7] LC_PAPER=en_US.UTF-8          LC_NAME=en_US.UTF-8
>> LC_ADDRESS=en_US.UTF-8
>> [10] LC_TELEPHONE=en_US.UTF-8      LC_MEASUREMENT=en_US.UTF-8
>> LC_IDENTIFICATION=en_US.UTF-8
>>
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods base
>>
>> other attached packages:
>> [1] xlsx_0.5.7     xlsxjars_0.6.1 rJava_0.9-8    limma_3.30.7
>>
>> loaded via a namespace (and not attached):
>> [1] tools_3.3.0
>> >
>> > ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>> M=matrix(rnorm(300), 30,10)))
>> > #sweep(ma, 2, c(1:10), "*")
>> >
>> > f = match.fun("*")
>> > f
>> function (e1, e2)  .Primitive("*")
>>
>> > f(ma, ma$M)
>>
>> ----> crash to command line with segfault.
>>
>> Best regards,
>> Hilmar
>>
>

I think this is a known issue with Java messing with the stack, see
e.g.
http://r.789695.n4.nabble.com/Error-memory-exhausted-limit-reached-td4729708.html.

I'll fix the infinite recursion caused by the methods package.

Michael

On Wed, Apr 19, 2017 at 1:12 AM, Wolfgang Huber <wolfgang.huber at
embl.de> wrote:
> Dear Hilmar
>
> Perhaps this gives an indication of why the infinite recursion happens:
>
> ## after calling `*` on ma and a matrix:
>
>> showMethods(classes=class(ma), includeDefs=TRUE, inherited = TRUE)
>
>
> Function: * (package base)
> e1="FOOCLASS", e2="matrix"
>     (inherited from: e1="vector", e2="structure")
>     (definition from function "Ops")
> function (e1, e2)
> {
>     value <- callGeneric(e1, e2 at .Data)
>     if (length(value) == length(e2)) {
>         e2 at .Data <- value
>         e2
>     }
>     else value
> }
>
>
>
>> is(ma, "vector")
>
> [1] TRUE
>
> I got that in a fresh session of
>>
>> sessionInfo()
>
> R Under development (unstable) (2017-04-18 r72542)
> Platform: x86_64-apple-darwin16.5.0 (64-bit)
> Running under: macOS Sierra 10.12.4
>
> Best wishes
> Wolfgang
>
> 19.4.17 10:01, Hilmar Berger scripsit:
>>
>> Hi,
>>
>> following up on my own question, I found smaller example that does not
>> require LIMMA:
>>
>> setClass("FOOCLASS",
>>          representation("list")
>> )
>> ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>>
>>> ma * ma$M
>>
>> Error: C stack usage  7970512 is too close to the limit
>>
>>> library(xlsx)
>>
>> Loading required package: rJava
>> Loading required package: xlsxjars
>>>
>>> ma * ma$M
>>
>> ---> Crash
>>
>> xlsx seems to act like a catalyst here, with the product operator
>> running in a deep nested iteration, exhausting the stack. Valgrind
shows
>> thousands of invalid stack accesses when loading xslx, which might
>> contribute to the problem. Package xlsx has not been updated since
2014,
>> so it might fail with more current versions of R or Java (I'm using
>> Oracle Java 8).
>>
>> Still, even if xlsx was the package to be blamed for the crash, I fail
>> to understand what exactly the product operator is trying to do in the
>> multiplication of the matrix with the object.
>>
>> Best regards,
>> Hilmar
>>
>> On 18/04/17 18:57, Hilmar Berger wrote:
>>>
>>> Hi,
>>>
>>> this is a problem that occurs in the presence of two libraries
(limma,
>>> xlsx) and leads to a crash of R. The problematic code is the wrong
>>> application of sweep or the product ("*") function on an
LIMMA MAList
>>> object. To my knowledge, limma does not define a "*"
method for MAList
>>> objects.
>>>
>>> If only LIMMA is loaded but not package xlsx, the code does not
crash
>>> but rather produces an error ("Error: C stack usage  7970512
is too
>>> close to the limit"). Loading only package rJava instead of
xlsx does
>>> also not produce the crash but the error message instead. Note that
>>> xlsx functions are not explicitly used.
>>>
>>> It could be reproduced on two different Linux machines running
>>> R-3.2.5, R-3.3.0 and R-3.3.2.
>>>
>>> Code to reproduce the problem:
>>> ---------------------------------
>>> library(limma)
>>> library(xlsx)
>>>
>>> # a MAList
>>> ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>>> M=matrix(rnorm(300), 30,10)))
>>>
>>> # This should actually be sweep(ma$M, ...) for functional code, but
I
>>> omitted the $M...
>>> #sweep(ma, 2, c(1:10), "*")
>>> # sweep will crash when doing the final operation of applying the
>>> function over the input matrix, which in this case is function
"*"
>>>
>>> f = match.fun("*")
>>> # This is not exactly the same as in sweep but it also tries to
>>> multiply the MAList object with a matrix of same size and leads to
the
>>> crash
>>> f(ma, ma$M)
>>> # ma * ma$M has the same effect
>>> ---------------------------------
>>>
>>> My output:
>>>
>>> R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
>>> Copyright (C) 2016 The R Foundation for Statistical Computing
>>> Platform: x86_64-pc-linux-gnu (64-bit)
>>>
>>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>>> You are welcome to redistribute it under certain conditions.
>>> Type 'license()' or 'licence()' for distribution
details.
>>>
>>>   Natural language support but running in an English locale
>>>
>>> R is a collaborative project with many contributors.
>>> Type 'contributors()' for more information and
>>> 'citation()' on how to cite R or R packages in
publications.
>>>
>>> Type 'demo()' for some demos, 'help()' for on-line
help, or
>>> 'help.start()' for an HTML browser interface to help.
>>> Type 'q()' to quit R.
>>>
>>> > library(limma)
>>> > library(xlsx)
>>> Loading required package: rJava
>>> Loading required package: xlsxjars
>>> >
>>> > sessionInfo()
>>> R version 3.3.0 (2016-05-03)
>>> Platform: x86_64-pc-linux-gnu (64-bit)
>>> Running under: Ubuntu 14.04.5 LTS
>>>
>>> locale:
>>>  [1] LC_CTYPE=en_US.UTF-8          LC_NUMERIC=C LC_TIME=en_US.UTF-8
>>>  [4] LC_COLLATE=en_US.UTF-8        LC_MONETARY=en_US.UTF-8
>>> LC_MESSAGES=en_US.UTF-8
>>>  [7] LC_PAPER=en_US.UTF-8          LC_NAME=en_US.UTF-8
>>> LC_ADDRESS=en_US.UTF-8
>>> [10] LC_TELEPHONE=en_US.UTF-8      LC_MEASUREMENT=en_US.UTF-8
>>> LC_IDENTIFICATION=en_US.UTF-8
>>>
>>> attached base packages:
>>> [1] stats     graphics  grDevices utils     datasets  methods base
>>>
>>> other attached packages:
>>> [1] xlsx_0.5.7     xlsxjars_0.6.1 rJava_0.9-8    limma_3.30.7
>>>
>>> loaded via a namespace (and not attached):
>>> [1] tools_3.3.0
>>> >
>>> > ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>>> M=matrix(rnorm(300), 30,10)))
>>> > #sweep(ma, 2, c(1:10), "*")
>>> >
>>> > f = match.fun("*")
>>> > f
>>> function (e1, e2)  .Primitive("*")
>>>
>>> > f(ma, ma$M)
>>>
>>> ----> crash to command line with segfault.
>>>
>>> Best regards,
>>> Hilmar
>>>
>>
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

We're working on a workaround for the JVM issue, it should be available 
in rJava soon.
(the JVM issue is only on Linux and it turns infinite/deep recursion 
into a crash of R; it also effectively reduces the R stack size)

Best
Tomas

On 04/19/2017 02:56 PM, Michael Lawrence wrote:
> I think this is a known issue with Java messing with the stack, see
> e.g.
http://r.789695.n4.nabble.com/Error-memory-exhausted-limit-reached-td4729708.html.
>
> I'll fix the infinite recursion caused by the methods package.
>
> Michael
>
> On Wed, Apr 19, 2017 at 1:12 AM, Wolfgang Huber <wolfgang.huber at
embl.de> wrote:
>> Dear Hilmar
>>
>> Perhaps this gives an indication of why the infinite recursion happens:
>>
>> ## after calling `*` on ma and a matrix:
>>
>>> showMethods(classes=class(ma), includeDefs=TRUE, inherited = TRUE)
>>
>> Function: * (package base)
>> e1="FOOCLASS", e2="matrix"
>>      (inherited from: e1="vector", e2="structure")
>>      (definition from function "Ops")
>> function (e1, e2)
>> {
>>      value <- callGeneric(e1, e2 at .Data)
>>      if (length(value) == length(e2)) {
>>          e2 at .Data <- value
>>          e2
>>      }
>>      else value
>> }
>>
>>
>>
>>> is(ma, "vector")
>> [1] TRUE
>>
>> I got that in a fresh session of
>>> sessionInfo()
>> R Under development (unstable) (2017-04-18 r72542)
>> Platform: x86_64-apple-darwin16.5.0 (64-bit)
>> Running under: macOS Sierra 10.12.4
>>
>> Best wishes
>> Wolfgang
>>
>> 19.4.17 10:01, Hilmar Berger scripsit:
>>> Hi,
>>>
>>> following up on my own question, I found smaller example that does
not
>>> require LIMMA:
>>>
>>> setClass("FOOCLASS",
>>>           representation("list")
>>> )
>>> ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>>>
>>>> ma * ma$M
>>> Error: C stack usage  7970512 is too close to the limit
>>>
>>>> library(xlsx)
>>> Loading required package: rJava
>>> Loading required package: xlsxjars
>>>> ma * ma$M
>>> ---> Crash
>>>
>>> xlsx seems to act like a catalyst here, with the product operator
>>> running in a deep nested iteration, exhausting the stack. Valgrind
shows
>>> thousands of invalid stack accesses when loading xslx, which might
>>> contribute to the problem. Package xlsx has not been updated since
2014,
>>> so it might fail with more current versions of R or Java (I'm
using
>>> Oracle Java 8).
>>>
>>> Still, even if xlsx was the package to be blamed for the crash, I
fail
>>> to understand what exactly the product operator is trying to do in
the
>>> multiplication of the matrix with the object.
>>>
>>> Best regards,
>>> Hilmar
>>>
>>> On 18/04/17 18:57, Hilmar Berger wrote:
>>>> Hi,
>>>>
>>>> this is a problem that occurs in the presence of two libraries
(limma,
>>>> xlsx) and leads to a crash of R. The problematic code is the
wrong
>>>> application of sweep or the product ("*") function on
an LIMMA MAList
>>>> object. To my knowledge, limma does not define a "*"
method for MAList
>>>> objects.
>>>>
>>>> If only LIMMA is loaded but not package xlsx, the code does not
crash
>>>> but rather produces an error ("Error: C stack usage 
7970512 is too
>>>> close to the limit"). Loading only package rJava instead
of xlsx does
>>>> also not produce the crash but the error message instead. Note
that
>>>> xlsx functions are not explicitly used.
>>>>
>>>> It could be reproduced on two different Linux machines running
>>>> R-3.2.5, R-3.3.0 and R-3.3.2.
>>>>
>>>> Code to reproduce the problem:
>>>> ---------------------------------
>>>> library(limma)
>>>> library(xlsx)
>>>>
>>>> # a MAList
>>>> ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>>>> M=matrix(rnorm(300), 30,10)))
>>>>
>>>> # This should actually be sweep(ma$M, ...) for functional code,
but I
>>>> omitted the $M...
>>>> #sweep(ma, 2, c(1:10), "*")
>>>> # sweep will crash when doing the final operation of applying
the
>>>> function over the input matrix, which in this case is function
"*"
>>>>
>>>> f = match.fun("*")
>>>> # This is not exactly the same as in sweep but it also tries to
>>>> multiply the MAList object with a matrix of same size and leads
to the
>>>> crash
>>>> f(ma, ma$M)
>>>> # ma * ma$M has the same effect
>>>> ---------------------------------
>>>>
>>>> My output:
>>>>
>>>> R version 3.3.0 (2016-05-03) -- "Supposedly
Educational"
>>>> Copyright (C) 2016 The R Foundation for Statistical Computing
>>>> Platform: x86_64-pc-linux-gnu (64-bit)
>>>>
>>>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>>>> You are welcome to redistribute it under certain conditions.
>>>> Type 'license()' or 'licence()' for
distribution details.
>>>>
>>>>    Natural language support but running in an English locale
>>>>
>>>> R is a collaborative project with many contributors.
>>>> Type 'contributors()' for more information and
>>>> 'citation()' on how to cite R or R packages in
publications.
>>>>
>>>> Type 'demo()' for some demos, 'help()' for
on-line help, or
>>>> 'help.start()' for an HTML browser interface to help.
>>>> Type 'q()' to quit R.
>>>>
>>>>> library(limma)
>>>>> library(xlsx)
>>>> Loading required package: rJava
>>>> Loading required package: xlsxjars
>>>>> sessionInfo()
>>>> R version 3.3.0 (2016-05-03)
>>>> Platform: x86_64-pc-linux-gnu (64-bit)
>>>> Running under: Ubuntu 14.04.5 LTS
>>>>
>>>> locale:
>>>>   [1] LC_CTYPE=en_US.UTF-8          LC_NUMERIC=C
LC_TIME=en_US.UTF-8
>>>>   [4] LC_COLLATE=en_US.UTF-8        LC_MONETARY=en_US.UTF-8
>>>> LC_MESSAGES=en_US.UTF-8
>>>>   [7] LC_PAPER=en_US.UTF-8          LC_NAME=en_US.UTF-8
>>>> LC_ADDRESS=en_US.UTF-8
>>>> [10] LC_TELEPHONE=en_US.UTF-8      LC_MEASUREMENT=en_US.UTF-8
>>>> LC_IDENTIFICATION=en_US.UTF-8
>>>>
>>>> attached base packages:
>>>> [1] stats     graphics  grDevices utils     datasets  methods
base
>>>>
>>>> other attached packages:
>>>> [1] xlsx_0.5.7     xlsxjars_0.6.1 rJava_0.9-8    limma_3.30.7
>>>>
>>>> loaded via a namespace (and not attached):
>>>> [1] tools_3.3.0
>>>>> ma = new("MAList", list(A=matrix(rnorm(300),
30,10),
>>>> M=matrix(rnorm(300), 30,10)))
>>>>> #sweep(ma, 2, c(1:10), "*")
>>>>>
>>>>> f = match.fun("*")
>>>>> f
>>>> function (e1, e2)  .Primitive("*")
>>>>
>>>>> f(ma, ma$M)
>>>> ----> crash to command line with segfault.
>>>>
>>>> Best regards,
>>>> Hilmar
>>>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel