R devel - Apr 2017 - Crash after (wrongly) applying product operator on object from LIMMA package

Hi Hilmar,

weird. The memory problem seems be due to recursion (my R, version 3.3.3,
says: Error: evaluation nested too deeply: infinite recursion /
options(expressions=)?, just write traceback() to see how it happens), but
why does it segfault with xlsx? Nb xlsx is the culprit: neither rJava nor
xlsxjars cause the problem.

On the other hand, quick googling for r+xlsx+segfault returns tons of
reports of how xlsx crashes in dozens of situations. See for example
http://r.789695.n4.nabble.com/segfault-in-gplots-heatmap-2-td4641808.html.
Also, the problem might be platform-specific. It would be interesting to
see whether anyone with a Mac can reproduce it.

kind regards,

j.





On 19 April 2017 at 10:01, Hilmar Berger <berger at mpiib-berlin.mpg.de>
wrote:
> Hi,
>
> following up on my own question, I found smaller example that does not
> require LIMMA:
>
> setClass("FOOCLASS",
>          representation("list")
> )
> ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>
> > ma * ma$M
> Error: C stack usage  7970512 is too close to the limit
>
> > library(xlsx)
> Loading required package: rJava
> Loading required package: xlsxjars
> > ma * ma$M
> ---> Crash
>
> xlsx seems to act like a catalyst here, with the product operator running
> in a deep nested iteration, exhausting the stack. Valgrind shows thousands
> of invalid stack accesses when loading xslx, which might contribute to the
> problem. Package xlsx has not been updated since 2014, so it might fail
> with more current versions of R or Java (I'm using Oracle Java 8).
>
> Still, even if xlsx was the package to be blamed for the crash, I fail to
> understand what exactly the product operator is trying to do in the
> multiplication of the matrix with the object.
>
> Best regards,
> Hilmar
>
>
> On 18/04/17 18:57, Hilmar Berger wrote:
>
>> Hi,
>>
>> this is a problem that occurs in the presence of two libraries (limma,
>> xlsx) and leads to a crash of R. The problematic code is the wrong
>> application of sweep or the product ("*") function on an
LIMMA MAList
>> object. To my knowledge, limma does not define a "*" method
for MAList
>> objects.
>>
>> If only LIMMA is loaded but not package xlsx, the code does not crash
but
>> rather produces an error ("Error: C stack usage  7970512 is too
close to
>> the limit"). Loading only package rJava instead of xlsx does also
not
>> produce the crash but the error message instead. Note that xlsx
functions
>> are not explicitly used.
>>
>> It could be reproduced on two different Linux machines running R-3.2.5,
>> R-3.3.0 and R-3.3.2.
>>
>> Code to reproduce the problem:
>> ---------------------------------
>> library(limma)
>> library(xlsx)
>>
>> # a MAList
>> ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
M=matrix(rnorm(300),
>> 30,10)))
>>
>> # This should actually be sweep(ma$M, ...) for functional code, but I
>> omitted the $M...
>> #sweep(ma, 2, c(1:10), "*")
>> # sweep will crash when doing the final operation of applying the
>> function over the input matrix, which in this case is function
"*"
>>
>> f = match.fun("*")
>> # This is not exactly the same as in sweep but it also tries to
multiply
>> the MAList object with a matrix of same size and leads to the crash
>> f(ma, ma$M)
>> # ma * ma$M has the same effect
>> ---------------------------------
>>
>> My output:
>>
>> R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
>> Copyright (C) 2016 The R Foundation for Statistical Computing
>> Platform: x86_64-pc-linux-gnu (64-bit)
>>
>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>> You are welcome to redistribute it under certain conditions.
>> Type 'license()' or 'licence()' for distribution
details.
>>
>>   Natural language support but running in an English locale
>>
>> R is a collaborative project with many contributors.
>> Type 'contributors()' for more information and
>> 'citation()' on how to cite R or R packages in publications.
>>
>> Type 'demo()' for some demos, 'help()' for on-line
help, or
>> 'help.start()' for an HTML browser interface to help.
>> Type 'q()' to quit R.
>>
>> > library(limma)
>> > library(xlsx)
>> Loading required package: rJava
>> Loading required package: xlsxjars
>> >
>> > sessionInfo()
>> R version 3.3.0 (2016-05-03)
>> Platform: x86_64-pc-linux-gnu (64-bit)
>> Running under: Ubuntu 14.04.5 LTS
>>
>> locale:
>>  [1] LC_CTYPE=en_US.UTF-8          LC_NUMERIC=C LC_TIME=en_US.UTF-8
>>  [4] LC_COLLATE=en_US.UTF-8        LC_MONETARY=en_US.UTF-8
>> LC_MESSAGES=en_US.UTF-8
>>  [7] LC_PAPER=en_US.UTF-8          LC_NAME=en_US.UTF-8
>> LC_ADDRESS=en_US.UTF-8
>> [10] LC_TELEPHONE=en_US.UTF-8      LC_MEASUREMENT=en_US.UTF-8
>> LC_IDENTIFICATION=en_US.UTF-8
>>
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods base
>>
>> other attached packages:
>> [1] xlsx_0.5.7     xlsxjars_0.6.1 rJava_0.9-8    limma_3.30.7
>>
>> loaded via a namespace (and not attached):
>> [1] tools_3.3.0
>> >
>> > ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>> M=matrix(rnorm(300), 30,10)))
>> > #sweep(ma, 2, c(1:10), "*")
>> >
>> > f = match.fun("*")
>> > f
>> function (e1, e2)  .Primitive("*")
>>
>> > f(ma, ma$M)
>>
>> ----> crash to command line with segfault.
>>
>> Best regards,
>> Hilmar
>>
>>
> --
> Dr. Hilmar Berger, MD
> Max Planck Institute for Infection Biology
> Charit?platz 1
> D-10117 Berlin
> GERMANY
>
> Phone:  + 49 30 28460 430
> Fax:    + 49 30 28460 401
>  E-Mail: berger at mpiib-berlin.mpg.de
> Web   : www.mpiib-berlin.mpg.de
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>


-- 
-------- January Weiner --------------------------------------

	[[alternative HTML version deleted]]

Hi January,

I believe the root of the xlsx issue has been identified and a fix 
suggested by Tomas Kalibera (see https://github.com/s-u/rJava/pull/102). 
In a nutshell, Oracle Java on Linux modifies the stack in a way that 
makes it smaller and and the same time makes it impossible for R to 
detect this change, leading to segfaults. It is not clear to me that the 
same problem would occur on Mac, since the behavior of Oracle seems to 
be Linux specific. Possibly even Linux users on OpenJDK might not 
encounter any problems (not tested).

So possibly the next release of rJava should also fix the xlsx problems 
with other packages.

Best regards,
Hilmar


On 24/04/17 11:46, January W. wrote:
> Hi Hilmar,
>
> weird. The memory problem seems be due to recursion (my R, version 
> 3.3.3, says: Error: evaluation nested too deeply: infinite recursion / 
> options(expressions=)?, just write traceback() to see how it happens), 
> but why does it segfault with xlsx? Nb xlsx is the culprit: neither 
> rJava nor xlsxjars cause the problem.
>
> On the other hand, quick googling for r+xlsx+segfault returns tons of 
> reports of how xlsx crashes in dozens of situations. See for example 
> http://r.789695.n4.nabble.com/segfault-in-gplots-heatmap-2-td4641808.html. 
> Also, the problem might be platform-specific. It would be interesting 
> to see whether anyone with a Mac can reproduce it.
>
> kind regards,
>
> j.
>
>
>
>
>
> On 19 April 2017 at 10:01, Hilmar Berger <berger at mpiib-berlin.mpg.de 
> <mailto:berger at mpiib-berlin.mpg.de>> wrote:
>
>     Hi,
>
>     following up on my own question, I found smaller example that does
>     not require LIMMA:
>
>     setClass("FOOCLASS",
>              representation("list")
>     )
>     ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>
>     > ma * ma$M
>     Error: C stack usage  7970512 is too close to the limit
>
>     > library(xlsx)
>     Loading required package: rJava
>     Loading required package: xlsxjars
>     > ma * ma$M
>     ---> Crash
>
>     xlsx seems to act like a catalyst here, with the product operator
>     running in a deep nested iteration, exhausting the stack. Valgrind
>     shows thousands of invalid stack accesses when loading xslx, which
>     might contribute to the problem. Package xlsx has not been updated
>     since 2014, so it might fail with more current versions of R or
>     Java (I'm using Oracle Java 8).
>
>     Still, even if xlsx was the package to be blamed for the crash, I
>     fail to understand what exactly the product operator is trying to
>     do in the multiplication of the matrix with the object.
>
>     Best regards,
>     Hilmar
>
>
>     On 18/04/17 18:57, Hilmar Berger wrote:
>
>         Hi,
>
>         this is a problem that occurs in the presence of two libraries
>         (limma, xlsx) and leads to a crash of R. The problematic code
>         is the wrong application of sweep or the product ("*")
>         function on an LIMMA MAList object. To my knowledge, limma
>         does not define a "*" method for MAList objects.
>
>         If only LIMMA is loaded but not package xlsx, the code does
>         not crash but rather produces an error ("Error: C stack usage 
>         7970512 is too close to the limit"). Loading only package
>         rJava instead of xlsx does also not produce the crash but the
>         error message instead. Note that xlsx functions are not
>         explicitly used.
>
>         It could be reproduced on two different Linux machines running
>         R-3.2.5, R-3.3.0 and R-3.3.2.
>
>         Code to reproduce the problem:
>         ---------------------------------
>         library(limma)
>         library(xlsx)
>
>         # a MAList
>         ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>         M=matrix(rnorm(300), 30,10)))
>
>         # This should actually be sweep(ma$M, ...) for functional
>         code, but I omitted the $M...
>         #sweep(ma, 2, c(1:10), "*")
>         # sweep will crash when doing the final operation of applying
>         the function over the input matrix, which in this case is
>         function "*"
>
>         f = match.fun("*")
>         # This is not exactly the same as in sweep but it also tries
>         to multiply the MAList object with a matrix of same size and
>         leads to the crash
>         f(ma, ma$M)
>         # ma * ma$M has the same effect
>         ---------------------------------
>
>         My output:
>
>         R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
>         Copyright (C) 2016 The R Foundation for Statistical Computing
>         Platform: x86_64-pc-linux-gnu (64-bit)
>
>         R is free software and comes with ABSOLUTELY NO WARRANTY.
>         You are welcome to redistribute it under certain conditions.
>         Type 'license()' or 'licence()' for distribution
details.
>
>           Natural language support but running in an English locale
>
>         R is a collaborative project with many contributors.
>         Type 'contributors()' for more information and
>         'citation()' on how to cite R or R packages in
publications.
>
>         Type 'demo()' for some demos, 'help()' for on-line
help, or
>         'help.start()' for an HTML browser interface to help.
>         Type 'q()' to quit R.
>
>         > library(limma)
>         > library(xlsx)
>         Loading required package: rJava
>         Loading required package: xlsxjars
>         >
>         > sessionInfo()
>         R version 3.3.0 (2016-05-03)
>         Platform: x86_64-pc-linux-gnu (64-bit)
>         Running under: Ubuntu 14.04.5 LTS
>
>         locale:
>          [1] LC_CTYPE=en_US.UTF-8          LC_NUMERIC=C
>         LC_TIME=en_US.UTF-8
>          [4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
>         LC_MESSAGES=en_US.UTF-8
>          [7] LC_PAPER=en_US.UTF-8          LC_NAME=en_US.UTF-8
>         LC_ADDRESS=en_US.UTF-8
>         [10] LC_TELEPHONE=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8
>         LC_IDENTIFICATION=en_US.UTF-8
>
>         attached base packages:
>         [1] stats     graphics  grDevices utils     datasets methods base
>
>         other attached packages:
>         [1] xlsx_0.5.7     xlsxjars_0.6.1 rJava_0.9-8 limma_3.30.7
>
>         loaded via a namespace (and not attached):
>         [1] tools_3.3.0
>         >
>         > ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>         M=matrix(rnorm(300), 30,10)))
>         > #sweep(ma, 2, c(1:10), "*")
>         >
>         > f = match.fun("*")
>         > f
>         function (e1, e2)  .Primitive("*")
>
>         > f(ma, ma$M)
>
>         ----> crash to command line with segfault.
>
>         Best regards,
>         Hilmar
>
>
>     -- 
>     Dr. Hilmar Berger, MD
>     Max Planck Institute for Infection Biology
>     Charit?platz 1
>     D-10117 Berlin
>     GERMANY
>
>     Phone: + 49 30 28460 430 <tel:%2B%2049%2030%2028460%20430>
>     Fax: + 49 30 28460 401 <tel:%2B%2049%2030%2028460%20401>
>      E-Mail: berger at mpiib-berlin.mpg.de
>     <mailto:berger at mpiib-berlin.mpg.de>
>     Web   : www.mpiib-berlin.mpg.de <http://www.mpiib-berlin.mpg.de>
>
>     ______________________________________________
>     R-devel at r-project.org <mailto:R-devel at r-project.org>
mailing list
>     https://stat.ethz.ch/mailman/listinfo/r-devel
>     <https://stat.ethz.ch/mailman/listinfo/r-devel>
>
>
>
>
> -- 
> -------- January Weiner --------------------------------------
-- 
Dr. Hilmar Berger, MD
Max Planck Institute for Infection Biology
Charit?platz 1
D-10117 Berlin
GERMANY

Phone:  + 49 30 28460 430
Fax:    + 49 30 28460 401
  
E-Mail: berger at mpiib-berlin.mpg.de
Web   : www.mpiib-berlin.mpg.de


	[[alternative HTML version deleted]]

Yes, the mentioned pull request for rJava is to workaround a JVM problem 
that is Linux-only.
I am not aware of any related problem on OSX.

In general, you can currently get two kinds of messages with infinite 
recursion:

Error: evaluation nested too deeply: infinite recursion / 
options(expressions=)?
Error: C stack usage XXX is too close to the limit

but in some situations you could get a segfault, an obvious one is 
infinite recursion in native code of a package.

If you're getting a segfault it is probably best to check with a 
debugger - infinite recursion would be easily identified in the stacktrace.

Best
Tomas


On 04/24/2017 11:58 AM, Hilmar Berger wrote:
> Hi January,
>
> I believe the root of the xlsx issue has been identified and a fix
> suggested by Tomas Kalibera (see https://github.com/s-u/rJava/pull/102).
> In a nutshell, Oracle Java on Linux modifies the stack in a way that
> makes it smaller and and the same time makes it impossible for R to
> detect this change, leading to segfaults. It is not clear to me that the
> same problem would occur on Mac, since the behavior of Oracle seems to
> be Linux specific. Possibly even Linux users on OpenJDK might not
> encounter any problems (not tested).
>
> So possibly the next release of rJava should also fix the xlsx problems
> with other packages.
>
> Best regards,
> Hilmar
>
>
> On 24/04/17 11:46, January W. wrote:
>> Hi Hilmar,
>>
>> weird. The memory problem seems be due to recursion (my R, version
>> 3.3.3, says: Error: evaluation nested too deeply: infinite recursion /
>> options(expressions=)?, just write traceback() to see how it happens),
>> but why does it segfault with xlsx? Nb xlsx is the culprit: neither
>> rJava nor xlsxjars cause the problem.
>>
>> On the other hand, quick googling for r+xlsx+segfault returns tons of
>> reports of how xlsx crashes in dozens of situations. See for example
>>
http://r.789695.n4.nabble.com/segfault-in-gplots-heatmap-2-td4641808.html.
>> Also, the problem might be platform-specific. It would be interesting
>> to see whether anyone with a Mac can reproduce it.
>>
>> kind regards,
>>
>> j.
>>
>>
>>
>>
>>
>> On 19 April 2017 at 10:01, Hilmar Berger <berger at
mpiib-berlin.mpg.de
>> <mailto:berger at mpiib-berlin.mpg.de>> wrote:
>>
>>      Hi,
>>
>>      following up on my own question, I found smaller example that does
>>      not require LIMMA:
>>
>>      setClass("FOOCLASS",
>>               representation("list")
>>      )
>>      ma = new("FOOCLASS", list(M=matrix(rnorm(300), 30,10)))
>>
>>      > ma * ma$M
>>      Error: C stack usage  7970512 is too close to the limit
>>
>>      > library(xlsx)
>>      Loading required package: rJava
>>      Loading required package: xlsxjars
>>      > ma * ma$M
>>      ---> Crash
>>
>>      xlsx seems to act like a catalyst here, with the product operator
>>      running in a deep nested iteration, exhausting the stack. Valgrind
>>      shows thousands of invalid stack accesses when loading xslx, which
>>      might contribute to the problem. Package xlsx has not been updated
>>      since 2014, so it might fail with more current versions of R or
>>      Java (I'm using Oracle Java 8).
>>
>>      Still, even if xlsx was the package to be blamed for the crash, I
>>      fail to understand what exactly the product operator is trying to
>>      do in the multiplication of the matrix with the object.
>>
>>      Best regards,
>>      Hilmar
>>
>>
>>      On 18/04/17 18:57, Hilmar Berger wrote:
>>
>>          Hi,
>>
>>          this is a problem that occurs in the presence of two libraries
>>          (limma, xlsx) and leads to a crash of R. The problematic code
>>          is the wrong application of sweep or the product
("*")
>>          function on an LIMMA MAList object. To my knowledge, limma
>>          does not define a "*" method for MAList objects.
>>
>>          If only LIMMA is loaded but not package xlsx, the code does
>>          not crash but rather produces an error ("Error: C stack
usage
>>          7970512 is too close to the limit"). Loading only package
>>          rJava instead of xlsx does also not produce the crash but the
>>          error message instead. Note that xlsx functions are not
>>          explicitly used.
>>
>>          It could be reproduced on two different Linux machines running
>>          R-3.2.5, R-3.3.0 and R-3.3.2.
>>
>>          Code to reproduce the problem:
>>          ---------------------------------
>>          library(limma)
>>          library(xlsx)
>>
>>          # a MAList
>>          ma = new("MAList", list(A=matrix(rnorm(300), 30,10),
>>          M=matrix(rnorm(300), 30,10)))
>>
>>          # This should actually be sweep(ma$M, ...) for functional
>>          code, but I omitted the $M...
>>          #sweep(ma, 2, c(1:10), "*")
>>          # sweep will crash when doing the final operation of applying
>>          the function over the input matrix, which in this case is
>>          function "*"
>>
>>          f = match.fun("*")
>>          # This is not exactly the same as in sweep but it also tries
>>          to multiply the MAList object with a matrix of same size and
>>          leads to the crash
>>          f(ma, ma$M)
>>          # ma * ma$M has the same effect
>>          ---------------------------------
>>
>>          My output:
>>
>>          R version 3.3.0 (2016-05-03) -- "Supposedly
Educational"
>>          Copyright (C) 2016 The R Foundation for Statistical Computing
>>          Platform: x86_64-pc-linux-gnu (64-bit)
>>
>>          R is free software and comes with ABSOLUTELY NO WARRANTY.
>>          You are welcome to redistribute it under certain conditions.
>>          Type 'license()' or 'licence()' for
distribution details.
>>
>>            Natural language support but running in an English locale
>>
>>          R is a collaborative project with many contributors.
>>          Type 'contributors()' for more information and
>>          'citation()' on how to cite R or R packages in
publications.
>>
>>          Type 'demo()' for some demos, 'help()' for
on-line help, or
>>          'help.start()' for an HTML browser interface to help.
>>          Type 'q()' to quit R.
>>
>>          > library(limma)
>>          > library(xlsx)
>>          Loading required package: rJava
>>          Loading required package: xlsxjars
>>          >
>>          > sessionInfo()
>>          R version 3.3.0 (2016-05-03)
>>          Platform: x86_64-pc-linux-gnu (64-bit)
>>          Running under: Ubuntu 14.04.5 LTS
>>
>>          locale:
>>           [1] LC_CTYPE=en_US.UTF-8          LC_NUMERIC=C
>>          LC_TIME=en_US.UTF-8
>>           [4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
>>          LC_MESSAGES=en_US.UTF-8
>>           [7] LC_PAPER=en_US.UTF-8          LC_NAME=en_US.UTF-8
>>          LC_ADDRESS=en_US.UTF-8
>>          [10] LC_TELEPHONE=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8
>>          LC_IDENTIFICATION=en_US.UTF-8
>>
>>          attached base packages:
>>          [1] stats     graphics  grDevices utils     datasets methods
base
>>
>>          other attached packages:
>>          [1] xlsx_0.5.7     xlsxjars_0.6.1 rJava_0.9-8 limma_3.30.7
>>
>>          loaded via a namespace (and not attached):
>>          [1] tools_3.3.0
>>          >
>>          > ma = new("MAList", list(A=matrix(rnorm(300),
30,10),
>>          M=matrix(rnorm(300), 30,10)))
>>          > #sweep(ma, 2, c(1:10), "*")
>>          >
>>          > f = match.fun("*")
>>          > f
>>          function (e1, e2)  .Primitive("*")
>>
>>          > f(ma, ma$M)
>>
>>          ----> crash to command line with segfault.
>>
>>          Best regards,
>>          Hilmar
>>
>>
>>      --
>>      Dr. Hilmar Berger, MD
>>      Max Planck Institute for Infection Biology
>>      Charit?platz 1
>>      D-10117 Berlin
>>      GERMANY
>>
>>      Phone: + 49 30 28460 430 <tel:%2B%2049%2030%2028460%20430>
>>      Fax: + 49 30 28460 401 <tel:%2B%2049%2030%2028460%20401>
>>       E-Mail: berger at mpiib-berlin.mpg.de
>>      <mailto:berger at mpiib-berlin.mpg.de>
>>      Web   : www.mpiib-berlin.mpg.de
<http://www.mpiib-berlin.mpg.de>
>>
>>      ______________________________________________
>>      R-devel at r-project.org <mailto:R-devel at r-project.org>
mailing list
>>      https://stat.ethz.ch/mailman/listinfo/r-devel
>>      <https://stat.ethz.ch/mailman/listinfo/r-devel>
>>
>>
>>
>>
>> -- 
>> -------- January Weiner --------------------------------------