R devel - Nov 2013 - problem using rJava with parallel::mclapply

Dear all,

I got an issue trying to parse excel files in parallel using XLConnect, the
process hangs forever.
Martin Studer, the maintainer of XLConnect kindly investigated the issue,
identified rJava as a possible cause of the problem:

This does not work (hangs):
library(parallel)
require(rJava)
.jinit()
res <- mclapply(1:2, function(i) {
      J("java.lang.Runtime")$getRuntime()$gc()
      1
  }, mc.cores = 2)

but this works:
library(parallel)
res <- mclapply(1:2, function(i) {
  require(rJava)
  .jinit()
  J("java.lang.Runtime")$getRuntime()$gc()
  1
}, mc.cores = 2)

To cite Martin, it seems to work with mclapply when the JVM process is
initialized after forking.

Is this a bug or a limitation of rJava ?
Or is there a good practice for rJava clients to avoid this problem ?

Best,
Karl

P.S.
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=C                 LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] tools_3.0.1

	[[alternative HTML version deleted]]

Karl,

I have the following notes to self that may be pertinent:

options(java.parameters         ## Must preceed `library(XLConnect)` in order to
prevent "Java
         ## requested System.exit(130), closing R." which happens when
         ## rJava quits R upon trapping INT (control-c), as is done by
         ## XLConnect (and playwith?), below. (c.f.:
         ## https://www.rforge.net/bugzilla/show_bug.cgi?id=237)
         "-Xrs")


~Malcolm



 >-----Original Message-----
 >From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at
r-project.org] On Behalf Of Karl Forner
 >Sent: Monday, November 11, 2013 11:41 AM
 >To: r-devel at r-project.org
 >Cc: Martin Studer
 >Subject: [Rd] problem using rJava with parallel::mclapply
 >
 >Dear all,
 >
 >I got an issue trying to parse excel files in parallel using XLConnect, the
 >process hangs forever.
 >Martin Studer, the maintainer of XLConnect kindly investigated the issue,
 >identified rJava as a possible cause of the problem:
 >
 >This does not work (hangs):
 >library(parallel)
 >require(rJava)
 >.jinit()
 >res <- mclapply(1:2, function(i) {
 >      J("java.lang.Runtime")$getRuntime()$gc()
 >      1
 >  }, mc.cores = 2)
 >
 >but this works:
 >library(parallel)
 >res <- mclapply(1:2, function(i) {
 >  require(rJava)
 >  .jinit()
 >  J("java.lang.Runtime")$getRuntime()$gc()
 >  1
 >}, mc.cores = 2)
 >
 >To cite Martin, it seems to work with mclapply when the JVM process is
 >initialized after forking.
 >
 >Is this a bug or a limitation of rJava ?
 >Or is there a good practice for rJava clients to avoid this problem ?
 >
 >Best,
 >Karl
 >
 >P.S.
 >> sessionInfo()
 >R version 3.0.1 (2013-05-16)
 >Platform: x86_64-unknown-linux-gnu (64-bit)
 >
 >locale:
 > [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 > [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 > [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 > [7] LC_PAPER=C                 LC_NAME=C
 > [9] LC_ADDRESS=C               LC_TELEPHONE=C
 >[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
 >
 >attached base packages:
 >[1] stats     graphics  grDevices utils     datasets  methods   base
 >
 >loaded via a namespace (and not attached):
 >[1] tools_3.0.1
 >
 >	[[alternative HTML version deleted]]
 >
 >______________________________________________
 >R-devel at r-project.org mailing list
 >https://stat.ethz.ch/mailman/listinfo/r-devel

On Nov 11, 2013, at 12:40 PM, Karl Forner <karl.forner at gmail.com>
wrote:
> Dear all,
> 
> I got an issue trying to parse excel files in parallel using XLConnect, the
> process hangs forever.
> Martin Studer, the maintainer of XLConnect kindly investigated the issue,
> identified rJava as a possible cause of the problem:
> 
> This does not work (hangs):
> library(parallel)
> require(rJava)
> .jinit()
> res <- mclapply(1:2, function(i) {
>      J("java.lang.Runtime")$getRuntime()$gc()
>      1
>  }, mc.cores = 2)
> 
> but this works:
> library(parallel)
> res <- mclapply(1:2, function(i) {
>  require(rJava)
>  .jinit()
>  J("java.lang.Runtime")$getRuntime()$gc()
>  1
> }, mc.cores = 2)
> 
> To cite Martin, it seems to work with mclapply when the JVM process is
> initialized after forking.
> 
> Is this a bug or a limitation of rJava ?
It?s a limitation of the Java runtime - you cannot fork a JVM. This is true for
most libraries that use long-lived threads or any kind if UI (see the warnings
in mcfork).

Cheers,
Simon


> Or is there a good practice for rJava clients to avoid this problem ?
> 
> Best,
> Karl
> 
> P.S.
>> sessionInfo()
> R version 3.0.1 (2013-05-16)
> Platform: x86_64-unknown-linux-gnu (64-bit)
> 
> locale:
> [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
> [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
> [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
> [7] LC_PAPER=C                 LC_NAME=C
> [9] LC_ADDRESS=C               LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> 
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
> 
> loaded via a namespace (and not attached):
> [1] tools_3.0.1
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>