search for: mcfork

Dear R developers, with the inclusion of the package "parallel" in the upcoming release of R, users and package developers are likely to make increasing usage of parallelization features. In part, these features rely on forking the R process. As ?mcfork points out, fork()ing in a GUI process is typically a bad idea. In RKWard, we "only" seem to have problems with signals arriving in the wrong threads, and occasional failure to collect the results from child processes. I haven't entirely given up the hope to fix this, eventually, b...

...metimes NaN or 10^300ish), which can be seen by examining X right before FUN returns (to mclapply()'s environment) and comparing to the "same" X after mclapply() returns to the calling environment. Part of svn commit r58219 is this hunk diff --git a/src/library/parallel/R/unix/mcfork.R b/src/library/parallel/R/unix/mcfork.R index 8e27534..4f92193 100644 --- a/src/library/parallel/R/unix/mcfork.R +++ b/src/library/parallel/R/unix/mcfork.R @@ -82,7 +82,8 @@ mckill <- function(process, signal = 2L) ## used by mcparallel, mclapply sendMaster <- function(what) { - i...

...0 to signify that we're leaving */ size_t len = 0; /* assign result for Fedora security settings */ ssize_t n = write(master_fd, &len, sizeof(len)); ... ... } So a pipe write is made in mc_exit, and here's how this function is used in src/library/parallel/R/unix/mcfork.R: mcexit <- function(exit.code = 0L, send = NULL) { if (!is.null(send)) try(sendMaster(send), silent = TRUE) .Call(C_mc_exit, as.integer(exit.code)) } Between sendMaster() and mc_exit() calls, which are made in the child process, the master process may call readChild() followed by rmC...

...n R [1]. You could add the > following to your ~/.Rprofile startup file: > > ## Warn when forked processing is used in the RStudio Console > if (Sys.getenv("RSTUDIO") == "1" && !nzchar(Sys.getenv("RSTUDIO_TERM"))) { > invisible(trace(parallel:::mcfork, tracer = > quote(warning("parallel::mcfork() was used. Note that forked > processes, e.g. parallel::mclapply(), may be unstable when used from > the RStudio Console > [https://github.com/rstudio/rstudio/issues/2597#issuecomment-482187011]", > call.=FALSE)))) > } > &g...

Yes I am running on Rstudio 1.2.5033. I was also running this code without error on Ubuntu in Rstudio. Checking again on the terminal and it does indeed work fine even with large data.frames. Any idea as to what interaction between Rstudio and mclapply causes this? Thanks, Shian On 28 Apr 2020, at 7:29 pm, Simon Urbanek <simon.urbanek at R-project.org<mailto:simon.urbanek at

...ble forked processing in R [1]. You could add the following to your ~/.Rprofile startup file: ## Warn when forked processing is used in the RStudio Console if (Sys.getenv("RSTUDIO") == "1" && !nzchar(Sys.getenv("RSTUDIO_TERM"))) { invisible(trace(parallel:::mcfork, tracer = quote(warning("parallel::mcfork() was used. Note that forked processes, e.g. parallel::mclapply(), may be unstable when used from the RStudio Console [https://github.com/rstudio/rstudio/issues/2597#issuecomment-482187011]", call.=FALSE)))) } to detect when forked processed is u...

...; /* assign result for Fedora security settings */ > > ssize_t n = write(master_fd, &len, sizeof(len)); > > ... ... > > } > > > > So a pipe write is made in mc_exit, and here's how this function is > > used in src/library/parallel/R/unix/mcfork.R: > > > > mcexit <- function(exit.code = 0L, send = NULL) > > { > > if (!is.null(send)) try(sendMaster(send), silent = TRUE) > > .Call(C_mc_exit, as.integer(exit.code)) > > } > > > > Between sendMaster() and mc_exit() calls, which are ma...

...out=1000) then in R-3.1.0 some child processes are not terminated and some pipes remain open. This does not happen in R-3.0.3. Remark: If the line mccollect(p) is omitted at all then "NULL!" is never printed but R aborts after 510 iterations of the loop with the message "Error in mcfork(detached) : unable to create a pipe". Then none of the child processes is terminated and number of possible open pipes is exhausted. Is this behavior intended or a bug? > version _ platform x86_64-pc-linux-gnu arch x86_64...

...the command line version of R? I would not be surprised to see such a crash if executed from a multi-threaded application, say from some GUI or frontend that runs multiple threads, or from some other R session where a third party library (curl?) already started some threads. In such situations mcfork/mclapply is unsafe (?mcfork warns against GUI and frontends and I've now expanded slightly) and and it could not be fixed without being turned into something like parLapply(). parLapply() on a non-FORK cluster should work fine even with such applications. Best Tomas On 09/19/2018 11:19 PM,...

...; following to your ~/.Rprofile startup file: >>> >>> ## Warn when forked processing is used in the RStudio Console >>> if (Sys.getenv("RSTUDIO") == "1" && !nzchar(Sys.getenv("RSTUDIO_TERM"))) { >>> invisible(trace(parallel:::mcfork, tracer = >>> quote(warning("parallel::mcfork() was used. Note that forked >>> processes, e.g. parallel::mclapply(), may be unstable when used from >>> the RStudio Console >>> [https://github.com/rstudio/rstudio/issues/2597#issuecomment-482187011]", &gt...

...add the >> following to your ~/.Rprofile startup file: >> >> ## Warn when forked processing is used in the RStudio Console >> if (Sys.getenv("RSTUDIO") == "1" && !nzchar(Sys.getenv("RSTUDIO_TERM"))) { >> invisible(trace(parallel:::mcfork, tracer = >> quote(warning("parallel::mcfork() was used. Note that forked >> processes, e.g. parallel::mclapply(), may be unstable when used from >> the RStudio Console >> [https://github.com/rstudio/rstudio/issues/2597#issuecomment-482187011]", >> call.=FALS...

.../R/unix/mcparallel.R (revision 77648) +++ src/library/parallel/R/unix/mcparallel.R (working copy) @@ -20,6 +20,7 @@ mcparallel <- function(expr, name, mc.set.seed = TRUE, silent = FALSE, mc.affinity = NULL, mc.interactive = FALSE, detached = FALSE) { + allowFork(assert = TRUE) f <- mcfork(detached) env <- parent.frame() if (isTRUE(mc.set.seed)) mc.advance.stream() Index: src/library/parallel/R/unix/pvec.R =================================================================== --- src/library/parallel/R/unix/pvec.R (revision 77648) +++ src/library/parallel/R/unix/pvec.R (wor...

...0 to signify that we're leaving */ size_t len = 0; /* assign result for Fedora security settings */ ssize_t n = write(master_fd, &len, sizeof(len)); ... ... } So a pipe write is made in mc_exit, and here's how this function is used in src/library/parallel/R/unix/mcfork.R: mcexit <- function(exit.code = 0L, send = NULL) { if (!is.null(send)) try(sendMaster(send), silent = TRUE) .Call(C_mc_exit, as.integer(exit.code)) } Between sendMaster() and mc_exit() calls, which are made in the child process, the master process may call readChild() followed by rmC...

...; size_t len = 0; > /* assign result for Fedora security settings */ > ssize_t n = write(master_fd, &len, sizeof(len)); > ... ... > } > > So a pipe write is made in mc_exit, and here's how this function is > used in src/library/parallel/R/unix/mcfork.R: > > mcexit <- function(exit.code = 0L, send = NULL) > { > if (!is.null(send)) try(sendMaster(send), silent = TRUE) > .Call(C_mc_exit, as.integer(exit.code)) > } > > Between sendMaster() and mc_exit() calls, which are made in the child > process, the master...

...~/.Rprofile startup file: > >>> > >>> ## Warn when forked processing is used in the RStudio Console > >>> if (Sys.getenv("RSTUDIO") == "1" && !nzchar(Sys.getenv("RSTUDIO_TERM"))) { > >>> invisible(trace(parallel:::mcfork, tracer = > >>> quote(warning("parallel::mcfork() was used. Note that forked > >>> processes, e.g. parallel::mclapply(), may be unstable when used from > >>> the RStudio Console > >>> [https://github.com/rstudio/rstudio/issues/2597#issuecomment-4...

...ary/parallel/R/unix/mcparallel.R (working copy) >> @@ -20,6 +20,7 @@ >> mcparallel <- function(expr, name, mc.set.seed = TRUE, silent = >> FALSE, mc.affinity = NULL, mc.interactive = FALSE, detached = FALSE) >> { >> + allowFork(assert = TRUE) >> f <- mcfork(detached) >> env <- parent.frame() >> if (isTRUE(mc.set.seed)) mc.advance.stream() >> Index: src/library/parallel/R/unix/pvec.R >> =================================================================== >> --- src/library/parallel/R/unix/pvec.R (revision 77648)...

When using multicore-forking of the parallel package, is it possible for a child process to know that it is a fork? Something like: parallel::mclapply(1:10, FUN = function(i) { test_if_running_in_a_fork() }) I'm looking into ways to protect against further parallel processes (including threads), which not necessarily are created via the parallel:mc* API, are being spawned off recursively.

Dear all, I got an issue trying to parse excel files in parallel using XLConnect, the process hangs forever. Martin Studer, the maintainer of XLConnect kindly investigated the issue, identified rJava as a possible cause of the problem: This does not work (hangs): library(parallel) require(rJava) .jinit() res <- mclapply(1:2, function(i) {

...+++ src/library/parallel/R/unix/mcparallel.R (working copy) > @@ -20,6 +20,7 @@ > > mcparallel <- function(expr, name, mc.set.seed = TRUE, silent = > FALSE, mc.affinity = NULL, mc.interactive = FALSE, detached = FALSE) > { > + allowFork(assert = TRUE) > f <- mcfork(detached) > env <- parent.frame() > if (isTRUE(mc.set.seed)) mc.advance.stream() > Index: src/library/parallel/R/unix/pvec.R > =================================================================== > --- src/library/parallel/R/unix/pvec.R (revision 77648) > +++ src/lib...

...>>>> @@ -20,6 +20,7 @@ >>>> mcparallel <- function(expr, name, mc.set.seed = TRUE, silent = >>>> FALSE, mc.affinity = NULL, mc.interactive = FALSE, detached = FALSE) >>>> { >>>> + allowFork(assert = TRUE) >>>> f <- mcfork(detached) >>>> env <- parent.frame() >>>> if (isTRUE(mc.set.seed)) mc.advance.stream() >>>> Index: src/library/parallel/R/unix/pvec.R >>>> =================================================================== >>>> --- src/library/pa...