similar to: Is there a way to disable / warn about forking?

Hi, I am experiencing a problem related to serialization behavior in 2.15.0 beta (binary installed from Debian unstable) and 2.16.0 (from svn) that is not present in 2.14.2 (binary from Debian testing). I don't fully understand the problem. Also, I tried but have not yet been able to create a small, self-contained example that reproduces the problem. However, I do have a large, not

Henrik, the whole point and only purpose of mc* functions is to fork. That's what the multicore package was about, so if you don't want to fork, don't use mc* functions - they don't have any other purpose. I really fail to see the point - if you use mc* functions you're very explicitly asking for forking - so your argument is like saying that print() should have an option to

When using multicore-forking of the parallel package, is it possible for a child process to know that it is a fork? Something like: parallel::mclapply(1:10, FUN = function(i) { test_if_running_in_a_fork() }) I'm looking into ways to protect against further parallel processes (including threads), which not necessarily are created via the parallel:mc* API, are being spawned off recursively.

If I understand the thread correctly this is an RStudio issue and I would suggest that the developers consider using pthread_atfork() so RStudio can handle forking as they deem fit (bail out with an error or make RStudio work). Note that in principle the functionality requested here can be easily implemented in a package so R doesn?t need to be modified. Cheers, Simon Sent from my iPhone

I'd like to pick up this thread started on 2019-04-11 (https://hypatia.math.ethz.ch/pipermail/r-devel/2019-April/077632.html). Modulo all the other suggestions in this thread, would my proposal of being able to disable forked processing via an option or an environment variable make sense? I've prototyped a working patch that works like: > options(fork.allowed = FALSE) >

Henrik, the example from the post works just fine in CRAN R for me - the post was about homebrew build so it's conceivably a bug in their libraries. That's exactly why I was proposing a more general solution where you can simply define a function in user-space that will issue a warning or stop on fork, it doesn't have to be part of core R, there are other packages that use fork() as

Thanks Henrik, That clears things up significantly. I did see the warning but failed to include it my initial email. It sounds like an RStudio issue, and it seems like that it?s quite intrinsic to how forks interact with RStudio. Given this code is eventually going to be a part of a package, should I expect it to fail mysteriously in RStudio for my users? Is the best solution here to migrate all

Thanks Simon, I will take note of the sensible default for core usage. I?m trying to achieve small scale parallelism, where tasks take 1-5 seconds and make fuller use of consumer hardware. Its not a HPC-worthy computation but even laptops these days come with 4 cores and I don?t see a reason to not make use of it. The goal for the current piece of code I?m working on is to bootstrap many

> On Jan 10, 2020, at 3:10 PM, G?bor Cs?rdi <csardi.gabor at gmail.com> wrote: > > On Fri, Jan 10, 2020 at 7:23 PM Simon Urbanek > <simon.urbanek at r-project.org> wrote: >> >> Henrik, >> >> the example from the post works just fine in CRAN R for me - the post was about homebrew build so it's conceivably a bug in their libraries. > > I

Hi Inaki, > "Performant"... in terms of what. If the cost of copying the data > predominates over the computation time, maybe you didn't need > parallelization in the first place. Performant in terms of speed. There's no copying in that example using `mclapply` and so it is significantly faster than other alternatives. It is a very simple and contrived example, but

Dear all, I?m working as an administrator of a High-Performance Computing (HPC) Cluster which runs on Linux. A lot of people are using R on this Linux cluster and, of course, the *parallel* package to speed up their computations. It has been our collective experience, that using |makeForkCluster| yields an overall better experience /on Linux/ than the |makePSOCKcluster|, for whatever definition

On Mon, 15 Apr 2019 at 08:44, Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > > On 4/13/19 12:05 PM, I?aki Ucar wrote: > > On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com> wrote: > >> I think it's worth saying that mclapply() works as documented > > Mostly, yes. But it says nothing about fork's copy-on-write and memory >

Dear all, I got an issue trying to parse excel files in parallel using XLConnect, the process hangs forever. Martin Studer, the maintainer of XLConnect kindly investigated the issue, identified rJava as a possible cause of the problem: This does not work (hangs): library(parallel) require(rJava) .jinit() res <- mclapply(1:2, function(i) {

On Fri, Jan 10, 2020 at 11:23 AM Simon Urbanek <simon.urbanek at r-project.org> wrote: > > Henrik, > > the example from the post works just fine in CRAN R for me - the post was about homebrew build so it's conceivably a bug in their libraries. Thanks for ruling that example out. > That's exactly why I was proposing a more general solution where you can simply define

On 1/10/20 7:33 AM, Henrik Bengtsson wrote: > I'd like to pick up this thread started on 2019-04-11 > (https://hypatia.math.ethz.ch/pipermail/r-devel/2019-April/077632.html). > Modulo all the other suggestions in this thread, would my proposal of > being able to disable forked processing via an option or an > environment variable make sense? I don't think R should be doing

Yes I am running on Rstudio 1.2.5033. I was also running this code without error on Ubuntu in Rstudio. Checking again on the terminal and it does indeed work fine even with large data.frames. Any idea as to what interaction between Rstudio and mclapply causes this? Thanks, Shian On 28 Apr 2020, at 7:29 pm, Simon Urbanek <simon.urbanek at R-project.org<mailto:simon.urbanek at

> On Jan 10, 2020, at 3:10 PM, G?bor Cs?rdi <csardi.gabor at gmail.com> wrote: > > On Fri, Jan 10, 2020 at 7:23 PM Simon Urbanek > <simon.urbanek at r-project.org> wrote: >> >> Henrik, >> >> the example from the post works just fine in CRAN R for me - the post was about homebrew build so it's conceivably a bug in their libraries. > > I

The RStudio GUI was just one example. AFAIK, and please correct me if I'm wrong, another example is where multi-threaded code is used in forked processing and that's sometimes unstable. Yes another, which might be multi-thread related or not, is https://stat.ethz.ch/pipermail/r-devel/2018-September/076845.html: res <- parallel::mclapply(urls, function(url) { download.file(url,

On Fri, Jan 10, 2020 at 7:23 PM Simon Urbanek <simon.urbanek at r-project.org> wrote: > > Henrik, > > the whole point and only purpose of mc* functions is to fork. That's what the multicore package was about, so if you don't want to fork, don't use mc* functions - they don't have any other purpose. But, with that same argument I'm surprised we have fake

On Fri, Jan 10, 2020 at 7:23 PM Simon Urbanek <simon.urbanek at r-project.org> wrote: > > Henrik, > > the example from the post works just fine in CRAN R for me - the post was about homebrew build so it's conceivably a bug in their libraries. I think it works now, because Apple switched to a different SSL library for libcurl. It usually crashes or fails on older macOS