similar to: SUGGESTION: Proposal to mitigate problem with stray processes left behind by parallel::makeCluster()

The problem causing the stray worker processes when the master fails to open a server socket to listen to connections from workers is not related to timeout in socketConnection(), because socketConnection() will fail right away. It is caused by a bug in checking the setup timeout (PR 17391). Fixed in 76275. Best Tomas On 3/18/19 2:23 AM, Henrik Bengtsson wrote: > (Bcc: CRAN) > >

Hi all, The call parallel::makeCluster(1L) hangs infinitely on my MacOS machine which seems to be already reported by some people (e.g., https://stat.ethz.ch/pipermail/r-devel/2018-February/075565.html). However, the solutions posted on SO, GH or R-devel do not work in my case. So far, I unsuccessfully tested ? 1. Couple of reboots 2. Adding 192.0.0.1 to /etc/hosts 3. Using R.app

Hi all, Is it possible to use the 'renice' option together with parallel clusters of type 'PSOCK'? The help page for parallel::makeCluster is not specific about which options are supported on which types and I am getting the following message when passing renice = 19 : > cl <- parallel::makeCluster(2, renice = 19) nice: ?+19?: No such file or directory Kind regards,

Hi Henrik, Thanks for the detailed in fast reply! My guess would be that the confusion comes from the different use of nice and renice. The workraund you provided work fine! Thanks a lot. Best, Andreas Henrik Bengtsson <henrik.bengtsson at gmail.com> writes: > Looks like a bug to me due to wrong assumptions about 'nice' > arguments, but could be because a

Hi Thomas, thanks for your reply (and thanks for your patience...). I am now using the following minimal reprex: > library(parallel) > cl <- makeCluster(2L) I freshly started the machine and did not open any other app. Just R.app (3.6.1). After executing the second line of code, R seems to hang infinitely and does not respond. The R process itself uses almost no CPU. Unfortunately,

BACKGROUND: While troubleshooting random, occasionally occurring, errors from parallel::makePSOCKcluster("localhost", port = 11000); Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE, : cannot open the connection I had another look at parallel:::newPSOCKnode(), which is used internally to set up each background worker. It is designed to,

A solution is to have parallel:::.slaveRSOCK() attempt to connect multiple times before failing, e.g. makeSOCKmaster <- function(master, port, timeout, useXDR, maxTries = 10L, interval = 1.0) { port <- as.integer(port) for (i in seq_len(maxTries)) { con <- tryCatch({ socketConnection(master, port = port, blocking = TRUE,

Looks like a bug to me due to wrong assumptions about 'nice' arguments, but could be because a "non-standard" 'nice' is used. If we do: > trace(system, tracer = quote(print(command))) Tracing function "system" in package "base" we see that the system call used is: > cl <- parallel::makePSOCKcluster(2L, renice = 19) Tracing system(cmd, wait

Great. For the record of this thread, I've submitted patch PR17391 (https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=17391). I've patched it against the latest R-devel on the SVN, passes 'make check-all', and I've verified it works with the above tests. /Henrik On Fri, Mar 9, 2018 at 4:37 AM, <luke-tierney at uiowa.edu> wrote: > I'm happy to look at a

Hi all, I am quite new in the world of parallelization and I wonder if there is a way to increase the speed of creation of a parallel socket cluster. The time spend to include threads increase exponentially with the number of thread considered and I use of computer with two 8 cores CPU and thus showing a total of 32 threads in windows 7. Currently, I use the default parameters (type =

I'm not sure the user would know ;). This is very system-specific issue just because the Linux network stack behaves so differently from other OSes (for purely historical reasons). That makes it hard to abstract as a "feature" for the R sockets that are supposed to be platform-independent. At least TCP_NODELAY is actually part of POSIX so it is on better footing, and disabling

On Mon, 2 Nov 2020 at 02:22, Simon Urbanek <simon.urbanek at r-project.org> wrote: > > It looks like R sockets on Linux could do with TCP_NODELAY -- without (status quo): How many network packets are generated with and without it? If there are many small writes and thus setting TCP_NODELAY causes many small packets to be sent, it might make more sense to set TCP_QUICKACK instead.

Hi Dominik, from the output, the master process could not "listen" on the port where it expects a connection from the worker. We need to find out why. I'd recommend first to create a minimal reproducible example (and one that does not use future, only parallel, and a minimal number of threads, ideally just 2). Then I'd recommend to check if the problem still exists with

On Mon, 31 Oct 2016, Henrik Bengtsson wrote: > Thank you for looking into this Luke. > > On Thu, Oct 27, 2016 at 9:26 AM, <luke-tierney at uiowa.edu> wrote: >> On unix, unless event polling is enabled Sys.sleep just waits in a >> select() call (with a SIGINT handler in place) so the elapsed time >> isn't checked until after the select call is complete.

I just noticed that parallel:::.slaveRSOCK() passes 'timeout' to socketConnection() as a character, i.e. there's a missing timeout <- as.integer(timeout), cf. port <- as.integer(port) and useXDR <- as.logical(value): > parallel:::.slaveRSOCK function () { makeSOCKmaster <- function(master, port, timeout, useXDR) { port <- as.integer(port) con

I'm happy to look at a patch that does this. I'd start with a small interval and increase it by 50%, say, on each try wit a max retry time limit. This isn't eliminating the problem,only reducing the probability, but still worth it. I had considered doing something like this but it didn't seem necessary at the time. You don't want to retry indefinitely since the connection

Dear list, i have some problems using the snow package to create a SOCK cluster. The errors just occour irregularly but it seems to me that they occour when I try to create more than one cluster on the same machine via different R instances started via submitting LSF jobs to a cluster. Does anyone have an idea how to solve this or where to start digging for solutions? The error messages

I'm exploring latency overhead of parallel PSOCK workers and noticed that serializing/unserializing data back to the main R session is significantly slower on Linux than it is on Windows/MacOS with similar hardware. Is there a reason for this difference and is there a way to avoid the apparent additional Linux overhead? I attempted to isolate the behavior with a test that simply returns

Hi Inaki, > "Performant"... in terms of what. If the cost of copying the data > predominates over the computation time, maybe you didn't need > parallelization in the first place. Performant in terms of speed. There's no copying in that example using `mclapply` and so it is significantly faster than other alternatives. It is a very simple and contrived example, but

Hi! I recognized that the argument matching of clusterApply (and therefore parLapply) goes wrong when one of the arguments of the function is called "c". In this case, the argument "c" is used as cluster and the functions give the following error message "Error in checkCluster(cl) : not a valid cluster". Of course, "c" is for many reasons an unfortunate