search for: psock

...expr min lq mean median uq max neval clusterEvalQ(cl, iris) 156.125 166.41 180.8806 170.247 174.298 5322.234 1000 Cheers, Simon > On 2/11/2020, at 3:39 AM, Jeff <jeff at vtkellers.com> wrote: > > I'm exploring latency overhead of parallel PSOCK workers and noticed that serializing/unserializing data back to the main R session is significantly slower on Linux than it is on Windows/MacOS with similar hardware. Is there a reason for this difference and is there a way to avoid the apparent additional Linux overhead? > > I attempted to...

Hi all, Is it possible to use the 'renice' option together with parallel clusters of type 'PSOCK'? The help page for parallel::makeCluster is not specific about which options are supported on which types and I am getting the following message when passing renice = 19 : > cl <- parallel::makeCluster(2, renice = 19) nice: ?+19?: No such file or directory Kind regards, Andreas

...(cl, iris) 156.125 166.41 180.8806 170.247 174.298 >> 5322.234 1000 >> >> Cheers, >> Simon >> >> >> > On 2/11/2020, at 3:39 AM, Jeff <jeff at vtkellers.com> wrote: >> > >> > I'm exploring latency overhead of parallel PSOCK workers and >> noticed that serializing/unserializing data back to the main R >> session is significantly slower on Linux than it is on Windows/MacOS >> with similar hardware. Is there a reason for this difference and is >> there a way to avoid the apparent additional Li...

I'm exploring latency overhead of parallel PSOCK workers and noticed that serializing/unserializing data back to the main R session is significantly slower on Linux than it is on Windows/MacOS with similar hardware. Is there a reason for this difference and is there a way to avoid the apparent additional Linux overhead? I attempted to isolat...

...s about 'nice' arguments, but could be because a "non-standard" 'nice' is used. If we do: > trace(system, tracer = quote(print(command))) Tracing function "system" in package "base" we see that the system call used is: > cl <- parallel::makePSOCKcluster(2L, renice = 19) Tracing system(cmd, wait = FALSE) on entry [1] "nice +19 '/usr/lib/R/bin/Rscript' --default-packages=datasets,utils,grDevices,graphics,stats,methods -e 'parallel:::.slaveRSOCK()' MASTER=localhost PORT=11146 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE" ni...

On Sat, 13 Apr 2019 at 18:41, Simon Urbanek <simon.urbanek at r-project.org> wrote: > > Sure, but that a completely bogus argument because in that case it would fail even more spectacularly with any other method like PSOCK because you would *have to* allocate n times as much memory so unlike mclapply it is guaranteed to fail. With mclapply it is simply much more efficient as it will share memory as long as possible. It is rather obvious that any new objects you create can no longer be shared as they now exist separat...

...ut could be because a "non-standard" 'nice' is used. If > we do: > >> trace(system, tracer = quote(print(command))) > Tracing function "system" in package "base" > > we see that the system call used is: > >> cl <- parallel::makePSOCKcluster(2L, renice = 19) > Tracing system(cmd, wait = FALSE) on entry > [1] "nice +19 '/usr/lib/R/bin/Rscript' > --default-packages=datasets,utils,grDevices,graphics,stats,methods -e > 'parallel:::.slaveRSOCK()' MASTER=localhost PORT=11146 OUT=/dev/null > TIMEOUT=...

...(cl, iris) 156.125 166.41 180.8806 170.247 174.298 5322.234 1000 > >>> Cheers, > >>> Simon > >>> > On 2/11/2020, at 3:39 AM, Jeff <jeff at vtkellers.com> wrote: > >>> > > >>> > I'm exploring latency overhead of parallel PSOCK workers and noticed that serializing/unserializing data back to the main R session is significantly slower on Linux than it is on Windows/MacOS with similar hardware. Is there a reason for this difference and is there a way to avoid the apparent additional Linux overhead? > >>> > &gt...

...median uq max neval > clusterEvalQ(cl, iris) 156.125 166.41 180.8806 170.247 174.298 5322.234 1000 > > Cheers, > Simon > > > > On 2/11/2020, at 3:39 AM, Jeff <jeff at vtkellers.com> wrote: > > > > I'm exploring latency overhead of parallel PSOCK workers and noticed that serializing/unserializing data back to the main R session is significantly slower on Linux than it is on Windows/MacOS with similar hardware. Is there a reason for this difference and is there a way to avoid the apparent additional Linux overhead? > > > > I atte...

...ved example, but there are lots of applications that depend on processing of large data and benefit from multithreading. For example, if I read in large sequencing data with `Rsamtools` and want to check sequences for a set of motifs. > I don't see why mclapply could not be rewritten using PSOCK clusters. Because it would be much slower. > To implement copy-on-write, Linux overcommits virtual memory, and this > is what causes scripts to break unexpectedly: everything works fine, > until you change a small unimportant bit and... boom, out of memory. > And in general, running...

(Bcc: CRAN) This is a proposal helping CRAN and alike as well as individual developers to avoid stray R processes being left behind that might be produced when an example or a package test fails to set up a parallel::makeCluster(). ISSUE If a package test sets up a PSOCK cluster and then the master process dies for one reason or the other, the PSOCK worker processes will remain running for 30 days ('timeout') until they timeout and terminate that way. When this happens on CRAN servers, where many packages are checked all the time, this will result in a lot...

...matically detects variables to export as in the amazing `future` package), the lack of copy-on-write functionality would cause scripts everywhere to break. A simple example illustrating these two points: `x <- 5e8; mclapply(1:24, sum, x, 8)` Using fork, `mclapply` takes 5 seconds. Using "psock", `clusterApply` does not complete. Travers On Fri, Apr 12, 2019 at 2:32 AM I?aki Ucar <iucar at fedoraproject.org> wrote: > > On Thu, 11 Apr 2019 at 22:07, Henrik Bengtsson > <henrik.bengtsson at gmail.com> wrote: > > > > ISSUE: > > Using *forks* for p...

On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com> wrote: > > I think it's worth saying that mclapply() works as documented Mostly, yes. But it says nothing about fork's copy-on-write and memory overcommitment, and that this means that it may work nicely or fail spectacularly depending on whether, e.g., you operate on a long vector. -- I?aki ?car

...ase the speed of creation of a parallel socket cluster. The time spend to include threads increase exponentially with the number of thread considered and I use of computer with two 8 cores CPU and thus showing a total of 32 threads in windows 7. Currently, I use the default parameters (type = "PSOCK"), but is there any fine tuning parameters that I can use to take advantage of this system ? Thanks in advance for your help ! Arnaud R version 3.0.1 (2013-05-16) Platform: x86_64-w64-mingw32/x64 (64-bit) [[alternative HTML version deleted]]

...gt;>> clusterEvalQ(cl, iris) 156.125 166.41 180.8806 170.247 174.298 5322.234 1000 >>> Cheers, >>> Simon >>> > On 2/11/2020, at 3:39 AM, Jeff <jeff at vtkellers.com> wrote: >>> > >>> > I'm exploring latency overhead of parallel PSOCK workers and noticed that serializing/unserializing data back to the main R session is significantly slower on Linux than it is on Windows/MacOS with similar hardware. Is there a reason for this difference and is there a way to avoid the apparent additional Linux overhead? >>> > >>...

...arallel packages in R end up calling multicore or snow deep down. I think one of the great advantages of mclapply is that it defaults to lapply when running on a single thread, this makes it much easier to maintain code with optional parallelism. I?m already running into trouble with the fact that PSOCK doesn?t seem to retain loaded packages in spawned processes. I would love to know if there reliable options in R that allow a similar interface to mclapply but use a different and more RStudio-stable mode of parallelisation? Thanks, Shian > On 29 Apr 2020, at 1:33 pm, Simon Urbanek <simon.u...

...> This is a proposal helping CRAN and alike as well as individual > developers to avoid stray R processes being left behind that might be > produced when an example or a package test fails to set up a > parallel::makeCluster(). > > > ISSUE > > If a package test sets up a PSOCK cluster and then the master process > dies for one reason or the other, the PSOCK worker processes will > remain running for 30 days ('timeout') until they timeout and > terminate that way. When this happens on CRAN servers, where many > packages are checked all the time, this...

...sses. _In practice_, you do not need signal handling. The startup time stamp method is completely fine, because it is practically impossible to have two processes with the same pid and the same (high precision) startup time. This method also works for any process (not just child processes), so for PSOCK clusters as well. Gabor [...]

...ackages in R end up calling multicore or snow deep down. > > I think one of the great advantages of mclapply is that it defaults to lapply when running on a single thread, this makes it much easier to maintain code with optional parallelism. I?m already running into trouble with the fact that PSOCK doesn?t seem to retain loaded packages in spawned processes. I would love to know if there reliable options in R that allow a similar interface to mclapply but use a different and more RStudio-stable mode of parallelisation? If you use parLapply(cl, ...) and gives the end-users the control over th...

Hi, I'd like to test whether a (localhost) PSOCK cluster node is still running or not by its PID, e.g. it may have crashed / core dumped. I'm ok with getting false-positive results due to *another* process with the same PID has since started. I can the PID of each cluster nodes by querying them for their Sys.getpid(), e.g. pids <- pa...