similar to: Race condition on parallel package's mcexit and rmChild

This issue has already been addressed in 76462 (R-devel) and also ported to R-patched. In fact rmChild() is used in mccollect(wait=FALSE). Best Tomas On 5/19/19 11:39 AM, Sun Yijiang wrote: > I've been hacking with parallel package for some time and built a > parallel processing framework with it. However, although very rarely, > I did notice "ignoring SIGPIPE signal"

I've been hacking with parallel package for some time and built a parallel processing framework with it. However, although very rarely, I did notice "ignoring SIGPIPE signal" error every now and then. After a deep dig into the source code, I think I found something worth noticing. In short, wring to pipe in the C function mc_exit(SEXP sRes) may cause a SIGPIPE. Code from

Have read the latest code, but I still don't understand why mc_exit needs to write zero on exit. If a child closes its pipe, parent will know that on next select. Best, Yijiang Tomas Kalibera <tomas.kalibera at gmail.com> ?2019?5?20??? ??10:52??? > > This issue has already been addressed in 76462 (R-devel) and also ported > to R-patched. In fact rmChild() is used in

Hi all, With the help of Tomas, I was able to track the issue down: Prior to R v3.6.0 the parallel package passes an uninitialized variable as the file descriptor argument to the close system call. In my particular R session this uninitialized variable (reproducibly) was holding the value 7, which corresponded to the file descriptor of the write end of the pipe the second child would use to

Hi all, With the help of Tomas, I was able to track the issue down: Prior to R v3.6.0 the parallel package passes an uninitialized variable as the file descriptor argument to the close system call. In my particular R session this uninitialized variable (reproducibly) was holding the value 7, which corresponded to the file descriptor of the write end of the pipe the second child would use to

Hi Andreas, thank you very much, good job finding it was EBADF. Now the question is why the pipe has been closed prematurely; it could be accidentally by R (a race condition in the cleanup code in fork.c) or possibly by some other code running in the same process (maybe the R program itself or some other code it runs). Maybe we can take this off the list and come back when we know the cause

Hi Tomas, Thanks for your prompt reply and your offer to help. I might need to get back to this since I am not too experienced in debugging these kinds of issues. Anyway, I gave it a try and I think I have found the immediate cause: I installed the debug symbols (r-base-core-dbg), placed https://github.com/wch/r-source/blob/tags/R-3-5-2/src/library/parallel/src/fork.c in cwd and changed the

Hi Andreas, the error is reported when some child process cannot send results to the master process, which originates from an error returned by write() - when write() returns -1 or 0. The logic around the writing has not changed since R 3.5.2. It should not be related to the printing in the child, only to returning the value. The problem may be originating from the execution environment,

Dear fellow R users, I am experiencing some problem using function mclapply from recommended R package parallel. This problem is a bit "deep in source code" (maybe r-devel list would be more appropriate ?). Actually, I have noticed on a specific Linux HPC system that the various readChild() calls in mclapply did not end the children processes, as would be expected (ie compared to what

I stumbled upon the following: f <- parallel::mcparallel(raw(0L)) parallel::mccollect(f) # $`77083` # NULL but f <- parallel::mcparallel(raw(1L)) parallel::mccollect(f) # Error in unserialize(r) : read error traceback() # 2: unserialize(r) # 1: parallel::mccollect(f) (restarting because the above appears to corrupt the R session) f <- parallel::mcparallel(raw(2L))

Hi again, One important correction of my first message: I misinterpreted the output. Actually in that R session 2 input files were processed one after the other in a loop. The first (with 88 parts went fine). The second (with 85 parts) produced the sendMaster errors and failed. If (in a new session via Rscript) I only process the second input file it will work. The other observations on R vs

Hi, I am facing a very weird problem with parallel::mclapply. I have a script which does some data wrangling on an input dataset in parallel and then writes the results to disk. I have been using this script daily for more than one year always on an EC2 instance launched from the same AMI (no updates installed after launch) and processed thousands of different input data sets successfully. I now

Moritz Muehlenhoff writes ("Re: Xen package security updates for jessie 4.4, XSA-213, XSA-214"): > Yes, the distribution line should be jessie-security, but please send > a debdiff to team at security.debian.org for a quick review before > uploading (I have no idea whether dgit supports security-master). Here is the proposed debdiff (actually, a git diff) for xen in jessie. My

The original implementation utilies select(). In Linux select() typically supports up to 1024 file descriptors. This can be a problem when user tries to boot up many guests. Switching to poll() has minimum impact on existing code and has better scalibility. Up to 8192 file descriptors are supported in the current implementation. Signed-off-by: Wei Liu <wei.liu2@citrix.com> ---

Dear R developers: ... 7: lapply(seq_len(cores), inner.do) 8: FUN(1:3[[3]], ...) 9: sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE)) Selection: .....................Error in sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE)) : long vectors not supported yet: memory.c:3100 admittedly, my outcome will be a very big list, with 30,000 elements, each containing data frames

Hello everyone, I'm trying to parallelize an R script I have written. To do this, I am first trying to use the multicore package, because I've had some previous success with that. The function I'm trying to parallelize is illumqc. I'd like to create a separate process for each of 8 files, contained in the vector "files". Below is my code: for(i in

Hi! When running dovecot from inetd (as per the instructions on http://wiki.dovecot.org/InetdInstall), imap-login and pop3-login thinks that TLS is enabled when in fact it isn't (we're connecting to port 110 and port 143, so the --ssl parameter isn't present). The reason for that is found in login-common/main.c:main(): > if (fd != -1) > (void)client_create(fd, &ip,

Hi, I am experiencing a problem related to serialization behavior in 2.15.0 beta (binary installed from Debian unstable) and 2.16.0 (from svn) that is not present in 2.14.2 (binary from Debian testing). I don't fully understand the problem. Also, I tried but have not yet been able to create a small, self-contained example that reproduces the problem. However, I do have a large, not

Thanks for the report, but unfortunately I cannot reproduce on my system (either macOS nor Linux, from the command line) to debug. Did you run this in the command line version of R? I would not be surprised to see such a crash if executed from a multi-threaded application, say from some GUI or frontend that runs multiple threads, or from some other R session where a third party library

This code actually happens to work for me on macOS, but I think in general you cannot rely on performing HTTP requests in fork clusters, i.e. with mclapply(). Fork clusters create worker processes by forking the R process and then _not_ executing another R binary. (Which is often convenient, because the new processes will inherit the memory image of the parent process.) Fork without exec is not