Hi,
I am facing a very weird problem with parallel::mclapply. I have a script which
does some data wrangling on an input dataset in parallel and then writes the
results to disk. I have been using this script daily for more than one year
always on an EC2 instance launched from the same AMI (no updates installed after
launch) and processed thousands of different input data sets successfully. I now
have an input dataset for which I face the following bug:
The basic outline of the problematic section of the script:
# parts is a data.table with 88 rows
mc_ret <- parallel::mclapply(sample.int(nrow(parts)), function(i) {
# do some data wrangling and write the result to a file
# ...
print(paste0("part ", i, " written successfully."))
return(TRUE)
}, mc.preschedule = FALSE, mc.cores = 2L)
str(mc_ret)
Expected output: "part i written successfully." is printed 88 times,
once for each value of i. mc_ret is a list of length 88, each element being
TRUE. Its structure is printed once. All outputs are created successfully.
Actual output (see end of the message): "part i written successfully."
is printed 88 times, once for each value of i. mc_ret is a list of length 88,
each element being TRUE. Its structure is printed. All outputs are created
successfully. So far so good.
But then "part i written successfully." it is printed another X times,
for values of i for which it was already printed. This output is intermingled
with X-1 times the following error message:
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
and Y times the message "Execution halted". mc_ret is printed again,
now being a list of length 85, with the first element being TRUE and all other
elements being NULL. X and Y vary from run to run.
Now to the main problem: I tried very hard to create a reproducible example, but
I failed. What I observed:
- The output is (and has always been) written to path which is on an NFS share.
If I instead write to a path on a local disk it will work.
- The script is invoked using Rscript. If I instead source it from an
interactive R session it works. There are at least two more people who have
observed this:
https://stackoverflow.com/questions/51986674/mclapply-sendmaster-error-only-with-rscript
- Before the call to mclapply the code acquires an exclusive file lock on a
dedicated lock file, not written to but also on the NFS share. If I remove the
code acquiring the lock, the whole script will also work if called using
Rscript.
- The problem also occurs for mc.preschedule = TRUE.
- There is no error if I set mc.cores to 1.
- And stressing again: the code works without any changes from Rscript for
thousands of other data sets.
Rscript -e "sessionInfo()":
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.5.2
I know this is a fairly old R version. I have not been able to reproduce the bug
with a more recent version, but since it is so difficult to trigger, this does
not mean much, I guess. I have looked through the changes made to the code of
mclapply since that version and could not find something directly related. I am
not even sure if it is a problem in the parallel package or some other (memory)
bug. What strikes me is that others have observed a very similar error when
using Rscript but not when using an interactive R session, just like I do.
I am not expecting a fix based on the information I provide, but maybe someone
has some thoughts on this!?
Regards,
Andreas
Actual output:
[1] "part 51 written successfully."
[1] "part 30 written successfully."
[1] "part 32 written successfully."
[1] "part 48 written successfully."
[1] "part 63 written successfully."
[1] "part 5 written successfully."
[1][1] "part 14 written successfully." "part 18 written
successfully."
[1] "part 38 written successfully."
[1] "part 11 written successfully."
[1] "part 68 written successfully."
[1] "part 45 written successfully."
[1] "part 88 written successfully."
[1] "part 36 written successfully."
[1] "part 44 written successfully."
[1] "part 55 written successfully."
[1] "part 26 written successfully."
[1] "part 37 written successfully."
[1] "part 22 written successfully."
[1] "part 13 written successfully."
[1] "part 67 written successfully."
[1] "part 10 written successfully."
[1] "part 24 written successfully."
[1] "part 20 written successfully."
[1] "part 74 written successfully."
[1] "part 50 written successfully."
[1] "part 2 written successfully."
[1] "part 17 written successfully."
[1] "part 66 written successfully."
[1] "part 61 written successfully."
[1][1] "part 7 written successfully." "part 31 written
successfully."
[1] "part 70 written successfully."
[1] "part 71 written successfully."
[1][1] "part 76 written successfully."
"part 42 written successfully."
[1][1] "part 72 written successfully." "part 12 written
successfully."
[1] "part 39 written successfully."
[1] "part 58 written successfully."
[1] "part 87 written successfully."
[1] "part 81 written successfully."
[1] "part 3 written successfully."
[1] "part 29 written successfully."
[1] "part 82 written successfully."
[1] "part 54 written successfully."
[1] "part 23 written successfully."
[1] "part 56 written successfully."
[1] "part 73 written successfully."
[1] "part 27 written successfully."
[1] "part 64 written successfully."
[1] "part 46 written successfully."
[1] "part 80 written successfully."
[1] "part 53 written successfully."
[1] "part 43 written successfully."
[1] "part 47 written successfully."
[1] "part 57 written successfully."
[1] "part 75 written successfully."
[1] "part 84 written successfully."
[1] "part 85 written successfully."
[1][1] "part 86 written successfully."
"part 19 written successfully."
[1][1] "part 33 written successfully." "part 34 written
successfully."
[1] "part 21 written successfully."
[1] "part 40 written successfully."
[1][1] "part 4 written successfully."
"part 28 written successfully."
[1] "part 62 written successfully."
[1] "part 16 written successfully."
[1] "part 79 written successfully."
[1] "part 6 written successfully."
[1] "part 77 written successfully."
[1] "part 65 written successfully."
[1] "part 60 written successfully."
[1] "part 78 written successfully."
[1] "part 1 written successfully."
[1] "part 83 written successfully."
[1] "part 9 written successfully."
[1] "part 15 written successfully."
[1][1] "part 41 written successfully."
"part 35 written successfully."
[1][1] "part 8 written successfully." "part 69 written
successfully."
[1] "part 59 written successfully."
[1] "part 25 written successfully."
[1] "part 49 written successfully."
[1] "part 52 written successfully."
List of 88
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
$ : logi TRUE
[1] "part 31 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> lapply -> FUN -> mcparallel
-> sendMaster
[1] "part 71 written successfully."
[1] "part 35 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 42 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
[1] "part 4 written successfully."
Execution halted
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1][1] "part 30 written successfully." "part 40 written
successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
[1] "part 41 written successfully."
[1] "part 80 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 45 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
[1] "part 82 written successfully."
[1] "part 54 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 59 written successfully."
Execution halted
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 27 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
[1] "part 36 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
[1] "part 44 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
Execution halted
[1] "part 83 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 61 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 70 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 66 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 10 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
Execution halted
[1] "part 53 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 55 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 15 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 50 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 5 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
Execution halted
[1] "part 48 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 32 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 69 written successfully."
Execution halted
[1] "part 52 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 77 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
Execution halted
[1] "part 24 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 56 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 6 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 62 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 33 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
[1] "part 16 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 34 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 74 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
[1] "part 2 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 19 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
[1] "part 9 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 11 written successfully."
[1] "part 28 written successfully."
[1] "part 29 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 17 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 65 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
Execution halted
Execution halted
[1] "part 68 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 43 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 38 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 1 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 57 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 8 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1] "part 25 written successfully."
Error in sendMaster(try(eval(expr, env), silent = TRUE)) :
write error, closing pipe to the master
Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel
-> sendMaster
[1]List of 85
$ : logi TRUE
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
$ : NULL
Hi again, One important correction of my first message: I misinterpreted the output. Actually in that R session 2 input files were processed one after the other in a loop. The first (with 88 parts went fine). The second (with 85 parts) produced the sendMaster errors and failed. If (in a new session via Rscript) I only process the second input file it will work. The other observations on R vs Rscript, NFS share etc. still hold. Sorry for this! Regards, Andreas 2019-11-27 12:10 GMT+01:00 Andreas Kersting<r-devel at akersting.de>:> Hi, > > I am facing a very weird problem with parallel::mclapply. I have a script which does some data wrangling on an input dataset in parallel and then writes the results to disk. I have been using this script daily for more than one year always on an EC2 instance launched from the same AMI (no updates installed after launch) and processed thousands of different input data sets successfully. I now have an input dataset for which I face the following bug: > > The basic outline of the problematic section of the script: > > # parts is a data.table with 88 rows > mc_ret <- parallel::mclapply(sample.int(nrow(parts)), function(i) { > # do some data wrangling and write the result to a file > # ... > > print(paste0("part ", i, " written successfully.")) > return(TRUE) > }, mc.preschedule = FALSE, mc.cores = 2L) > > str(mc_ret) > > > Expected output: "part i written successfully." is printed 88 times, once for each value of i. mc_ret is a list of length 88, each element being TRUE. Its structure is printed once. All outputs are created successfully. > > Actual output (see end of the message): "part i written successfully." is printed 88 times, once for each value of i. mc_ret is a list of length 88, each element being TRUE. Its structure is printed. All outputs are created successfully. So far so good. > > But then "part i written successfully." it is printed another X times, for values of i for which it was already printed. This output is intermingled with X-1 times the following error message: > > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > > and Y times the message "Execution halted". mc_ret is printed again, now being a list of length 85, with the first element being TRUE and all other elements being NULL. X and Y vary from run to run. > > > Now to the main problem: I tried very hard to create a reproducible example, but I failed. What I observed: > - The output is (and has always been) written to path which is on an NFS share. If I instead write to a path on a local disk it will work. > - The script is invoked using Rscript. If I instead source it from an interactive R session it works. There are at least two more people who have observed this: https://stackoverflow.com/questions/51986674/mclapply-sendmaster-error-only-with-rscript > - Before the call to mclapply the code acquires an exclusive file lock on a dedicated lock file, not written to but also on the NFS share. If I remove the code acquiring the lock, the whole script will also work if called using Rscript. > - The problem also occurs for mc.preschedule = TRUE. > - There is no error if I set mc.cores to 1. > - And stressing again: the code works without any changes from Rscript for thousands of other data sets. > > > Rscript -e "sessionInfo()": > R version 3.5.2 (2018-12-20) > Platform: x86_64-pc-linux-gnu (64-bit) > Running under: Ubuntu 18.04.2 LTS > > Matrix products: default > BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 > LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so > > locale: > [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8 > [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8 > [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C > [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > loaded via a namespace (and not attached): > [1] compiler_3.5.2 > > > I know this is a fairly old R version. I have not been able to reproduce the bug with a more recent version, but since it is so difficult to trigger, this does not mean much, I guess. I have looked through the changes made to the code of mclapply since that version and could not find something directly related. I am not even sure if it is a problem in the parallel package or some other (memory) bug. What strikes me is that others have observed a very similar error when using Rscript but not when using an interactive R session, just like I do. > > I am not expecting a fix based on the information I provide, but maybe someone has some thoughts on this!? > > Regards, > Andreas > > > > > Actual output: > > [1] "part 51 written successfully." > [1] "part 30 written successfully." > [1] "part 32 written successfully." > [1] "part 48 written successfully." > [1] "part 63 written successfully." > [1] "part 5 written successfully." > [1][1] "part 14 written successfully." "part 18 written successfully." > > [1] "part 38 written successfully." > [1] "part 11 written successfully." > [1] "part 68 written successfully." > [1] "part 45 written successfully." > [1] "part 88 written successfully." > [1] "part 36 written successfully." > [1] "part 44 written successfully." > [1] "part 55 written successfully." > [1] "part 26 written successfully." > [1] "part 37 written successfully." > [1] "part 22 written successfully." > [1] "part 13 written successfully." > [1] "part 67 written successfully." > [1] "part 10 written successfully." > [1] "part 24 written successfully." > [1] "part 20 written successfully." > [1] "part 74 written successfully." > [1] "part 50 written successfully." > [1] "part 2 written successfully." > [1] "part 17 written successfully." > [1] "part 66 written successfully." > [1] "part 61 written successfully." > [1][1] "part 7 written successfully." "part 31 written successfully." > > [1] "part 70 written successfully." > [1] "part 71 written successfully." > [1][1] "part 76 written successfully." > "part 42 written successfully." > [1][1] "part 72 written successfully." "part 12 written successfully." > > [1] "part 39 written successfully." > [1] "part 58 written successfully." > [1] "part 87 written successfully." > [1] "part 81 written successfully." > [1] "part 3 written successfully." > [1] "part 29 written successfully." > [1] "part 82 written successfully." > [1] "part 54 written successfully." > [1] "part 23 written successfully." > [1] "part 56 written successfully." > [1] "part 73 written successfully." > [1] "part 27 written successfully." > [1] "part 64 written successfully." > [1] "part 46 written successfully." > [1] "part 80 written successfully." > [1] "part 53 written successfully." > [1] "part 43 written successfully." > [1] "part 47 written successfully." > [1] "part 57 written successfully." > [1] "part 75 written successfully." > [1] "part 84 written successfully." > [1] "part 85 written successfully." > [1][1] "part 86 written successfully." > "part 19 written successfully." > [1][1] "part 33 written successfully." "part 34 written successfully." > > [1] "part 21 written successfully." > [1] "part 40 written successfully." > [1][1] "part 4 written successfully." > "part 28 written successfully." > [1] "part 62 written successfully." > [1] "part 16 written successfully." > [1] "part 79 written successfully." > [1] "part 6 written successfully." > [1] "part 77 written successfully." > [1] "part 65 written successfully." > [1] "part 60 written successfully." > [1] "part 78 written successfully." > [1] "part 1 written successfully." > [1] "part 83 written successfully." > [1] "part 9 written successfully." > [1] "part 15 written successfully." > [1][1] "part 41 written successfully." > "part 35 written successfully." > [1][1] "part 8 written successfully." "part 69 written successfully." > > [1] "part 59 written successfully." > [1] "part 25 written successfully." > [1] "part 49 written successfully." > [1] "part 52 written successfully." > List of 88 > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > $ : logi TRUE > [1] "part 31 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> lapply -> FUN -> mcparallel -> sendMaster > [1] "part 71 written successfully." > [1] "part 35 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 42 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > [1] "part 4 written successfully." > Execution halted > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1][1] "part 30 written successfully." "part 40 written successfully." > > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > [1] "part 41 written successfully." > [1] "part 80 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 45 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > [1] "part 82 written successfully." > [1] "part 54 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 59 written successfully." > Execution halted > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 27 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > [1] "part 36 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > [1] "part 44 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > Execution halted > [1] "part 83 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 61 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 70 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 66 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 10 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > Execution halted > [1] "part 53 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 55 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 15 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 50 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 5 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > Execution halted > [1] "part 48 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 32 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 69 written successfully." > Execution halted > [1] "part 52 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 77 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > Execution halted > [1] "part 24 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 56 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 6 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 62 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 33 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > [1] "part 16 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 34 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 74 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > [1] "part 2 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 19 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > [1] "part 9 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 11 written successfully." > [1] "part 28 written successfully." > [1] "part 29 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 17 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 65 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > Execution halted > Execution halted > [1] "part 68 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 43 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 38 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 1 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 57 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 8 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1] "part 25 written successfully." > Error in sendMaster(try(eval(expr, env), silent = TRUE)) : > write error, closing pipe to the master > Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster > [1]List of 85 > $ : logi TRUE > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > $ : NULL > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >
Hi Andreas, the error is reported when some child process cannot send results to the master process, which originates from an error returned by write() - when write() returns -1 or 0. The logic around the writing has not changed since R 3.5.2. It should not be related to the printing in the child, only to returning the value. The problem may be originating from the execution environment, virtualization, and/or possibly from a lack of robustness in R. To resolve this we need to find out which error was returned and why. Either you can try to create a reproducible example (something I could use to trigger an error on my system and then debug) or to debug on your system (build R from source, ensure the bug is still triggered, then instrument to print the exact error from the OS and where it was detected, etc). In principle you could also try without code instrumentation just using strace. Just from looking at the code in R around the writing I am not seeing any bug there. If you choose to debug on your system I can help with the instrumentation. Best Tomas On 11/27/19 12:40 PM, Andreas Kersting wrote:> Hi again, > > One important correction of my first message: I misinterpreted the output. Actually in that R session 2 input files were processed one after the other in a loop. The first (with 88 parts went fine). The second (with 85 parts) produced the sendMaster errors and failed. If (in a new session via Rscript) I only process the second input file it will work. The other observations on R vs Rscript, NFS share etc. still hold. > > Sorry for this! Regards, > Andreas > > 2019-11-27 12:10 GMT+01:00 Andreas Kersting<r-devel at akersting.de>: >> Hi, >> >> I am facing a very weird problem with parallel::mclapply. I have a script which does some data wrangling on an input dataset in parallel and then writes the results to disk. I have been using this script daily for more than one year always on an EC2 instance launched from the same AMI (no updates installed after launch) and processed thousands of different input data sets successfully. I now have an input dataset for which I face the following bug: >> >> The basic outline of the problematic section of the script: >> >> # parts is a data.table with 88 rows >> mc_ret <- parallel::mclapply(sample.int(nrow(parts)), function(i) { >> # do some data wrangling and write the result to a file >> # ... >> >> print(paste0("part ", i, " written successfully.")) >> return(TRUE) >> }, mc.preschedule = FALSE, mc.cores = 2L) >> >> str(mc_ret) >> >> >> Expected output: "part i written successfully." is printed 88 times, once for each value of i. mc_ret is a list of length 88, each element being TRUE. Its structure is printed once. All outputs are created successfully. >> >> Actual output (see end of the message): "part i written successfully." is printed 88 times, once for each value of i. mc_ret is a list of length 88, each element being TRUE. Its structure is printed. All outputs are created successfully. So far so good. >> >> But then "part i written successfully." it is printed another X times, for values of i for which it was already printed. This output is intermingled with X-1 times the following error message: >> >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> >> and Y times the message "Execution halted". mc_ret is printed again, now being a list of length 85, with the first element being TRUE and all other elements being NULL. X and Y vary from run to run. >> >> >> Now to the main problem: I tried very hard to create a reproducible example, but I failed. What I observed: >> - The output is (and has always been) written to path which is on an NFS share. If I instead write to a path on a local disk it will work. >> - The script is invoked using Rscript. If I instead source it from an interactive R session it works. There are at least two more people who have observed this: https://stackoverflow.com/questions/51986674/mclapply-sendmaster-error-only-with-rscript >> - Before the call to mclapply the code acquires an exclusive file lock on a dedicated lock file, not written to but also on the NFS share. If I remove the code acquiring the lock, the whole script will also work if called using Rscript. >> - The problem also occurs for mc.preschedule = TRUE. >> - There is no error if I set mc.cores to 1. >> - And stressing again: the code works without any changes from Rscript for thousands of other data sets. >> >> >> Rscript -e "sessionInfo()": >> R version 3.5.2 (2018-12-20) >> Platform: x86_64-pc-linux-gnu (64-bit) >> Running under: Ubuntu 18.04.2 LTS >> >> Matrix products: default >> BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 >> LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so >> >> locale: >> [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8 >> [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8 >> [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C >> [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> loaded via a namespace (and not attached): >> [1] compiler_3.5.2 >> >> >> I know this is a fairly old R version. I have not been able to reproduce the bug with a more recent version, but since it is so difficult to trigger, this does not mean much, I guess. I have looked through the changes made to the code of mclapply since that version and could not find something directly related. I am not even sure if it is a problem in the parallel package or some other (memory) bug. What strikes me is that others have observed a very similar error when using Rscript but not when using an interactive R session, just like I do. >> >> I am not expecting a fix based on the information I provide, but maybe someone has some thoughts on this!? >> >> Regards, >> Andreas >> >> >> >> >> Actual output: >> >> [1] "part 51 written successfully." >> [1] "part 30 written successfully." >> [1] "part 32 written successfully." >> [1] "part 48 written successfully." >> [1] "part 63 written successfully." >> [1] "part 5 written successfully." >> [1][1] "part 14 written successfully." "part 18 written successfully." >> >> [1] "part 38 written successfully." >> [1] "part 11 written successfully." >> [1] "part 68 written successfully." >> [1] "part 45 written successfully." >> [1] "part 88 written successfully." >> [1] "part 36 written successfully." >> [1] "part 44 written successfully." >> [1] "part 55 written successfully." >> [1] "part 26 written successfully." >> [1] "part 37 written successfully." >> [1] "part 22 written successfully." >> [1] "part 13 written successfully." >> [1] "part 67 written successfully." >> [1] "part 10 written successfully." >> [1] "part 24 written successfully." >> [1] "part 20 written successfully." >> [1] "part 74 written successfully." >> [1] "part 50 written successfully." >> [1] "part 2 written successfully." >> [1] "part 17 written successfully." >> [1] "part 66 written successfully." >> [1] "part 61 written successfully." >> [1][1] "part 7 written successfully." "part 31 written successfully." >> >> [1] "part 70 written successfully." >> [1] "part 71 written successfully." >> [1][1] "part 76 written successfully." >> "part 42 written successfully." >> [1][1] "part 72 written successfully." "part 12 written successfully." >> >> [1] "part 39 written successfully." >> [1] "part 58 written successfully." >> [1] "part 87 written successfully." >> [1] "part 81 written successfully." >> [1] "part 3 written successfully." >> [1] "part 29 written successfully." >> [1] "part 82 written successfully." >> [1] "part 54 written successfully." >> [1] "part 23 written successfully." >> [1] "part 56 written successfully." >> [1] "part 73 written successfully." >> [1] "part 27 written successfully." >> [1] "part 64 written successfully." >> [1] "part 46 written successfully." >> [1] "part 80 written successfully." >> [1] "part 53 written successfully." >> [1] "part 43 written successfully." >> [1] "part 47 written successfully." >> [1] "part 57 written successfully." >> [1] "part 75 written successfully." >> [1] "part 84 written successfully." >> [1] "part 85 written successfully." >> [1][1] "part 86 written successfully." >> "part 19 written successfully." >> [1][1] "part 33 written successfully." "part 34 written successfully." >> >> [1] "part 21 written successfully." >> [1] "part 40 written successfully." >> [1][1] "part 4 written successfully." >> "part 28 written successfully." >> [1] "part 62 written successfully." >> [1] "part 16 written successfully." >> [1] "part 79 written successfully." >> [1] "part 6 written successfully." >> [1] "part 77 written successfully." >> [1] "part 65 written successfully." >> [1] "part 60 written successfully." >> [1] "part 78 written successfully." >> [1] "part 1 written successfully." >> [1] "part 83 written successfully." >> [1] "part 9 written successfully." >> [1] "part 15 written successfully." >> [1][1] "part 41 written successfully." >> "part 35 written successfully." >> [1][1] "part 8 written successfully." "part 69 written successfully." >> >> [1] "part 59 written successfully." >> [1] "part 25 written successfully." >> [1] "part 49 written successfully." >> [1] "part 52 written successfully." >> List of 88 >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> $ : logi TRUE >> [1] "part 31 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> lapply -> FUN -> mcparallel -> sendMaster >> [1] "part 71 written successfully." >> [1] "part 35 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 42 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> [1] "part 4 written successfully." >> Execution halted >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1][1] "part 30 written successfully." "part 40 written successfully." >> >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> [1] "part 41 written successfully." >> [1] "part 80 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 45 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> [1] "part 82 written successfully." >> [1] "part 54 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 59 written successfully." >> Execution halted >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 27 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> [1] "part 36 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> [1] "part 44 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> Execution halted >> [1] "part 83 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 61 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 70 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 66 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 10 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> Execution halted >> [1] "part 53 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 55 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 15 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 50 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 5 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> Execution halted >> [1] "part 48 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 32 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 69 written successfully." >> Execution halted >> [1] "part 52 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 77 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> Execution halted >> [1] "part 24 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 56 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 6 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 62 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 33 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> [1] "part 16 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 34 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 74 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> [1] "part 2 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 19 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> [1] "part 9 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 11 written successfully." >> [1] "part 28 written successfully." >> [1] "part 29 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 17 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 65 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> Execution halted >> Execution halted >> [1] "part 68 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 43 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 38 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 1 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 57 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 8 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1] "part 25 written successfully." >> Error in sendMaster(try(eval(expr, env), silent = TRUE)) : >> write error, closing pipe to the master >> Calls: lapply ... <Anonymous> -> <Anonymous> -> mcparallel -> sendMaster >> [1]List of 85 >> $ : logi TRUE >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> $ : NULL >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel >> > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel