R help - Oct 2020 - exiting mclapply early on error

Hey folks,

Is there any way to exit an mclapply early on error?

For example, in the following mclapply loop, I have to wait for all the
processes to finish before the error is returned.

```
mclapply(X = 1:12, FUN = function(x) {Sys.sleep(0.1); if(x == 4) stop()},
mc.cores = 4, mc.preschedule = F)
```

When there are many calculations in FUN, it takes a long time before the error
is returned. It would be nice if there were an option to exit (all child
processes) early on error. Is there any way to do this?

Thanks,
Giovanni Righi

Hello Giovanni,

I don't know if my workflow would suit you but I tend to want the
opposite when I launch a parallel process. I tend to want to keep the
processes alive as long as they can. If the computation time is long I
would not want to lose everything.

lapply..8 <- function(X,FUN,...){
    max..cores <- as.numeric(system("grep ^processor /proc/cpuinfo
2>/dev/null | wc -l",TRUE))
    mclapply(X,FUN,mc.cores=min(max..cores,8))
}

lapply_with_error..8 <- function(X,FUN,...) {    
    lapply..8(X, function(x, ...) tryCatch(FUN(x, ...),
                                            error=function(e){
                                       print(e)
                                       e}
                                       ))
}

res <- lapply_with_error..8(X = 1:12, FUN = function(x) {Sys.sleep(0.1); if(x
== 4) stop()})

Then we can investigate the problem for the element that generated
errors. It is even better if we could anticipate the errors and avoid
surprises  by well _testing_ the function before launching a long
process. 

If you want the processes to fail fast, I fear that you want to launch the
parallel process too soon without
having tested your function enough.  

HTH,

Jeremie

Thanks for the response, Jeremie. I wholeheartedly agree about testing.

In my case, this feature would be used purely to reduce computation time. I?m
calculating
an expensive (and embarrassingly parallel) likelihood function, and for some
parameter combinations
my objective function diverges to infinity. In those instances, I?d like to
throw an error that stops
subsequent computations and that I could catch to return a value of infinity. I
agree, though,
that the default should allow all processes to finish.

What I?ve suggested does not seem possible with mclapply(). Do you know how I
could suggest
this as a feature request to the maintainer of the parallel package?

Thanks,
Giovanni