similar to: R intermittently crashes across cluster

Dear R-Devel List, **TL;DR:** The function **parLapplyLB** of the parallel package has [reportedly][1] (see also attached RRD output) not been doing its job, i.e. not actually balancing the load. My colleague Dirk Sarpe and I found the cause of the problem and we also have a patch to fix it (attached). A similar fix has also been provided [here][2]. [1]:

Looking at parLapplyLB, one sees that it takes in X and then passes splitList(X, length(cl)) to clusterApplyLB, which then calls dynamicClusterApply. Thus while dynamicClusterApply does handle tasks in a load-balancing fashion, sending out individual tasks as previous tasks complete, parLapplyLB preempts that by splitting up the tasks in advance into as many groups of tasks as there are cluster

Hi, I'm trying to understand the rationale for your proposed amount of splitting and more precisely why that one is THE one. If I put labels on your example numbers in one of your previous post: nbrOfElements <- 97 nbrOfWorkers <- 5 With these, there are two extremes in how you can split up the processing in chunks such that all workers are utilized: (A) Each worker, called

Dear Christian and Henrik, thank you for spotting the problem and suggestions for a fix. We'll probably add a chunk.size argument to parLapplyLB and parLapply to follow OpenMP terminology, which has already been an inspiration for the present code (parLapply already implements static scheduling via internal function staticClusterApply, yet with a fixed chunk size; parLapplyLB already

Hi all, I am trying to use snow package to do a parallel MCMC. I have read a few guides and articles, the following is that I came up with. When I run it I got the error message: Error in checkForRemoteErrors(lapply(cl, recvResult)) : 4 nodes produced errors; first error: could not find function "ui.Next" The data is a longitudinal data with few repeated readings on a number of

Hello, Is the first time I am using SNOW package and I am trying to tune the cost parameter for a linear SVM, where the cost (variable cost1) takes 10 values between 0.5 and 30. I have a large dataset and a pc which is not very powerful, so I need to tune the parameters using both CPUs of the pc. Somehow I cannot manage to do it. It seems that both CPUs are fitting the model for the same values

Here is the code, assuming 8 cores in the cpu. library('modeest') library('snow') cl = makeCluster(rep('localhost', 8), 'SOCK') x = vector(length=50) x = sapply(x, function(i) i=sample(c(1,0), 1)) pastK = function(n, x, k) { if (n>k) { return(x[(n-k):(n-1)]) } else {return(NA)} } predR = function(x, k) { pastList = lapply(1:length(x), function(n)

Hi, The following function use to work, but now it doesn't giving the error "> CallSnow(, 100) Using snow package, asking for 2 nodes 2 slaves are spawned successfully. 0 failed. Error in checkForRemoteErrors(val) : 2 nodes produced errors; first error: no applicable method for 'lapply' applied to an object of class "list" ". Where this is the

Dear R-Devel List, I have installed R 3.4.3 with the patch applied on our cluster and ran a *real-world* job of one of our users to confirm that the patch works to my satisfaction. Here are the results. The original was a series of jobs, all essentially doing the same stuff using bootstrapped data, so for the original there is more data and I show the arithmetic mean with standard deviation. The

Hello all, I am re-posting my previous question with a simpler, more transparent, commented code. I have been ramming my head against this problem, and I wondered if anyone could lend a hand. I want to make parallel a bootstrap of a linear mixed model on my 8-core mac. Below is the process that I want to make parallel (namely, the boot.out<-boot(dat.res,boot.fun, R = nboot) command).

I have this function and want to run it parallel with different sets of data. Using SNOW and clusterApplyLB. system.time(out <- mclapply(cData, plotGraph)) #each cData contains 100X6000 doubles system.time(out <- mclapply(cData2, plotGraph)) system.time(out <- mclapply(cData3, plotGraph)) system.time(out <- mclapply(cData4, plotGraph)) system.time(out <- mclapply(cData5,

Dear Henrik, The rationale is just that it is within these extremes and that it is really simple to calculate, without making any assumptions and knowing that it won't be perfect. The extremes A and B you are mentioning are special cases based on assumptions. Case A is based on the assumption that the function has a long runtime or varying runtime, then you are likely to get the best load

Dear Tomas, Thanks for your commitment to fix this issue and also to add the chunk size as an argument. If you want our input, let us know ;) Best Regards On 02/26/2018 04:01 PM, Tomas Kalibera wrote: > Dear Christian and Henrik, > > thank you for spotting the problem and suggestions for a fix. We'll probably add a chunk.size argument to parLapplyLB and parLapply to follow OpenMP

Hi, I am currently building an R package and I am facing a peculiar problem where some of the functions does not work within the package. However, if I source the script the function works. For example, in a method for parallelization of analysis on each chromosome simultaneously I am receiving error at the following position of the code: # this profile the information chromosome wise and

Hello, I have a program that used to run well in October, it uses library snow. Since then, one change has ocurred (snow library has been updated) and another could have ocurred (I've unadvertently modified something). Anyway, now when I make the call: parallel.model.results <- clusterApply(cl,processors.struct,MCexe) exactly as I used to do, where MCexe is my function and

I'm working with snow and created a local cluster. So far, the same code has always worked (please see below). However, now I receive a message that the connection with the nodes cannot be opened. I restarted my workstation but that didn't help. Is there a known solution for this problem? Thanks a lot for any help. bram foubert library(snow) cl =

Hi R-Core, i am using nnet and DEoptim, Xcc=matrix(rnorm(100,0.5,0.08),50,2) Ycr=matrix(rnorm(50,0.2,0.05),50,1) pred_regm1 <- function(A) { A1=A[1] A2=A[2] A3=A[3] regm1 <- nnet(Xcc,Ycr,entropy=T,size=A1,decay=A2,maxit=2000,trace=F,Hess=T,rang=A3,skip=T) dif=sum((predict(regm1,Xcc)-Ycr)^2) return(dif) } somar=DEoptim(pred_regm1,c(1,0.00001,0.01), c(25,0.999,0.95),

I'm trying to use snow in my dual-core (hopefully later this is going to run in a cluster). So, at this moment I create a cluster using SOCK connection (MPI in the future). However when I try to use clusterApplyLB I got "Error in socketSelect(socklist) : not a socket connection". Any ideas ? Do you know if that is going to be an isuue too when I swith from SOCK to MPI ? Sample code

Dear All, I have a question of how to export S4 class specification to clusters/workers in parallel computing. The package I used is snowfall. The problem is reproducible as follows. Any hint is greatly appreciated. Edwin Sun === begin =========== library(snowfall) sfInit(parallel = TRUE, cpus = 2) setClass("catt", representation(aa = "numeric")) setClass("dogg",

Dear R Users, I am attempting to use snow with rmpi. My configuration is: - R Version 2.7.2, rmpi - RMPI 0.5-6 - DeinoMPI 1.1.0 - Windows Xp SP2 I can sucessfully create a cluster and execute simple commands but for some reason, I cannot change the working directory in my nodes. > noclusters<-2 > cl <- makeCluster(noclusters, type = "MPI") 2 slaves are spawned