search for: clusterevalq

Dear R-users, we are using library(snow) for computation on a linux cluster with RMPI. We have a problem with clusterEvalQ: after launching clusterEvalQ it seems loading the required library on each node but if we type a function belonging to the loaded package R doesn't find it. > library(snow) # making cluster with 3 nodes > cl <- makeCluster(3, type = "MPI") Loading required package: Rmpi...

...ll writes and thus setting TCP_NODELAY causes many small >> packets to be sent, it might make more sense to set TCP_QUICKACK >> instead. >> I?aki >>> Unit: microseconds >>> expr min lq mean median uq max >>> clusterEvalQ(cl, iris) 1449.997 43991.99 43975.21 43997.1 44001.91 48027.83 >>> neval >>> 1000 >>> exactly the same machine + R but with TCP_NODELAY enabled in R_SockConnect(): >>> Unit: microseconds >>> expr min lq mean median...

...there a way to avoid the apparent additional Linux overhead? I attempted to isolate the behavior with a test that simply returns an existing object from the worker back to the main R session. library(parallel) library(microbenchmark) gcinfo(TRUE) cl <- makeCluster(1) (x <- microbenchmark(clusterEvalQ(cl, iris), times = 1000, unit = "us")) plot(x$time, ylab = "microseconds") head(x$time, n = 10) On Windows/MacOS, the test runs in 300-500 microseconds depending on hardware. A few of the 1000 runs are an order of magnitude slower but this can probably be attributed to garbag...

...erated with and without it? If there are many small writes and thus setting TCP_NODELAY causes many small packets to be sent, it might make more sense to set TCP_QUICKACK instead. I?aki > Unit: microseconds > expr min lq mean median uq max > clusterEvalQ(cl, iris) 1449.997 43991.99 43975.21 43997.1 44001.91 48027.83 > neval > 1000 > > exactly the same machine + R but with TCP_NODELAY enabled in R_SockConnect(): > > Unit: microseconds > expr min lq mean median uq max neval > clust...

Hi: I try to construct a Document-Term Meatrix from a corpus. The commands I used are: > library(parallel)> library(tm)> library(RWeka)> library(topicmodels)> library(RTextTools)> cl=makeCluster(detectCores())> invisible(clusterEvalQ(cl, library(tm)))> invisible(clusterEvalQ(cl, library(RWeka))) > invisible(clusterEvalQ(cl, library(topicmodels)))> invisible(clusterEvalQ(cl, library(RTextTools)))> myCorpus <-Corpus(DirSource("/home/neeph/Test/DMOZ_Business"), encoding="UTF-8", readerControl=lis...

...e the problem. See the example below ``` ## Create a temporary file filePath <- tempfile() con <- file(filePath, "wrb") writeBin(rep(0.0,10),con) close(con) library(simplemmap) library(parallel) cl <- makeCluster(1) x <- mmap(filePath, "double") ## Turn gctorture on clusterEvalQ(cl, gctorture()) clusterExport(cl, "x") ## x is an 0-length vector on the worker clusterEvalQ(cl, x) stopCluster(cl) ``` you can find more info on the problem if you manually build a connection between two R processes and export the ALTREP object. See output below ``` > con <- sock...

Please, check a tcpdump session on localhost while running the following script: library(parallel) library(tictoc) cl <- makeCluster(1) Sys.sleep(1) for (i in 1:10) { tic() x <- clusterEvalQ(cl, iris) toc() } The initialization phase comprises 7 packets. Then, the 1-second sleep will help you see where the evaluation starts. Each clusterEvalQ generates 6 packets: 1. main -> worker PSH, ACK 1026 bytes 2. worker -> main ACK 66 bytes 3. worker -> main PSH, ACK 3758 bytes 4. m...

...le() > con <- file(filePath, "wrb") > writeBin(rep(0.0,10),con) > close(con) > > library(simplemmap) > library(parallel) > cl <- makeCluster(1) > x <- mmap(filePath, "double") > ## Turn gctorture on > clusterEvalQ(cl, gctorture()) > clusterExport(cl, "x") > ## x is an 0-length vector on the worker > clusterEvalQ(cl, x) > stopCluster(cl) > ``` > > you can find more info on the problem if you manually build a connection > between two R processes...

It looks like R sockets on Linux could do with TCP_NODELAY -- without (status quo): Unit: microseconds expr min lq mean median uq max clusterEvalQ(cl, iris) 1449.997 43991.99 43975.21 43997.1 44001.91 48027.83 neval 1000 exactly the same machine + R but with TCP_NODELAY enabled in R_SockConnect(): Unit: microseconds expr min lq mean median uq max neval clusterEvalQ(cl, iris) 156.125 166.41 180.8...

...l writes and thus setting TCP_NODELAY causes many small > packets to be sent, it might make more sense to set TCP_QUICKACK > instead. > > I?aki > >> Unit: microseconds >> expr min lq mean median uq >> max >> clusterEvalQ(cl, iris) 1449.997 43991.99 43975.21 43997.1 44001.91 >> 48027.83 >> neval >> 1000 >> >> exactly the same machine + R but with TCP_NODELAY enabled in >> R_SockConnect(): >> >> Unit: microseconds >> expr min...

...correct result, but did not speed things up. Not sure why... I also tried snowfall and snow. While I can create a cluster and run simple processes (e.g., provided example from literature), I can't get the bootstrap to run. For example, using snow: cl <- makeCluster(8) clusterSetupRNG(cl) clusterEvalQ(cl,library(boot)) clusterEvalQ(cl,library(lme4)) boot.out<-clusterCall(cl,boot(dat.res,boot.fun, R = nboot)) stopCluster() returns the following error: Error in checkForRemoteErrors(lapply(cl, recvResult)) : 8 nodes produced errors; first error: could not find function "fun" I am...

...ed from or put into? Second, how closely does the evaluation environment in the browser/debugger match what you would get in the function at the same point? I ask because if I evaluate a statement in the browser it seems to work one way, but if I execute it it works another way. The statement is clusterEvalQ(cl, crossval.setup(x, y, groups, theta.fit, theta.predict)) This evaluates the crossval.setup function across the cluster cl. crossval.setup is a function which puts its arguments in a list g (a local variable) and then does gcv <<- g. The intent is to stuff the data into a global variable...

...emporary file > filePath <- tempfile() > con <- file(filePath, "wrb") > writeBin(rep(0.0,10),con) > close(con) > > library(simplemmap) > library(parallel) > cl <- makeCluster(1) > x <- mmap(filePath, "double") > ## Turn gctorture on > clusterEvalQ(cl, gctorture()) > clusterExport(cl, "x") > ## x is an 0-length vector on the worker > clusterEvalQ(cl, x) > stopCluster(cl) > ``` > > you can find more info on the problem if you manually build a connection > between two R processes and export the ALTREP object. S...

Dear group, does anybody know how to export libraries/functions to all nodes when launching snow? I want to use a function from fBasics (dstable) for a rGenoud optimization routine, but I fail "making the function accessible" to the nodes created. I know how it works for variables, I know how it works in snowfall(which cant be used in that case), but I dont know how it culd work in

...correct result, but did not speed things up. Not sure why... I also tried snowfall and snow. While I can create a cluster and run simple processes (e.g., provided example from literature), I can't get the bootstrap to run. For example, using snow: cl <- makeCluster(8) clusterSetupRNG(cl) clusterEvalQ(cl,library(boot)) clusterEvalQ(cl,library(lme4)) m2.ph.rlm.boot<-clusterCall(cl,boot(m2.ph,m2.ph.fun, R = nboot)) stopCluster() returns the following error: Error in checkForRemoteErrors(lapply(cl, recvResult)) : 8 nodes produced errors; first error: could not find function "fun"...

...node is still running or not by its PID, e.g. it may have crashed / core dumped. I'm ok with getting false-positive results due to *another* process with the same PID has since started. I can the PID of each cluster nodes by querying them for their Sys.getpid(), e.g. pids <- parallel::clusterEvalQ(cl, Sys.getpid()) Is there a function in core R for testing whether a process with a given PID exists or not? From trial'n'error, I found that on Linux: pid_exists <- function(pid) as.logical(tools::pskill(pid, signal = 0L)) returns TRUE for existing processes and FALSE otherwise, b...

...h, "wrb") > > writeBin(rep(0.0,10),con) > > close(con) > > > > library(simplemmap) > > library(parallel) > > cl <- makeCluster(1) > > x <- mmap(filePath, "double") > > ## Turn gctorture on > > clusterEvalQ(cl, gctorture()) > > clusterExport(cl, "x") > > ## x is an 0-length vector on the worker > > clusterEvalQ(cl, x) > > stopCluster(cl) > > ``` > > > > you can find more info on the problem if you manually build a connection >...

I am trying to run a monte carlo process using snow with a MPI cluster. I have ~thirty processors to run the algorithm on and I want to run it 5000 times and take the average of the output. A very simple way to do this is to divide 5000 by the number of processors to get a number n and tell each processor to run the algorithm n times. I realize there are more efficient ways to manage the

...d,])) svm.lin <- svm(hogTrain$X,hogTrain$Y, kernel="linear",cost=c[i], cross=5) results.lin <- predict(svm.lin, hogTest$X) e.test.lin <- sqrt(sum((results.lin-hogTest$Y)^2)/length(hogTest$Y)) return(e.test.lin) } } cl<- makeCluster(10, type="SOCK" ) clusterEvalQ(cl,library(e1071)) clusterExport(cl,c("data.X","data.Y","NR","cost1")) RMSEP<-clusterApplyLB(cl,cost1,sv.lin) stopCluster(cl) -- View this message in context: http://old.nabble.com/SVM-Param-Tuning-with-using-SNOW-package-tp26399401p26399401.html Se...

...quot;\"", comment = "", trim_ws = TRUE, skip = 0, n_max = Inf, guess_max = min(1000, n_max), progress = FALSE)) } # C.2) parallel Package: Environment Settings no_cores <- detectCores() c1 <- makeCluster(no_cores) invisible(clusterEvalQ(c1, library(readr))) setDefaultCluster(c1) # C.3) parRapply Function Application: DISPOIN_CSV_List <- parRapply(c1, DISPOIN_DIR_REL, parRaplly_Function) suppressWarnings(stopCluster(c1)) # D) List's Tibbles Compilation into a single Tibble: DISPOIN_CSV <- do.call(rbind, DI...