search for: clusterexport

We got some errors and eventually figured out that parallel::clusterExport second argument is "varlist" while in snow::clusterExport it is "list". The user had loaded parallel first, but did something else which inadvertently loaded snow, then clusterExport failed because we had "varlist" and not "list". Are these different on pur...

I am trying to do parallel programming and I tried this library(doSNOW) library(foreach) testfunc<-function(x){ x<-x+1 x } noc<-2 cl <- makeCluster(do.call(rbind,rep(list("localhost"),noc)), type = "SOCK") registerDoSNOW(cl) clusterExport(cl=cl,c("testfunc.r")) testl<-foreach(pp=1:2) %dopar% { testfunc(pp) } And this works but if I try to enclose my commands inside a text file to be sourced it doesn't work noc<-2 testfunc<-function(x){ x<-x+1 x } cl <- makeCluster(do.call(rbind,rep(list("loca...

...mn matrix had to be shipped along with the item in the list and the traffic time was actually longer than the computing time. However, if I export the 500,000 object first across the spawned processes as in this mock script cl <- makeCluster(nnodes,method) mArrayData <- getData(experiments) clusterExport(cl, 'mArrayData') Results <- parLapply(cl, theMapList, function(x) t.testFnc(x)) With a function that define the mArrayData argument as a default parameter as in t.testFnc <- function(probeList, array=mArrayData){ x <- array[probeList$A,] y <- array[probeList$B,]...

...below ``` ## Create a temporary file filePath <- tempfile() con <- file(filePath, "wrb") writeBin(rep(0.0,10),con) close(con) library(simplemmap) library(parallel) cl <- makeCluster(1) x <- mmap(filePath, "double") ## Turn gctorture on clusterEvalQ(cl, gctorture()) clusterExport(cl, "x") ## x is an 0-length vector on the worker clusterEvalQ(cl, x) stopCluster(cl) ``` you can find more info on the problem if you manually build a connection between two R processes and export the ALTREP object. See output below ``` > con <- socketConnection(port = 1234,server...

Hi I try parallelising some code using the snow package and the following lines: cl <- makeSOCKcluster(8) pfunc <- function (x) (if(x <= (-th)) 1 else 0) ###correlation coefficient clusterExport(cl,c("pfunc","th")) cor.c.f <- parApply(cl,tms,c(1,2),FUN=pfunc) The parApply results in the error message: > cor.c.f <- parApply(cl,tms,c(1,2),FUN=pfunc) Error in do.call("fun", lapply(args, enquote)) : could not find function "fun". Any ideas?...

...#dbConn : active db connection formed via odbcDriverConnect #name : a char string sqlQuery(dbConn, paste("select id from table where name='", name, "'", sep="")) } mc<-detectCores() cl<-makeCluster(mc) clusterExport(cl, c("sqlQuery", "dbConn")) parSapply(cl, names, nameToID, dbConn=dbConn) #incorrect passing of nameToID's second argument ### As in the comment, this is not the correct way to assign the second argument to nameToID. I have also tried the following: parSapply(cl, name...

...that I changed my previous R script so I could use the function lapply().This script apparently worked fine. Finally, I tried to parallelize the problem by using snow and parLapply(). The resulting script looks like this: ## Not run # library(snow)cl <- makeCluster(3, type = "SOCK") clusterExport(cl,list('param.esp','copy.files','for12.template','program.executor')) parLapply(cl,a.list,a.function))stopCluster(cl) # ##End not run Although it runs, the parallelized version is messing up the input parameters to pass to the executable (see table below, where par...

..."wrb") > writeBin(rep(0.0,10),con) > close(con) > > library(simplemmap) > library(parallel) > cl <- makeCluster(1) > x <- mmap(filePath, "double") > ## Turn gctorture on > clusterEvalQ(cl, gctorture()) > clusterExport(cl, "x") > ## x is an 0-length vector on the worker > clusterEvalQ(cl, x) > stopCluster(cl) > ``` > > you can find more info on the problem if you manually build a connection > between two R processes and export the ALTREP object. See outpu...

If the code you are running in parallel is complicated, maybe foreach is not sophisticated enough to find all the variables you refer to. Maybe use parallel::clusterExport yourself? But be a aware that passing parameters is much safer than directly accessing globals in parallel processing, so this might just be your warning to not do that anyway. -- Sent from my phone. Please excuse my brevity. On March 9, 2018 7:50:44 AM PST, Nik Tuzov <ntuzov at beacon.partek...

...gTrain$X,hogTrain$Y, kernel="linear",cost=c[i], cross=5) results.lin <- predict(svm.lin, hogTest$X) e.test.lin <- sqrt(sum((results.lin-hogTest$Y)^2)/length(hogTest$Y)) return(e.test.lin) } } cl<- makeCluster(10, type="SOCK" ) clusterEvalQ(cl,library(e1071)) clusterExport(cl,c("data.X","data.Y","NR","cost1")) RMSEP<-clusterApplyLB(cl,cost1,sv.lin) stopCluster(cl) -- View this message in context: http://old.nabble.com/SVM-Param-Tuning-with-using-SNOW-package-tp26399401p26399401.html Sent from the R help mailing list ar...

...t; In my simulation I need to run a function several times with an different variable each time. All the invocations on the functions are independent of the others. I start the simulation on one node, create a cluster of several nodes, load my custom package and snow on all of them, use clusterExport(cl, "x1") to export the variable x1(among other variable I need), then I call my simulation on the cluster using clusterApplyLB(cl, 2:S, simClust) where cl is the cluster and S is a constant defined above as 500. Using print statements (since snow, or R for that matter, has next...

Hello all: Please help me with this "can't find object" issue. I'm trying to get leave-one-out predicted values for Beta-binomial regression. It may be the gamlss issue because the code seems to work when %do% is used. I have searched for similar issues, but haven't managed to figure it out. This is on Windows 10 platform. Thanks in advance, Nik #

...ework (RPF) to deploy and use distributed computational resources (non R based, native libraries with JNI support or java code) - Use R for parallel computing via a Java API or Web Services. - Create and use Remote R Instances from within R (snow' like fucntions : makeCluster, clusterEvalQ, clusterExport, clusterApply, stopCluster ..) - Use the R API from within an applet (book, use and release a Remote R Instance via HTTP Tunneling) - Use R to generate Graphics on the fly for thin web clients - Use R from within a Workbench that includes an advanced script editor, a Spreadsheet View fully conn...

...empfile() > con <- file(filePath, "wrb") > writeBin(rep(0.0,10),con) > close(con) > > library(simplemmap) > library(parallel) > cl <- makeCluster(1) > x <- mmap(filePath, "double") > ## Turn gctorture on > clusterEvalQ(cl, gctorture()) > clusterExport(cl, "x") > ## x is an 0-length vector on the worker > clusterEvalQ(cl, x) > stopCluster(cl) > ``` > > you can find more info on the problem if you manually build a connection > between two R processes and export the ALTREP object. See output below > ``` >> co...

...ry(doSNOW) > library(panel) > > cl<-makeMPIcluster(3) > registerDoSNOW(cl) > > clusterEvalQ(cl,library(panel)) > > res<-clusterCall(cl, function(){Sys.info()["nodename"]}) > print(do.call(rbind,res)) > > sme<- matrix(rnorm(100),10,10) > clusterExport(cl,"sme") > > myfun<-function() > { > for(i in 1:1000) > { > x<-eddcmp(sme) > } > } > > ged<-0 > > system.time({ > ged<-foreach(i=1:10) %dopar% > { > myfun() > } > }) > > system.time({ > ged<-foreac...

I'm parallelizing some computation on hierarchical data, and would find it natural to do something like this (where a call to parLapply is embedded in outer call to parLapply): cl <- makeCluster(rep.int('localhost', 5), type='SOCK') clusterExport(cl, 'cl') parLapply(cl, 1:5, function(i) parLapply(cl, 1:5, function(j) i * j)) Snow executes without error, but produces NULL; is embedded parallelization, in principle, possible?

...in the cluster. Normally I allocate 20000M and operate on 5 X 20000. Even this is taking lot of time. Is there any way I can find cl = cor(cdata) utilizing the computers in the clusters(I am using 32 nodes ). I am using the following code to get the cluster.. library(snow) cl <- makeCluster(64) clusterExport(cl, "fakeData") clusterEvalQ(cl, library(boot)) system.time(out2 <- clusterApplyLB(cl, pair, geneCor)) stopCluster(cl) But here the geneCor and pair is calculated as, pair <- combn(1:nrow(fakeData), 2, simplify = F) geneCor <- function(x, gene = fakeData) { cor(t(gene[x[1], ]),...

...m, m, m) # write wrapper function that computes results fun <- function(mat) { s <- summary(mat ~ edges + dsp(1)) return(s) } cl <- makePSOCKcluster(2) # create cluster object test1 <- sapply(l, fun) # works! test2 <- parSapply(cl, l, fun) # problem: results look weird! clusterExport(cl, varlist = "summary.statistics") test3 <- parSapply(cl, l, fun) # error! ########## I found a solution somewhere in a forum: clusterCall(cl, function() library("ergm")) However, as I'd like to use the code in a package, the CRAN check tells me that using library...

Hi, I need to understand when is it best to use /rmpi/ and when is it best to use /snow/ for parallel programming in R? I understand snow can be used for a group of non-clustered work stations also. But I wish to understand from the point of view of using both on clusters for a problem which has few chunks of straightforward data-parallelism interleaved with some communication. Since both are

...ramework (RPF) to deploy and use distributed computational resources (non R based, native libraries with JNI support or java code) - Use R for parallel computing via a Java API or Web Services. - Create and use Remote R Instances from within R (snow' like fucntions : makeCluster, clusterEvalQ, clusterExport, clusterApply, stopCluster ..) - Use the R API from within an applet (book, use and release a Remote R Instance via HTTP Tunneling) - Use R to generate Graphics on the fly for thin web clients - Use R from within a Workbench that includes an advanced script editor, a Spreadsheet View fully connec...