search for: parrapply

Dear R Users, I have developed the following code for importing a series of zipped CSV by parallel computing. My problems are that: A) Some ZIP Files (Which contain CSVs inside) are corrupted, and cannot be opened. B) After executing parRapply I can only see the last.warning variable error, for knowing which CSV have failed in each node, but I cannot see all warnings, only 1 at a time. So: * For showing a list of all warnings in all nodes, I was thinking of using the following function in the code: warnings(DISPOIN_CSV_List <-...

According to ?parCapply: parRapply and parCapply always return a vector. This appears not to be the case in the following minimal reproducible example: > library(parallel) > nslaves <- 2 > cl <- makeCluster(nslaves) > X <- matrix(2,nrow=3,ncol=4) > X <- rbind(c(1,1,0,...

...nction (cl = NULL, fun, ..., MoreArgs = NULL, RECYCLE = TRUE, SIMPLIFY = FALSE, USE.NAMES = TRUE, .scheduling = c("static", "dynamic")) NULL > args(snow::clusterMap) function (cl, fun, ..., MoreArgs = NULL, RECYCLE = TRUE) NULL Same problem with parRapply and parCapply and the naming of the x/X argument: > args(apply) function (X, MARGIN, FUN, ...) NULL > args(parApply) function (cl = NULL, X, MARGIN, FUN, ...) NULL > args(parRapply) function (cl = NULL, x, FUN, ...) NULL > args(parCapply) function (cl =...

...tried to convert my data frame in a data table but it didn't improve things (probably read/write of this DT will be much faster) - the only way I found was to remove 1/3 of the data frame with the strings of lowest frequency which speed up the process by a factor x10 ! - didn't try yet parRapply and with a machine with multicore I can get another factor. I did use parLapply for some other code but I had many issue with memory (crashing my Mac). I had to sub-divide the dataset to have it working correctly but I didn't manage to fully understand the issue. I am sure their is...

...tried to convert my data frame in a data table but it didn't improve things (probably read/write of this DT will be much faster) - the only way I found was to remove 1/3 of the data frame with the strings of lowest frequency which speed up the process by a factor x10 ! - didn't try yet parRapply and with a machine with multicore I can get another factor. I did use parLapply for some other code but I had many issue with memory (crashing my Mac). I had to sub-divide the dataset to have it working correctly but I didn't manage to fully understand the issue. I am sure their is...

...frame in a data table but it didn't improve > things (probably read/write of this DT will be much faster) > > - the only way I found was to remove 1/3 of the data frame with the > strings of lowest frequency which speed up the process by a factor x10 ! > > - didn't try yet parRapply and with a machine with multicore I can get > another factor. > I did use parLapply for some other code but I had many issue with > memory (crashing my Mac). > I had to sub-divide the dataset to have it working correctly but I > didn't manage to fully understand the iss...