similar to: Error: serialization is too large to store in a raw vector

Hi, I am experiencing a problem related to serialization behavior in 2.15.0 beta (binary installed from Debian unstable) and 2.16.0 (from svn) that is not present in 2.14.2 (binary from Debian testing). I don't fully understand the problem. Also, I tried but have not yet been able to create a small, self-contained example that reproduces the problem. However, I do have a large, not

I have an lapply function call that I want to parallelize. Below is a very simplified version of the code: url_base <- "https://cloud.r-project.org/src/contrib/" files <- c("A3_1.0.0.tar.gz", "ABC.RAP_0.9.0.tar.gz") res <- parallel::mclapply(files, function(s) download.file(paste0(url_base, s), s)) Instead of download a couple of files in parallel, I get a

Hi Andreas, the error is reported when some child process cannot send results to the master process, which originates from an error returned by write() - when write() returns -1 or 0. The logic around the writing has not changed since R 3.5.2. It should not be related to the printing in the child, only to returning the value. The problem may be originating from the execution environment,

Hi, I am facing a very weird problem with parallel::mclapply. I have a script which does some data wrangling on an input dataset in parallel and then writes the results to disk. I have been using this script daily for more than one year always on an EC2 instance launched from the same AMI (no updates installed after launch) and processed thousands of different input data sets successfully. I now

Hi Andreas, thank you very much, good job finding it was EBADF. Now the question is why the pipe has been closed prematurely; it could be accidentally by R (a race condition in the cleanup code in fork.c) or possibly by some other code running in the same process (maybe the R program itself or some other code it runs). Maybe we can take this off the list and come back when we know the cause

Dear R developers: ... 7: lapply(seq_len(cores), inner.do) 8: FUN(1:3[[3]], ...) 9: sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE)) Selection: .....................Error in sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE)) : long vectors not supported yet: memory.c:3100 admittedly, my outcome will be a very big list, with 30,000 elements, each containing data frames

Dear all, While I am executing my code I receive the error below Error in sort.int(x, na.last = na.last, decreasing = decreasing, ...) :   'x' must be atomic the weird thing that I am not calling anywhere sort function nor do I rely on anyh sorting. How I can discover where this comes from (inside which function?). I would like to thank you in advance for your help B.R Alex

Hello everyone, I'm trying to parallelize an R script I have written. To do this, I am first trying to use the multicore package, because I've had some previous success with that. The function I'm trying to parallelize is illumqc. I'd like to create a separate process for each of 8 files, contained in the vector "files". Below is my code: for(i in

Dear all I am using lapply (actually mclapply that share the same syntax). I want to call the same function that takes as input a vector. My initial data structure is a matrix that I want to cut it to multiple vectors (one vector for every row of the matrix) and then feed that to the function by using mclapply. Could you please help me converting the matrices to nrow times vectors. I would

Dear all I have wrote the following line return(as.vector(lapply(as.data.frame(data),min,simplify=TRUE))); I want the lapply to return a vector as it returns a list with elements as shown below List of 30001 $ V1 : num -131 $ V2 : num -131 $ V3 : num -137 $ V4 : num -129 $ V5 : num -130 as you can see I have already tried the simplify=TRUE and also the as.vector() but both

Dear all I would like to ask you if an assignment can be done inside a lapply statement. For example I would like to covert a double nested for loop for (i in c(1:dimx)){ for (j in c(1:dimy)){ Powermap[i,j] <- Pr(c(i,j),c(PRX,PRY),f) } } to something like that: ij<-expand.grid(i=seq(1:dimx),j=(1:dimy)) unlist(lapply(1:nrow(ij),function(rowId) { return

Dear all. I am using the mclapply function to split my code to the many cores my system has. It seems that is working fine. This is the parallel version of lcapply. The only problem that I seem to have is that the printf cannot print messages. The ideal to me is to have fro my function an output of the form Shadowlist<-mclapply(1:dimz, function(i) { print(sprintf('Creating the

Dear all, I would like to ask your help understand the subsequent steps for making my program faster. The following code: Gauslist<-array(data=NA,dim=c(dimx,dimy,dimz)) for (i in c(1:dimz)){ print(sprintf('Creating the %d map',i)); Gauslist[,,i]<-f <- GaussRF(x=x, y=y, model=model, grid=TRUE,param=c(mean,variance,nugget,scale,Whit.alpha)) } creates 100 GaussMaps (each

Dear all, I was reading last night the lm and the Formula manual page, and 'I have to admit that I had tough time to understand their syntax. Is there a simpler guide for the dummies like me to start with? I would like to thank you in advance for your help Regards Alex [[alternative HTML version deleted]]

Dear all, I have a function that recognizes the following format for timestamps "%Y-%m-%d %H:%M:%S" my function takes two input arguments the TimeStart and TimeEnd I would like to help me create the right list with pairs of TimeStart and TimeEnd which I can feed to lapply (I am using mclapply actually).  For every lapply I want two inputs to be fed to my function. I only know how to

Hi, I've been doing some tests using serialize() to a raw vector: df <- data.frame(runif(50e6,1,10)) ser <- serialize(df,NULL) In this example the data frame and the serialized raw vector occupy ~400MB each, for a total of ~800M. However the memory peak during serialize() is ~1.2GB: $ cat /proc/15155/status |grep Vm ... VmHWM: 1207792 kB VmRSS: 817272 kB We work with very

Hello, R version 2.4.1 (2006-12-18) i386-pc-mingw32 Calling serialize() with a NULL connection serializes it to a raw vector. However, when the object to be serialized is large, it takes a very long time: > system.time( serialize(matrix(0, 1000, 1000), NULL) ) [1] 38.25 40.73 81.54 NA NA > system.time( serialize(matrix(0, 2000, 2000), NULL) ) [1] 609.72 664.75 1318.57 NA

Dear all, I am using my code the vgram.matrix of packets fields. I have around 500 matrices that I need to pass inside that function and then plot those results. Even though my system has 16 cores is quite clear that I am only using one of those. Would it be able to skip these 500 "tasks" to the 16 cores, with each processor having around 4 matrices to process? What would you suggest

Presumably one could stream over the data twice, the first to get the size, without storing the data. Slower but more memory efficient, unless I'm missing something. Michael On Tue, Mar 17, 2015 at 2:03 PM, Simon Urbanek <simon.urbanek at r-project.org> wrote: > Jorge, > > what you propose is not possible because the size of the output is > unknown, that's why a

Dear all, I have two questions regarding packaging. a. What is the inverse action of require(). I want to "detach" one of the two libraries I loaded as there are two functions with the same name and come from two different libraries b. If I do not "detach" the one library how I can refer to the one function and to the second one? B.R Alex [[alternative HTML version