similar to: Multicore package: sharing/modifying variable accross processes

I am using the library(bigmemory) to handle large datasets, say 1 GB, and facing following problems. Any hints from anybody can be helpful. _Problem-1: _ I am using "read.big.matrix" function to create a filebacked big matrix of my data and get the following warning: > x = read.big.matrix("/home/utkarsh.s/data.csv",header=T,type="double",shared=T,backingfile

Hi, I am trying to run the bigmemory example provided on the http://www.bigmemory.org/ The example runs on the "airline data" and generates summary of the csv files:- library(bigmemory) library(biganalytics) x <- read.big.matrix("2005.csv", type="integer", header=TRUE, backingfile="airline.bin", descriptorfile="airline.desc",

Hi, With reference to the mail below, I have large datasets, coming from various different sources, which I can read into filebacked big.matrix using library bigmemory. I want to merge them all into one 'big.matrix' object. (Later, I want to run regression using library 'biglm'). I am unsuccessfully trying to do this from quite some time now. Can you please

I work with problems that have rather large data requirements -- typically a bunch of multigig arrays. Given how generous R is with using memory, the only way for me to work with R has been to use bigmatrices from bigmemory package. One thing that is missing a bit is interoperability of bigmatrices with C++ and possibly java. What i mean by that is API that would allow read and write filebacked

I recently learned about the bigmemory and foreach packages and am trying to use them to help me create a very large matrix. Without those packages, I can create the type of matrix that I want with 10 columns and 5e6 rows. I would like to be able to scale up to 5e9 rows, or more, if possible. I have created a simplified example of what I'm trying to do, below. The first part of the

My long-promised Rdsm package is now on CRAN. Some of you may recall that I made a prototype available on my own Web page last July. This is the official version, much evolved since I released the prototype. The CRAN description states: Provides a threads-like programming environment for R, usable both on a multicore machine and across a network of multiple machines. The package

Hi, in R 2.12.1, R CMD check hangs when building a vignette that uses a foreach loop with the doMC parallel backend. This does not happen in R 2.13.1, nor if I use doSEQ instead of doMC. All versions of multicore, doMC and foreach are the same on both my R installations. Has anybody encountered a similar issue? Thank you. Renaud ### UNIVERSITY OF CAPE TOWN This e-mail is subject to the

Dear friends, I have to use a very large matrix. Something of the sort of matrix(80000,80000,n) .... where n is something numeric of the sort 0.xxxxxx I have not found a way of doing it. I keep getting the error Error in matrix(nrow = 80000, ncol = 80000, 0.2) : too many elements specified Any suggestions? I have searched the mailing list, but to no avail. Best, -- Corrado Topi Global

Dear All, I would like to have an advice for designing an R library, and thought that R-devel may be the best place to ask given so many people who are highly expert in R are around. We are at an early stage of designing an R library, which is effectively an interface to a C++ library providing fast access to large matrices stored on HDD as binary files. The core of the C++ library is

Sir, I want to apply mclapply() function for my analysis. So, I have to install multicore package. But I can not install the package. >install.packages("multicore") It gives that package multicore is not available. Can you help me? Regards, Suman Dhara [[alternative HTML version deleted]]

I have pretty big data sizes, like matrices of .5 to 1.5GB so once i need to juggle several of them i am in need of disk cache. I am trying to use bigmemory package but getting problems that are hard to understand. I am getting seg faults and machine just hanging. I work by the way on Red Hat Linux, 64 bit R version 10. Simplest problem is just saving matrices. When i do something like

It is so difficult ... Which FE? It need BE support? I didn't get it. 2010/8/4 vijay kumar <vijaygbvv at gmail.com> > Yeah OpenMP support. I read that it has a front end support but not the > back end. So are there any projects or teams looking at this issue. > > On Wed, Aug 4, 2010 at 7:24 AM, Liu <proljc at gmail.com> wrote: > >> Multicore? >> You

Hi, I'm playing around with the 'bigmemory' package, and I have finally managed to create some really big matrices. However, only now I realize that there may not be functions made for what I want to do with the matrices... I would like to perform a cluster analysis based on a big.matrix. Googling around I have found indications that a certain kmeans.big.matrix() function should

Hi all, I am new to this LLVM. I went through the documenation of LLVM but I didn't find any support for Multicore. Is there any such possibility where multicore architecture can be exploited using LLVM. Thanks Vijay -------------- next part -------------- An HTML attachment was scrubbed... URL:

On Aug 3, 2010, at 8:48 PM, Liu wrote: > It is so difficult ... > Which FE? It need BE support? I didn't get it. > > 2010/8/4 vijay kumar <vijaygbvv at gmail.com> > Yeah OpenMP support. I read that it has a front end support but not the back end. So are there any projects or teams looking at this issue. > > On Wed, Aug 4, 2010 at 7:24 AM, Liu <proljc at

Hello All, In general when we use Rprof for performance evaluation on Multicore systems the output provides the time on the basis of the "user" time and the sampling time is equal to the the user time as reported by system.time. This does not seem right behavior when R is linked to BLAS/Lapack or other libraries which are optimized for parallel or multicore architectures as

Hello All, In general when we use Rprof for performance evaluation on Multicore systems the output provides the time on the basis of the "user" time and the sampling time is equal to the the user time as reported by system.time. This does not seem right behavior when R is linked to BLAS/Lapack or other libraries which are optimized for parallel or multicore architectures as

Dear all, bigKRLS, which has been on CRAN for a couple of years, had to be pulled recently due to what seems to be a sanitizer issue stemming from its use of bigmemory. bigKRLS works fine (we?ve used it ourselves on many different platforms and have had over 15,000 downloads without an end user reporting difficulties because of this issue). Unfortunately, we have been unable to reproduce the

Is there any reason to expect a problem ? i'm running this script on the cluster down the hall: module load R/2.11.0 R library(multicore) fxx<-function(ll) runif(1) mclapply(1:10,fxx) i get: Error in fork() : Unable to fork. less /proc/cpuinfo yields: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU

This is a 'rather than re-invent the wheel' post. Has anyone out there re-written combn so that it can be parallelized - with multicore, snow, or otherwise? I have a job that requires large numbers of combinations, and rather than get all of the index values, then crank it through mclapply, I was wondering if there was a way to just do this natively within a function. Just curious.