similar to: adding more columns in big.matrix object of bigmemory package

Hi, With reference to the mail below, I have large datasets, coming from various different sources, which I can read into filebacked big.matrix using library bigmemory. I want to merge them all into one 'big.matrix' object. (Later, I want to run regression using library 'biglm'). I am unsuccessfully trying to do this from quite some time now. Can you please

I am using the library(bigmemory) to handle large datasets, say 1 GB, and facing following problems. Any hints from anybody can be helpful. _Problem-1: _ I am using "read.big.matrix" function to create a filebacked big matrix of my data and get the following warning: > x = read.big.matrix("/home/utkarsh.s/data.csv",header=T,type="double",shared=T,backingfile

Hi, I am trying to run the bigmemory example provided on the http://www.bigmemory.org/ The example runs on the "airline data" and generates summary of the csv files:- library(bigmemory) library(biganalytics) x <- read.big.matrix("2005.csv", type="integer", header=TRUE, backingfile="airline.bin", descriptorfile="airline.desc",

I recently learned about the bigmemory and foreach packages and am trying to use them to help me create a very large matrix. Without those packages, I can create the type of matrix that I want with 10 columns and 5e6 rows. I would like to be able to scale up to 5e9 rows, or more, if possible. I have created a simplified example of what I'm trying to do, below. The first part of the

Hi all, I am trying to read a large csv file (~11 Gb - ~900,000 columns, 3000 rows) using the read.big.matrix command from the bigmemory package. I am using the following command: x<-read.big.matrix('data.csv', sep=',', header=TRUE, type='char', backingfile='data.bin', descriptorfile='data.desc') When the command starts, everything seems to be fine,

I have pretty big data sizes, like matrices of .5 to 1.5GB so once i need to juggle several of them i am in need of disk cache. I am trying to use bigmemory package but getting problems that are hard to understand. I am getting seg faults and machine just hanging. I work by the way on Red Hat Linux, 64 bit R version 10. Simplest problem is just saving matrices. When i do something like

Hi, all, I have a really big matrix that I want to run k-means on. I tried: >data <- read.big.memory('mydata.csv',type='double',backingfile='mydata.bin',descriptorfile='mydata.desc') I'm using doMC to register multicore. >library(doMC) >registerDoMC(cores=8) >ans<-bigkmeans(data,k) In system monitor, it seems only one thread running R. Is

I work with problems that have rather large data requirements -- typically a bunch of multigig arrays. Given how generous R is with using memory, the only way for me to work with R has been to use bigmatrices from bigmemory package. One thing that is missing a bit is interoperability of bigmatrices with C++ and possibly java. What i mean by that is API that would allow read and write filebacked

Dear all, bigKRLS, which has been on CRAN for a couple of years, had to be pulled recently due to what seems to be a sanitizer issue stemming from its use of bigmemory. bigKRLS works fine (we?ve used it ourselves on many different platforms and have had over 15,000 downloads without an end user reporting difficulties because of this issue). Unfortunately, we have been unable to reproduce the

The re-engineered bigmemory package is now available (Version 3.5 and above) on CRAN. We strongly recommend you cease using the older versions at this point. bigmemory now offers completely platform-independent support for the big.matrix class in shared memory and, optionally, as filebacked matrices for larger-than-RAM applications. We're working on updating the package vignette, and a

The re-engineered bigmemory package is now available (Version 3.5 and above) on CRAN. We strongly recommend you cease using the older versions at this point. bigmemory now offers completely platform-independent support for the big.matrix class in shared memory and, optionally, as filebacked matrices for larger-than-RAM applications. We're working on updating the package vignette, and a

I'm trying to read CODA/mcmc files (see the coda package), as generated by jags/WinBUGS/OpenBUGS, into a big.matrix. I can't load the whole mcmc object produced by read.coda() into memory since I'm using a laptop for this analysis (currently I'm unfunded). Right now I'm doing it by creating the filebacked.big.matrix, reading a chunk of data at a time from the chain

Hi, Does the backing file of a big.matrix store the contents of entire matrix? Or does it store the portion of it that is not stored in RAM? In other words, can the backing file be treated as a file containing the matrix's full data? I have been writing my big.matrix objects to disk (write.big.matrix), and other programs that want to access this matrix then just read it in (read.big.matrix).

Hi all, I am performing a stepwise regression by running the "step" function on an "lm" object. Now I want to save the intermediate iterations. I know the argument trace=T will print it on the console, but I rather want to assign it to some R object or may be output it in a CSV or text file. Any help will be appreciated. Regards Utkarsh

Hi All, I have vector of length 52, say, x=sample(30,52,replace=T). I want to sort x and split into five *nearly equal groups*. Note that the observations are repeated in x so in case of a tie I want both the observations to fall in same group. This seems a very common task to do, but still I couldn't find an R function to do this. Any help would be highly appreciated. Regards Utkarsh

Hi All, I am currently using R version 2.8.1 on linux cent os 4.4 (i386) and want to upgrade to version 2.9.1. It seems to me that version-2.9.1 is it not for my OS. Am I right? Regards Utkarsh [[alternative HTML version deleted]]

Dear R-list I'm on my way to start a new project on a rather big panel, consisting of approximately 8 million observations in 30 waves of data and about 15 variables. I have a similar data set that is approximately 7 gigabytes in size. Until now I have done my data management in SAS, and Stata, mostly identifying spells, counting events in intervals, and a like, but I would like to

Dear all, I am new to network analysis, but since I have good data I started to read about it and learned how to use the ergm and related packages. I generally get interesting results, but when I run a model including sociality and selective mixing effects for different groups, the model runs (and converges) but I get a warning as follows: mod <- ergm(network ~ edges + gwesp(0, fixed=T) +

I am trying to add the bigmemory packages but I get the following error message: In file included from bigmemory.cpp:14:0: ../inst/include/bigmemory/isna.hpp: In function 'bool neginf(double)': ../inst/include/bigmemory/isna.hpp:22:57: error: 'isinf' was not declared in this scope make: *** [bigmemory.o] Error 1 ERROR: compilation failed for package 'bigmemory' * removing

Hi All, I am currently doing the following to compute summary statistics of aggregated data: a = aggregate(warpbreaks$breaks, warpbreaks[,-1], mean) b = aggregate(warpbreaks$breaks, warpbreaks[,-1], sum) c = aggregate(warpbreaks$breaks, warpbreaks[,-1], length) ans = cbind(a, b[,3], c[,3]) This seems unnecessarily complex to me so I tried > aggregate(warpbreaks$breaks, warpbreaks[,-1],