similar to: bigmemory - extracting submatrix from big.matrix object

Hi, With reference to the mail below, I have large datasets, coming from various different sources, which I can read into filebacked big.matrix using library bigmemory. I want to merge them all into one 'big.matrix' object. (Later, I want to run regression using library 'biglm'). I am unsuccessfully trying to do this from quite some time now. Can you please

Hi, I am trying to run the bigmemory example provided on the http://www.bigmemory.org/ The example runs on the "airline data" and generates summary of the csv files:- library(bigmemory) library(biganalytics) x <- read.big.matrix("2005.csv", type="integer", header=TRUE, backingfile="airline.bin", descriptorfile="airline.desc",

I recently learned about the bigmemory and foreach packages and am trying to use them to help me create a very large matrix. Without those packages, I can create the type of matrix that I want with 10 columns and 5e6 rows. I would like to be able to scale up to 5e9 rows, or more, if possible. I have created a simplified example of what I'm trying to do, below. The first part of the

Hi all, Is there any way I can add more columns to an existing filebacked big.matrix object. In general, I want a way to modify an existing big.matrix object, i.e., add rows/columns, rename colnames, etc. I tried the following: > library(bigmemory) > x = read.big.matrix("test.csv",header=T,type="double",shared=T,backingfile="test

I work with problems that have rather large data requirements -- typically a bunch of multigig arrays. Given how generous R is with using memory, the only way for me to work with R has been to use bigmatrices from bigmemory package. One thing that is missing a bit is interoperability of bigmatrices with C++ and possibly java. What i mean by that is API that would allow read and write filebacked

I have pretty big data sizes, like matrices of .5 to 1.5GB so once i need to juggle several of them i am in need of disk cache. I am trying to use bigmemory package but getting problems that are hard to understand. I am getting seg faults and machine just hanging. I work by the way on Red Hat Linux, 64 bit R version 10. Simplest problem is just saving matrices. When i do something like

Dear all, bigKRLS, which has been on CRAN for a couple of years, had to be pulled recently due to what seems to be a sanitizer issue stemming from its use of bigmemory. bigKRLS works fine (we?ve used it ourselves on many different platforms and have had over 15,000 downloads without an end user reporting difficulties because of this issue). Unfortunately, we have been unable to reproduce the

The re-engineered bigmemory package is now available (Version 3.5 and above) on CRAN. We strongly recommend you cease using the older versions at this point. bigmemory now offers completely platform-independent support for the big.matrix class in shared memory and, optionally, as filebacked matrices for larger-than-RAM applications. We're working on updating the package vignette, and a

The re-engineered bigmemory package is now available (Version 3.5 and above) on CRAN. We strongly recommend you cease using the older versions at this point. bigmemory now offers completely platform-independent support for the big.matrix class in shared memory and, optionally, as filebacked matrices for larger-than-RAM applications. We're working on updating the package vignette, and a

Hi, I'm playing around with the 'bigmemory' package, and I have finally managed to create some really big matrices. However, only now I realize that there may not be functions made for what I want to do with the matrices... I would like to perform a cluster analysis based on a big.matrix. Googling around I have found indications that a certain kmeans.big.matrix() function should

Dear all, I am new to network analysis, but since I have good data I started to read about it and learned how to use the ergm and related packages. I generally get interesting results, but when I run a model including sociality and selective mixing effects for different groups, the model runs (and converges) but I get a warning as follows: mod <- ergm(network ~ edges + gwesp(0, fixed=T) +

I am reading into Java dput output for a matrix, more specifically for a file backed big-matrix. I basically need to lift dimnames for a matrix from dput output. It's no big deal, but the code is very 'hackish' due to the need to get rid of quotes, endlines, parenthesis, etc. I was wondering if i could manipulate to an extent dput output with some options that define it, for example,

Hi, all, I have a really big matrix that I want to run k-means on. I tried: >data <- read.big.memory('mydata.csv',type='double',backingfile='mydata.bin',descriptorfile='mydata.desc') I'm using doMC to register multicore. >library(doMC) >registerDoMC(cores=8) >ans<-bigkmeans(data,k) In system monitor, it seems only one thread running R. Is

Hi all, I am trying to read a large csv file (~11 Gb - ~900,000 columns, 3000 rows) using the read.big.matrix command from the bigmemory package. I am using the following command: x<-read.big.matrix('data.csv', sep=',', header=TRUE, type='char', backingfile='data.bin', descriptorfile='data.desc') When the command starts, everything seems to be fine,

Dear friends, I have to use a very large matrix. Something of the sort of matrix(80000,80000,n) .... where n is something numeric of the sort 0.xxxxxx I have not found a way of doing it. I keep getting the error Error in matrix(nrow = 80000, ncol = 80000, 0.2) : too many elements specified Any suggestions? I have searched the mailing list, but to no avail. Best, -- Corrado Topi Global

Dear list, I have a table where first 3 columns are identical if the name in the first column is the same, and the number in N4 is slightly different for all identical names, like this: ------------------------------------------------------------------- 29 Mm.1_at 3 + 93649936 30 Mm.1_at 3 + 93649990 31 Mm.1_at 3 + 93649993 32

Hi, I want to parallelize some computations when it's possible on multicore machines. Each computation produces a big objects that I don't want to store if not necessary: in the end only the object that best fits my data have to be returned. In non-parallel mode, a single gloabl object is updated if the current computation gets a better result than the best previously found. My plan

Hi R-Wizards: I've looking through the R docs and have yet to find what I'm looking for and have tried a few intermediate steps to now avail yet and rather than spend another few hours looking for the solution, I figured I would post a message. I have a matrix (actually a set of them) that I want to pull all the names of the non-zero columns into a vector/list for further processing:

Hi All, I want to "merge" two datasets by column "ID" and I don't want the result to be sorted by "ID". I am doing the following: > z = merge(x, y, by = "ID", sort=F) The result is not sorted by "ID". But (as oppose to what I expected) it is not even in the original order of either "x" or "y". Can

Hi, I have a matrix of say 1024x1024 and I want to look at it in chunks. That is I'd like to divide into a series of submatrices of order 2x2. | 1 2 3 4 5 6 7 8 ... | | 1 2 3 4 5 6 7 8 ... | | 1 2 3 4 5 6 7 8 ... | | 1 2 3 4 5 6 7 8 ... | ... So the first submatrix would be | 1 2 | | 1 2 | the second one would be | 3 4 | | 3 4 | and so on. That is I want the matrix to be evenly divided