search for: filebacked

I am using the library(bigmemory) to handle large datasets, say 1 GB, and facing following problems. Any hints from anybody can be helpful. _Problem-1: _ I am using "read.big.matrix" function to create a filebacked big matrix of my data and get the following warning: > x = read.big.matrix("/home/utkarsh.s/data.csv",header=T,type="double",shared=T,backingfile = "backup", backingpath = "/home/utkarsh.s") Warning message: In filebacked.big.matrix(nrow = numRows, n...

Hi, With reference to the mail below, I have large datasets, coming from various different sources, which I can read into filebacked big.matrix using library bigmemory. I want to merge them all into one 'big.matrix' object. (Later, I want to run regression using library 'biglm'). I am unsuccessfully trying to do this from quite some time now. Can you please suggest some way? Am I missing some already...

...ames somehow. Another great thing wiould be to make dput dump rownames and colnames on two separate lines, but i don't think it's possible. To give a specific example, instead of dput output like **new("big.matrix.descriptor" , description = structure(list(sharedType = "FileBacked", filename = "res", totalRows = 1528, totalCols = 53040, rowOffset = c(0, 1528), colOffset = c(0, 53040), nrow = 1528, ncol = 53040, rowNames = c("A", "AA", "RNT.A", "ADVA", "AAPL", "AAS", "ABFS", &quot...

...er", header=TRUE, backingfile="airline.bin", descriptorfile="airline.desc", extraCols="Age") summary(x) This runs fine for the provided csv for year 1987 (size=121MB). However, for big files like for year 2005 (size=639MB), it gives following errors:- Error in filebacked.big.matrix(nrow = nrow, ncol = ncol, type = type, : Problem creating filebacked matrix. Error: object 'x' not found Error in summary(x) : error in evaluating the argument 'object' in selecting a method for function 'summary' Here is the output from running the memory....

...ultigig arrays. Given how generous R is with using memory, the only way for me to work with R has been to use bigmatrices from bigmemory package. One thing that is missing a bit is interoperability of bigmatrices with C++ and possibly java. What i mean by that is API that would allow read and write filebacked matrices from C++, and ideally java without being called from R. Having ability to save armadillo matrices into filebacked matrices and load them back into armadillo would be another very useful thing. This would allow really smooth cooperation between various pieces of software. I would prefer to...

...[i]:end.i[i], 1] <- runif(n*info$p[i], info$a1[i], info$a2[i]) m[start.i[i]:end.i[i], 2] <- rnorm(n*info$p[i], info$a1[i], info$a2[i]) } # example getting ready to scale up to large matrix n <- 50 end.i <- cumsum(n*info$p) start.i <- c(0, end.i[-nrowz]) + 1 m <- filebacked.big.matrix(nrow=n, ncol=2, backingfile="test3.bin", descriptorfile="test3.desc") m[start.i[1]:end.i[1], 1] <- foreach(i=start.i[1]:end.i[1], .combine=c) %do% runif(1, info$a1[1], info$a2[1]) m[start.i[2]:end.i[2], 1] <- foreach(i=start.i[2]:end.i[2], .combine=c) %do% r...

...9;t allow "clever" merges (for actually good reasons). However, it is still probably less painful (and faster) than other options, though we don't implement it: we leave it to the user because details may vary depending on the example and the code is trivial. - Allocate an empty new filebacked big.matrix of the proper size. - Fill it in chunks (typically a column at a time if you can afford the RAM overhead, or a portion of a column at a time). Column operations are more efficient than row operations (again, because of the internals of the data structure). - Because you'll be using...

...A/mcmc files (see the coda package), as generated by jags/WinBUGS/OpenBUGS, into a big.matrix. I can't load the whole mcmc object produced by read.coda() into memory since I'm using a laptop for this analysis (currently I'm unfunded). Right now I'm doing it by creating the filebacked.big.matrix, reading a chunk of data at a time from the chain file using read.table() with "skip" and "nrows" set, and storing it into the big.matrix. While this is memory efficient, the processing overhead seems be related to the size of the skip value, so that the time...

I would like some advice. I am considering the best way to restructure my backup scheme on my NT network. I would like to see if it would be possible/feasible to use cron, smbmount, some SCSI tape drives, and some type of archive tool to backup my NT shares to a RedHat box using Samba. What do all of you do out there? I would envision a situation where each night a script is run by cron that

I have pretty big data sizes, like matrices of .5 to 1.5GB so once i need to juggle several of them i am in need of disk cache. I am trying to use bigmemory package but getting problems that are hard to understand. I am getting seg faults and machine just hanging. I work by the way on Red Hat Linux, 64 bit R version 10. Simplest problem is just saving matrices. When i do something like

Dear friends, I have to use a very large matrix. Something of the sort of matrix(80000,80000,n) .... where n is something numeric of the sort 0.xxxxxx I have not found a way of doing it. I keep getting the error Error in matrix(nrow = 80000, ncol = 80000, 0.2) : too many elements specified Any suggestions? I have searched the mailing list, but to no avail. Best, -- Corrado Topi Global

...ulties because of this issue). Unfortunately, we have been unable to reproduce the UndefinedBehaviorSanitizer error locally, and are unsure how to proceed. We corresponded with Michael Kane, author of bigmemory, who seems to think the issue may be on the CRAN side. At his suggestion, we switched to filebacked big matrices, which didn?t resolve the issue. We also tried adding MAIN_LDFLAGS = -fsanitize=undefined,address to the MAKEVARS, which didn?t do the trick either. Here are some links: 1. bigKRLS repo: https://github.com/rdrr1990/bigKRLS 2. bigKRLS file backs all big matrices with a...

The re-engineered bigmemory package is now available (Version 3.5 and above) on CRAN. We strongly recommend you cease using the older versions at this point. bigmemory now offers completely platform-independent support for the big.matrix class in shared memory and, optionally, as filebacked matrices for larger-than-RAM applications. We're working on updating the package vignette, and a draft is available upon request (just send me an email if you're interested). The user interface is largely unchanged. Feedback, bug reports, etc... are welcome. Jay Emerson & Michael Ka...

The re-engineered bigmemory package is now available (Version 3.5 and above) on CRAN. We strongly recommend you cease using the older versions at this point. bigmemory now offers completely platform-independent support for the big.matrix class in shared memory and, optionally, as filebacked matrices for larger-than-RAM applications. We're working on updating the package vignette, and a draft is available upon request (just send me an email if you're interested). The user interface is largely unchanged. Feedback, bug reports, etc... are welcome. Jay Emerson & Michael Ka...

Hi, I want to parallelize some computations when it's possible on multicore machines. Each computation produces a big objects that I don't want to store if not necessary: in the end only the object that best fits my data have to be returned. In non-parallel mode, a single gloabl object is updated if the current computation gets a better result than the best previously found. My plan

Hi all, Is there any way I can add more columns to an existing filebacked big.matrix object. In general, I want a way to modify an existing big.matrix object, i.e., add rows/columns, rename colnames, etc. I tried the following: > library(bigmemory) > x = read.big.matrix("test.csv",header=T,type="double",shared=T,backingfile=...

Hi, I'm playing around with the 'bigmemory' package, and I have finally managed to create some really big matrices. However, only now I realize that there may not be functions made for what I want to do with the matrices... I would like to perform a cluster analysis based on a big.matrix. Googling around I have found indications that a certain kmeans.big.matrix() function should

Dear all, I am new to network analysis, but since I have good data I started to read about it and learned how to use the ergm and related packages. I generally get interesting results, but when I run a model including sociality and selective mixing effects for different groups, the model runs (and converges) but I get a warning as follows: mod <- ergm(network ~ edges + gwesp(0, fixed=T) +

Dear All, I would like to have an advice for designing an R library, and thought that R-devel may be the best place to ask given so many people who are highly expert in R are around. We are at an early stage of designing an R library, which is effectively an interface to a C++ library providing fast access to large matrices stored on HDD as binary files. The core of the C++ library is

Dear R users, I am using R right now for a simulation of a model that needs a lot of memory. Therefore I use the *bigmemory* package and - to make it faster - the *doMC* package. See my code posted on http://pastebin.com/dFRGdNrG Now, if I use the foreach loop with the addon %do% (for sequential run) I have no problems at all - only here and there some singularities in regressor matrices which