similar to: Pass By Value Questions

To answer your first question about read.big.matrix(), we don't know what your acc3.dat file is, but it doesn't appear to have been detected as a standard file (like a CSV file) or -- perhaps -- doesn't even exist (or doesn't exist in your current directory)? Next: > In addition, I am planning to do a multiple imputation with MICE package > using the data read by bigmemory

Dear R developers, It would be great if you could implement the two minor code changes suggested below, which would help processing large objects in R. Jens Oehlschl?gel # Wish no. 1: let [.AsIs return the class AFTER subsetting, not the class of the original object # Wish no. 2: adjust write.csv and write.csv2 for multiple calls in chunked writing # Rationale no. 1: a couple of packages

Dear R-list I'm on my way to start a new project on a rather big panel, consisting of approximately 8 million observations in 30 waves of data and about 15 variables. I have a similar data set that is approximately 7 gigabytes in size. Until now I have done my data management in SAS, and Stata, mostly identifying spells, counting events in intervals, and a like, but I would like to

Hi Folks, I've been bumping my head against the 4GB limit for 32-bit R. I can't go to 64-bit R due to package compatibility issues (ROBDC - possible but painful, xlsReadWrite - not possible, and others). I have a number of big dataframes whose columns all sorts of data types - factor, character, integer, etc. I run and save models that keep copies of the modeled data inside the model

Hi, I'm playing around with the 'bigmemory' package, and I have finally managed to create some really big matrices. However, only now I realize that there may not be functions made for what I want to do with the matrices... I would like to perform a cluster analysis based on a big.matrix. Googling around I have found indications that a certain kmeans.big.matrix() function should

Hi, I am trying to run the bigmemory example provided on the http://www.bigmemory.org/ The example runs on the "airline data" and generates summary of the csv files:- library(bigmemory) library(biganalytics) x <- read.big.matrix("2005.csv", type="integer", header=TRUE, backingfile="airline.bin", descriptorfile="airline.desc",

Dear R-helpers I have a data set of roughly 10 million records, 7 columns. It has only about 500MB as a csv, so it fits in the memory. It's painfully slow to do anything with it, but it's possible. I also have another dataset of covariates that I would like to explore - with about 4GB of data... I would like to merge the two datasets and use lmer to build a mixed effects model. Is

Hi everyone, Any ideas on troubleshooting this memory issue: > d1<-read.csv("arrears.csv") Error: cannot allocate vector of size 77.3 Mb In addition: Warning messages: 1: In class(data) <- "data.frame" : Reached total allocation of 1535Mb: see help(memory.size) 2: In class(data) <- "data.frame" : Reached total allocation of 1535Mb: see

I have recently been using R - more speciifcally the GUI packages Rattle and Rcmdr. I like these products a lot and want to use them for some projects - the problem that I run into is when I start to try and run large datasets through them.  The data sets are 10-15 million in record quantity and usually have 15-30 fields (both numerical and categorical). I saw that there were some packages

Hi, I have two questions on using the ff package and wonder if anyone who used ff can share some thoughts. I need to save a matrix as a memory-mapped file and load it back later. To save the matrix, I use mat = matrix(1:20, 4, 5) matFF = ff(mat, dim=dim(mat), filename="~/a.mat", overwrite=TRUE, dimnames = dimnames(mat)) To load it back, I use matFF2 = ff(vmode = "double",

Hi, With reference to the mail below, I have large datasets, coming from various different sources, which I can read into filebacked big.matrix using library bigmemory. I want to merge them all into one 'big.matrix' object. (Later, I want to run regression using library 'biglm'). I am unsuccessfully trying to do this from quite some time now. Can you please

I am trying to add the bigmemory packages but I get the following error message: In file included from bigmemory.cpp:14:0: ../inst/include/bigmemory/isna.hpp: In function 'bool neginf(double)': ../inst/include/bigmemory/isna.hpp:22:57: error: 'isinf' was not declared in this scope make: *** [bigmemory.o] Error 1 ERROR: compilation failed for package 'bigmemory' * removing

Dear all, what is the proper way to make the assignment operator generic and define methods depending on the class of the assigned value? Best regards Jens Oehlschl?gel P.S. I vaguely remember that this was possible in S+. In R I tried to no avail: # using this like h<-1:3 gives Error: in `<-.default`(h, 1:3) : invalid (do_set) left-hand side to assignment

Hi, Is there any official way to determine the colClasses of a data.frame? Why has POSIXct such a strange class structure? Why is colClasses "ordered" not allowed (and doesn't work)? Background ========== I am writing a chunked csv reader that provides the functionality of read.table for large files (in the next version of package ff). In chunked reading, one wants to learn the

Hello All, I tried to intall the bigmemory package from a CRAN mirror site and received the following output while installing. Any idea what's going on and how to fix it? The system details are provided below. --------------------- begin error messages ----------------------- * installing *source* package 'bigmemory' ... checking for Sun Studio compiler...no checking for

Dear Experienced R Practitioners, I have 4GB .txt data called "dataset.txt" and have attempted to use *ff, bigmemory, filehash and sqldf *packages to import it, but have had no success. The readLines output of this data is: readLines("dataset.txt",n=20) [1] " "

Hi, I'm trying to introduce a (spatial) exponential correlation structure (with range=200 and nugget.effet of 0.3) in a lme model of this form: lme(ARBUS~YEAR, random=~1|IDSOUS). The structure of the data is "IDSOUS" "XMIN" "YMAX" "YEAR" "ARBUS" with 2 years of data and 5600 points for each year. I do:

My long-promised Rdsm package is now on CRAN. Some of you may recall that I made a prototype available on my own Web page last July. This is the official version, much evolved since I released the prototype. The CRAN description states: Provides a threads-like programming environment for R, usable both on a multicore machine and across a network of multiple machines. The package

Full_Name: Jens Oehlschl?gel Version: 2.9.2 OS: Win32 Submission from: (NULL) (85.181.152.156) # Obviously an empty basename is allowed (if the filepath is a pure path) > basename("/") [1] "" # but here we get the dot from the path > basename("./") [1] "." > version _ platform i386-pc-mingw32

Hello all. I wish to read a large data set into R. My current issue is in getting the data so that R would be able to access it. Using read.table won't work since the data is over 1GB in size (and I am using windows XP), so my plan was to read the file chunk by chunk and each time move it into bigmemory (I'll play with that when the time will come, maybe ff is better ?!). I encountered