similar to: read.table() with quoted integers

I've spent some time trying to wrap my head around reading in large csv files with the ff-package. I think I know how to do it, but am bumping into some problems. I've tried to recreate the issues as best as I can with a smaller example and maybe someone can help explain the problems. The following code just creates a csv file with an integer column, character column and logical column.

*Dear R users, Ive just started using the ff package. There is a csv file (~4Gb) with 7 columns and 6e+7 rows. I want to read only column from the file, skipping the first 100 rows. Below Ive provided different outcomes, which will clarify my problem * > sessionInfo() R version 2.14.2 (2012-02-29) Platform: x86_64-pc-mingw32/x64 (64-bit) locale: ... attached base packages: [1] tools

Hi everyone, my apologies if I'm overlooking something obvious in the documentation. I'm relatively inexperienced with the (awesome) ff package. My goal is to use the read.table.ffdf() function to call the read.fwf() function and pass through the colClasses and comment.char arguments. The code below shows exactly what doesn't work for me. If the colClasses and comment.char

Greetings useRs, Can anyone provide an example how to use ff to feed a very large data frame to glm? The data.frame cannot be loaded in R using conventional read.csv as it is too big. glm(...,data=ff.file) ?? Thank you Stephen B

Hello fellow R users, I am trying to read a 6.9 million row text file with 26 columns separated by spaces into R using ff. When I specify a small number for first.rows, next.rows and nrows it is read with no issue. However, when I try to specify larger next.rows values and no nrows parameter to read the entire file, I keep getting errors. Please see code below. I am trying to this on a m1.large

Hello, I frequently have to export a large quantity of data from some source (for example, a database, or a hand-written perl script) and then read it into R. This occasionally takes a lot of time; I'm usually using read.table("filename",comment.char="",quote="") to read the data once it is written to disk. However, I *know* that the program that generates

 Hi there, Sorry to bother those who are not interested in this problem. I'm dealing with a large data set, more than 6 GB file, and doing regression test with those data. I was wondering are there any efficient ways to read those data? Instead of just using read.table()? BTW, I'm using a 64bit version desktop and a 64bit version R, and the memory for the desktop is enough for

One of our winXP users intermittently can't process the login script. sometimes it works and maps his drives, sometimes it doesn't. I have the netlogon share configured correctly. Looking at the logs for the user in question, I see this when the user logs in: "couldn't find service netlogo" "closed connection to service netlogon" it truncates netlogo in the log

Dear Experienced R Practitioners, I have 4GB .txt data called "dataset.txt" and have attempted to use *ff, bigmemory, filehash and sqldf *packages to import it, but have had no success. The readLines output of this data is: readLines("dataset.txt",n=20) [1] " "

I'm trying to import a table into R the file is about 700MB. Here's my first try: > DD<-read.table("01uklicsam-20070301.dat",header=TRUE) Error: cannot allocate vector of size 15.6 Mb In addition: Warning messages: 1: In scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings, : Reached total allocation of 1535Mb: see help(memory.size) 2: In scan(file, what,

Hi, The read.csv.ffdf function in package ff will create the ff object physical file in the default directories, I am trying to let the files created in the paths users specify, I think the point is to make use of the asffdf_args parameter, I have a test CSV file named D:\rtemp\fftest.csv, the content of the file is as following: col1,col2,col3 1,"amber",2.4 2,"linda",4.5

Hi helpeRs, I have inherited a set of data files that use the file system as a sort of poor man's database, i.e., the data files are nested in directories that indicate which city they come from. For example: dir.create("deleteme") for(i in paste("deleteme", c("New York", "Los Angeles"), sep="/")) { dir.create(i) for(j in

Hello everyone, I need to create and work with some big matrices that actually have somewhat over 2 million columns and 117 rows. To do some calculations on such big matrices R just needs too much memory for my PC (4GB installed). So I need a solution to work with large datasets. I'm trying to use the ff-package but I don't think I really understand the whole functionality of the

Hi My aim is to read a large .csv file into R. I ran the following code and am using R version 10.1 on Windows. >library(ff) > read.csv.ffdf(x=NULL,"file.csv",fileEncoding="",nrows=-1,first.rows=NULL,next.rows=NULL,levels=NULL,appendLevels=TRUE,FUN="read.table",transFUN=NULL,asffdf_args=list(),BATCHBYTES=getOption("ffbatchbytes"),VERBOSE=FALSE)

I have a big data set that includes character variables of many different values. I'm trying to use ff to read the data and then use biglm.big.matrix to build linear models. However, since big.matrix will convert all character vectors to factors and the character labels will be lost. I decided to create a lookup table outside of R for my character columns and use numbers to represent different

Hello. I'm using as.ffdf(mydataframe) to create ffdf objects inside an lapply loop and returning that. I then use crbind to combine the lapply results into allData. So...simplified flow looks like this. res <- lapply(1:nchunks, function(n) { blah blah with nth chunk mydataframe <- data.frame(blah blah) dat <-

If read.csv's colClasses= argument is NOT used then read.csv accepts double quoted numerics: 1: > read.csv(stdin()) 0: A,B 1: "1",1 2: "2",2 3: A B 1 1 1 2 2 2 However, if colClasses is used then it seems that it does not: > read.csv(stdin(), colClasses = "numeric") 0: A,B 1: "1",1 2: "2",2 3: Error in scan(file, what, nmax, sep,

Suppose I have a data file (possibly with a huge number of columns), where the columns with factors are coded as "1", "2", "3", etc ... The default behavior of read.table is to convert these columns to integer vectors. Is there a way to get read.table to recognize that columns of quoted numbers represent factors (while unquoted numbers are interpreted as

I'm having trouble getting ffdfindexget to work right in Windows. Even the most trivial of examples gives me problems. > myVec = ff(1:5) > another = ff(10:14) > littleFrame = ffdf(myVec, another) > posVec = ff(c(2, 4), vmode = 'integer') > ffdfindexget(littleFrame, posVec) Error in if (any(B < 1)) stop("B too small") : missing value where TRUE/FALSE

Dear folks ? I have a large (26 gig) ASCII flat file in fixed-width format with about 10 million observations of roughly 400 variables. (It is 51 years of Current Population Survey micro data from IPUMS, roughly half the fields for each record). The file was produced by automatic process in response to a data request of mine. The file is not accompanied by a human-readable file giving the