similar to: ffbase, help with %in%

Dear R friends. After having some troubles learning how to create a ffdf object, now I find myself having problems saving it. this is the data i´d like to save: str(DATA) List of 3 $ virtual: 'data.frame': 6 obs. of 7 variables: .. $ VirtualVmode : chr "double" "short" "integer" "integer" ... .. $ AsIs : logi FALSE FALSE FALSE

I've noticed that ff uses the unzip utility available on it's host OS to load datasets via ffload. It seems to work fine in linux, but when I try to use the package in Windows (hence dling Windows unzip utils) I get 2 errors, one telling me that the options aren't being passed to unzip correctly and another indicating that my filename is incorrect. First, does anyone know of an unzip

Dear folks ? I have a large (26 gig) ASCII flat file in fixed-width format with about 10 million observations of roughly 400 variables. (It is 51 years of Current Population Survey micro data from IPUMS, roughly half the fields for each record). The file was produced by automatic process in response to a data request of mine. The file is not accompanied by a human-readable file giving the

I've spent some time trying to wrap my head around reading in large csv files with the ff-package. I think I know how to do it, but am bumping into some problems. I've tried to recreate the issues as best as I can with a smaller example and maybe someone can help explain the problems. The following code just creates a csv file with an integer column, character column and logical column.

Hi, The read.csv.ffdf function in package ff will create the ff object physical file in the default directories, I am trying to let the files created in the paths users specify, I think the point is to make use of the asffdf_args parameter, I have a test CSV file named D:\rtemp\fftest.csv, the content of the file is as following: col1,col2,col3 1,"amber",2.4 2,"linda",4.5

Hello, I frequently have to export a large quantity of data from some source (for example, a database, or a hand-written perl script) and then read it into R. This occasionally takes a lot of time; I'm usually using read.table("filename",comment.char="",quote="") to read the data once it is written to disk. However, I *know* that the program that generates

I'm having trouble getting ffdfindexget to work right in Windows. Even the most trivial of examples gives me problems. > myVec = ff(1:5) > another = ff(10:14) > littleFrame = ffdf(myVec, another) > posVec = ff(c(2, 4), vmode = 'integer') > ffdfindexget(littleFrame, posVec) Error in if (any(B < 1)) stop("B too small") : missing value where TRUE/FALSE

Hello everyone, I need to create and work with some big matrices that actually have somewhat over 2 million columns and 117 rows. To do some calculations on such big matrices R just needs too much memory for my PC (4GB installed). So I need a solution to work with large datasets. I'm trying to use the ff-package but I don't think I really understand the whole functionality of the

Hi, I have two questions on using the ff package and wonder if anyone who used ff can share some thoughts. I need to save a matrix as a memory-mapped file and load it back later. To save the matrix, I use mat = matrix(1:20, 4, 5) matFF = ff(mat, dim=dim(mat), filename="~/a.mat", overwrite=TRUE, dimnames = dimnames(mat)) To load it back, I use matFF2 = ff(vmode = "double",

Hi, I am running the examples in page 70 of the ff package document, but it failed with the following error > cat("let's create some ff objects\n") let's create some ff objects > n <- 8e3 > a <- ff(sample(n, n, TRUE), vmode="integer", length=n, filename="d:/tmp/a.ff") > b <- ff(sample(255, n, TRUE), vmode="ubyte", length=n,

Hi, I am using version 2.1-1 of the ff package. I have a data set with 80 million rows and I need to create a new ffdf object, subseting by values in one of the original ffdf's columns. Here is my code: bigData <- read.table.ffdf(file="/data/demodata/data/smallData.txt", next.rows=1e5, head=TRUE, sep="|") dim(bigData) N <- nrow(bigData);N select <- ff(

I'm trying to import a table into R the file is about 700MB. Here's my first try: > DD<-read.table("01uklicsam-20070301.dat",header=TRUE) Error: cannot allocate vector of size 15.6 Mb In addition: Warning messages: 1: In scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings, : Reached total allocation of 1535Mb: see help(memory.size) 2: In scan(file, what,

Dear list, I am trying to understand and use the ff package. As I had some problems saving some ff objects, and as I did not fully manage to understand the whole concept of *.ff, *.ffData and *.RData with the help of the documentation, I tried to reproduce the examples from the help of ffsave. When I ran, however : (copied from the help) message("let's create some ff objects")

Greetings useRs, Can anyone provide an example how to use ff to feed a very large data frame to glm? The data.frame cannot be loaded in R using conventional read.csv as it is too big. glm(...,data=ff.file) ?? Thank you Stephen B

Dear R Devel, In a thread this morning Luke Tierney mentioned that R's way of garbage collecting is going to change soon in 3.4.0. I couldn't find this info on Google but I wanted to share what I had been discussing in another forum, in case now is not too late to raise considerations which could affect the design of planned changes to R's garbage collection facilities. I ran into a

Hello all! I have a problem with ggplot2 library. I want to do an heat map and the y variables are the year months. If I use the following code, he y values are in alphabetical order, but I want it in month order. The code is: library(reshape) library(ggplot2) library(scales) p <- ggplot(data.m, aes(variable, Month)) + geom_tile(aes(fill = value),

*Dear R users, Ive just started using the ff package. There is a csv file (~4Gb) with 7 columns and 6e+7 rows. I want to read only column from the file, skipping the first 100 rows. Below Ive provided different outcomes, which will clarify my problem * > sessionInfo() R version 2.14.2 (2012-02-29) Platform: x86_64-pc-mingw32/x64 (64-bit) locale: ... attached base packages: [1] tools

Really old subject?, so, all my apologizes for digging up but, since I also ran into this? maybe this hack can be useful to someone I propose monkey patching here: library(ff) my.as.ffdf.data.frame <- function (x, vmode = NULL, col_args = list(), ...) { rnam <- attr(x, "row.names") if (is.integer(rnam)) { if (all(rnam == seq_along(rnam))) rnam <- NULL else

Dear list, This is a bit of an off-topic question, but I'm hoping to get some advice from more experienced people. I've used the website "Web of Science" to manually collect publication counts responding to several keywords as a function of date, since the 1960s. http://apps.isiknowledge.com/RAMore.do?product=UA&search_mode=&SID=P1g9lFJp9 at

Hi My aim is to read a large .csv file into R. I ran the following code and am using R version 10.1 on Windows. >library(ff) > read.csv.ffdf(x=NULL,"file.csv",fileEncoding="",nrows=-1,first.rows=NULL,next.rows=NULL,levels=NULL,appendLevels=TRUE,FUN="read.table",transFUN=NULL,asffdf_args=list(),BATCHBYTES=getOption("ffbatchbytes"),VERBOSE=FALSE)