similar to: Looking for a good tutorial on ff package

Hi, The read.csv.ffdf function in package ff will create the ff object physical file in the default directories, I am trying to let the files created in the paths users specify, I think the point is to make use of the asffdf_args parameter, I have a test CSV file named D:\rtemp\fftest.csv, the content of the file is as following: col1,col2,col3 1,"amber",2.4 2,"linda",4.5

*Dear R users, Ive just started using the ff package. There is a csv file (~4Gb) with 7 columns and 6e+7 rows. I want to read only column from the file, skipping the first 100 rows. Below Ive provided different outcomes, which will clarify my problem * > sessionInfo() R version 2.14.2 (2012-02-29) Platform: x86_64-pc-mingw32/x64 (64-bit) locale: ... attached base packages: [1] tools

Hi My aim is to read a large .csv file into R. I ran the following code and am using R version 10.1 on Windows. >library(ff) > read.csv.ffdf(x=NULL,"file.csv",fileEncoding="",nrows=-1,first.rows=NULL,next.rows=NULL,levels=NULL,appendLevels=TRUE,FUN="read.table",transFUN=NULL,asffdf_args=list(),BATCHBYTES=getOption("ffbatchbytes"),VERBOSE=FALSE)

Hello everyone, I need to create and work with some big matrices that actually have somewhat over 2 million columns and 117 rows. To do some calculations on such big matrices R just needs too much memory for my PC (4GB installed). So I need a solution to work with large datasets. I'm trying to use the ff-package but I don't think I really understand the whole functionality of the

I've spent some time trying to wrap my head around reading in large csv files with the ff-package. I think I know how to do it, but am bumping into some problems. I've tried to recreate the issues as best as I can with a smaller example and maybe someone can help explain the problems. The following code just creates a csv file with an integer column, character column and logical column.

Hello. I'm using as.ffdf(mydataframe) to create ffdf objects inside an lapply loop and returning that. I then use crbind to combine the lapply results into allData. So...simplified flow looks like this. res <- lapply(1:nchunks, function(n) { blah blah with nth chunk mydataframe <- data.frame(blah blah) dat <-

Hi everyone, my apologies if I'm overlooking something obvious in the documentation. I'm relatively inexperienced with the (awesome) ff package. My goal is to use the read.table.ffdf() function to call the read.fwf() function and pass through the colClasses and comment.char arguments. The code below shows exactly what doesn't work for me. If the colClasses and comment.char

I'm having trouble getting ffdfindexget to work right in Windows. Even the most trivial of examples gives me problems. > myVec = ff(1:5) > another = ff(10:14) > littleFrame = ffdf(myVec, another) > posVec = ff(c(2, 4), vmode = 'integer') > ffdfindexget(littleFrame, posVec) Error in if (any(B < 1)) stop("B too small") : missing value where TRUE/FALSE

Hi, I want to use map("state") and have the ecoregion shape (please see link) file projected onto this. ftp://ftp.epa.gov/wed/ecoregions/cec_na/NA_CEC_Eco_Level3.zip ftp://ftp.epa.gov/wed/ecoregions/cec_na/NA_CEC_Eco_Level3.zip Could someone please show me how; I have never messed with this sort of thing. Thanks. -- View this message in context:

Hi Does anybody know how to create ff objects with data reading from stream objects, such as data reading from PostgreSQL database through RPostgreSQL. For this purpose although we can save the data to a csv file through external tools and then read it through csv readers, but it requires one more data read and write operation, which is of high I/O cost for large datasets. Xiaobo.Gu

Greetings useRs, Can anyone provide an example how to use ff to feed a very large data frame to glm? The data.frame cannot be loaded in R using conventional read.csv as it is too big. glm(...,data=ff.file) ?? Thank you Stephen B

Dear Experienced R Practitioners, I have 4GB .txt data called "dataset.txt" and have attempted to use *ff, bigmemory, filehash and sqldf *packages to import it, but have had no success. The readLines output of this data is: readLines("dataset.txt",n=20) [1] " "

Hi, I have two questions on using the ff package and wonder if anyone who used ff can share some thoughts. I need to save a matrix as a memory-mapped file and load it back later. To save the matrix, I use mat = matrix(1:20, 4, 5) matFF = ff(mat, dim=dim(mat), filename="~/a.mat", overwrite=TRUE, dimnames = dimnames(mat)) To load it back, I use matFF2 = ff(vmode = "double",

Hi, I am using version 2.1-1 of the ff package. I have a data set with 80 million rows and I need to create a new ffdf object, subseting by values in one of the original ffdf's columns. Here is my code: bigData <- read.table.ffdf(file="/data/demodata/data/smallData.txt", next.rows=1e5, head=TRUE, sep="|") dim(bigData) N <- nrow(bigData);N select <- ff(

Hi, I am running the examples in page 70 of the ff package document, but it failed with the following error > cat("let's create some ff objects\n") let's create some ff objects > n <- 8e3 > a <- ff(sample(n, n, TRUE), vmode="integer", length=n, filename="d:/tmp/a.ff") > b <- ff(sample(255, n, TRUE), vmode="ubyte", length=n,

On Thu, 4 Aug 2016 at 17:53 Michael Sumner <mdsumner at gmail.com> wrote: > On Thu, 4 Aug 2016 at 11:17 Paul Murrell <paul at stat.auckland.ac.nz> wrote: > >> Hi >> >> Just to clarify, I think this IS a problem with grid.path() as well as >> polypath(). >> >> > Hi, oh dear - sorry about that > > I appreciate the deeper explanation, I

I have a problem with duplicated date_time stamps that I do not see as duplicated. I read a file with observations taken every 30 minutes: > aur2009=read.csv(paste(datadir,"AUR_ECPP_2009.csv",sep="/"),sep=";",stringsAsFactors=F) > aur2009[1:3,1:5] Date.Time E_filled E_filled_flag LE_filled LE_filled_flag 1 1/1/2009 0:00 0 NaN 5.86

Hola R, tengo las siguientes preguntas: Pregunta 1: Cargar las tablas de los datos de peliculas en R usando `ff`. Cómo se construye una columna nueva que de, para cada cliente y cada evaluación, de el número de días que han pasado desde la primera evaluación del cliente? Qué función se utiliza para verifica el consumo de memoria en las operaciones? Pregunta 2: Cómo se corre un modelo de

I have recently been using R - more speciifcally the GUI packages Rattle and Rcmdr. I like these products a lot and want to use them for some projects - the problem that I run into is when I start to try and run large datasets through them.  The data sets are 10-15 million in record quantity and usually have 15-30 fields (both numerical and categorical). I saw that there were some packages

Hi Just to clarify, I think this IS a problem with grid.path() as well as polypath(). For the example you give, grid.path() diverts to drawing a polygon (because there is no 'id' specified), and the NAs in 'x' generate two separate polygons, which get drawn one on top of the other. The correct analogy to the polypath() example is ... x2 <- matrix(x[!is.na(x)], ncol=2)