R help - Mar 2012 - ff usage for glm

Greetings useRs,

Can anyone provide an example how to use ff to feed a very large data frame to
glm?
The data.frame cannot be loaded in R using conventional read.csv as it is too
big.

glm(...,data=ff.file) ??

Thank you
Stephen B

you want to check the ff man page (?ff), there is an example described
there with biglm. b

On 30 March 2012 21:05, Bond, Stephen <Stephen.Bond@cibc.com> wrote:
> Greetings useRs,
>
> Can anyone provide an example how to use ff to feed a very large data
> frame to glm?
> The data.frame cannot be loaded in R using conventional read.csv as it is
> too big.
>
> glm(...,data=ff.file) ??
>
> Thank you
> Stephen B
	[[alternative HTML version deleted]]

On Sat, Mar 31, 2012 at 9:05 AM, Bond, Stephen <Stephen.Bond at cibc.com>
wrote:
> Greetings useRs,
>
> Can anyone provide an example how to use ff to feed a very large data frame
to glm?
> The data.frame cannot be loaded in R using conventional read.csv as it is
too big.
>
> glm(...,data=ff.file) ??
>
I shouldn't think glm() will work on data that are too big to read
into R.  However, bigglm() from the biglm package should work.  You
just need to write a function that supplies chunks of data from
ff.file as requested (see the example on ?bigglm).  I haven't used ff,
but it looks from the documentation as though chunk() will do all the
difficult parts.

  -thomas

-- 
Thomas Lumley
Professor of Biostatistics
University of Auckland

this works:
bigcl <- read.table(file="bigCL1.csv",sep=',',header=T ,
colClasses=c(rep("factor",3),"numeric","NULL","integer",
                       
"numeric","integer","NULL","numeric","NULL"),nrow=1000)

this doesn't:
bigcl <- read.table.ffdf(file="bigCL1.csv",sep=',',header=T
,
                         
colClasses=c(TERMGROUP="factor",SN="factor",
                           
INS="factor",INCENTIVE="numeric",SRC="character",FLAG="integer",
                           
RELAGE="numeric",BURNSUM="integer",prod="character",BAL="numeric",origbal="NULL")
                          )
Error in ff(initdata = initdata, length = length, levels = levels, ordered =
ordered,  :
  vmode 'character' not implemented

neither this:
> bigcl <-
read.table.ffdf(file="bigCL1.csv",sep=',',header=T  ,
+                         
colClasses=c(TERMGROUP="factor",SN="factor",
+                           
INS="factor",INCENTIVE="numeric",SRC="NULL",FLAG="integer",
+                           
RELAGE="numeric",BURNSUM="integer",prod="NULL",BAL="numeric",origbal="NULL")
+                          )
Error in repnam(colClasses, colnames(x), default = NA) :
  the following argument names do not
match'SRC','prod','origbal'


if I load as is without classes then:
> bigcl <-
read.table.ffdf(file="bigCL1.csv",sep=',',header=T)
> bigcl$term <- factor(bigcl$term)
Error in sort.list(y) : 'x' must be atomic for 'sort.list'
Have you called 'sort' on a list?

attempting to save it:
> ffsave(bigcl,file="fcl")
Error in system(cmd, input = filelist, intern = TRUE) : 'zip' not found

attempting to get rid of the unwanted columns:
> bigcl <- bigcl[,c(-5,-9,-11)]
Error: cannot allocate vector of size 129.8 Mb
In addition: Warning messages:
1: In class(df) <- "data.frame" :
  Reached total allocation of 1535Mb: see help(memory.size)
2: In class(df) <- "data.frame" :
  Reached total allocation of 1535Mb: see help(memory.size)
3: In class(df) <- "data.frame" :
  Reached total allocation of 1535Mb: see help(memory.size)
4: In class(df) <- "data.frame" :
  Reached total allocation of 1535Mb: see help(memory.size)

 ff was supposed to be gentle on RAM usage??

I am attaching the first 1000 lines from the csv file.
thanks a lot.


Stephen