R help - Jan 2010 - A question about the ff package

Hi,
I am using version 2.1-1 of the ff package.
I have a data set with 80 million rows and I need to create a new ffdf
object, subseting by values in one of the original ffdf's columns. Here is
my code:

bigData <-
read.table.ffdf(file="/data/demodata/data/smallData.txt",
next.rows=1e5, head=TRUE, sep="|")
dim(bigData)
N <- nrow(bigData);N
select <- ff( vmode='logical', length=N, update=FALSE)
for (i in chunk(bigData, from=1, to=N, by=1e5))
  select[i] <- ifelse(bigData[i,]$C.2 > 0.5, TRUE, FALSE)

bd <- bigData[select,]
  Error in UseMethod("as.hi") :
    no applicable method for 'as.hi' applied to an object of class
"c('ff_vector', 'ff')"

I tried something like the example in ?as.hi but I can not convert the
select vector into a RAM object and then use as.hi() b/c I do not have
enough RAM.  What are my options for subseting this ffdf object by the
values of one of its columns?

More generally, is there a conversion guide / vignette showing how to
convert standard operations done on RAM data frames to operations done on
ffdf objects?

Here is my R version information
> R.Version()
$platform
[1] "x86_64-pc-linux-gnu"
$`svn rev`
[1] "50720"
$version.string
[1] "R version 2.10.1 (2009-12-14)"

Thank you for your help.
OS

	[[alternative HTML version deleted]]

Peter,

ff objects are not allowed as subscripts to ff objects. You can take several
routes

1) use bit objects instead of logical or ff logical. This is fast and takes
factor 32 less RAM than logicals (BTW bit objects can be coerced to ff via
as.ff() and as.bit() but they convert to vmode "boolean" (1 bit), not
"logical" (2 bits). Examples for working with bit are on
http://ff.r-forge.r-project.org/ff&bit_UseR!2009.pdf

2) convert your logicals into positive integer subscripts (assuming that there
are not too many elements selected, as you assume if writing bigData[select,]

3) keep your logical in a ff logical or ff boolean and then do chunked looping
over both - the ff with the subscripts and the ffdf - and in each chunk convert
the logical selection to integers, see 2)

HTH


Jens Oehlschl?gel


P.S. you  might want to try the newer version on r-forge. It has several
improvements but is not yet on CRAN because there is currently some issue with
snow leopard.