R help - Apr 2006 - splitting and saving a large dataframe

Hi,

I searched for this in the mailing list, but found no results.

I have a large dataframe ( dim(mydata)= 1297059      16,
object.size(mydata145280576) ) , and I want to perform some calculations which
can be done by
a factor's levels, say, mydata$myfactor. So what I want is to split this
dataframe into nlevels(mydata$myfactor) = 80 levels. But I must do this
efficiently, that is, I can't actually split the dataframe and then save the
resulting 80 smaller ones, as I would run out of memory. I don't know how to
do it though.

So, can anyone give a hint on how to get to get one smaller dataframe, save
it into a file, remove it from the workspace, get the next smaller
dataframe, save it into another file, remoe it from the workspace, and so
on?

Thank you,

Dimitri

	[[alternative HTML version deleted]]

for( i in levels(mydata$myfactor) ) {
   d = mydata[mydata$myfactor==i,]
   write.table(d, paste('d', i, '.data', sep='') )
   }

Jean Coursol


Quoting Dimitri Szerman <dimitrijoe at ipea.gov.br>:
> Hi,
>
> I searched for this in the mailing list, but found no results.
>
> I have a large dataframe ( dim(mydata)= 1297059      16,
object.size(mydata> 145280576) ) , and I want to perform some calculations
which can be done by
> a factor's levels, say, mydata$myfactor. So what I want is to split
this
> dataframe into nlevels(mydata$myfactor) = 80 levels. But I must do this
> efficiently, that is, I can't actually split the dataframe and then
save the
> resulting 80 smaller ones, as I would run out of memory. I don't know
how to
> do it though.
>
> So, can anyone give a hint on how to get to get one smaller dataframe, save
> it into a file, remove it from the workspace, get the next smaller
> dataframe, save it into another file, remoe it from the workspace, and so
> on?
>
> Thank you,
>
> Dimitri
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
>