R help - Dec 2009 - how to append new data to saved data on disk efficiently

Hi,

I currently combine multiple processed data (data frame) into a list and
save the list as ".rda" using the save command. When new data come, I
load
the rda file, process the new data into a data frame, append the data
frame to the end of the list, and save the whole list to the disk. The
loading and saving steps are quite time consuming. Since I don't need to
change the old data in the list when new data come, I wonder if there is
an efficient way to update the rda efficiently, e.g. appending the new
data to the rda file directly without loading the whole rda file?

Any suggestions would be appreciated.

thanks

Jeff

You can keep each dataframe as a separate file and process them that way.
You can look into storing in a relational database or using filehash.  It
all depends on how you want to process the data later.

On Tue, Dec 29, 2009 at 8:40 AM, Hao Cen <hcen@andrew.cmu.edu> wrote:
> Hi,
>
> I currently combine multiple processed data (data frame) into a list and
> save the list as ".rda" using the save command. When new data
come, I load
> the rda file, process the new data into a data frame, append the data
> frame to the end of the list, and save the whole list to the disk. The
> loading and saving steps are quite time consuming. Since I don't need
to
> change the old data in the list when new data come, I wonder if there is
> an efficient way to update the rda efficiently, e.g. appending the new
> data to the rda file directly without loading the whole rda file?
>
> Any suggestions would be appreciated.
>
> thanks
>
> Jeff
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
>
http://www.R-project.org/posting-guide.html<http://www.r-project.org/posting-guide.html>
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem that you are trying to solve?

	[[alternative HTML version deleted]]