R devel - Apr 1997 - R-beta: Re: S Compatibility

Z. Todd Taylor writes:
> I have a question/comment on this topic.  If I understand what
> I've read, R's lexical scoping rules are what require that all
> of R's data be held in memory (as opposed to S's method of
> storing each object on disk and reading it as often as
> necessary).
> My question is whether there will ever be a way around having to
> store everything in memory.  I make heavy use of S's
> user-defined database mechanism to gain transparent access to
> some huge datasets.  I store tens of thousands of
> individually-compressed S objects that I can access
> transparently via my user-defined database.
> In S, I attach my user-defined database(s) and all those objects
> are available; but they don't use any memory until I actually
> use them.  Obviously, I rarely access all of them in one
> session, but usually access more than I could hold in memory
> at once.
> So, how could I do this in R?

You can't (yet).  Here is a possible approach I have been thinking
about (thinking not doing).  Lazy evaluation is implemented using
"promise-to-evaluate" objects.  We could use something quite similar
(perhaps identical) to implement what you want.  We will introduce
"promise-to-fetch" kind of object which contains information on how to
fetch an object and a slot which would point to the real object if is
currently in memory.  When the evaluator encounters such an object it
checks the value slot to see if the value is present.  If it is it
returns it, otherwise it uses the "fetch" recipe to first bring it
into
memory and then it returns it.

When memory runs low, the real objects could be freed and the recipes
retained.  How this would work (timestamps?) and how it would work with
the garbage collector is an open question.

Because the objects are referenced through these "promise" gizmos,
scoping is no problem.

Its something we definitely need (almost as much as REAL libraries).
If only there were another 24 hours in each day ...
	Ross
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-r-help
mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Ross Ihaka <ihaka@stat.auckland.ac.nz> writes:

[on out-of-memory data]
> 
> Its something we definitely need (almost as much as REAL libraries).
> If only there were another 24 hours in each day ...
No problem. You just need someone as smart as you are to chip in with
the development. Or four people half as smart ;^)

There might also be some more "brutal" approaches, such as having the
vector heap as a memory-mapped file, leaving the cacheing and similar
work to the operating system. OS's are already quite good at handling
files with holes in them, so it's not like allocating a huge file just
for a handful af variables. Some special routines for defragmentation
might be necessary, though. Not that I really know what I'm talking
about...

At any rate, it would probably be a good thing to start thinking in
terms of modularity, database backends and such.

I've moved the thread to r-devel, it seemed to be getting too
technical for r-help.

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard@biostat.ku.dk)             FAX: (+45) 35327907
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To:
r-devel-request@stat.math.ethz.ch
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-