R help - May 2000 - environments and lexical scoping (conceptual issues)

Dear R people,

I have been reading about R in V&R''s `S programming'',
specifically Section
3.4, entitled `Databases, frames, and environments''.

I have found it useful in helping me understand things which I had
previously found puzzling. However, there are still some things I don''t
fully understand.  Could some kind person help me clarify them? For
instance, I am having difficulties with the latter part of the
following passage (pgs 63,64), which is entirely about R. Quoting:

VR> `Lexical scoping'' comes into play when we consider how R finds
objects
VR> to associate with names when evaluating a function. In principle the
VR> answer is easy: it searches the `environment'',that is in the
frame of
VR> the environment, then in the frame of the enclosing environment, then
VR> in the frame of the enclosing environment of the enclosing environment,
VR> and so on. In less precise language, it searches up the tree of 
VR> environments until it finds a match or reaches the root, which will be
VR> the user''s workspace, and then along the search path.

Ok, clear so far.

VR> So, we need to know where the evaluation environment is on the tree of
VR> environments. When the evaluation environment is a function body, the
VR> enclosing environment is that of the function, which is normally the
VR> environment within which the function was defined. This is what is
VR> meant by lexical scoping.

Still clear...

VR> The enclosing environment need not be the parent environment, the 
VR> environment from which the evaluation environment was created, and
VR> [the enclosing environment (my addition)] is usually the environment
VR> within which the function was created, often the workspace.

?? I don''t understand. If "the enclosing environment need not be
the
parent environment, from which the evaluation environment was created"
then this seems to contradict the previous para, where it says "When the
evaluation environment is a function body, the enclosing environment is
that of the function, which is normally the environment within which the
function was defined." Does the last para refer to situations other than
functions defined within functions? If so, an example of this would be
helpful in understanding what is meant. And if the enclosing environment
is not the parent environment, then how is the enclosing environment
chosen? 

VR> Notice that the end effect is somewhat similar to that of S; the 
VR> frames of the parents of the function are not normally searched, so
VR> the search goes from the local environment to the user''s
workplace, and
VR> then along the search path given by search() (as the enclosing
VR> environment of all functions in packages is set to the workspace).
VR> However, the scope is different for functions defined within other
VR> functions. (A function''s environment can be changed by using
VR> environment as a replacement function, a complication we will not
VR> explore here.)

This seems to confirm that functions defined within other functions *are*
treated differently, ie. by lexical scoping, whereas environments created
by other means are not necessarily. 

The phrase "(as the enclosing environment of all functions in packages is
set to the workspace)" is also confusing me. Is the enclosing environment
in this case being forced to be the user''s workplace, and is this an
example of the enclosing environment not being the parent environment?

The discussion in `S Programming'' is by far the most detailed of these
important concepts that I have seen, but it would be even more useful if
it were made more wordy and more (simple) examples were given, so as to
make misunderstandings more difficult. I have read the preceding passage a
number of times, but as you can see its meaning is still not fully clear
to me, and I have not been able to find any documentation that sheds
further light on the concepts discussed here.
                                                      Faheem. 


-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Faheem Mitha <faheem at email.unc.edu> writes:
> VR> The enclosing environment need not be the parent environment, the 
> VR> environment from which the evaluation environment was created, and
> VR> [the enclosing environment (my addition)] is usually the environment
> VR> within which the function was created, often the workspace.
> 
> ?? I don''t understand. If "the enclosing environment need not
be the
> parent environment, from which the evaluation environment was created"
> then this seems to contradict the previous para, where it says "When
the
> evaluation environment is a function body, the enclosing environment is
> that of the function, which is normally the environment within which the
> function was defined." Does the last para refer to situations other
than
> functions defined within functions? If so, an example of this would be
> helpful in understanding what is meant. And if the enclosing environment
> is not the parent environment, then how is the enclosing environment
> chosen? 
This is crucial. You have to distinguish between the *defining*
environment and the *calling* environment. 

 x<-5
 f<-function()x 
 g<-function(){x<-2;f()}
 g() # == 5

The converse would be

 g<-function(){x<-2;function()x}
 h<-g()
 x<-5
 h() # == 2

There are situations beyond defining functions inside other functions,
for instance 

 k <- evalq(function(n)x+n, environment(h))
 k(2) # == 4

> VR> Notice that the end effect is somewhat similar to that of S; the 
> VR> frames of the parents of the function are not normally searched, so
> VR> the search goes from the local environment to the user''s
workplace, and
> VR> then along the search path given by search() (as the enclosing
> VR> environment of all functions in packages is set to the workspace).
> VR> However, the scope is different for functions defined within other
> VR> functions. (A function''s environment can be changed by
using
> VR> environment as a replacement function, a complication we will not
> VR> explore here.)
> 
> This seems to confirm that functions defined within other functions *are*
> treated differently, ie. by lexical scoping, whereas environments created
> by other means are not necessarily.
Not really. Environments are always lexically scoped. The enclosure is
the environment in which the environment was defined. Functions are
special only in the sense that they carry the environment in which
they are defined (this is the concept of "function closures"). When
you are defining a function within another function that environment
is the execution environment of the function that calls "function",
unless you explicitly set the environment differently using e.g. eval().
> The phrase "(as the enclosing environment of all functions in packages
is
> set to the workspace)" is also confusing me. Is the enclosing
environment
> in this case being forced to be the user''s workplace, and is this
an
> example of the enclosing environment not being the parent environment?
Exactly. Have a look at library(). Notice that packages are loaded
with

 sys.source(file, env, keep.source = keep.source) 

This makes all loaded functions be defined in "env", but it has the
side effect that their enclosing environment is also "env". That is
not practical because then a function in a package cannot "see"
definitions in the global environment or for that matter in later
packages, hence the chasing

 .Internal(lib.fixup(env, .GlobalEnv))
> The discussion in `S Programming'' is by far the most detailed of
these
> important concepts that I have seen, but it would be even more useful if
> it were made more wordy and more (simple) examples were given, so as to
> make misunderstandings more difficult. I have read the preceding passage a
> number of times, but as you can see its meaning is still not fully clear
> to me, and I have not been able to find any documentation that sheds
> further light on the concepts discussed here.
Also have a look at:

Ihaka & Gentleman (1996), "R: A Language for Data Analysis and
Graphics", Journal of Computational and Graphical Statistics, 5,
299-314.

There''s an R language manual on the drawing board, and it is supposed
to contain material of this caliber, but writing it is a slow process.
(As you will have noticed, it is necessary to be both comprehensive
and comprehensible, and achieving both goals can be difficult.)

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /''_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)             FAX: (+45) 35327907
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

On Thu, 18 May 2000, Faheem Mitha wrote:
> 
> VR> The enclosing environment need not be the parent environment, the 
> VR> environment from which the evaluation environment was created, and
> VR> [the enclosing environment (my addition)] is usually the environment
> VR> within which the function was created, often the workspace.
> 
> ?? I don''t understand. If "the enclosing environment need not
be the
> parent environment, from which the evaluation environment was created"
> then this seems to contradict the previous para, where it says "When
the
> evaluation environment is a function body, the enclosing environment is
> that of the function, which is normally the environment within which the
> function was defined." Does the last para refer to situations other
than
> functions defined within functions? If so, an example of this would be
> helpful in understanding what is meant. And if the enclosing environment
> is not the parent environment, then how is the enclosing environment
> chosen? 
There are two environments: the one that the function was defined in
("enclosing"), and the one it was invoked in ("parent")

If you create a function at the command line or load it in a package
it''s
enclosing environment is the global workspace.   If you define a function
f() inside another function g() its enclosing environment
is the environment inside g(). The enclosing environment for a function is
fixed when the function is created.  You can find out the enclosing
environment for a function f() using 
	environment(f)

The "parent" environment, on the other hand, is defined when you
invoke a
function.  If you invoke lm() at the command line its parent environment
is the global workspace, if you invoke it inside a function f() then its
parent environment is the environment inside f().  You can find out the
parent environment for an invocation of a function by using
sys.frame(sys.parent()).


So for most user-visible functions the enclosing environment will be the
global workspace, since that is where most functions are defined.  The
parent environment will be wherever the function happens to be called
from.

If a function f() is defined inside another function g() it will probably
be used inside g() as well, so in fact its parent environment and
enclosing environment will be the same.


Parent environments are important is that things like model formulas need
to be evaluated in the environment the function was called from, since
that''s where all the variables will be available. This relies on the
parent environment being potentially different with each invocation.

Enclosing environments are important because a function can use variables
in the enclosing environment to share information with other functions or
with other invocations of itself. This relies on the enclosing environment
being the same each time the function is invoked.

Scoping *is* hard.  Looking at examples helps, and I would particularly
recommend playing around with all the examples in that chapter of VR. It
is particularly instructive to look at examples that work differently in
R and S and try to see why they differ.

	-thomas

Thomas Lumley
Assistant Professor, Biostatistics
University of Washington, Seattle

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

On Thu, 18 May 2000, Faheem Mitha wrote:
> Dear R people,
> 
> I have been reading about R in V&R''s `S programming'',
specifically Section
> 3.4, entitled `Databases, frames, and environments''.
> The discussion in `S Programming'' is by far the most detailed of
these
> important concepts that I have seen, but it would be even more useful if
> it were made more wordy and more (simple) examples were given, so as to
> make misunderstandings more difficult. I have read the preceding passage a
> number of times, but as you can see its meaning is still not fully clear
> to me, and I have not been able to find any documentation that sheds
> further light on the concepts discussed here.
Well, page 55 does say it is hard work, and there are extensive examples
in that chapter and in chapter 7.  Thomas Lumley and Peter Dalgaard
have I hope shed some further light for you.

One problem with writing about R is that it changes, and it changed a lot
while that section was being written (including in notation and 
in part because of difficulties we found in describing it).

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._