R devel - May 2005 - How to understand packages, namespaces, environments

I would be very glad of pointers to information on how the concepts of 
packages, namespaces and environments are interrelated in R.

I am trying to get a handle on this both so I can delve further into 
understanding other people's code and so I can organize my own in a 
more coherent manner.

 From my reading about environments it seems they function as what I 
would intuitively call namespaces. However, the documentation for the 
'library' function implies that "namespace" has a specific
meaning in R
that I have so far failed to grasp. What is that meaning (or where 
should I look to read up on it) ?

I can think of the following more specific questions. Perhaps they may 
most usefully serve to reveal my misconceptions, so corrections would 
be very helpful.

Does saying the following load package 'foo' into its own environment ?
 > library(foo)

Do environments have names?

Of what does the list returned by search() actually consist? Is it a 
list of environments, a list that may include environments, or 
something else?

What is actually designated by a character vector of form
"package:foo"?

In what ways can I use the character vector "package:foo" when 
interacting with R?

Many thanks,
Alex Stoddard

P.S. It took me a long time of flailing about to discover the 'search' 
function. Perhaps it could be included in the "See Also" section of
the
help for 'ls'.

On Fri, 6 May 2005, Alexander Stoddard wrote:
> I would be very glad of pointers to information on how the concepts of 
> packages, namespaces and environments are interrelated in R.
>
> I am trying to get a handle on this both so I can delve further into 
> understanding other people's code and so I can organize my own in a
more
> coherent manner.
>
> From my reading about environments it seems they function as what I would 
> intuitively call namespaces. However, the documentation for the
'library'
> function implies that "namespace" has a specific meaning in R
that I have so
> far failed to grasp. What is that meaning (or where should I look to read
up
> on it) ?
Look in `Writing R Extensions' and on developer.r-project.org.  I don't 
know about `intuitively' but they are similar to other systems (e.g. 
Perl's) namespaces.

More user-level documentation of namespaces (e.g. in R-lang) is on the 
WOULD-BE-NICE list, but there is enough in `Writing R Extensions' for many 
package writers and we are talking about more details of what happens 
under the skin.
> I can think of the following more specific questions. Perhaps they may most
> usefully serve to reveal my misconceptions, so corrections would be very 
> helpful.
It seems you have revealed that you have not studied the manuals nor the 
help pages.  Please do so before posting (see the posting guide).
> Does saying the following load package 'foo' into its own
environment ?
>> library(foo)
>
> Do environments have names?
Not in general.  You will see <environment: some-hex-address> printed 
quite often.  I am not at all sure you have grasped what an environment is 
in R: see the R-lang manual.  (Look it up in the index, as at present the 
term is used before it is defined.)
> Of what does the list returned by search() actually consist? Is it a list
of
> environments, a list that may include environments, or something else?
search() does not return a list: it returns a character vector.  It seems 
you have not read its help page, which tells you that quite explicitly.
> What is actually designated by a character vector of form
"package:foo"?
That is actually a character string, not a character vector.  It is a name 
that can be used to refer to attach'ed objects, as the search() help page 
says.
> In what ways can I use the character vector "package:foo" when
interacting
> with R?
I am aware of attach(), detach() and as.environment() (which is used by 
quite a few functions, e.g. get()).  (Asking what you can do it an 
extensible system such as R is not very helpful.)  Generally it is a name 
used when a package or another attach'ed object might be meant: it is not 
used if a package is all that can be specified.
> Many thanks,
> Alex Stoddard
>
> P.S. It took me a long time of flailing about to discover the
'search'
> function. Perhaps it could be included in the "See Also" section
of the help
> for 'ls'.
This *is* in `An Introduction to R', something we ask all R users to read.

I (and I am biased) suggest the right way to get to grips with things like 
this is to read a good book on the S/R language.

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

Alexander Stoddard wrote:
> I would be very glad of pointers to information on how the concepts of 
> packages, namespaces and environments are interrelated in R.
I don't think that documentation exists yet, but I did update the 
documentation on environments for 2.1.0.  Look in the section (2.1.10 I 
think) on Environment Objects in the R language definition manual. 
Basically environments are things where you can ask to look up names and 
R will return R objects to you.  They're a bit more complicated than 
just a list of (name, object) pairs, but can (almost) be used as that 
(and I'm hoping will be able to be used that way in 2.2.0, but that 
remains to be seen).
> I am trying to get a handle on this both so I can delve further into 
> understanding other people's code and so I can organize my own in a
more
> coherent manner.
> 
>  From my reading about environments it seems they function as what I 
> would intuitively call namespaces. However, the documentation for the 
> 'library' function implies that "namespace" has a
specific meaning in R
> that I have so far failed to grasp. What is that meaning (or where 
> should I look to read up on it) ?
A namespace is a more abstract concept, which happens to be implemented 
in R using environments. Exactly how is a bit of a mystery to me, I'm 
afraid!  However, the basic idea is that a namespace defines a fixed set 
of meanings for names, which the code in a package can assume won't 
change.  It may also hide some names from outsiders.
> 
> I can think of the following more specific questions. Perhaps they may 
> most usefully serve to reveal my misconceptions, so corrections would be 
> very helpful.
> 
> Does saying the following load package 'foo' into its own
environment ?
>  > library(foo)
This loads some of the (name, object) pairs from the package into two 
environments:  the public one the user can see, and the namespace one 
that the code in the package can see.  They're related, you don't get 
two copies of the objects.
> 
> Do environments have names?
They are R objects, so some of them have names, and some of them are 
anonymous parts of other objects.
> 
> Of what does the list returned by search() actually consist? Is it a 
> list of environments, a list that may include environments, or something 
> else?
It's a list of strings that can be used to get corresponding 
environments.  You don't get the actual
environments.
> 
> What is actually designated by a character vector of form
"package:foo"?
That string is used by the as.environment() function to go through the 
search list and obtain the environment.  For instance, you can do

x <- as.environment("package:foo")

and x will be a reference to the environment.  Then

ls(x)

will list all the objects there, get('bar', envir=x)  will extract the 
object named 'bar' from it, etc.
> 
> In what ways can I use the character vector "package:foo" when 
> interacting with R?
I can't think of any other uses than in as.environment(), but there 
might be some, or some automatic calls to it.  I'd have to check the man 
pages to be sure, but perhaps some functions that accept envir=x type 
arguments would automatically call as.environment on x first.
> Many thanks,
> Alex Stoddard
> 
> P.S. It took me a long time of flailing about to discover the
'search'
> function. Perhaps it could be included in the "See Also" section
of the
> help for 'ls'.
That's a good suggestion.  Could you do me a favour, and as you're 
reading up on this stuff identify a whole list of deficiencies in the 
documentation and suggested fixes for them?  Making a change to a man 
page has a certain fixed overhead (making sure there are no syntax 
errors, committing, possibly porting the change to the patch branch), so 
batching them is efficient.

Duncan Murdoch

[Alexander Stoddard]
> Subject: Re: [Rd] How to understand packages, namespaces, environments
> 
> > 
> > Does saying the following load package 'foo' into its own 
> environment ?
> >  > library(foo)
> 
[Duncan Murdoch]
> This loads some of the (name, object) pairs from the package into two
> environments:  the public one the user can see, and the 
> namespace one that the code in the package can see.  They're 
> related, you don't get two copies of the objects.
That's interesting-- I thought there really were two copies. In my debug
package, I've taken the approach of changing both copies. Is one of the
copies a "master", and the other one something like an activeBinding?
Can I get away with changing just one of them?

Mark Bravington

> On 5/8/05, Mark.Bravington@csiro.au <Mark.Bravington@csiro.au> wrote:
> > [Alexander Stoddard]
> > > Subject: Re: [Rd] How to understand packages, namespaces, 
> > > environments
> > >
> > > >
> > > > Does saying the following load package 'foo' into
its own
> > > environment ?
> > > >  > library(foo)
> > >
> > [Duncan Murdoch]
> > > This loads some of the (name, object) pairs from the package into
> > > two
> > > environments:  the public one the user can see, and the namespace
> > > one that the code in the package can see.  They're related,
you
> > > don't get two copies of the objects.
> >
[MVB] 
> > That's interesting-- I thought there really were two copies. In my
> > debug package, I've taken the approach of changing both copies.
> 
[Gabor Grothendieck]
> How does one refer to the two different copies? Thanks.
> 
> 
The help for fun.locator (in the debug package) contains my
interpretation of what's going on (which might be wrong, but seems to
work). A slightly simplified version of the guts of fun.locator is as
follows:

  # function to check for something called fname
  is.here <- function( env) exists( fname, env=env, inherits=FALSE)

  # Search path
  search.envs <- lapply( 1:length( search()), pos.to.env)
  ff <- search.envs[ sapply( search.envs, is.here)]
 
  # Hidden namespace environments
  ln <- lapply( loadedNamespaces(), asNamespace)
  ff <- c( ff, ln[ sapply( ln, is.here)]) 
  
  # S3 methods:
  S3 <- lapply( ln, function( x) if( exists( '.__S3MethodsTable__.',
x,
inherits=FALSE)) 
    get( '.__S3MethodsTable__.', x) else 0)
  S3 <- S3[ !sapply( S3, is.numeric)]
  ff <- c( ff, S3[ sapply( S3, is.here)])

Then ff is a list of environments where a copy (?) of fname exists-- and
you can use
 ff[[i]][[fname]] 
or 
 get(fname, env=ff[[i]]) & assign( fname, ..., env=ff[[i]])

I should point out that the doco for asNamespace says "not intended to
be called directly"-- but I couldn't see an alternative, and anyway it
seems to work (for now).

Mark