R devel - Jan 2015 - RFC: getifexists() {was [Bug 16065] "exists" ...}

On Thu, Jan 8, 2015 at 11:57 AM, <luke-tierney at uiowa.edu> wrote:
> On Thu, 8 Jan 2015, Michael Lawrence wrote:
>
>  If we do add an argument to get(), then it should be named consistently
>> with the ifnotfound argument of mget(). As mentioned, the possibility
of a
>> NULL value is problematic. One solution is a sentinel value that
indicates
>> an unbound value (like R_UnboundValue).
>>
>
> A null default is fine -- it's a default; if it isn't right for a
> particular case you can provide something else.
>
>
>> But another idea (and one pretty similar to John's) is to follow
the
>> SYMSXP
>> design at the C level, where there is a structure that points to the
name
>> and a value. We already have SYMSXPs at the R level of course (name
>> objects) but they do not provide access to the value, which is
typically
>> R_UnboundValue. But this does not even need to be implemented with
SYMSXP.
>> The design would allow something like:
>>
>> binding <- getBinding("x", env)
>> if (hasValue(binding)) {
>>  x <- value(binding) # throws an error if none
>>  message(name(binding), "has value", x)
>> }
>>
>> That I think it is a bit verbose but readable and could be made fast.
And
>> I
>> think binding objects would be useful in other ways, as they are
>> essentially a "named object". For example, when iterating
over an
>> environment.
>>
>
> This would need a lot more thought. Directly exposing the internals is
> definitely not something we want to do as we may well want to change
> that design. But there are lots of other corner issues that would have
> to be thought through before going forward, such as what happens if an
> rm occurs between obtaining a binding object and doing something with
> it. Serialization would also need thinking through. This doesn't seem
> like a worthwhile place to spend our efforts to me.
>
>
Just wanted to be clear that I was not suggesting to expose any internals.
We could implement the behavior using SYMSXP, or not. Nor would the binding
need to be mutable. The binding would be considered independent of the
environment from which it was retrieved. As Pete has mentioned, it could be
a useful abstraction to have in general.

> Adding getIfExists, or .get, or get0, or whatever seems fine. Adding
> an argument to get() with missing giving current behavior may be OK
> too. Rewriting exists and get as .Primitives may be sufficient though.
>
> Best,
>
> luke
>
>
>  Michael
>>
>>
>>
>>
>> On Thu, Jan 8, 2015 at 6:03 AM, John Nolan <jpnolan at
american.edu> wrote:
>>
>>  Adding an optional argument to get (and mget) like
>>>
>>> val <- get(name, where, ..., value.if.not.found=NULL )   (*)
>>>
>>> would be useful for many.  HOWEVER, it is possible that there could
be
>>> some confusion here: (*) can give a NULL because either x exists
and
>>> has value NULL, or because x doesn't exist.   If that matters,
the user
>>> would need to be careful about specifying a value.if.not.found that
>>> cannot
>>> be confused with a valid value of x.
>>>
>>> To avoid this difficulty, perhaps we want both: have Martin's
>>> getifexists(
>>> )
>>> return a list with two values:
>>>   - a boolean variable 'found'  # = value returned by
exists( )
>>>   - a variable 'value'
>>>
>>> Then implement get( ) as:
>>>
>>> get <- function(x,...,value.if.not.found ) {
>>>
>>>   if( missing(value.if.not.found) ) {
>>>     a <- getifexists(x,... )
>>>     if (!a$found) error("x not found")
>>>   } else {
>>>     a <- getifexists(x,...,value.if.not.found )
>>>   }
>>>   return(a$value)
>>> }
>>>
>>> Note that value.if.not.found has no default value in above.
>>> It behaves exactly like current get does if value.if.not.found
>>> is not specified, and if it is specified, it would be faster
>>> in the common situation mentioned below:
>>>      if(exists(x,...)) { get(x,...) }
>>>
>>> John
>>>
>>> P.S. if you like dromedaries call it valueIfNotFound ...
>>>
>>>  ..............................................................
>>>  John P. Nolan
>>>  Math/Stat Department
>>>  227 Gray Hall,   American University
>>>  4400 Massachusetts Avenue, NW
>>>  Washington, DC 20016-8050
>>>
>>>  jpnolan at american.edu       voice: 202.885.3140
>>>  web: academic2.american.edu/~jpnolan
>>>  ..............................................................
>>>
>>>
>>> -----"R-devel" <r-devel-bounces at r-project.org>
wrote: -----
>>> To: Martin Maechler <maechler at stat.math.ethz.ch>, R-devel
at r-project.org
>>> From: Duncan Murdoch
>>> Sent by: "R-devel"
>>> Date: 01/08/2015 06:39AM
>>> Subject: Re: [Rd] RFC: getifexists() {was [Bug 16065]
"exists" ...}
>>>
>>> On 08/01/2015 4:16 AM, Martin Maechler wrote:
>>> > In November, we had a "bug repository conversation"
>>> > with Peter Hagerty and myself:
>>> >
>>> >   https://bugs.r-project.org/bugzilla/show_bug.cgi?id=16065
>>> >
>>> > where the bug report title started with
>>> >
>>> >  --->>  "exists" is a bottleneck for dispatch
and package loading, ...
>>> >
>>> > Peter proposed an extra simplified and henc faster version of
exists(),
>>> > and I commented
>>> >
>>> >     > --- Comment #2 from Martin Maechler <maechler at
stat.math.ethz.ch>
>>> ---
>>> >     > I'm very grateful that you've started
exploring the bottlenecks
>>> of
>>> loading
>>> >     > packages with many S4 classes (and methods)...
>>> >     > and I hope we can make real progress there rather
sooner than
>>> later.
>>> >
>>> >     > OTOH, your `summaryRprof()` in your vignette
indicates that
>>> exists() may use
>>> >     > upto 10% of the time spent in
library(reportingTools),  and your
>>> speedup
>>> >     > proposals of exist()  may go up to ca 30%  which is
good and well
>>> worth
>>> >     > considering,  but still we can only expect 2-3%
speedup for
>>> package loading
>>> >     > which unfortunately is not much.
>>> >
>>> >     > Still I agree it is worth looking at exists() as you
did  ... and
>>> >     > consider providing a fast simplified version of it in
addition to
>>> current
>>> >     > exists() [I think].
>>> >
>>> >     > BTW, as we talk about enhancements here, maybe
consider a further
>>> possibility:
>>> >     > My subjective guess is that probably more than half
of exists()
>>> uses are of the
>>> >     > form
>>> >
>>> >     > if(exists(name, where, .......)) {
>>> >     >    get(name, whare, ....)
>>> >     >    ..
>>> >     > } else {
>>> >     >     NULL / error() / .. or similar
>>> >     > }
>>> >
>>> >     > i.e. many exists() calls when returning TRUE are
immediately
>>> followed by the
>>> >     > corresponding get() call which repeats quite a bit of
the lookup
>>> that exists()
>>> >     > has done.
>>> >
>>> >     > Instead, I'd imagine a function, say 
getifexists(name, ...) that
>>> does both at
>>> >     > once in the "exists is TRUE" case but in a
way we can easily keep
>>> the if(.) ..
>>> >     > else clause above.  One already existing approach
would use
>>> >
>>> >     > if(!inherits(tryCatch(xx <- get(name, where, ...),
>>> error=function(e)e), "error")) {
>>> >
>>> >     >   ... (( work with xx )) ...
>>> >
>>> >     > } else  {
>>> >     >    NULL / error() / .. or similar
>>> >     > }
>>> >
>>> >     > but of course our C implementation would be more
efficient and
>>> use
>>> more concise
>>> >     > syntax {which should not look like error handling}.  
Follow ups
>>> to this idea
>>> >     > should really go to R-devel (the mailing list).
>>> >
>>> > and now I do follow up here myself :
>>> >
>>> > I found that  'getifexists()' is actually very simple
to implement,
>>> > I have already tested it a bit, but not yet committed to
R-devel
>>> > (the "R trunk" aka "master branch")
because I'd like to get
>>> > public comments {RFC := Request For Comments}.
>>> >
>>>
>>> I don't like the name -- I'd prefer getIfExists.  As Baath
(2012, R
>>> Journal) pointed out, R names are very inconsistent in naming
>>> conventions, but lowerCamelCase is the most common choice.  Second
most
>>> common is period.separated, so an argument could be made for
>>> get.if.exists, but there's still the possibility of confusion
with S3
>>> methods, and users of other languages where "." is an
operator find it a
>>> little strange.
>>>
>>> If you don't like lowerCamelCase (and a lot of people
don't), then I
>>> think underscore_separated is the next best choice, so would use
>>> get_if_exists.
>>>
>>> Another possibility is to make no new name at all, and just add an
>>> optional parameter to get() (which if present acts as your
value.if.not
>>> parameter, if not present keeps the current "object not
found" error).
>>>
>>> Duncan Murdoch
>>>
>>>
>>> > My version of the help file {for both exists() and
getifexists()}
>>> > rendered in text is
>>> >
>>> > ---------------------- help(getifexists)
------------------------------
>>> -
>>> > Is an Object Defined?
>>> >
>>> > Description:
>>> >
>>> >      Look for an R object of the given name and possibly
return it
>>> >
>>> > Usage:
>>> >
>>> >      exists(x, where = -1, envir = , frame, mode =
"any",
>>> >             inherits = TRUE)
>>> >
>>> >      getifexists(x, where = -1, envir = as.environment(where),
>>> >                  mode = "any", inherits = TRUE,
value.if.not = NULL)
>>> >
>>> > Arguments:
>>> >
>>> >        x: a variable name (given as a character string).
>>> >
>>> >    where: where to look for the object (see the details
section); if
>>> >           omitted, the function will search as if the name of
the
>>> >           object appeared unquoted in an expression.
>>> >
>>> >    envir: an alternative way to specify an environment to look
in, but
>>> >           it is usually simpler to just use the ?where?
argument.
>>> >
>>> >    frame: a frame in the calling list.  Equivalent to giving
?where? as
>>> >           ?sys.frame(frame)?.
>>> >
>>> >     mode: the mode or type of object sought: see the ?Details?
section.
>>> >
>>> > inherits: should the enclosing frames of the environment be
searched?
>>> >
>>> > value.if.not: the return value of ?getifexists(x, *)? when ?x?
does not
>>> >           exist.
>>> >
>>> > Details:
>>> >
>>> >      The ?where? argument can specify the environment in which
to look
>>> >      for the object in any of several ways: as an integer (the
position
>>> >      in the ?search? list); as the character string name of an
element
>>> >      in the search list; or as an ?environment? (including
using
>>> >      ?sys.frame? to access the currently active function
calls).  The
>>> >      ?envir? argument is an alternative way to specify an
environment,
>>> >      but is primarily there for back compatibility.
>>> >
>>> >      This function looks to see if the name ?x? has a value
bound to it
>>> >      in the specified environment.  If ?inherits? is ?TRUE?
and a value
>>> >      is not found for ?x? in the specified environment, the
enclosing
>>> >      frames of the environment are searched until the name ?x?
is
>>> >      encountered.  See ?environment? and the ?R Language
Definition?
>>> >      manual for details about the structure of environments
and their
>>> >      enclosures.
>>> >
>>> >      *Warning:* ?inherits = TRUE? is the default behaviour for
R but
>>> >      not for S.
>>> >
>>> >      If ?mode? is specified then only objects of that type are
sought.
>>> >      The ?mode? may specify one of the collections
?"numeric"? and
>>> >      ?"function"? (see ?mode?): any member of the
collection will
>>> >      suffice.  (This is true even if a member of a collection
is
>>> >      specified, so for example ?mode = "special"?
will seek any type of
>>> >      function.)
>>> >
>>> > Value:
>>> >
>>> >      ?exists():? Logical, true if and only if an object of the
correct
>>> >      name and mode is found.
>>> >
>>> >      ?getifexists():? The object-as from ?get(x, *)?- if
?exists(x, *)?
>>> >      is true, otherwise ?value.if.not?.
>>> >
>>> > Note:
>>> >
>>> >    With ?getifexists()?, instead of the easy to read but
somewhat
>>> >    inefficient
>>> >
>>> >        if (exists(myVarName, envir = myEnvir)) {
>>> >          r <- get(myVarName, envir = myEnvir)
>>> >          ## ... deal with r ...
>>> >        }
>>> >
>>> >    you now can use the more efficient (and slightly harder to
read)
>>> >
>>> >        if (!is.null(r <- getifexists(myVarName, envir =
myEnvir))) {
>>> >          ## ... deal with r ...
>>> >        }
>>> >
>>> > References:
>>> >
>>> >      Becker, R. A., Chambers, J. M. and Wilks, A. R. (1988)
_The New S
>>> >      Language_.  Wadsworth & Brooks/Cole.
>>> >
>>> > See Also:
>>> >
>>> >      ?get?.  For quite a different kind of ?existence?
checking, namely
>>> >      if function arguments were specified, ?missing?; and for
yet a
>>> >      different kind, namely if a file exists, ?file.exists?.
>>> >
>>> > Examples:
>>> >
>>> >      ##  Define a substitute function if necessary:
>>> >      if(!exists("some.fun", mode =
"function"))
>>> >        some.fun <- function(x) {
cat("some.fun(x)\n"); x }
>>> >      search()
>>> >      exists("ls", 2) # true even though ls is in pos
= 3
>>> >      exists("ls", 2, inherits = FALSE) # false
>>> >
>>> >      ## These are true (in most circumstances):
>>> >      identical(ls,   getifexists("ls"))
>>> >      identical(NULL, getifexists(".foo.bar.")) #
default value.if.not >>> NULL(!)
>>> >
>>> > ----------------- end[ help(getifexists) ]
>>> -----------------------------
>>> >
>>> > ______________________________________________
>>> > R-devel at r-project.org mailing list
>>> > https://stat.ethz.ch/mailman/listinfo/r-devel
>>> >
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>
>>> ______________________________________________
>>> R-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>
>>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> --
> Luke Tierney
> Ralph E. Wareham Professor of Mathematical Sciences
> University of Iowa                  Phone:             319-335-3386
> Department of Statistics and        Fax:               319-335-3017
>    Actuarial Science
> 241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
	[[alternative HTML version deleted]]

>>>>> Michael Lawrence <lawrence.michael at gene.com>
>>>>>     on Thu, 8 Jan 2015 14:02:26 -0800 writes:
    > On Thu, Jan 8, 2015 at 11:57 AM, <luke-tierney at uiowa.edu>
wrote:
    >> On Thu, 8 Jan 2015, Michael Lawrence wrote:
    >> 
    >> If we do add an argument to get(), then it should be named
consistently
    >>> with the ifnotfound argument of mget(). As mentioned, the
possibility of a
    >>> NULL value is problematic. One solution is a sentinel value
that indicates
    >>> an unbound value (like R_UnboundValue).
    >>> 
    >> 
    >> A null default is fine -- it's a default; if it isn't right
for a
    >> particular case you can provide something else.
    >> 
    >> 
    >>> But another idea (and one pretty similar to John's) is to
follow the
    >>> SYMSXP
    >>> design at the C level, where there is a structure that points
to the name
    >>> and a value. We already have SYMSXPs at the R level of course
(name
    >>> objects) but they do not provide access to the value, which is
typically
    >>> R_UnboundValue. But this does not even need to be implemented
with SYMSXP.
    >>> The design would allow something like:
    >>> 
    >>> binding <- getBinding("x", env)
    >>> if (hasValue(binding)) {
    >>>   x <- value(binding) # throws an error if none
    >>>   message(name(binding), "has value", x)
    >>> }
    >>> 
    >>> That I think it is a bit verbose but readable and could be made
fast. And
    >>> I
    >>> think binding objects would be useful in other ways, as they
are
    >>> essentially a "named object". For example, when
iterating over an
    >>> environment.
    >>> 
    >> 
    >> This would need a lot more thought. Directly exposing the internals
is
    >> definitely not something we want to do as we may well want to
change
    >> that design. But there are lots of other corner issues that would
have
    >> to be thought through before going forward, such as what happens if
an
    >> rm occurs between obtaining a binding object and doing something
with
    >> it. Serialization would also need thinking through. This
doesn't seem
    >> like a worthwhile place to spend our efforts to me.
    >> 
    >> 

    > Just wanted to be clear that I was not suggesting to expose any
internals.
    > We could implement the behavior using SYMSXP, or not. Nor would the
binding
    > need to be mutable. The binding would be considered independent of the
    > environment from which it was retrieved. As Pete has mentioned, it
could be
    > a useful abstraction to have in general.

It could be, indeed.   Luke's advice (above) and my own gut
feeling do tell me that this is a much larger step than solving
the "getIfExists()" problem.  
In the R development cycle I'd think that it should go to the
next (2015-2016) "3.3" cycle, rather than the current "3.2"
one
with goal in April.

    >> Adding getIfExists, or .get, or get0, or whatever seems fine.
Adding
    >> an argument to get() with missing giving current behavior may be OK
    >> too. Rewriting exists and get as .Primitives may be sufficient
though.

Thank you, Luke.  Given that, Duncan's and the other inputs,
I think we should go for a new function -- .Internal() for now.

To Pete's point about arguments, I did drop 'frame' on purpose 
and indeed we could try to do away with 'where/pos' as well and
have the environment only specified by 'envir'.

Name: I like  get0() for its brevity and prefer it to .get().

Let me expose my current implementation on R-devel ... and start
using it in the 'methods' package so we (Pete H. :-) can start
measuring its impact.

Martin

>>>>> Martin Maechler <maechler at stat.math.ethz.ch>
>>>>>     on Fri, 9 Jan 2015 14:00:38 +0100 writes:
>>>>> Michael Lawrence <lawrence.michael at gene.com>
>>>>>     on Thu, 8 Jan 2015 14:02:26 -0800 writes:
    >> On Thu, Jan 8, 2015 at 11:57 AM, <luke-tierney at uiowa.edu>
wrote:
    >>> On Thu, 8 Jan 2015, Michael Lawrence wrote:
    >>> 
    >>> If we do add an argument to get(), then it should be named
consistently
    >>>> with the ifnotfound argument of mget(). 

You are right... I forgot to say so earlier in the thread.

The definition now is

get0 <- function (x, envir = pos.to.env(-1L), mode = "any",
inherits = TRUE,
                  ifnotfound = NULL)
    .Internal(get0(x, envir, mode, inherits, ifnotfound))



    >>>> As mentioned, the possibility of a
    >>>> NULL value is problematic. One solution is a sentinel value
that indicates
    >>>> an unbound value (like R_UnboundValue).
    >>>> 
    >>> 
    >>> A null default is fine -- it's a default; if it isn't
right for a
    >>> particular case you can provide something else.
    >>> 

    [..................]

    >>> Adding getIfExists, or .get, or get0, or whatever seems fine.
Adding
    >>> an argument to get() with missing giving current behavior may
be OK
    >>> too. Rewriting exists and get as .Primitives may be sufficient
though.

    > Thank you, Luke.  Given that, Duncan's and the other inputs,
    > I think we should go for a new function -- .Internal() for now.

    > To Pete's point about arguments, I did drop 'frame' on
purpose
    > and indeed we could try to do away with 'where/pos' as well and
    > have the environment only specified by 'envir'.

    > Name: I like  get0() for its brevity and prefer it to .get().

    > Let me expose my current implementation on R-devel ... and start
    > using it in the 'methods' package so we (Pete H. :-) can start
    > measuring its impact.

I have now committed  get0() to R-devel  (svn rev 67386) 

which is already using it in quite a few places:
in 'base', notably in base/R/namespace.R   where it may speedup, also
in 'methods' in quite a few places also in the hope of some S4
speedup.

{{Now I feel having deserved some weekend break ...}}

Martin