R devel - Jan 2018 - Coping with non-standard evaluation in R program analysis

Hello R experts,


I plan to develop a tool for dynamic analysis of R programs. I would like to
trace function calls at runtime, capturing argument and return values. Following
a suggestion made some time ago on this list, my high-level implementation
strategy is to rewrite the AST, augmenting call expressions with pre-call and
post-call shims to capture the arguments and return value, respectively.


I can think of only one fundamental conceptual obstacle to this approach: R
functions are not necessarily referentially transparent. The arguments received
by a function are not values but promises. They can be evaluated directly
("standard evaluation"), after applying arbitrary syntactic
transformations ("non-standard evaluation", aka NSE), or not at all.
Therefore, if you peek at the values of function arguments before evaluating the
function, you risk altering the semantics of the program, possibly fatally.


I'm looking for general advice about how to cope with NSE in this context. I
also have some specific questions:


1) Is it possible to determine whether a given function (primitive, in R, or
external) uses NSE on some or all of its arguments?


2) Is it possible to inspect the promise objects received by functions, say to
determine whether they have been evaluated, without actually evaluating them?
The R manual is not encouraging in this area:


https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Promise-objects


Thank you,


Evan


	[[alternative HTML version deleted]]

For 2), it is not exposed in R's standard library but it is exposed in
the Rinternals API. A promise that is forced in normal evaluation will
have PRENV set to NULL.

Peter

On Tue, Jan 2, 2018 at 4:19 PM, Evan James Patterson
<epatters at stanford.edu> wrote:
> Hello R experts,
>
>
> I plan to develop a tool for dynamic analysis of R programs. I would like
to trace function calls at runtime, capturing argument and return values.
Following a suggestion made some time ago on this list, my high-level
implementation strategy is to rewrite the AST, augmenting call expressions with
pre-call and post-call shims to capture the arguments and return value,
respectively.
>
>
> I can think of only one fundamental conceptual obstacle to this approach: R
functions are not necessarily referentially transparent. The arguments received
by a function are not values but promises. They can be evaluated directly
("standard evaluation"), after applying arbitrary syntactic
transformations ("non-standard evaluation", aka NSE), or not at all.
Therefore, if you peek at the values of function arguments before evaluating the
function, you risk altering the semantics of the program, possibly fatally.
>
>
> I'm looking for general advice about how to cope with NSE in this
context. I also have some specific questions:
>
>
> 1) Is it possible to determine whether a given function (primitive, in R,
or external) uses NSE on some or all of its arguments?
>
>
> 2) Is it possible to inspect the promise objects received by functions, say
to determine whether they have been evaluated, without actually evaluating them?
The R manual is not encouraging in this area:
>
>
>
https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Promise-objects
>
>
> Thank you,
>
>
> Evan
>
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

Hi Evan,

You may find some parts of what we are doing with genthat useful.  Genthat is a
tool
for creating unit tests by recording argument and return values of calls. This
is
done by instrumentation of the source code.

The git repo with the code is here https://github.com/PRL-PRG/genthat

We don?t really deal with NSE though. It could be something worth thinking about
for us.
The contact for genthtat is Filip.


We are also writing a tool for analyzing promises and generating execution
traces
within the R VM.  This is available in 
    https://github.com/PRL-PRG/R-dyntrace

Aviral and Konrad are the contacts there.


Best,

jan






Jan Vitek, Professor 
Computer Science, 
Northeastern University
> On Jan 3, 2018, at 7:18 PM, Peter Meilstrup <peter.meilstrup at
gmail.com> wrote:
> 
> For 2), it is not exposed in R's standard library but it is exposed in
> the Rinternals API. A promise that is forced in normal evaluation will
> have PRENV set to NULL.
> 
> Peter
> 
> On Tue, Jan 2, 2018 at 4:19 PM, Evan James Patterson
> <epatters at stanford.edu> wrote:
>> Hello R experts,
>> 
>> 
>> I plan to develop a tool for dynamic analysis of R programs. I would
like to trace function calls at runtime, capturing argument and return values.
Following a suggestion made some time ago on this list, my high-level
implementation strategy is to rewrite the AST, augmenting call expressions with
pre-call and post-call shims to capture the arguments and return value,
respectively.
>> 
>> 
>> I can think of only one fundamental conceptual obstacle to this
approach: R functions are not necessarily referentially transparent. The
arguments received by a function are not values but promises. They can be
evaluated directly ("standard evaluation"), after applying arbitrary
syntactic transformations ("non-standard evaluation", aka NSE), or not
at all. Therefore, if you peek at the values of function arguments before
evaluating the function, you risk altering the semantics of the program,
possibly fatally.
>> 
>> 
>> I'm looking for general advice about how to cope with NSE in this
context. I also have some specific questions:
>> 
>> 
>> 1) Is it possible to determine whether a given function (primitive, in
R, or external) uses NSE on some or all of its arguments?
>> 
>> 
>> 2) Is it possible to inspect the promise objects received by functions,
say to determine whether they have been evaluated, without actually evaluating
them? The R manual is not encouraging in this area:
>> 
>> 
>>
https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Promise-objects
>> 
>> 
>> Thank you,
>> 
>> 
>> Evan
>> 
>> 
>>       [[alternative HTML version deleted]]
>> 
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

Evan,

You can look at what CodeDepends (on cran/ github.com/duncantl/CodeDepends
) does for nonstandard evaluation. It is a static code analyzer which
allows for custom handler functions for calls to specific functions. The
default handlers provided are "NSE-aware" (many of) the common
contexts
where that occurs. It then consumes arguments to those functions as NSE and
annotates them accordingly in the result. This is a heuristic, and we don't
currently do the harder bit of attempting to figure out where those
non-standard symbols will be resolved currently, but it's potentially a
start.

As for manipulating the AST, do be somewhat careful of this. You may run
afoul of other things that are doing this under the hood based on
particular assumptions, notably the rlang stuff that backs the tidyverse
these days.  You might be safe though, it depends on what you want.

Another approach is what I in the histry package, which handles top-level
expressions in both interactive session and dynamic document weaving
contexts. Happy to talk more about this off list as I have done a fair bit
of work in the space it sounds like you're in.

Best,
~G

On Thu, Jan 4, 2018 at 12:20 AM, jan Vitek <vitekj at icloud.com> wrote:
> Hi Evan,
>
> You may find some parts of what we are doing with genthat useful.  Genthat
> is a tool
> for creating unit tests by recording argument and return values of calls.
> This is
> done by instrumentation of the source code.
>
> The git repo with the code is here https://github.com/PRL-PRG/genthat
>
> We don?t really deal with NSE though. It could be something worth thinking
> about for us.
> The contact for genthtat is Filip.
>
>
> We are also writing a tool for analyzing promises and generating execution
> traces
> within the R VM.  This is available in
>     https://github.com/PRL-PRG/R-dyntrace
>
> Aviral and Konrad are the contacts there.
>
>
> Best,
>
> jan
>
>
>
>
>
>
> Jan Vitek, Professor
> Computer Science,
> Northeastern University
>
> > On Jan 3, 2018, at 7:18 PM, Peter Meilstrup <peter.meilstrup at
gmail.com>
> wrote:
> >
> > For 2), it is not exposed in R's standard library but it is
exposed in
> > the Rinternals API. A promise that is forced in normal evaluation will
> > have PRENV set to NULL.
> >
> > Peter
> >
> > On Tue, Jan 2, 2018 at 4:19 PM, Evan James Patterson
> > <epatters at stanford.edu> wrote:
> >> Hello R experts,
> >>
> >>
> >> I plan to develop a tool for dynamic analysis of R programs. I
would
> like to trace function calls at runtime, capturing argument and return
> values. Following a suggestion made some time ago on this list, my
> high-level implementation strategy is to rewrite the AST, augmenting call
> expressions with pre-call and post-call shims to capture the arguments and
> return value, respectively.
> >>
> >>
> >> I can think of only one fundamental conceptual obstacle to this
> approach: R functions are not necessarily referentially transparent. The
> arguments received by a function are not values but promises. They can be
> evaluated directly ("standard evaluation"), after applying
arbitrary
> syntactic transformations ("non-standard evaluation", aka NSE),
or not at
> all. Therefore, if you peek at the values of function arguments before
> evaluating the function, you risk altering the semantics of the program,
> possibly fatally.
> >>
> >>
> >> I'm looking for general advice about how to cope with NSE in
this
> context. I also have some specific questions:
> >>
> >>
> >> 1) Is it possible to determine whether a given function
(primitive, in
> R, or external) uses NSE on some or all of its arguments?
> >>
> >>
> >> 2) Is it possible to inspect the promise objects received by
functions,
> say to determine whether they have been evaluated, without actually
> evaluating them? The R manual is not encouraging in this area:
> >>
> >>
> >> https://cran.r-project.org/doc/manuals/r-release/R-lang.
> html#Promise-objects
> >>
> >>
> >> Thank you,
> >>
> >>
> >> Evan
> >>
> >>
> >>       [[alternative HTML version deleted]]
> >>
> >> ______________________________________________
> >> R-devel at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-devel
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>


-- 
Gabriel Becker, PhD
Scientist (Bioinformatics)
Genentech Research

	[[alternative HTML version deleted]]