R help - Jan 2019 - Function in default parameter value closing over variables defined later in the enclosing function

Hi!

I needed to generalize a loss function being optimized inside another
function, so I made it a function argument with a default value. It
worked without problems, but later I noticed that the inner function,
despite being defined in the function arguments, somehow closes over a
variable belonging to the outer function, which is defined later.

Example:

outside <- function(inside = function() print(secret)) {
	secret <- 'secret'
	inside()
}
outside()

I'm used to languages that have both lambdas and variable declaration
(like perl5 -Mstrict or C++11), so I was a bit surprised.

Does this work because R looks up the variable by name late enough at
runtime for the `secret` variable to exist in the parent environment of
the `inside` function? Can I rely on it? Is this considered bad style? 
Should I rewrite it (and how)?

-- 
Best regards,
Ivan

Hi Ivan & All,

R's scoping system basically goes to all environments along the call
stack when trying to resolve an unbound variable, see the language
definition [1], section 4.3.4, and perhaps also 2.1.5.

Generally, unbound variables should be used with care. It's a bit
difficult to decide whether and how the code should be rewritten,
I'd say that depends on the underlying intentions / purposes. As it
is, the code could be simplified to just

    print("secret");

but that's probably missing the point.

Best regards, Jan


[1] https://cran.r-project.org/doc/manuals/r-release/R-lang.html

On Wed, Jan 23, 2019 at 12:53:01PM +0300, Ivan Krylov
wrote:
> Hi!
> 
> I needed to generalize a loss function being optimized inside another
> function, so I made it a function argument with a default value. It
> worked without problems, but later I noticed that the inner function,
> despite being defined in the function arguments, somehow closes over a
> variable belonging to the outer function, which is defined later.
> 
> Example:
> 
> outside <- function(inside = function() print(secret)) {
> 	secret <- 'secret'
> 	inside()
> }
> outside()
> 
> I'm used to languages that have both lambdas and variable declaration
> (like perl5 -Mstrict or C++11), so I was a bit surprised.
> 
> Does this work because R looks up the variable by name late enough at
> runtime for the `secret` variable to exist in the parent environment of
> the `inside` function? Can I rely on it? Is this considered bad style? 
> Should I rewrite it (and how)?
> 
> -- 
> Best regards,
> Ivan
> 
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

On 23/01/2019 4:53 a.m., Ivan Krylov wrote:
> Hi!
> 
> I needed to generalize a loss function being optimized inside another
> function, so I made it a function argument with a default value. It
> worked without problems, but later I noticed that the inner function,
> despite being defined in the function arguments, somehow closes over a
> variable belonging to the outer function, which is defined later.
> 
> Example:
> 
> outside <- function(inside = function() print(secret)) {
> 	secret <- 'secret'
> 	inside()
> }
> outside()
> 
> I'm used to languages that have both lambdas and variable declaration
> (like perl5 -Mstrict or C++11), so I was a bit surprised.
Defaults of variables are evaluated in the evaluation frame of the call.
So the inside() function is created in the evaluation frame, and it's 
environment will be that frame.

When it is called it will create a new evaluation frame (empty in your 
example), with a parent being its environment, i.e. the evaluation frame 
from when it was created, so it will be able to see your secret variable.

If it made an assignment to secret using standard "<-" assignment,
it
would create a new variable in its own evaluation frame, but if it used 
superassignment "<<-", it would modify the original secret
variable.
> 
> Does this work because R looks up the variable by name late enough at
> runtime for the `secret` variable to exist in the parent environment of
> the `inside` function? Can I rely on it? Is this considered bad style?
> Should I rewrite it (and how)?
I would consider it bad style if the inside() function had anything 
other than a trivial definition as in your example.  However, in my 
opinion it would be fine to write it as

  outside <- function(inside = defaultInsideFn) {
     defaultInsideFn <- function() print(secret)
     secret <- 'secret'
     inside()
  }

which is essentially equivalent, other than having a shorter header on 
the outside() function.

Duncan Murdoch

On 23/01/2019 5:27 a.m., Jan T Kim wrote:
> Hi Ivan & All,
> 
> R's scoping system basically goes to all environments along the call
> stack when trying to resolve an unbound variable, see the language
> definition [1], section 4.3.4, and perhaps also 2.1.5.
You are misinterpreting that section.  It's not the call stack that is 
searched, it's the chain of environments that starts with the evaluation 
frame of the current function.  Those are very different.  For example,


g <- function() {
   print(secret)
}

f <- function() {
   secret <- "secret"
   g()
}

would fail, because even though secret is defined in the caller of g() 
and is therefore in the call stack, that's irrelevant:  it's not in
g's
evaluation frame (which has no variables) or its parent (which is the 
global environment if those definitions were evaluated there).

Duncan Murdoch
> 
> Generally, unbound variables should be used with care. It's a bit
> difficult to decide whether and how the code should be rewritten,
> I'd say that depends on the underlying intentions / purposes. As it
> is, the code could be simplified to just
> 
>      print("secret");
> 
> but that's probably missing the point.
> 
> Best regards, Jan
> 
> 
> [1] https://cran.r-project.org/doc/manuals/r-release/R-lang.html
> 
> On Wed, Jan 23, 2019 at 12:53:01PM +0300, Ivan Krylov wrote:
>> Hi!
>>
>> I needed to generalize a loss function being optimized inside another
>> function, so I made it a function argument with a default value. It
>> worked without problems, but later I noticed that the inner function,
>> despite being defined in the function arguments, somehow closes over a
>> variable belonging to the outer function, which is defined later.
>>
>> Example:
>>
>> outside <- function(inside = function() print(secret)) {
>> 	secret <- 'secret'
>> 	inside()
>> }
>> outside()
>>
>> I'm used to languages that have both lambdas and variable
declaration
>> (like perl5 -Mstrict or C++11), so I was a bit surprised.
>>
>> Does this work because R looks up the variable by name late enough at
>> runtime for the `secret` variable to exist in the parent environment of
>> the `inside` function? Can I rely on it? Is this considered bad style?
>> Should I rewrite it (and how)?
>>
>> -- 
>> Best regards,
>> Ivan
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
> 
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

Dear Jan & Duncan,

Thanks for your replies!

On Wed, 23 Jan 2019 09:56:25 -0500
Duncan Murdoch <murdoch.duncan at gmail.com> wrote:
> Defaults of variables are evaluated in the evaluation frame of the
> call. So the inside() function is created in the evaluation frame,
> and it's environment will be that frame.
 
> When it is called it will create a new evaluation frame (empty in
> your example), with a parent being its environment, i.e. the
> evaluation frame from when it was created, so it will be able to see
> your secret variable.
Nice explanation about closures in R inheriting not only their
explicitly captured variables, but whole environments of evaluation
(not stack) frames where they have been created.
> in my opinion it would be fine to write it as
> 
>   outside <- function(inside = defaultInsideFn) {
>      defaultInsideFn <- function() print(secret)
>      secret <- 'secret'
>      inside()
>   }
I like this idea; I'm going to use it.

-- 
Best regards,
Ivan