R devel - Sep 2016 - Finalizer execution order question

Given an externalptr object 'pool' which protects an R object
'prot':

  # SEXP prot = (a dynamically updated list with handles)
  SEXP pool =  R_MakeExternalPtr(p, R_NilValue, prot);
  R_RegisterCFinalizerEx(pool, fin_pool, TRUE);

WRE explains that 'prot' remains in existence as long as 'pool'
is
around. Does this also mean 'prot' still exists when the finalizer of
'pool' gets executed?

Long story: I am running into an issue with the next generation of the
curl package which uses dynamic pools of (many) http request handles.
Both the pool and the handles are externalptr objects with finalizers
that clean up after themselves.

When the pool goes out of scope, its finalizer has to loop over
pending handles to release them. However, to my surprise, the
finalizers of the handles get executed *before* the finalizer of
'pool'. Therefore the handles have destroyed themselves before I can
properly take hem out of the pool. This is exactly what I was trying
to prevent by putting the handles in a 'prot' list.

I wrote a simple package [1] that illustrates this issue. To run the example:

  devtools::install_github("jeroenooms/test")
  library(test)
  pool <- make_pool()
  show_handles(pool)
  rm(pool)
  gc()

What this example is supposed to illustrate is that even though 'prot'
gets protected from GC while the externalptr is around, the finalizers
in 'prot' have already executed when the externalptr gets finalized.

What is even stranger (to me) is that the SEXPs in 'prot' still seem
in tact during the finalizer of externalptr (ASAN is not giving
use-after-free warnings either). It's just that their finalizers have
already executed. So that leaves the pool finalizer in the odd
position of seeing only the post-mortem SEXP contents of the handles.
This does provide a workaround in my case, but can we rely on this?

So my question: is this expected behavior? What would be an
alternative approach to ensure that handles stay protected until the
pool has been cleaned and finalized (rather than running all
finalizers in the order they were registered)?


[1] https://github.com/jeroenooms/test

On Thu, 15 Sep 2016, Jeroen Ooms wrote:
> Given an externalptr object 'pool' which protects an R object
'prot':
>
>  # SEXP prot = (a dynamically updated list with handles)
>  SEXP pool =  R_MakeExternalPtr(p, R_NilValue, prot);
>  R_RegisterCFinalizerEx(pool, fin_pool, TRUE);
>
> WRE explains that 'prot' remains in existence as long as
'pool' is
> around. Does this also mean 'prot' still exists when the finalizer
of
> 'pool' gets executed?
prot is still reachable and valid. If it also has a finalizer on it
then that finalizer might have been run before the one on prot -- you
should not assume anything about the order in which finalizers are run.
> Long story: I am running into an issue with the next generation of the
> curl package which uses dynamic pools of (many) http request handles.
> Both the pool and the handles are externalptr objects with finalizers
> that clean up after themselves.
>
> When the pool goes out of scope, its finalizer has to loop over
> pending handles to release them. However, to my surprise, the
> finalizers of the handles get executed *before* the finalizer of
> 'pool'. Therefore the handles have destroyed themselves before I
can
> properly take hem out of the pool. This is exactly what I was trying
> to prevent by putting the handles in a 'prot' list.
If your handles are always in a pool, then put a finalizer on the pool
but not on the handles?

Lists of weak references might also be useful to consider for this
sort of thing. I've used those to make sure finalizers are run before
a package DLL is inloaded (since running them later would not be a
good idea).

Best,

luke
>
> I wrote a simple package [1] that illustrates this issue. To run the
example:
>
>  devtools::install_github("jeroenooms/test")
>  library(test)
>  pool <- make_pool()
>  show_handles(pool)
>  rm(pool)
>  gc()
>
> What this example is supposed to illustrate is that even though
'prot'
> gets protected from GC while the externalptr is around, the finalizers
> in 'prot' have already executed when the externalptr gets
finalized.
>
> What is even stranger (to me) is that the SEXPs in 'prot' still
seem
> in tact during the finalizer of externalptr (ASAN is not giving
> use-after-free warnings either). It's just that their finalizers have
> already executed. So that leaves the pool finalizer in the odd
> position of seeing only the post-mortem SEXP contents of the handles.
> This does provide a workaround in my case, but can we rely on this?
>
> So my question: is this expected behavior? What would be an
> alternative approach to ensure that handles stay protected until the
> pool has been cleaned and finalized (rather than running all
> finalizers in the order they were registered)?
>
>
> [1] https://github.com/jeroenooms/test
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu