R devel - Aug 2018 - SIGSEGV in R_RunWeakRefFinalizer, object allocated with Rcpp

Thanks, Tomas, Luke, for the clarifications. Then, I have another question.

But first, let me introduce how I ended up here, because obviously I
just don't go around dyn.unloading things that I've just compiled. I
was testing a package with valgrind. Everything ok, no leaks. Great.
But I'm always suspicious (probably unjustifiably) of all the memory
that is reported as "still reachable", so I wanted to check whether
there was any difference if I detach(unload=TRUE) the package after
all the tests.

In a nutshell, I ended up discovering that the following code:

```
library(simmer)
simmer() # allocates a C++ object, as in my initial example
detach("package:simmer", unload=TRUE)
```

segfaults on Windows, but not on Linux (then I built the example in my
initial email to confirm it wasn't simmer's fault). So given that,
from your explanation, I should expect a segfault here, the question
is: what on Earth does (or does not) R on Linux do to avoid
segfaulting compared to Windows? :) And a corolary would be, can't R
on Windows do the same?

Regards,
I?aki

El jue., 9 ago. 2018 a las 21:13, <luke-tierney at uiowa.edu>
escribi?:
>
> On Thu, 9 Aug 2018, Dirk Eddelbuettel wrote:
>
> >
> > On 9 August 2018 at 20:37, Tomas Kalibera wrote:
> > | So to answer your original question, this could probably be handled
in
> > | Rcpp,
> >
> > Hm. Why do you say that / what did you have in mind?
>
> We say it because it is true. Rcpp registers C finalizers and running
> them after unloading will segfault. For now it would be better for Rcpp
> (and everyone else) to explicitly discourage unloading as it is
> unreliable on many levels.
>
> What Rcpp could do to avoid segfaulting is to keep a weak list of all
> objects to which it attaches C finalizers and arrange for those to be
> cleaned up in an R_unload_<dllname> routine. Not clear it is worth
the
> trouble. At the R level we could provide more support for this since
> we already have a weak list of objects with finalizers, but again not
> clear it is worth the trouble.
>
> Best,
>
> luke
>
> > Recall that we do not alter SEXPs or introduce additional additional
> > reference counters -- because we do not think that altering the basic
R API
> > for such calls would be a wise strategy.  So we do more or less what
is done
> > in C for R, with some additional hand-holding which circumvents a
number of
> > common errors.
> >
> > | but in either case I would not use dyn.unload() in the first
> > | place. This problem may be just one of many.
> >
> > I think I'd second that. I never had much unloading packages or
dynamic
> > libraries and tend to "just say no". Both short-lived
processes (ie via 'r')
> > as well as long sessions (ie R via ESS, running for weeks) work for my
> > workflows.
> >
> > Dirk
> >
> >
>
> --
> Luke Tierney
> Ralph E. Wareham Professor of Mathematical Sciences
> University of Iowa                  Phone:             319-335-3386
> Department of Statistics and        Fax:               319-335-3017
>     Actuarial Science
> 241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu


--
I?aki ?car
http://www.enchufa2.es
@Enchufa2

Hi I?aki,

I think that "still reachable" memory is potentially a problem only if
you cared about (frequent) package unloading, and if package unloading 
did not have correctness problems in the first place. I would only worry 
about "memory leaks" reported by valgrind.

That your example (unloading a library while there is still an object 
with a finalizer implemented in that library) segfaults on one system 
but not another is not too surprising. Depending on the OS, the dynamic 
linker, the current state of the system, etc - when the GC tries to run 
the finalizer, it may point to inaccessible memory (segfault), but also 
still to the code of the unloaded finalizer (possibly no segfault) or 
something else. There is no way to distinguish between all these cases 
before running the finalizer - neither the OS nor R can do it - we could 
only in principle, with a lot of other problems and platform-dependent 
hacks, detect when the memory is inaccessible, but that may not be worth it.

Best
Tomas


On 08/10/2018 12:46 AM, I?aki ?car wrote:
> Thanks, Tomas, Luke, for the clarifications. Then, I have another question.
>
> But first, let me introduce how I ended up here, because obviously I
> just don't go around dyn.unloading things that I've just compiled.
I
> was testing a package with valgrind. Everything ok, no leaks. Great.
> But I'm always suspicious (probably unjustifiably) of all the memory
> that is reported as "still reachable", so I wanted to check
whether
> there was any difference if I detach(unload=TRUE) the package after
> all the tests.
>
> In a nutshell, I ended up discovering that the following code:
>
> ```
> library(simmer)
> simmer() # allocates a C++ object, as in my initial example
> detach("package:simmer", unload=TRUE)
> ```
>
> segfaults on Windows, but not on Linux (then I built the example in my
> initial email to confirm it wasn't simmer's fault). So given that,
> from your explanation, I should expect a segfault here, the question
> is: what on Earth does (or does not) R on Linux do to avoid
> segfaulting compared to Windows? :) And a corolary would be, can't R
> on Windows do the same?
>
> Regards,
> I?aki
>
> El jue., 9 ago. 2018 a las 21:13, <luke-tierney at uiowa.edu>
escribi?:
>> On Thu, 9 Aug 2018, Dirk Eddelbuettel wrote:
>>
>>> On 9 August 2018 at 20:37, Tomas Kalibera wrote:
>>> | So to answer your original question, this could probably be
handled in
>>> | Rcpp,
>>>
>>> Hm. Why do you say that / what did you have in mind?
>> We say it because it is true. Rcpp registers C finalizers and running
>> them after unloading will segfault. For now it would be better for Rcpp
>> (and everyone else) to explicitly discourage unloading as it is
>> unreliable on many levels.
>>
>> What Rcpp could do to avoid segfaulting is to keep a weak list of all
>> objects to which it attaches C finalizers and arrange for those to be
>> cleaned up in an R_unload_<dllname> routine. Not clear it is
worth the
>> trouble. At the R level we could provide more support for this since
>> we already have a weak list of objects with finalizers, but again not
>> clear it is worth the trouble.
>>
>> Best,
>>
>> luke
>>
>>> Recall that we do not alter SEXPs or introduce additional
additional
>>> reference counters -- because we do not think that altering the
basic R API
>>> for such calls would be a wise strategy.  So we do more or less
what is done
>>> in C for R, with some additional hand-holding which circumvents a
number of
>>> common errors.
>>>
>>> | but in either case I would not use dyn.unload() in the first
>>> | place. This problem may be just one of many.
>>>
>>> I think I'd second that. I never had much unloading packages or
dynamic
>>> libraries and tend to "just say no". Both short-lived
processes (ie via 'r')
>>> as well as long sessions (ie R via ESS, running for weeks) work for
my
>>> workflows.
>>>
>>> Dirk
>>>
>>>
>> --
>> Luke Tierney
>> Ralph E. Wareham Professor of Mathematical Sciences
>> University of Iowa                  Phone:             319-335-3386
>> Department of Statistics and        Fax:               319-335-3017
>>      Actuarial Science
>> 241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
>> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
>
>
> --
> I?aki ?car
> http://www.enchufa2.es
> @Enchufa2

El vie., 10 ago. 2018 a las 14:17, Tomas Kalibera
(<tomas.kalibera at gmail.com>) escribi?:
>
> Hi I?aki,
>
> I think that "still reachable" memory is potentially a problem
only if
> you cared about (frequent) package unloading, and if package unloading
> did not have correctness problems in the first place. I would only worry
> about "memory leaks" reported by valgrind.
>
> That your example (unloading a library while there is still an object
> with a finalizer implemented in that library) segfaults on one system
> but not another is not too surprising. Depending on the OS, the dynamic
> linker, the current state of the system, etc - when the GC tries to run
> the finalizer, it may point to inaccessible memory (segfault), but also
> still to the code of the unloaded finalizer (possibly no segfault) or
> something else. There is no way to distinguish between all these cases
> before running the finalizer - neither the OS nor R can do it - we could
> only in principle, with a lot of other problems and platform-dependent
> hacks, detect when the memory is inaccessible, but that may not be worth
it.
Ok, I get it. And I agree: it's not worth it. Thanks again.

I?aki
>
> Best
> Tomas
>
>
> On 08/10/2018 12:46 AM, I?aki ?car wrote:
> > Thanks, Tomas, Luke, for the clarifications. Then, I have another
question.
> >
> > But first, let me introduce how I ended up here, because obviously I
> > just don't go around dyn.unloading things that I've just
compiled. I
> > was testing a package with valgrind. Everything ok, no leaks. Great.
> > But I'm always suspicious (probably unjustifiably) of all the
memory
> > that is reported as "still reachable", so I wanted to check
whether
> > there was any difference if I detach(unload=TRUE) the package after
> > all the tests.
> >
> > In a nutshell, I ended up discovering that the following code:
> >
> > ```
> > library(simmer)
> > simmer() # allocates a C++ object, as in my initial example
> > detach("package:simmer", unload=TRUE)
> > ```
> >
> > segfaults on Windows, but not on Linux (then I built the example in my
> > initial email to confirm it wasn't simmer's fault). So given
that,
> > from your explanation, I should expect a segfault here, the question
> > is: what on Earth does (or does not) R on Linux do to avoid
> > segfaulting compared to Windows? :) And a corolary would be, can't
R
> > on Windows do the same?
> >
> > Regards,
> > I?aki
> >
> > El jue., 9 ago. 2018 a las 21:13, <luke-tierney at uiowa.edu>
escribi?:
> >> On Thu, 9 Aug 2018, Dirk Eddelbuettel wrote:
> >>
> >>> On 9 August 2018 at 20:37, Tomas Kalibera wrote:
> >>> | So to answer your original question, this could probably be
handled in
> >>> | Rcpp,
> >>>
> >>> Hm. Why do you say that / what did you have in mind?
> >> We say it because it is true. Rcpp registers C finalizers and
running
> >> them after unloading will segfault. For now it would be better for
Rcpp
> >> (and everyone else) to explicitly discourage unloading as it is
> >> unreliable on many levels.
> >>
> >> What Rcpp could do to avoid segfaulting is to keep a weak list of
all
> >> objects to which it attaches C finalizers and arrange for those to
be
> >> cleaned up in an R_unload_<dllname> routine. Not clear it is
worth the
> >> trouble. At the R level we could provide more support for this
since
> >> we already have a weak list of objects with finalizers, but again
not
> >> clear it is worth the trouble.
> >>
> >> Best,
> >>
> >> luke
> >>
> >>> Recall that we do not alter SEXPs or introduce additional
additional
> >>> reference counters -- because we do not think that altering
the basic R API
> >>> for such calls would be a wise strategy.  So we do more or
less what is done
> >>> in C for R, with some additional hand-holding which
circumvents a number of
> >>> common errors.
> >>>
> >>> | but in either case I would not use dyn.unload() in the first
> >>> | place. This problem may be just one of many.
> >>>
> >>> I think I'd second that. I never had much unloading
packages or dynamic
> >>> libraries and tend to "just say no". Both
short-lived processes (ie via 'r')
> >>> as well as long sessions (ie R via ESS, running for weeks)
work for my
> >>> workflows.
> >>>
> >>> Dirk
> >>>
> >>>
> >> --
> >> Luke Tierney
> >> Ralph E. Wareham Professor of Mathematical Sciences
> >> University of Iowa                  Phone:            
319-335-3386
> >> Department of Statistics and        Fax:              
319-335-3017
> >>      Actuarial Science
> >> 241 Schaeffer Hall                  email:   luke-tierney at
uiowa.edu
> >> Iowa City, IA 52242                 WWW: 
http://www.stat.uiowa.edu