R devel - Jul 2015 - Are downstream dependencies rebuilt when a package is updated on CRAN?

I was wondering: are the downstream dependencies of a package rebuilt
when a package is updated on CRAN? (I'm referring to the binary
packages, of course.)

The reason I ask is because there are cases where this can cause
problems. Suppose that when pkgB is built, it calls
pkgA::makeClosure(), which returns a closure that refers to a function
in pkgA. Suppose this code is in pkgA 1.0:
makeClosure <- function() {
  function() {
    funA()
  }
}
funA <- function() "funA was called!"

And this is the code in pkgB:

foo <- pkgA::makeClosure()


After building and installing pkgB, you can do this, which gives the
expected result:
> pkgB::foo()
[1] "funA was called!"


Now suppose pkgA is upgraded to 2.0, and it contains this:
makeClosure <- function() {
  function() {
    newFunA()
  }
}
newFunA <- function() "newFunA was called!"


After this upgrade, when you run the pkgB::foo(), you'll get an
error:
> pkgB::foo()
Error in foo() : could not find function "funA"


This is because the environment for pkgB::foo is the pkgA namespace,
but the contents of that namespace have changed underneath it. The
solution is to rebuild and reinstall pkgB against pkgA 2.0.

This is why I'm asking about downstream dependencies being rebuilt on
CRAN. If pkgB's binary packages are not automatically rebuilt when
pkgA is updated on CRAN, then there would be a broken combination of
pkgA and pkgB on CRAN. (Note that source packages aren't vulnerable to
this problem)


This is not a purely academic exercise -- something similar to this
has actually occurred. When R went from version 3.0.0 to 3.0.1, there
was a new function in the methods package named .setDummyField, and
reference class objects that were built against methods 3.0.1 received
closures that included calls to .setDummyField(). If these refclass
objects were saved in a package at build-time, then they would not
work on systems with R (and methods) 3.0.0. Because CRAN built
binaries on R 3.0.1 after its release, users who were on R 3.0.0
simply could not run some binary packages from CRAN. Their possible
solutions were to upgrade R, or install the packages from source. I
know this happened with Shiny, lme4, ROAuth, among others.
  https://groups.google.com/forum/#!msg/shiny-discuss/poNAAtyYuS4/n6iAVMI3mb0J
  https://github.com/lme4/lme4/issues/54
 
http://stackoverflow.com/questions/18258087/twitter-error-when-authorizing-token

It happens that the methods package is included with the base R
distribution, but this kind of problem can occur with other packages
as well.

Many of us have encountered hard-to-diagnose problems that are solved
by reinstalling R and all packages. I suspect that this is the root
cause in many of those cases.


At the very least, it would be nice to be able to specify in a pkgB
that it depends on pkgA in such a way that it must be rebuilt when
pkgA is upgraded. There are two important consequences of this: CRAN
would need to rebuild pkgB when a new pkgA is accepted; and when users
upgrade pkgA, they would need to receive a rebuilt pkgB as well.

-Winston

Winston,

see far below. ;-)



On 03.07.2015 06:36, Winston Chang wrote:
> I was wondering: are the downstream dependencies of a package rebuilt
> when a package is updated on CRAN? (I'm referring to the binary
> packages, of course.)
>
> The reason I ask is because there are cases where this can cause
> problems. Suppose that when pkgB is built, it calls
> pkgA::makeClosure(), which returns a closure that refers to a function
> in pkgA. Suppose this code is in pkgA 1.0:
> makeClosure <- function() {
>    function() {
>      funA()
>    }
> }
> funA <- function() "funA was called!"
>
> And this is the code in pkgB:
>
> foo <- pkgA::makeClosure()
>
>
> After building and installing pkgB, you can do this, which gives the
> expected result:
>> pkgB::foo()
> [1] "funA was called!"
>
>
> Now suppose pkgA is upgraded to 2.0, and it contains this:
> makeClosure <- function() {
>    function() {
>      newFunA()
>    }
> }
> newFunA <- function() "newFunA was called!"
>
>
> After this upgrade, when you run the pkgB::foo(), you'll get an error:
>> pkgB::foo()
> Error in foo() : could not find function "funA"
>
>
> This is because the environment for pkgB::foo is the pkgA namespace,
> but the contents of that namespace have changed underneath it. The
> solution is to rebuild and reinstall pkgB against pkgA 2.0.
>
> This is why I'm asking about downstream dependencies being rebuilt on
> CRAN. If pkgB's binary packages are not automatically rebuilt when
> pkgA is updated on CRAN, then there would be a broken combination of
> pkgA and pkgB on CRAN. (Note that source packages aren't vulnerable to
> this problem)
>
>
> This is not a purely academic exercise -- something similar to this
> has actually occurred. When R went from version 3.0.0 to 3.0.1, there
> was a new function in the methods package named .setDummyField, and
> reference class objects that were built against methods 3.0.1 received
> closures that included calls to .setDummyField(). If these refclass
> objects were saved in a package at build-time, then they would not
> work on systems with R (and methods) 3.0.0. Because CRAN built
> binaries on R 3.0.1 after its release, users who were on R 3.0.0
> simply could not run some binary packages from CRAN. Their possible
> solutions were to upgrade R, or install the packages from source. I
> know this happened with Shiny, lme4, ROAuth, among others.
>   
https://groups.google.com/forum/#!msg/shiny-discuss/poNAAtyYuS4/n6iAVMI3mb0J
>    https://github.com/lme4/lme4/issues/54
>   
http://stackoverflow.com/questions/18258087/twitter-error-when-authorizing-token
>
> It happens that the methods package is included with the base R
> distribution, but this kind of problem can occur with other packages
> as well.
>
> Many of us have encountered hard-to-diagnose problems that are solved
> by reinstalling R and all packages. I suspect that this is the root
> cause in many of those cases.
I am aware of this probelm. Another as with the environments happens if 
S4 stuff is changed.
>
> At the very least, it would be nice to be able to specify in a pkgB
> that it depends on pkgA in such a way that it must be rebuilt when
> pkgA is upgraded. There are two important consequences of this: CRAN
> would need to rebuild pkgB when a new pkgA is accepted;
This happens for Windows binary packages on CRAN.


 > and when users
> upgrade pkgA, they would need to receive a rebuilt pkgB as well.
This will not happen automatically, since the R package mechanism is 
based on version numbers tht are not changed on CRAN.

It is very hard to know when such a rebuild is really needed. In very 
few cases only, actually. Currently we always rebuild which causes a lot 
of overhead. But we do not always want to trigger updates since these 
are not needed in I guess > 99% of all package updates.

Best wishes,
Uwe

> -Winston