R devel - Dec 2016 - Upgrading a package to which other packages are LinkingTo

On 16 December 2016 at 10:14, Duncan Murdoch wrote:
| On 16/12/2016 8:37 AM, Dirk Eddelbuettel wrote:
| >
| > On 16 December 2016 at 08:20, Duncan Murdoch wrote:
| > | Perhaps the solution is to recommend that packages which export their
| > | C-level entry points either guarantee them not to change or offer
| > | (require?) version checks by user code.  So dplyr should start out by
| > | saying "I'm using Rcpp interface 0.12.8".  If Rcpp has a
new version
| > | with a compatible interface, it replies "that's fine". 
If Rcpp has
| > | changed its interface, it says "Sorry, I don't support that
any more."
| >
| > We try. But it's hard, and I'd argue, likely impossible.
| >
| > For example I even added a "frozen" package [1] in the sources
/ unit tests
| > to test for just this. In practice you just cannot hit every possible
access
| > point of the (rich, in our case) API so the tests pass too often.
| >
| > Which is why we relentlessly test against reverse-depends to _at least
ensure
| > buildability_ from our releases.

I meant to also add:  "... against a large corpus of other packages."
The intent is to empirically answer this.

| > As for seamless binary upgrade, I don't think in can work in
practice.  Ask
| > Uwe one day we he rebuilds everything every time on Windows. And for what
it
| > is worth, we essentially do the same in Debian.
| >
| > Sometimes you just need to rebuild.  That may be the price of admission
for
| > using the convenience of rich C++ interfaces.
| >
| 
| Okay, so would you say that Kirill's suggestion is not overkill?  Every 
| time package B uses LinkingTo: A, R should assume it needs to rebuild B 
| when A is updated?

Based on my experience is a "halting problem" -- i.e. cannot know ex
ante.

So "every time" would be overkill to me.  Sometimes you know you must
recompile (but try to be very prudent with public-facing API).  Many times
you do not. It is hard to pin down.

At work we have a bunch of servers with Rcpp and many packages against them
(installed system-wide for all users). We _very really_ needs rebuild.  

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org

On 16/12/2016 10:40 AM, Dirk Eddelbuettel wrote:
> On 16 December 2016 at 10:14, Duncan Murdoch wrote:
> | On 16/12/2016 8:37 AM, Dirk Eddelbuettel wrote:
> | >
> | > On 16 December 2016 at 08:20, Duncan Murdoch wrote:
> | > | Perhaps the solution is to recommend that packages which export
their
> | > | C-level entry points either guarantee them not to change or offer
> | > | (require?) version checks by user code.  So dplyr should start out
by
> | > | saying "I'm using Rcpp interface 0.12.8".  If Rcpp
has a new version
> | > | with a compatible interface, it replies "that's
fine".  If Rcpp has
> | > | changed its interface, it says "Sorry, I don't support
that any more."
> | >
> | > We try. But it's hard, and I'd argue, likely impossible.
> | >
> | > For example I even added a "frozen" package [1] in the
sources / unit tests
> | > to test for just this. In practice you just cannot hit every
possible access
> | > point of the (rich, in our case) API so the tests pass too often.
> | >
> | > Which is why we relentlessly test against reverse-depends to _at
least ensure
> | > buildability_ from our releases.
>
> I meant to also add:  "... against a large corpus of other
packages."
> The intent is to empirically answer this.
>
> | > As for seamless binary upgrade, I don't think in can work in
practice.  Ask
> | > Uwe one day we he rebuilds everything every time on Windows. And for
what it
> | > is worth, we essentially do the same in Debian.
> | >
> | > Sometimes you just need to rebuild.  That may be the price of
admission for
> | > using the convenience of rich C++ interfaces.
> | >
> |
> | Okay, so would you say that Kirill's suggestion is not overkill? 
Every
> | time package B uses LinkingTo: A, R should assume it needs to rebuild B
> | when A is updated?
>
> Based on my experience is a "halting problem" -- i.e. cannot know
ex ante.
>
> So "every time" would be overkill to me.  Sometimes you know you
must
> recompile (but try to be very prudent with public-facing API).  Many times
> you do not. It is hard to pin down.
>
> At work we have a bunch of servers with Rcpp and many packages against them
> (installed system-wide for all users). We _very really_ needs rebuild.
So that comes back to my suggestion:  you should provide a way for a 
dependent package to ask if your API has changed.  If you say it hasn't, 
the package is fine.  If you say it has, the package should abort, 
telling the user they need to reinstall it.  (Because it's a hard 
question to answer, you might get it wrong and say it's fine when it's 
not.  But that's easy to fix:  just make a new release that does require 
a rebuild.)

Duncan Murdoch

On 16 December 2016 at 11:00, Duncan Murdoch wrote:
| On 16/12/2016 10:40 AM, Dirk Eddelbuettel wrote:
| > On 16 December 2016 at 10:14, Duncan Murdoch wrote:
| > | On 16/12/2016 8:37 AM, Dirk Eddelbuettel wrote:
| > | >
| > | > On 16 December 2016 at 08:20, Duncan Murdoch wrote:
| > | > | Perhaps the solution is to recommend that packages which export
their
| > | > | C-level entry points either guarantee them not to change or
offer
| > | > | (require?) version checks by user code.  So dplyr should start
out by
| > | > | saying "I'm using Rcpp interface 0.12.8".  If Rcpp
has a new version
| > | > | with a compatible interface, it replies "that's
fine".  If Rcpp has
| > | > | changed its interface, it says "Sorry, I don't support
that any more."
| > | >
| > | > We try. But it's hard, and I'd argue, likely impossible.
| > | >
| > | > For example I even added a "frozen" package [1] in the
sources / unit tests
| > | > to test for just this. In practice you just cannot hit every
possible access
| > | > point of the (rich, in our case) API so the tests pass too often.
| > | >
| > | > Which is why we relentlessly test against reverse-depends to _at
least ensure
| > | > buildability_ from our releases.
| >
| > I meant to also add:  "... against a large corpus of other
packages."
| > The intent is to empirically answer this.
| >
| > | > As for seamless binary upgrade, I don't think in can work in
practice.  Ask
| > | > Uwe one day we he rebuilds everything every time on Windows. And
for what it
| > | > is worth, we essentially do the same in Debian.
| > | >
| > | > Sometimes you just need to rebuild.  That may be the price of
admission for
| > | > using the convenience of rich C++ interfaces.
| > | >
| > |
| > | Okay, so would you say that Kirill's suggestion is not overkill? 
Every
| > | time package B uses LinkingTo: A, R should assume it needs to rebuild B
| > | when A is updated?
| >
| > Based on my experience is a "halting problem" -- i.e. cannot
know ex ante.
| >
| > So "every time" would be overkill to me.  Sometimes you know
you must
| > recompile (but try to be very prudent with public-facing API).  Many
times
| > you do not. It is hard to pin down.
| >
| > At work we have a bunch of servers with Rcpp and many packages against
them
| > (installed system-wide for all users). We _very really_ needs rebuild.

Edit:  "We _very rarely_ need rebuilds" is what was meant there.
 
| So that comes back to my suggestion:  you should provide a way for a 
| dependent package to ask if your API has changed.  If you say it hasn't, 
| the package is fine.  If you say it has, the package should abort, 
| telling the user they need to reinstall it.  (Because it's a hard 
| question to answer, you might get it wrong and say it's fine when it's
| not.  But that's easy to fix:  just make a new release that does require 

Sure.

We have always increased the higher-order version number when that is needed.

One problem with your proposal is that the testing code may run after the
package load, and in the case where it matters ... that very code may not get
reached because the package didn't load.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org