R SIG Debian - Aug 2023 - pinning of binary r-cran-* packages from c2d4u / r2u on Ubuntu 22.04

Am 09.08.2023 um 18:17 schrieb Dirk Eddelbuettel:
> 
> Hi Thomas,
> 
> Thanks a lot for bringing this over from r-package-devel!
And thanks for the quick help from my side!
> As for your question, "there is a lot of meat on these bones" and
I really
> think we should discuss this here.
> 
> Quick comments inline, more recap at the bottom.
> 
> On 9 August 2023 at 17:29, Thomas Petzoldt wrote:
> | Hi,
> |
> | I am running a couple of shiny servers with several apps that are based
> | around own CRAN packages. It worked stable for years, but due to the
> | growing number of packages, the compile time for regular manual package
> | installation and updates became inconvenient.
> |
> | Therefore, I have been very happy to use pre-compiled packages from the
> | c2d4u repository  -- and from now on testing r2u following a hint from
> | Dirk Edelbuettel. @Dirk: installation worked like a charm.
> 
> Glad to hear!
Indeed. Just updated 58 packages and all apps continued working.

> It also generally works flawlessly (and repeatedly) for myself and others
in
> many context involving many packages.  Per the logs, we have shipped over
6.5
> million packages in the ~ 15 months it has been up and generally do more
than
> 10k each day now.
> 
> | The downside is, that the r-cran-* binaries are being installed
> | automatically, together with the system update. So experienced repeated
> | cases that crashed some of the shiny apps. The main reason with c2d4u
> | were conflicts between the binary packages and other packages installed
> | from sources. This may be less relevant with r2u, but in the interest of
> | reproducibility and stability, I would prefer conservative manual
updates.
> 
> Yes. If and when that happens, it is a bug!
Not necessarily. Breaking code can also be due to an incompatible 
feature update. We know that R has both, conservative and less 
conservative universes of packages ;-) Security is another issue.
> The correct (but laborious way) is to file and report them.
Hmm, depends. I am happy to contribute bug reports, but need evidence 
that it's a bug and not a feature - or configuration issue on my side.
> I think the overall issue here is the mix and match from 'distro'
packages,
> and added packages. Given the complete coverage, I prefer r2u for
> consistency.  Which is why it is pinned higher at 700.
Agreed, so I applied the r2u pinning as descibed on your
page.
> But one can have issue if because of ("semi-random") package
renmaing /
> versioning / use of epoch [a leading version digit overriding] the distro
> wins.
> 
> Eg we had a recent issue for r2u when following the Debian and the R 4.3.*
> catchup, several packages (all around the graphics API that needed a
rebuilt)
> got force-rebuilt and ended up with, say, 1.2.3-2 beating
1.2.3-1.c2u2204.1.
> So I quickly rebuilt that dozen. It was more of an issue for jammy (22.04)
> than focal as the (distro) focal packages are generally way behind what
CRAN
> and hence r2u has.
> 
> The only really reliable way to fix this is by reporting the bugs the old
> school way so that r2u can take care of it.  I would need help from users
and
> that is why I am asking for it here.
>   
> | My question: what is best practise, to disallow automatic updates for
> | all (or part of) r-cran-* packages? Uncommenting the complete package
> | source in the apt/sources.list.d/cd4u...list file? Fiddling around with
> | /etc/apt/preferences ?
> |
> | The ideal approach would be to put a plain textfile of all installed
> | r-cran packages somewhere to the system, where packages that are to be
> | upgraded (or oppositely: pinned) are just commented or outcommented.
> 
> I mentioned in the initial short reply on the other list I can see several
> options (now up to four)
> 
> 1) use dpkg to put packages on 'hold'. They will not be touched or
> updated. Reliable, manual, tedious for many. See 'man dpkg'.
This (or same with apt-mark) is probably the easiest: Write a shell (or 
R) script iterating over a text file of packages. Should work similar to 
a traditional "install.packsges()"-script if almost all packages are 
from r2u and only a few directly from Github.
> 2) look into 'apt pinning' via preferences and its wiki page at
> https://wiki.debian.org/AptConfiguration etc   This may work, I have not
had
> the patience to work the details out.
Same to me.
>  But there are other value beyond the
> '500 to 900' default range. This may work, and be quite elegant.
> 
> 3) as you hint, just 'hide' the sources.list for r2u or c2d4u but
it still
> risks updates from the distro versions :-/  Maybe 'easiest yet
worst'.
Good point. This is what I have been afraid of.
> 4) maybe go the other way use eg the Rocker Project container for shiny and
> run you sacred production in a shielded setting? This sounds like more
work,
> but it may be better. If it works for you depends.
Yes, I' just discussed a switch to docker with my colleague during 
lunch. Maybe the university will install a central Shiny service in the 
future or we buy the service ... For now, I am still happy with my very 
simplistic solution with one VM running a load balancing Apache proxy 
and one or more other VMs with multiple shiny server instances (the free 
version). It has its limitations, but is a pragmatic setting for 
research projects and university courses.
> Let's keep at this and see if we can make something 'already pretty
good'
> even better!
All right, I'll stay tuned and give feedback.
> 
> Cheers,  Dirk
> 
Thanks a lot,

Thomas

Thomas,

There is a fifth option I just realized.

I use a variant of that to keep for example an old version of roxygen2 I
prefer available. Recall that R packages are generally relocatable.  So
install the 58 you need, then do, say `mkdir /opt/shiny-r-lib` and copy them
in there. Setup your shiny deployment with whatever R_LIBS_USER or
R_LIBS_SITE works for, or simply add `.libPaths("/opt/shiny-r-lib")`.

Now those packages will be found first, used by Shiny and cannot be touched
by install.packages (be it with `apt` via `r2u` or not). Easy peasy "poor
man's virtual environemnt".

On 9 August 2023 at 18:58, Thomas Petzoldt wrote:
| Am 09.08.2023 um 18:17 schrieb Dirk Eddelbuettel:
| > Yes. If and when that happens, it is a bug!
| 
| Not necessarily. Breaking code can also be due to an incompatible 
| feature update. We know that R has both, conservative and less 
| conservative universes of packages ;-) Security is another issue.

I am talking more narrowly about the failure to install a package, or getting
the wrong package, or ... as a bug in the r2u setup and provision. Those bugs
we should squash. All others continues to exist, and we fight them in
different venues.

Dirk

-- 
dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org