R devel - Sep 2016 - Building R under Linux - library dependencies

> | Is there a way to overcome this problem? Precompiled versions of R can be
installed on various system configurations, so I guess that there should be a
way to compile it in a version-agnostic manner.
> 
> Yes, for example by
> 
>   -- using a Docker container which is portable across OSs (!!) and
versions
Docker R containers are north of 250 MB. I have checked experimentally that you
can trim R down to 16 MB (!) and you'll still be able to execute it (though
with warnings). That *is* quite a difference, especially when deploying small
applications.
>   -- relying on package management which is what every Linux distro does
> 
> (...)
> 
> PS For the latter point, our .deb based R package currently shows this:
> 
> (...)
> 
> Depends: zip, unzip, libpaper-utils, xdg-utils, libblas3 | libblas.so.3,
libbz2-1.0, libc6 (>= 2.23), libcairo2 (>= 1.6.0), libcurl3 (>=
7.28.0), libglib2.0-0 (>= 2.12.0), libgomp1 (>= 4.9), libjpeg8 (>= 8c),
liblapack3 | liblapack.so.3, liblzma5 (>= 5.1.1alpha+20120614),
libpango-1.0-0 (>= 1.14.0), libpangocairo-1.0-0 (>= 1.14.0), libpcre3,
libpng12-0 (>= 1.2.13-4), libreadline6 (>= 6.0), libtcl8.6 (>= 8.6.0),
libtiff5 (>= 4.0.3), libtk8.6 (>= 8.6.0), libx11-6, libxt6, zlib1g (>=
1:1.1.4), ucf (>= 3.0), ca-certificates
Sure, package dependencies would be great as well - at least you'd be sure
that users of, say, Debian-based distros will be able to run this portable R, as
long as they've installed the required libraries. But notice that in your
example package versions equal *or greater* than listed are required - so if
someone has upgraded their system, they still will be able to run that R. With a
version built from source you need *exactly* the same version as on the machine
where R was compiled. Hence my question: how come the precompiled distribution
of R has "less strict" library requirements than manually compiled
versions?

Best,
Pawel

On Wed, Sep 7, 2016 at 1:50 PM, Pawe? Pi?tkowski <cosi1 at tlen.pl> wrote:
> > | Is there a way to overcome this problem? Precompiled versions of R
can
> be installed on various system configurations, so I guess that there should
> be a way to compile it in a version-agnostic manner.
> >
> > Yes, for example by
> >
> >   -- using a Docker container which is portable across OSs (!!) and
> versions
>
> Docker R containers are north of 250 MB. I have checked experimentally
> that you can trim R down to 16 MB (!) and you'll still be able to
execute
> it (though with warnings). That *is* quite a difference, especially when
> deploying small applications.


... I would guesstimate the libraries required to run R with any useful set
of libraries is quite a bit bigger than the cited 16M .......


> >   -- relying on package management which is what every Linux distro
does
> >
> > (...)
> >
> > PS For the latter point, our .deb based R package currently shows
this:
> >
> > (...)
> >
> > Depends: zip, unzip, libpaper-utils, xdg-utils, libblas3 |
libblas.so.3,
> libbz2-1.0, libc6 (>= 2.23), libcairo2 (>= 1.6.0), libcurl3 (>=
7.28.0),
> libglib2.0-0 (>= 2.12.0), libgomp1 (>= 4.9), libjpeg8 (>= 8c),
liblapack3 |
> liblapack.so.3, liblzma5 (>= 5.1.1alpha+20120614), libpango-1.0-0
(>> 1.14.0), libpangocairo-1.0-0 (>= 1.14.0), libpcre3, libpng12-0
(>> 1.2.13-4), libreadline6 (>= 6.0), libtcl8.6 (>= 8.6.0), libtiff5
(>> 4.0.3), libtk8.6 (>= 8.6.0), libx11-6, libxt6, zlib1g (>=
1:1.1.4), ucf (>> 3.0), ca-certificates
>
> Sure, package dependencies would be great as well - at least you'd be
sure
> that users of, say, Debian-based distros will be able to run this portable
> R, as long as they've installed the required libraries. But notice that
in
> your example package versions equal *or greater* than listed are required -
> so if someone has upgraded their system, they still will be able to run
> that R. With a version built from source you need *exactly* the same
> version as on the machine where R was compiled. Hence my question: how come
> the precompiled distribution of R has "less strict" library
requirements
> than manually compiled versions?
>
Package managers don't usually cite 'less than' versions for
packages -
because how do you assert a version that won't work when it hasn't been
released yet?

You could go on a tear and build statically linked versions of
R-with-everything-you-need, and maybe avoid the library madness... but this
is sort of a fool's errand and a huge consumer of time.  OS vendors and
compiler developers have stopped doing things that way for reasons.... it's
much simpler to reduce duplication and make everything work - while
allowing for patching out security issues - when you are *just slightly*
more flexible.

ABI compatibility and library versioning are, I think, fairly well
understood....

Doing this stuff with a container is very much the easiest route, if you
actually want it to be completely portable.  You're certainly welcome to
start with an Alpine Linux base and add R on top and then start paring...
but I start to not understand the point, somewhere in there....  it's a lot
of time spent on something that doesn't seem that beneficial when you've
got (even fairly reasonably modern) hardware that can deal with a tiny bit
of extra bloat.  SD cards and USB sticks are pretty cheap everywhere, now,
aren't they?

I could say, maybe, putting time into this as some kind of retrocomputing
project... but probably not otherwise.

best,

--e

	[[alternative HTML version deleted]]

> > Docker R containers are north of 250 MB. I have checked experimentally
that you can trim R down to 16 MB (!) and you'll still be able to execute it
(though with warnings). That *is* quite a difference, especially when deploying
small applications.
?
> ... I would guesstimate the libraries required to run R with any useful set
of libraries is quite a bit bigger than the cited 16M .......?
?
Maybe. The minimal usable subset is about 37 MB, add a few custom libraries,
code of your application etc... But it's *still* much less than 250 MB.
?
> > Sure, package dependencies would be great as well - at least you'd
be sure that users of, say, Debian-based distros will be able to run this
portable R, as long as they've installed the required libraries. But notice
that in your example package versions equal *or greater* than listed are
required - so if someone has upgraded their system, they still will be able to
run that R. With a version built from source you need *exactly* the same version
as on the machine where R was compiled. Hence my question: how come the
precompiled distribution of R has "less strict" library requirements
than manually compiled versions?
?
> Package managers don't usually cite 'less than' versions for
packages - because how do you assert a version that won't work when it
hasn't been released yet?
I meant that manually built versions of R (at least those compiled by me) are
fixed at a certain version of dynamic libraries - the same as installed on the
machine R was compiled on. You can't run this compiled R on an upgraded
configuration.
?
> You could go on a tear and build statically linked versions of
R-with-everything-you-need, and maybe avoid the library madness... but this is
sort of a fool's errand and a huge consumer of time.? OS vendors and
compiler developers have stopped doing things that way for reasons.... it's
much simpler to reduce duplication and make everything work - while allowing for
patching out security issues - when you are *just slightly* more flexible.
Why link the libraries statically? Most Linux distributions make symlinks to
dynamically linked libraries - so you have for example libicuuc.so that links to
libicuuc.so.XX (where XX is the version number). Why not rely on these generic
names?
?
> Doing this stuff with a container is very much the easiest route, if you
actually want it to be completely portable.? You're certainly welcome to
start with an Alpine Linux base and add R on top and then start paring... but I
start to not understand the point, somewhere in there.... ?it's a lot of
time spent on something that doesn't seem that beneficial when you've
got (even fairly reasonably modern) hardware that can deal with a tiny bit of
extra bloat.? SD cards and USB sticks are pretty cheap everywhere, now,
aren't they?
> 
> I could say, maybe, putting time into this as some kind of retrocomputing
project... but probably not otherwise.
Potential users who would have to download 250 megabytes beg to differ ;-)

Best,
-p-

On 7 September 2016 at 20:50, Pawe? Pi?tkowski wrote:
| > | Is there a way to overcome this problem? Precompiled versions of R can
be installed on various system configurations, so I guess that there should be a
way to compile it in a version-agnostic manner.
| > 
| > Yes, for example by
| > 
| >   -- using a Docker container which is portable across OSs (!!) and
versions
| 
| Docker R containers are north of 250 MB. I have checked experimentally that
you can trim R down to 16 MB (!) and you'll still be able to execute it
(though with warnings). That *is* quite a difference, especially when deploying
small applications.

You are not enumerating your trade-offs very well. There are natural
conflicts. What is you really want?

- Being able to pre-build and distribute?  We have done that since the last
5C1990s with .deb packages.

- Being able to install with minimal size?  Have you queried your users?  I
note that among the Docker containers for R (in the "Rocker" project
Carl and
I run) the _larger_ ones containing RStudio plus optionally "lots from
hadley" plus optionally lots of rOpenSci tend to me _more_ popular (for
ease
of installation of the aggregate).

And while share the overall sentiment a little bit, you have to realize that
it is 2016 with the corresponding bandwith and storage:

  edd at max:~$ du -csh /usr/local/lib/R/site-library/
  1.5G    /usr/local/lib/R/site-library/
  1.5G    total
  edd at max:~$

And that it _outside_ of R itself, or the (numerous) other shared libraries.

| >   -- relying on package management which is what every Linux distro does
| > 
| > (...)
| > 
| > PS For the latter point, our .deb based R package currently shows this:
| > 
| > (...)
| > 
| > Depends: zip, unzip, libpaper-utils, xdg-utils, libblas3 | libblas.so.3,
libbz2-1.0, libc6 (>= 2.23), libcairo2 (>= 1.6.0), libcurl3 (>=
7.28.0), libglib2.0-0 (>= 2.12.0), libgomp1 (>= 4.9), libjpeg8 (>= 8c),
liblapack3 | liblapack.so.3, liblzma5 (>= 5.1.1alpha+20120614),
libpango-1.0-0 (>= 1.14.0), libpangocairo-1.0-0 (>= 1.14.0), libpcre3,
libpng12-0 (>= 1.2.13-4), libreadline6 (>= 6.0), libtcl8.6 (>= 8.6.0),
libtiff5 (>= 4.0.3), libtk8.6 (>= 8.6.0), libx11-6, libxt6, zlib1g (>=
1:1.1.4), ucf (>= 3.0), ca-certificates
| 
| Sure, package dependencies would be great as well - at least you'd be sure
that users of, say, Debian-based distros will be able to run this portable R, as
long as they've installed the required libraries. But notice that in your
example package versions equal *or greater* than listed are required - so if
someone has upgraded their system, they still will be able to run that R. With a
version built from source you need *exactly* the same version as on the machine
where R was compiled. Hence my question: how come the precompiled distribution
of R has "less strict" library requirements than manually compiled
versions?

This is not the list for internals of how Linux packaging works, but if you
took the question to debian-user or debian-devel you would like get a pretty
qualified answer.  That dependency resolution system has been refined for
well over 20 years, so don't expect one sentence answers.

Good luck,  Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org

> You are not enumerating your trade-offs very well. There are natural
> conflicts. What is you really want?
> 
> - Being able to pre-build and distribute?  We have done that since the last
> 5C1990s with .deb packages.
> 
> - Being able to install with minimal size?  Have you queried your users?  I
> note that among the Docker containers for R (in the "Rocker"
project Carl and
> I run) the _larger_ ones containing RStudio plus optionally "lots from
> hadley" plus optionally lots of rOpenSci tend to me _more_ popular
(for ease
> of installation of the aggregate).
> 
> And while share the overall sentiment a little bit, you have to realize
that
> it is 2016 with the corresponding bandwith and storage:
> 
>   edd at max:~$ du -csh /usr/local/lib/R/site-library/
>   1.5G    /usr/local/lib/R/site-library/
>   1.5G    total
>   edd at max:~$
> 
> And that it _outside_ of R itself, or the (numerous) other shared
libraries.
OK, to be honest, it was rather a proof-of-concept than a specific idea. Other
interpreted and VM-based languages have robust app deployment systems with
smaller footprint, so I thought that it would be nice to have something similar
in R.
Maybe you are right and neither R developers, nor users actually need it.

Thanks for the discussion,
-p-