R SIG Debian - Oct 2020 - Installing package fails at "testing if installed package can be loaded from temporary location"

Hi,

I can not install packages from source which links to RcppArmadillo on
Ubuntu 20.04 (after upgrading from 18.04). The following problem occurs:

** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for 'myPackage' in
dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object
'/usr/local/lib/R/site-library/00LOCK-my_package/00new/myPackage/libs/myPackage.so':
 
/usr/local/lib/R/site-library/00LOCK-my_package/00new/myPackge/libs/myPackage.so:
undefined symbol: dgesvx_
Error: loading failed
Execution halted
ERROR: loading failed

This dgesvx_ looked to me like a LAPACK problem (
https://www.netlib.org/lapack/explore-html/d7/d3b/group__double_g_esolve_ga9d90ccf6e340cacd08b7bbbb502ceb21.html#ga9d90ccf6e340cacd08b7bbbb502ceb21
).

I was playing with different BLAS/LAPACK versions on my system and
researching this for a couple of days now but without success.

I was first looking at which LAPACK and BLAS R is "using"

user at user:~$ ps aux | grep exec/R
user+   11663  1.9  1.7 1030952 415240 pts/0  Sl+  06:33   0:13
/usr/lib/R/bin/exec/R --vanilla --quiet
user+   13836  2.5  0.2 441664 52024 pts/1    Sl+  06:44   0:00
/usr/lib/R/bin/exec/R
user+   13932  0.0  0.0   8900   660 pts/2    S+   06:45   0:00 grep exec/R
user at user:~$ lsof -p 13836 | grep 'blas\|lapack'
R       13836 user  mem    REG    8,1 33562376  3961750
/usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
R       13836 user  mem    REG    8,1 27200552  3961749
/usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
user at user:~$ ls -l /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
-rw-r--r-- 1 root root 27200552 Jun  2 06:55
/usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
user at user:~$ ls -l /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
-rw-r--r-- 1 root root 33562376 Jun  2 06:55
/usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

And then tried different versions:

user at user:~$ sudo update-alternatives --config libblas.so.3-x86_64-linux-gnu
There are 4 choices for the alternative libblas.so.3-x86_64-linux-gnu
(providing /usr/lib/x86_64-linux-gnu/libblas.so.3).

  Selection    Path
Priority   Status
------------------------------------------------------------
* 0            /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
100       auto mode
  1            /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3              35
       manual mode
  2            /usr/lib/x86_64-linux-gnu/blas/libblas.so.3               10
       manual mode
  3            /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so.3    95
       manual mode
  4            /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
100       manual mode

Press <enter> to keep the current choice[*], or type selection number: 3
update-alternatives: using
/usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so.3 to provide
/usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in
manual mode
user at user:~$ sudo update-alternatives --config
liblapack.so.3-x86_64-linux-gnu
There are 4 choices for the alternative liblapack.so.3-x86_64-linux-gnu
(providing /usr/lib/x86_64-linux-gnu/liblapack.so.3).

  Selection    Path
Priority   Status
------------------------------------------------------------
* 0            /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
100       auto mode
  1            /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3
 35        manual mode
  2            /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3
10        manual mode
  3            /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so.3
 95        manual mode
  4            /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
100       manual mode

Press <enter> to keep the current choice[*], or type selection number: 3
update-alternatives: using
/usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so.3 to provide
/usr/lib/x86_64-linux-gnu/liblapack.so.3 (liblapack.so.3-x86_64-linux-gnu)
in manual mode
user at user:~$ sudo update-alternatives --config libblas.so.3-x86_64-linux-gnu
There are 4 choices for the alternative libblas.so.3-x86_64-linux-gnu
(providing /usr/lib/x86_64-linux-gnu/libblas.so.3).

  Selection    Path
Priority   Status
------------------------------------------------------------
  0            /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
100       auto mode
  1            /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3              35
       manual mode
  2            /usr/lib/x86_64-linux-gnu/blas/libblas.so.3               10
       manual mode
* 3            /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so.3    95
       manual mode
  4            /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
100       manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to
provide /usr/lib/x86_64-linux-gnu/libblas.so.3
(libblas.so.3-x86_64-linux-gnu) in manual mode
user at user:~$ sudo update-alternatives --config
liblapack.so.3-x86_64-linux-gnu
There are 4 choices for the alternative liblapack.so.3-x86_64-linux-gnu
(providing /usr/lib/x86_64-linux-gnu/liblapack.so.3).

  Selection    Path
Priority   Status
------------------------------------------------------------
  0            /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
100       auto mode
  1            /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3
 35        manual mode
  2            /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3
10        manual mode
* 3            /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so.3
 95        manual mode
  4            /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
100       manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3
to provide /usr/lib/x86_64-linux-gnu/liblapack.so.3
(liblapack.so.3-x86_64-linux-gnu) in manual mode
I have no idea what could be wrong here... Thanks in advance for hints and
help on what to do here.

Best regards,
Sam

	[[alternative HTML version deleted]]

Hi Sam,

Thanks for bringing this here. It is the right place.

And for what it is worth I also use 20.04, and update RcppArmadillo
frequently (currently working on a pre-release)

On 8 October 2020 at 14:37, Sam H wrote:
| Hi,
| 
| I can not install packages from source which links to RcppArmadillo on
| Ubuntu 20.04 (after upgrading from 18.04). The following problem occurs:
| 
| ** testing if installed package can be loaded from temporary location
| Error: package or namespace load failed for 'myPackage' in
| dyn.load(file, DLLpath = DLLpath, ...):
|  unable to load shared object
|
'/usr/local/lib/R/site-library/00LOCK-my_package/00new/myPackage/libs/myPackage.so':
|  
/usr/local/lib/R/site-library/00LOCK-my_package/00new/myPackge/libs/myPackage.so:
| undefined symbol: dgesvx_

Ok, this could be two things. Are you using the R package for Ubuntu from
CRAN, or directly from the Rutter PPA ?  All this works best with R being
compiled using shared libraries and external LAPACK/BLAS.  So in R itself,
when you type sessionInfo() you should see two lines like this:

  Matrix products: default
  BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
  LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

Second, for any locally compiled package, `ldd` on that .so file should
resolve LAPACK and BLAS:

  edd at rob:~$ ldd
/usr/local/lib/R/site-library/RcppZiggurat/libs/RcppZiggurat.so | grep
"blas\|lapack"
          libgslcblas.so.0 => /usr/lib/x86_64-linux-gnu/libgslcblas.so.0
(0x00007effdd3ab000)
          libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3
(0x00007effdb9ac000)
  edd at rob:~$ 

| Error: loading failed
| Execution halted
| ERROR: loading failed
| 
| This dgesvx_ looked to me like a LAPACK problem (
|
https://www.netlib.org/lapack/explore-html/d7/d3b/group__double_g_esolve_ga9d90ccf6e340cacd08b7bbbb502ceb21.html#ga9d90ccf6e340cacd08b7bbbb502ceb21
| ).
| 
| I was playing with different BLAS/LAPACK versions on my system and
| researching this for a couple of days now but without success.
| 
| I was first looking at which LAPACK and BLAS R is "using"
| 
| user at user:~$ ps aux | grep exec/R
| user+   11663  1.9  1.7 1030952 415240 pts/0  Sl+  06:33   0:13
| /usr/lib/R/bin/exec/R --vanilla --quiet
| user+   13836  2.5  0.2 441664 52024 pts/1    Sl+  06:44   0:00
| /usr/lib/R/bin/exec/R
| user+   13932  0.0  0.0   8900   660 pts/2    S+   06:45   0:00 grep exec/R
| user at user:~$ lsof -p 13836 | grep 'blas\|lapack'
| R       13836 user  mem    REG    8,1 33562376  3961750
| /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

Same as mine above, found via a more hardcore way :)

| R       13836 user  mem    REG    8,1 27200552  3961749
| /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
| user at user:~$ ls -l /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
| -rw-r--r-- 1 root root 27200552 Jun  2 06:55
| /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
| user at user:~$ ls -l
/usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
| -rw-r--r-- 1 root root 33562376 Jun  2 06:55
| /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
| 
| And then tried different versions:
| 
| user at user:~$ sudo update-alternatives --config
libblas.so.3-x86_64-linux-gnu
| There are 4 choices for the alternative libblas.so.3-x86_64-linux-gnu
| (providing /usr/lib/x86_64-linux-gnu/libblas.so.3).
| 
|   Selection    Path
| Priority   Status
| ------------------------------------------------------------
| * 0            /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
| 100       auto mode
|   1            /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3              35
|        manual mode
|   2            /usr/lib/x86_64-linux-gnu/blas/libblas.so.3               10
|        manual mode
|   3            /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so.3    95
|        manual mode
|   4            /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
| 100       manual mode

Also fine.
 
| Press <enter> to keep the current choice[*], or type selection number: 3
| update-alternatives: using
| /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so.3 to provide
| /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in
| manual mode
| user at user:~$ sudo update-alternatives --config
| liblapack.so.3-x86_64-linux-gnu
| There are 4 choices for the alternative liblapack.so.3-x86_64-linux-gnu
| (providing /usr/lib/x86_64-linux-gnu/liblapack.so.3).
| 
|   Selection    Path
| Priority   Status
| ------------------------------------------------------------
| * 0            /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
| 100       auto mode
|   1            /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3
|  35        manual mode
|   2            /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3
| 10        manual mode
|   3            /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so.3
|  95        manual mode
|   4            /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
| 100       manual mode
| 
| Press <enter> to keep the current choice[*], or type selection number: 3
| update-alternatives: using
| /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so.3 to provide
| /usr/lib/x86_64-linux-gnu/liblapack.so.3 (liblapack.so.3-x86_64-linux-gnu)
| in manual mode
| user at user:~$ sudo update-alternatives --config
libblas.so.3-x86_64-linux-gnu
| There are 4 choices for the alternative libblas.so.3-x86_64-linux-gnu
| (providing /usr/lib/x86_64-linux-gnu/libblas.so.3).
| 
|   Selection    Path
| Priority   Status
| ------------------------------------------------------------
|   0            /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
| 100       auto mode
|   1            /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3              35
|        manual mode
|   2            /usr/lib/x86_64-linux-gnu/blas/libblas.so.3               10
|        manual mode
| * 3            /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so.3    95
|        manual mode
|   4            /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
| 100       manual mode
| 
| Press <enter> to keep the current choice[*], or type selection number: 2
| update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to
| provide /usr/lib/x86_64-linux-gnu/libblas.so.3
| (libblas.so.3-x86_64-linux-gnu) in manual mode
| user at user:~$ sudo update-alternatives --config
| liblapack.so.3-x86_64-linux-gnu
| There are 4 choices for the alternative liblapack.so.3-x86_64-linux-gnu
| (providing /usr/lib/x86_64-linux-gnu/liblapack.so.3).
| 
|   Selection    Path
| Priority   Status
| ------------------------------------------------------------
|   0            /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
| 100       auto mode
|   1            /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3
|  35        manual mode
|   2            /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3
| 10        manual mode
| * 3            /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so.3
|  95        manual mode
|   4            /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
| 100       manual mode
| 
| Press <enter> to keep the current choice[*], or type selection number: 2
| update-alternatives: using /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3
| to provide /usr/lib/x86_64-linux-gnu/liblapack.so.3
| (liblapack.so.3-x86_64-linux-gnu) in manual mode
| I have no idea what could be wrong here... Thanks in advance for hints and
| help on what to do here.

It is possible to 'bust' one's system by playing with
update-alternatives. I
usually don't touch it, or make triple-sure I am doing it right.

I would check whether linear algebra currently works for you. One check:

   $ Rscript -e 'library(RcppArmadillo); example("fastLm")

Dirk

| Best regards,
| Sam
| 
| 	[[alternative HTML version deleted]]
| 
| _______________________________________________
| R-SIG-Debian mailing list
| R-SIG-Debian at r-project.org
| https://stat.ethz.ch/mailman/listinfo/r-sig-debian

-- 
https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org

Sam,

PS It could of course also be something else: Is you src/Makevars for your
test package correct?  Did you use `RcppArmadillo.packages.skeleton()` ? You
need the lapack/blas link instruction therein.  See

edd at rob:~$ cat /usr/local/lib/R/site-library/RcppArmadillo/skeleton/Makevars

## With R 3.1.0 or later, you can uncomment the following line to tell R to 
## enable compilation with C++11 (where available)
##
## Also, OpenMP support in Armadillo prefers C++11 support. However, for wider
## availability of the package we do not yet enforce this here.  It is however
## recommended for client packages to set it.
##
## And with R 3.4.0, and RcppArmadillo 0.7.960.*, we turn C++11 on as OpenMP
## support within Armadillo prefers / requires it
CXX_STD = CXX11

PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS) 
PKG_LIBS = $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
edd at rob:~$ 


Dirk

-- 
https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org

Dirk, thank you a thousand times.

Indeed, src/Makevars was wrong. I modified Makevars so that now looks like
the below and the package now compiled and linked properly.
CXX_STD = CXX11

PKG_LIBS += $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
$(shell ${R_HOME}/bin/Rscript -e "RcppParallel::RcppParallelLibs()")
PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS) -I../inst/include

Best,
Sam

On Thu, Oct 8, 2020 at 3:39 PM Dirk Eddelbuettel <edd at debian.org>
wrote:
>
> Sam,
>
> PS It could of course also be something else: Is you src/Makevars for your
> test package correct?  Did you use `RcppArmadillo.packages.skeleton()` ?
> You
> need the lapack/blas link instruction therein.  See
>
> edd at rob:~$ cat
> /usr/local/lib/R/site-library/RcppArmadillo/skeleton/Makevars
>
> ## With R 3.1.0 or later, you can uncomment the following line to tell R
> to
> ## enable compilation with C++11 (where available)
> ##
> ## Also, OpenMP support in Armadillo prefers C++11 support. However, for
> wider
> ## availability of the package we do not yet enforce this here.  It is
> however
> ## recommended for client packages to set it.
> ##
> ## And with R 3.4.0, and RcppArmadillo 0.7.960.*, we turn C++11 on as
> OpenMP
> ## support within Armadillo prefers / requires it
> CXX_STD = CXX11
>
> PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
> PKG_LIBS = $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
> edd at rob:~$
>
>
> Dirk
>
> --
> https://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
>

-- 
Best regards,
Sam

	[[alternative HTML version deleted]]