R SIG Debian - Mar 2010 - why no libRblas.so in ubuntu packages on CRAN?

Hi, everybody.

I notice debian packages for Ubuntu on CRAN are missing "libRblas.so"
and I'm trying to find out why.

Why do I care?  We are working in an hpc project in a CentOS Linux
cluster.  That led me to the question of "which BLAS is fastest?"
Dirk E referred me this morning to the R admin manual.  The standard R
compilation creates a shared blas library libRblas.so  and then one
can easily 'swap in a different shared library' to experiment with
GotoBLAS2 or Atlas or whatnot.  I found that quite amazing!

Then I booted up my Ubuntu laptop to try this out. I have the Ubuntu
deb packages from CRAN, version 2.10.  In r-core-base, there is no
libRblas.so.  I have:

ii  r-base                                     2.10.1-2karmic0
                             GNU R statistical computation and
graphics s
ii  r-base-core                                2.10.1-2karmic0
                             GNU R core of statistical computation and
gr
ii  r-base-dev                                 2.10.1-2karmic0
                             GNU R installation of auxiliary GNU R
packag

$ dpkg -L r-base-core r-base-dev r-base | grep .so
/etc/ld.so.conf.d
/usr/lib/R/library/splines/libs/splines.so
/usr/lib/R/library/grDevices/libs/grDevices.so
/usr/lib/R/library/methods/libs/methods.so
/usr/lib/R/library/stats/libs/stats.so
/usr/lib/R/library/tcltk/exec/console.tcl
/usr/lib/R/library/tcltk/libs/tcltk.so
/usr/lib/R/library/grid/libs/grid.so
/usr/lib/R/library/tools/libs/tools.so
/usr/lib/R/lib/libR.so
/usr/lib/R/modules/R_X11.so
/usr/lib/R/modules/vfonts.so
/usr/lib/R/modules/internet.so
/usr/lib/R/modules/lapack.so
/usr/share/R/doc/html/resources.html
/etc/ld.so.conf.d/libR.conf

As far as I can see, the Ubuntu CRAN repositories are not configured
with a deb-src option. So I can't  get the exact source code used to
create the Ubuntu packages.  Hopefully, the packager did not make
changes in the Debian source package.  The CRAN Debian repositories do
answer to deb-src requests for Debian "lenny."

I've been inspecting that for quitte a while.  The Debian build
script's configure statement does not explicitly invoke
--disable-BLAS-shlib.

       ./configure --prefix=/usr                       \
                    --with-cairo                        \
                    --with-jpeglib                      \
                    --with-pango                        \
                    --with-png                          \
                    --with-readline                     \
                    --with-tcltk                        \
                    --with-system-bzlib                 \
                    --with-system-pcre                  \
                    --with-system-zlib                  \
                    --mandir=/usr/share/man             \
                    --infodir=/usr/share/info           \
                    --datadir=/usr/share/R/share        \
                    --includedir=/usr/share/R/include   \
                    $(atlas)            \
                    $(lapack)           \
                    --without-gnome                     \
                    --enable-R-profiling                \
                    --enable-R-shlib                    \
                    --enable-memory-profiling           \
                    --without-recommended-packages      \
                    --build $(buildarch)

$(atlas) draws its value from the rules file

atlas           = --with-blas

Similarly, $(lapack)

lapack          = --with-lapack


I rebuilt the debian packages with the source from lennycran and, as
in Ubuntu, there's no libRblas.so.

$ dpkg -c r-base-core_2.10.1-2~lennycran.0_i386.deb  | grep libRblas.so

returns nothing.


I DO get libRblas.so if I make 3 changes in the debian/rules file. I deleted
1) $(atlas)
2) $(lapack), and
3) --enable-R-shlib.

I believe the R-admin manual discourages the --with-lapack option
altogether, and the discussion of that option is somewhat confusing.
--with-lapack does not mean "use lapack", it means by-pass the lapack
that R provides in its code and try to use lapack from an external
library (presumably, the lapack provided by the blas implementation).
I believe that putting --with-lapack without adding
=/some/lib/location may confuse the configure script.  I make change 3
because R-admin discourages the R-shlib option for most users.

After that, the package I build DOES include libRblas.so

$ dpkg -c r-base-core_2.10.1-2~lennycran.0_i386.deb  | grep libRblas.so
-rw-r--r-- root/root    165132 2010-03-02 22:02 ./usr/lib/R/lib/libRblas.so

I've chased this one around for a couple of hours, so I think I've
done my "due diligence."

pj
-- 
Paul E. Johnson
Professor, Political Science
1541 Lilac Lane, Room 504
University of Kansas

Paul,

On 2 March 2010 at 22:44, Paul Johnson wrote:
| Hi, everybody.
| 
| I notice debian packages for Ubuntu on CRAN are missing
"libRblas.so"
| and I'm trying to find out why.

Fortran compiler changes plus better upstream Lapack sources (3.0 was no
good) lead to changes in the configure call to R.  

| Why do I care?  We are working in an hpc project in a CentOS Linux
| cluster.  That led me to the question of "which BLAS is fastest?"
| Dirk E referred me this morning to the R admin manual.  The standard R
| compilation creates a shared blas library libRblas.so  and then one

No, the certainty in your statement in not correct.  It does that if you
build with the Blas in R --- which takes away the very plug & play I told
you
about yesterday.  Which is why we do what we do. Allow the plu & play.

| can easily 'swap in a different shared library' to experiment with
| GotoBLAS2 or Atlas or whatnot.  I found that quite amazing!

Of course you can do that here too. That happens at the SYSTEM level and
Debian's R and Octave package supported this 'amazing switch' for
maybe a
dozen years now (and I was the one initially adding to Octave and R based on
the awesome work of our maintainer of Atlas at the time).

Look at 

edd at joe:~$ ldd /usr/lib/R/bin/exec/R
        linux-gate.so.1 =>  (0x00ee9000)
        libR.so => /usr/lib/R/lib/libR.so (0x00788000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x00110000)
        libblas.so.3gf => /usr/lib/sse2/atlas/libblas.so.3gf (0x00f59000)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00da8000)
        libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0x00647000)
        libreadline.so.6 => /lib/libreadline.so.6 (0x00255000)
        libpcre.so.3 => /lib/libpcre.so.3 (0x0028e000)
        libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x003df000)
        libz.so.1 => /lib/libz.so.1 (0x002bf000)
        libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0x002d5000)
        /lib/ld-linux.so.2 (0x00f3c000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0041d000)
        libncurses.so.5 => /lib/libncurses.so.5 (0x002d9000)
edd at joe:~$


(for my machine with atlas-sse2)

and marvel how libblas comes from the _Atlas_ package because we defer 
to the OS.  Same for 

       /usr/lib/R/lib/libR.so

and

       /usr/lib/R/modules/lapack.so

| Then I booted up my Ubuntu laptop to try this out. I have the Ubuntu
| deb packages from CRAN, version 2.10.  In r-core-base, there is no

There is no package r-core-base. 

| libRblas.so.  I have:
| 
| ii  r-base                                     2.10.1-2karmic0
|                              GNU R statistical computation and
| graphics s
| ii  r-base-core                                2.10.1-2karmic0
|                              GNU R core of statistical computation and
| gr
| ii  r-base-dev                                 2.10.1-2karmic0
|                              GNU R installation of auxiliary GNU R
| packag
| 
| $ dpkg -L r-base-core r-base-dev r-base | grep .so
| /etc/ld.so.conf.d
| /usr/lib/R/library/splines/libs/splines.so
| /usr/lib/R/library/grDevices/libs/grDevices.so
| /usr/lib/R/library/methods/libs/methods.so
| /usr/lib/R/library/stats/libs/stats.so
| /usr/lib/R/library/tcltk/exec/console.tcl
| /usr/lib/R/library/tcltk/libs/tcltk.so
| /usr/lib/R/library/grid/libs/grid.so
| /usr/lib/R/library/tools/libs/tools.so
| /usr/lib/R/lib/libR.so
| /usr/lib/R/modules/R_X11.so
| /usr/lib/R/modules/vfonts.so
| /usr/lib/R/modules/internet.so
| /usr/lib/R/modules/lapack.so
| /usr/share/R/doc/html/resources.html
| /etc/ld.so.conf.d/libR.conf
| 
| As far as I can see, the Ubuntu CRAN repositories are not configured
| with a deb-src option. So I can't  get the exact source code used to
| create the Ubuntu packages.  Hopefully, the packager did not make
| changes in the Debian source package.  The CRAN Debian repositories do
| answer to deb-src requests for Debian "lenny."
| 
| I've been inspecting that for quitte a while.  The Debian build
| script's configure statement does not explicitly invoke
| --disable-BLAS-shlib.
| 
|        ./configure --prefix=/usr                       \
|                     --with-cairo                        \
|                     --with-jpeglib                      \
|                     --with-pango                        \
|                     --with-png                          \
|                     --with-readline                     \
|                     --with-tcltk                        \
|                     --with-system-bzlib                 \
|                     --with-system-pcre                  \
|                     --with-system-zlib                  \
|                     --mandir=/usr/share/man             \
|                     --infodir=/usr/share/info           \
|                     --datadir=/usr/share/R/share        \
|                     --includedir=/usr/share/R/include   \
|                     $(atlas)            \
|                     $(lapack)           \
|                     --without-gnome                     \
|                     --enable-R-profiling                \
|                     --enable-R-shlib                    \
|                     --enable-memory-profiling           \
|                     --without-recommended-packages      \
|                     --build $(buildarch)
| 
| $(atlas) draws its value from the rules file
| 
| atlas           = --with-blas
| 
| Similarly, $(lapack)
| 
| lapack          = --with-lapack

Correct. On most arches. We have to change that on some.
 
| I rebuilt the debian packages with the source from lennycran and, as
| in Ubuntu, there's no libRblas.so.
| 
| $ dpkg -c r-base-core_2.10.1-2~lennycran.0_i386.deb  | grep libRblas.so
| 
| returns nothing.

Correct. As it shouldn;t. System blas being used, not R blas.

| I DO get libRblas.so if I make 3 changes in the debian/rules file. I deleted
| 1) $(atlas)
| 2) $(lapack), and
| 3) --enable-R-shlib.

You don't want libRblas because you want the system wide blas. Change the
system wide blas to GOTO or whatever and then R, Octave, GSL, Gretl, .... 
__all__ get faster BLAS at once.

That is Debian. We provide an OS-level solution.
 
| I believe the R-admin manual discourages the --with-lapack option
| altogether, and the discussion of that option is somewhat confusing.
| --with-lapack does not mean "use lapack", it means by-pass the
lapack
| that R provides in its code and try to use lapack from an external
| library (presumably, the lapack provided by the blas implementation).
| I believe that putting --with-lapack without adding
| =/some/lib/location may confuse the configure script.  I make change 3
| because R-admin discourages the R-shlib option for most users.
| 
| After that, the package I build DOES include libRblas.so
| 
| $ dpkg -c r-base-core_2.10.1-2~lennycran.0_i386.deb  | grep libRblas.so
| -rw-r--r-- root/root    165132 2010-03-02 22:02 ./usr/lib/R/lib/libRblas.so
| 
| I've chased this one around for a couple of hours, so I think I've
| done my "due diligence."

R can only recommend narrowly within the R stack. It has no control over the
distro and doesn't recommend.

If you need a different r-base-core configuration for your _local_ purposes,
just build a different _locale_ r-base-core package.

I encourage you to do so. If you find significant timing advantages, let me
know and I would be glad to improve the package. So far, I do not believe you
have suggested an improvement.  But this is tricky stuff and I appreciate
that you did your due diligence.


Now, what you missed is the fact that (Revolution) R 2.9.2 in Ubuntu Karmic
comes shipped with the Intel MKL -- in a directory local to R -- which are
even faster than Goto BLAS, multithreaded and typically only available if you
pay monies to Intel. But as Intel is friendly to REvo, these became
available.  They are truly multithreaded and should be much faster.  I meant
to benchmark them more closely but didn't get around to it.  

Dirk

-- 
  Registration is open for the 2nd International conference R / Finance 2010
  See http://www.RinFinance.com for details, and see you in Chicago in April!