R devel - Apr 2016 - Fresh build from source of R-3.2.5 failing "make check" under 64-bit Ubuntu

Hi,

Let me justify building R from source. While I can obtain R built for me
under Ubuntu, I tend to build it from scratch as we use a NFS-shared
build as well as shared R libraries for all the bioinfo staff at this
company. Only one build and one set of packages to ensure uniformity
across all workstations.

My problem is that despite using a pretty standard build process, I am
failing at the "make check" step. I don't see what I might have
done
that is causing the failure.

My platform is:
> uname -a
Linux littlebourne 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 20:50:15
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
Ubuntu 14.04.4 LTS \n \l

I've downloaded the recent R-3.2.5 tar-gz file and gone through the
standard build, though in a non-standard location. I keep copies of the
tar-gzipped file in an "Archive" directory.
> gunzip -c Archive/R-3.2.5.tar.gz | tar xf -
> cd R-3.2.5/
> ./configure --prefix=/opt/apps/x86_64/R/R-3.2.5 LIBnn=lib
[ ... much output ...]
R is now configured for x86_64-pc-linux-gnu

  Source directory:          .
  Installation directory:    /opt/apps/x86_64/R/R-3.2.5

  C compiler:                gcc -std=gnu99  -g -O2
  Fortran 77 compiler:       gfortran  -g -O2

  C++ compiler:              g++  -g -O2
  C++ 11 compiler:           g++  -std=c++11 -g -O2
  Fortran 90/95 compiler:    gfortran -g -O2
  Obj-C compiler:         gcc -g -O2 -fobjc-exceptions

  Interfaces supported:      X11, tcltk
  External libraries:        readline, zlib, bzlib, lzma, PCRE
  Additional capabilities:   PNG, JPEG, TIFF, NLS, cairo
  Options enabled:           shared BLAS, R profiling

  Capabilities skipped:      ICU
  Options not enabled:       memory profiling

  Recommended packages:      yes

# Note: I use tcsh, not bash
> make >& make.out &
# There are no obvious errors and the 'make' proceeds finally through
some Java configuration
# and then exits with a zero status.
> make check >& make_check.out &
[ ... much output captured ...]
make[3]: Entering directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
running code in 'reg-tests-1a.R' ... OK
running code in 'reg-tests-1b.R' ... OK
running code in 'reg-tests-1c.R' ... OK
running code in 'reg-tests-2.R' ... OK
  comparing 'reg-tests-2.Rout' to './reg-tests-2.Rout.save' ...
OK
running code in 'reg-examples1.R' ... OK
running code in 'reg-examples2.R' ... OK
running code in 'reg-packages.R' ...make[3]: *** [reg-packages.Rout]
Error 1
make[3]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
make[2]: *** [test-Reg] Error 2
make[2]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
make[1]: *** [test-all-basics] Error 1
make[1]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
make: *** [check] Error 2
> cd tests
> tail -30 reg-packages.Rout.fail
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (pkgB)
Loading required package: pkgB
building package exNSS4 ...

installing package exNSS4 using file exNSS4_1.1.tar.gz ...
* installing *source* package 'exNSS4' ...
** R
** preparing package for lazy loading
Creating a generic function for 'plot' from package 'graphics'
in
package 'exNSS4'
** help
No man pages found in package  'exNSS4'
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (exNSS4)
Loading required package: exNSS4
> (res <- installed.packages(lib.loc = "myLib", priority =
"NA"))
      Package LibPath Version Priority Depends   Imports LinkingTo Suggests
myTst "myTst" "myLib" "1.0"   NA      
"methods" NA      NA        NA
      Enhances License                     License_is_FOSS
myTst NA       "What license is it under?" NA            
      License_restricts_use OS_type MD5sum NeedsCompilation Built 
myTst NA                    NA      NA     NA              
"3.2.5"
> stopifnot(identical(res[,"Package"], setNames(,sort(c(p.lis,
"myTst")))),
+       res[,"LibPath"] == "myLib")
Error: identical(res[, "Package"], setNames(, sort(c(p.lis,
"myTst"))))
is not TRUE
Execution halted
> ls myLib
exNSS4  myTst  pkgA  pkgB

So, it looks like the "installed.packages()" is not correctly
detecting
what is present in "myLib".

Has anyone else seen this? Has anyone got any ideas about what is going
wrong? My environment does not include R_LIBS, LD_LIBRARY, etc. The PATH
does include an older version of R, 3.1.1.

Regards,
Mark Dalphin

Hmm. I don't think this has been seen by other Ubuntu users, or other
Linuxen for that matter.

You have a gratuitous use of "&", but I don't really see how
that would give these symptoms, unless you accidentally run a command twice,
creating a race condition between two background processes.

More likely it is file system related, like NFS directory caching or maybe
permissions, although the latter seems unlikely if one package does install
properly. You might want to recheck whether  installed.packages(lib.loc =
"myLib", priority = "NA") gives different results a bit
later, and if not, check that the package descriptions are sane.

-pd

On 21 Apr 2016, at 00:50 , Mark Dalphin <mark.dalphin at
pacificedge.co.nz> wrote:
> Hi,
> 
> Let me justify building R from source. While I can obtain R built for me
> under Ubuntu, I tend to build it from scratch as we use a NFS-shared
> build as well as shared R libraries for all the bioinfo staff at this
> company. Only one build and one set of packages to ensure uniformity
> across all workstations.
> 
> My problem is that despite using a pretty standard build process, I am
> failing at the "make check" step. I don't see what I might
have done
> that is causing the failure.
> 
> My platform is:
> 
>> uname -a
> Linux littlebourne 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 20:50:15
> UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> 
>> cat /etc/issue
> Ubuntu 14.04.4 LTS \n \l
> 
> I've downloaded the recent R-3.2.5 tar-gz file and gone through the
> standard build, though in a non-standard location. I keep copies of the
> tar-gzipped file in an "Archive" directory.
> 
>> gunzip -c Archive/R-3.2.5.tar.gz | tar xf -
>> cd R-3.2.5/
>> ./configure --prefix=/opt/apps/x86_64/R/R-3.2.5 LIBnn=lib
> [ ... much output ...]
> R is now configured for x86_64-pc-linux-gnu
> 
>  Source directory:          .
>  Installation directory:    /opt/apps/x86_64/R/R-3.2.5
> 
>  C compiler:                gcc -std=gnu99  -g -O2
>  Fortran 77 compiler:       gfortran  -g -O2
> 
>  C++ compiler:              g++  -g -O2
>  C++ 11 compiler:           g++  -std=c++11 -g -O2
>  Fortran 90/95 compiler:    gfortran -g -O2
>  Obj-C compiler:         gcc -g -O2 -fobjc-exceptions
> 
>  Interfaces supported:      X11, tcltk
>  External libraries:        readline, zlib, bzlib, lzma, PCRE
>  Additional capabilities:   PNG, JPEG, TIFF, NLS, cairo
>  Options enabled:           shared BLAS, R profiling
> 
>  Capabilities skipped:      ICU
>  Options not enabled:       memory profiling
> 
>  Recommended packages:      yes
> 
> # Note: I use tcsh, not bash
>> make >& make.out &
> # There are no obvious errors and the 'make' proceeds finally
through
> some Java configuration
> # and then exits with a zero status.
> 
>> make check >& make_check.out &
> [ ... much output captured ...]
> make[3]: Entering directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
> running code in 'reg-tests-1a.R' ... OK
> running code in 'reg-tests-1b.R' ... OK
> running code in 'reg-tests-1c.R' ... OK
> running code in 'reg-tests-2.R' ... OK
>  comparing 'reg-tests-2.Rout' to './reg-tests-2.Rout.save'
... OK
> running code in 'reg-examples1.R' ... OK
> running code in 'reg-examples2.R' ... OK
> running code in 'reg-packages.R' ...make[3]: ***
[reg-packages.Rout] Error 1
> make[3]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
> make[2]: *** [test-Reg] Error 2
> make[2]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
> make[1]: *** [test-all-basics] Error 1
> make[1]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
> make: *** [check] Error 2
> 
>> cd tests
>> tail -30 reg-packages.Rout.fail
> *** installing help indices
> ** building package indices
> ** testing if installed package can be loaded
> * DONE (pkgB)
> Loading required package: pkgB
> building package exNSS4 ...
> 
> installing package exNSS4 using file exNSS4_1.1.tar.gz ...
> * installing *source* package 'exNSS4' ...
> ** R
> ** preparing package for lazy loading
> Creating a generic function for 'plot' from package
'graphics' in
> package 'exNSS4'
> ** help
> No man pages found in package  'exNSS4'
> *** installing help indices
> ** building package indices
> ** testing if installed package can be loaded
> * DONE (exNSS4)
> Loading required package: exNSS4
>> (res <- installed.packages(lib.loc = "myLib", priority =
"NA"))
>      Package LibPath Version Priority Depends   Imports LinkingTo Suggests
> myTst "myTst" "myLib" "1.0"   NA      
"methods" NA      NA        NA
>      Enhances License                     License_is_FOSS
> myTst NA       "What license is it under?" NA            
>      License_restricts_use OS_type MD5sum NeedsCompilation Built 
> myTst NA                    NA      NA     NA              
"3.2.5"
>> stopifnot(identical(res[,"Package"], setNames(,sort(c(p.lis,
"myTst")))),
> +       res[,"LibPath"] == "myLib")
> Error: identical(res[, "Package"], setNames(, sort(c(p.lis,
"myTst"))))
> is not TRUE
> Execution halted
> 
>> ls myLib
> exNSS4  myTst  pkgA  pkgB
> 
> So, it looks like the "installed.packages()" is not correctly
detecting
> what is present in "myLib".
> 
> Has anyone else seen this? Has anyone got any ideas about what is going
> wrong? My environment does not include R_LIBS, LD_LIBRARY, etc. The PATH
> does include an older version of R, 3.1.1.
> 
> Regards,
> Mark Dalphin
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
-- 
Peter Dalgaard, Professor,
Center for Statistics, Copenhagen Business School
Solbjerg Plads 3, 2000 Frederiksberg, Denmark
Phone: (+45)38153501
Office: A 4.23
Email: pd.mes at cbs.dk  Priv: PDalgd at gmail.com

Testing for an NFS effect on the failure of 'make check':

Try first on /usr/local/

    9:42	cd /usr/local/src/
    9:42	sudo mkdir R
    9:43	 sudo chown mdalphin:mdalphin R
    9:43	 cd R
    9:43	 gunzip -c /opt/apps/x86_64/R/Archive/R-3.2.5.tar.gz | tar xf -
    9:43	 cd R-3.2.5/
    9:44	 ./configure --prefix=/opt/apps/x86_64/R/R-3.2.5 LIBnn=lib
    9:44	 make
    9:57	 make check

This PASSES all tests. Exit status of ZERO. 
  
Now try on NFS:

10:04	 cd /opt/apps/x86_64/R/src/
10:05	 gunzip -c /opt/apps/x86_64/R/Archive/R-3.2.5.tar.gz | tar xf -
10:06	cd R-3.2.5/
10:06	./configure --prefix=/opt/apps/x86_64/R/R-3.2.5 LIBnn=lib
10:06	make
10:14	 make check
 This FAILS (see below). Exit status of NON-ZERO:

running code in 'reg-packages.R' ...make[3]: *** [reg-packages.Rout]
Error 1
make[3]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
make[2]: *** [test-Reg] Error 2
make[2]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
make[1]: *** [test-all-basics] Error 1
make[1]: Leaving directory `/opt/apps/x86_64/R/src/R-3.2.5/tests'
make: *** [check] Error 2

What might be different?
> ls -ld /opt/apps/x86_64/R/src/R-3.2.5/ /usr/local/src/R/R-3.2.5/
drwxr-xr-x 15 mdalphin mdalphin 4096 Apr 22 09:57
/opt/apps/x86_64/R/src/R-3.2.5/
drwxr-xr-x 15 mdalphin mdalphin 4096 Apr 22 09:50 /usr/local/src/R/R-3.2.5/
> mount | grep /opt/
myHost.science:/mnt/home/opt/apps on /opt/apps type nfs
(rw,soft,bg,nfsvers=3,addr=XXX.XXX.XXX.XXX)

Just the NFS, I guess. This is not good, but not an R-Devel problem.

Thanks for the help.

Cheers,

Mark Dalphin PhD

Director of Bioinformatics
mark.dalphin at pacificedge.co.nz
Ph: +64 3 479 4690
Cell: +64 21 156 7625
Skype: mark.dalphin.pel

________________________________

87 St David St, PO Box 56, Dunedin, New Zealand 9016    www.pacificedgedx.com


________________________________________
From: Mark Dalphin <mark.dalphin at pacificedge.co.nz>
Sent: Friday, 22 April 2016 9:19
To: peter dalgaard; Prof Brian Ripley
Subject: Re: [Rd] Fresh build from source of R-3.2.5 failing "make
check" under 64-bit Ubuntu

I have run it both ways; for the purposes of this email, it was easier
to capture both STDOUT and STDERR ( '>&') into one file and have
the
process run in the background (second '&'). That way I could more
readily capture the exact commands used and not have them scroll off the
top of the screen (make's output is quite large). The exact same error
occurs when I run the 'make' command as:

    ./configure PARAMETERS
    make
    make check

It is on an NFS file system; I'll look into building on a local file
system (/usr/local/) and see if I get the same errors.

Cheers,
Mark Dalphin


On 22/04/16 00:35, peter dalgaard wrote:
> On 21 Apr 2016, at 14:11 , Prof Brian Ripley <ripley at
stats.ox.ac.uk> wrote:
>
>> On 21/04/2016 12:20, peter dalgaard wrote:
>>> Hmm. I don't think this has been seen by other Ubuntu users, or
other Linuxen for that matter.
>>>
>>> You have a gratuitous use of "&", but I don't
really see how that would give these symptoms, unless you accidentally run a
command twice, creating a race condition between two background processes.
>> He did say he used tcsh, for which >& is legal and far more
memorable than (goes to look it up to get the order right) >foo 2>&1
> I know (old tcsh user, before bash got traction[*], in fact). It was the
"&" at the end that I was talking about:
>
> make check >& make_check.out &
>
> This runs make in the background and redirects stdout/err to a file. As I
said, it is most likely harmless unless you accidentally get two processes
running at the same time, but I think I'd try runnning in the foreground,
just in case.
>
> -pd
>
> [*] That'll be before Linux; the first versions of Slackware used bash
(ash for boot disks). So... like 1991, 25 years back. Yikes!
>
>>> More likely it is file system related, like NFS directory caching
or maybe permissions, although the latter seems unlikely if one package does
install properly. You might want to recheck whether  installed.packages(lib.loc
= "myLib", priority = "NA") gives different results a bit
later, and if not, check that the package descriptions are sane.
>>>
>>> -pd
>>>
>>> On 21 Apr 2016, at 00:50 , Mark Dalphin <mark.dalphin at
pacificedge.co.nz> wrote:
>>>
>>>> Hi,
>>>>
>>>> Let me justify building R from source. While I can obtain R
built for me
>>>> under Ubuntu, I tend to build it from scratch as we use a
NFS-shared
>>>> build as well as shared R libraries for all the bioinfo staff
at this
>>>> company. Only one build and one set of packages to ensure
uniformity
>>>> across all workstations.
>>>>
>>>> My problem is that despite using a pretty standard build
process, I am
>>>> failing at the "make check" step. I don't see
what I might have done
>>>> that is causing the failure.
>>>>
>>>> My platform is:
>>>>
>>>>> uname -a
>>>> Linux littlebourne 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17
20:50:15
>>>> UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>>> cat /etc/issue
>>>> Ubuntu 14.04.4 LTS \n \l
>>>>
>>>> I've downloaded the recent R-3.2.5 tar-gz file and gone
through the
>>>> standard build, though in a non-standard location. I keep
copies of the
>>>> tar-gzipped file in an "Archive" directory.
>>>>
>>>>> gunzip -c Archive/R-3.2.5.tar.gz | tar xf -
>>>>> cd R-3.2.5/
>>>>> ./configure --prefix=/opt/apps/x86_64/R/R-3.2.5 LIBnn=lib
>>>> [ ... much output ...]
>>>> R is now configured for x86_64-pc-linux-gnu
>>>>
>>>>  Source directory:          .
>>>>  Installation directory:    /opt/apps/x86_64/R/R-3.2.5
>>>>
>>>>  C compiler:                gcc -std=gnu99  -g -O2
>>>>  Fortran 77 compiler:       gfortran  -g -O2
>>>>
>>>>  C++ compiler:              g++  -g -O2
>>>>  C++ 11 compiler:           g++  -std=c++11 -g -O2
>>>>  Fortran 90/95 compiler:    gfortran -g -O2
>>>>  Obj-C compiler:         gcc -g -O2 -fobjc-exceptions
>>>>
>>>>  Interfaces supported:      X11, tcltk
>>>>  External libraries:        readline, zlib, bzlib, lzma, PCRE
>>>>  Additional capabilities:   PNG, JPEG, TIFF, NLS, cairo
>>>>  Options enabled:           shared BLAS, R profiling
>>>>
>>>>  Capabilities skipped:      ICU
>>>>  Options not enabled:       memory profiling
>>>>
>>>>  Recommended packages:      yes
>>>>
>>>> # Note: I use tcsh, not bash
>>>>> make >& make.out &
>>>> # There are no obvious errors and the 'make' proceeds
finally through
>>>> some Java configuration
>>>> # and then exits with a zero status.
>>>>
>>>>> make check >& make_check.out &
>>>> [ ... much output captured ...]
>>>> make[3]: Entering directory
`/opt/apps/x86_64/R/src/R-3.2.5/tests'
>>>> running code in 'reg-tests-1a.R' ... OK
>>>> running code in 'reg-tests-1b.R' ... OK
>>>> running code in 'reg-tests-1c.R' ... OK
>>>> running code in 'reg-tests-2.R' ... OK
>>>>  comparing 'reg-tests-2.Rout' to
'./reg-tests-2.Rout.save' ... OK
>>>> running code in 'reg-examples1.R' ... OK
>>>> running code in 'reg-examples2.R' ... OK
>>>> running code in 'reg-packages.R' ...make[3]: ***
[reg-packages.Rout] Error 1
>>>> make[3]: Leaving directory
`/opt/apps/x86_64/R/src/R-3.2.5/tests'
>>>> make[2]: *** [test-Reg] Error 2
>>>> make[2]: Leaving directory
`/opt/apps/x86_64/R/src/R-3.2.5/tests'
>>>> make[1]: *** [test-all-basics] Error 1
>>>> make[1]: Leaving directory
`/opt/apps/x86_64/R/src/R-3.2.5/tests'
>>>> make: *** [check] Error 2
>>>>
>>>>> cd tests
>>>>> tail -30 reg-packages.Rout.fail
>>>> *** installing help indices
>>>> ** building package indices
>>>> ** testing if installed package can be loaded
>>>> * DONE (pkgB)
>>>> Loading required package: pkgB
>>>> building package exNSS4 ...
>>>>
>>>> installing package exNSS4 using file exNSS4_1.1.tar.gz ...
>>>> * installing *source* package 'exNSS4' ...
>>>> ** R
>>>> ** preparing package for lazy loading
>>>> Creating a generic function for 'plot' from package
'graphics' in
>>>> package 'exNSS4'
>>>> ** help
>>>> No man pages found in package  'exNSS4'
>>>> *** installing help indices
>>>> ** building package indices
>>>> ** testing if installed package can be loaded
>>>> * DONE (exNSS4)
>>>> Loading required package: exNSS4
>>>>> (res <- installed.packages(lib.loc = "myLib",
priority = "NA"))
>>>>      Package LibPath Version Priority Depends   Imports
LinkingTo Suggests
>>>> myTst "myTst" "myLib" "1.0"   NA 
"methods" NA      NA        NA
>>>>      Enhances License                     License_is_FOSS
>>>> myTst NA       "What license is it under?" NA
>>>>      License_restricts_use OS_type MD5sum NeedsCompilation
Built
>>>> myTst NA                    NA      NA     NA              
"3.2.5"
>>>>> stopifnot(identical(res[,"Package"],
setNames(,sort(c(p.lis, "myTst")))),
>>>> +       res[,"LibPath"] == "myLib")
>>>> Error: identical(res[, "Package"], setNames(,
sort(c(p.lis, "myTst"))))
>>>> is not TRUE
>>>> Execution halted
>>>>
>>>>> ls myLib
>>>> exNSS4  myTst  pkgA  pkgB
>>>>
>>>> So, it looks like the "installed.packages()" is not
correctly detecting
>>>> what is present in "myLib".
>>>>
>>>> Has anyone else seen this? Has anyone got any ideas about what
is going
>>>> wrong? My environment does not include R_LIBS, LD_LIBRARY, etc.
The PATH
>>>> does include an older version of R, 3.1.1.
>>>>
>>>> Regards,
>>>> Mark Dalphin
>>>>
>>>> ______________________________________________
>>>> R-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>> --
>> Brian D. Ripley,                  ripley at stats.ox.ac.uk
>> Emeritus Professor of Applied Statistics, University of Oxford

> On 22 Apr 2016, at 00:25 , Mark Dalphin <mark.dalphin at
pacificedge.co.nz> wrote:
> 
>> 
>> mount | grep /opt/
> myHost.science:/mnt/home/opt/apps on /opt/apps type nfs
(rw,soft,bg,nfsvers=3,addr=XXX.XXX.XXX.XXX)
> 
> Just the NFS, I guess. This is not good, but not an R-Devel problem.
On a hunch: Check out what the "soft" mount option does. I seem to
recall some complications related to immediacy of operations.

-pd

-- 
Peter Dalgaard, Professor,
Center for Statistics, Copenhagen Business School
Solbjerg Plads 3, 2000 Frederiksberg, Denmark
Phone: (+45)38153501
Office: A 4.23
Email: pd.mes at cbs.dk  Priv: PDalgd at gmail.com