R devel - Feb 2017 - Test suite failures in R-devel_2017-02-25_r72256

Hi,

I tried compiling the latest pre-release for R 3.3.3 for the NixOS Linux
distribution [1], but the build fails during the "make check" phase
because of the following 2 issues:

1) The "tools" test in "tests/Examples" requires network
access, which
   it doesn't have in our build environment. Therefore, it fails as
   follows according to "tools-Ex.Rout.fail":

   | [...]
   | > set.seed(11)
   | > ## End(Don't show)
   | > pdb <- CRAN_package_db()
   | Warning in url(sprintf("%s/%s", cran, path), open =
"rb") :
   |   URL 'http://CRAN.R-project.org/web/packages/packages.rds': status
was 'Couldn't resolve host name'
   | Error in url(sprintf("%s/%s", cran, path), open = "rb")
:
   |   cannot open the connection to
'http://CRAN.R-project.org/web/packages/packages.rds'
   | Calls: CRAN_package_db -> as.data.frame -> read_CRAN_object ->
gzcon -> url
   | Execution halted

   I'm wondering whether it would be possible to extend the test suite
   with a configure-time flag that disable tests which depend on network
   access? My experience is that most modern Linux distributions run
   their builds in a restricted environment and therefore will run into
   trouble if the suite assumes that it can access the Internet.

2) When R is compiled with the --without-recommended-packages flag
   (which is our preferred configuration), the "base" test in
   "tests/Examples" fails, apparently because it depends on MASS.
I'm
   citing from "base-Ex.Rout.fail":

   | >  ## The string "foo" and the symbol 'foo' can be
used interchangably here:
   | >  stopifnot( identical(isNamespaceLoaded(  "foo"   ), FALSE),
   | +             identical(isNamespaceLoaded(quote(foo)), FALSE),
   | +             identical(isNamespaceLoaded(quote(stats)), statL))
   | >
   | > hasM <- isNamespaceLoaded("MASS") # (to restore if
needed)
   | > Mns <- asNamespace("MASS") # loads it if not already
   | Error in loadNamespace(name) : there is no package called 'MASS'
   | Calls: asNamespace ... tryCatch -> tryCatchList -> tryCatchOne ->
<Anonymous>
   | Execution halted

I hope this helps!

Best regards,
Peter



[1] http://nixos.org/

>>>>> Peter Simons <simons at nospf.cryp.to>
>>>>>     on Mon, 27 Feb 2017 10:30:39 +0100 writes:
    > Hi, I tried compiling the latest pre-release for R 3.3.3
    > for the NixOS Linux distribution [1], but the build fails
    > during the "make check" phase because of the following 2
    > issues:

    > 1) The "tools" test in "tests/Examples" requires
network
    > access, which it doesn't have in our build
    > environment. 

One may argue that the 'make check' (or even 'make check-all')
tests could / should be allowed more resources than the pure
build environment.

    > Therefore, it fails as follows according to
    > "tools-Ex.Rout.fail":

    > | [...]
    > | > set.seed(11)
    > | > ## End(Don't show)
    > | > pdb <- CRAN_package_db()
    > | Warning in url(sprintf("%s/%s", cran, path), open =
"rb") :
    > |   URL 'http://CRAN.R-project.org/web/packages/packages.rds':
status was 'Couldn't resolve host name'
    > | Error in url(sprintf("%s/%s", cran, path), open =
"rb") :
    > |   cannot open the connection to
'http://CRAN.R-project.org/web/packages/packages.rds'
    > | Calls: CRAN_package_db -> as.data.frame -> read_CRAN_object
-> gzcon -> url
    > | Execution halted

    > I'm wondering whether it would be possible to extend the test suite
    > with a configure-time flag that disable tests which depend on network
    > access? My experience is that most modern Linux distributions run
    > their builds in a restricted environment and therefore will run into
    > trouble if the suite assumes that it can access the Internet.

[see above] Is it necessary to also run the 'make check' part in
that restricted environment?  Or could that ('checking") not get
more priviledges?

Note that you can only run  "make check" if you don't install
recommended packages, whereas more thorough testing would
include
	make check-devel
or even
	make check-all

but these do have quite a bit more requirements including
recommended packages being present.


    > 2) When R is compiled with the --without-recommended-packages flag
    > (which is our preferred configuration), the "base" test in
    > "tests/Examples" fails, apparently because it depends on
MASS. I'm
    > citing from "base-Ex.Rout.fail":

    > | >  ## The string "foo" and the symbol 'foo' can
be used interchangably here:
    > | >  stopifnot( identical(isNamespaceLoaded(  "foo"   ),
FALSE),
    > | +             identical(isNamespaceLoaded(quote(foo)), FALSE),
    > | +             identical(isNamespaceLoaded(quote(stats)), statL))
    > | >
    > | > hasM <- isNamespaceLoaded("MASS") # (to restore if
needed)
    > | > Mns <- asNamespace("MASS") # loads it if not
already
    > | Error in loadNamespace(name) : there is no package called
'MASS'
    > | Calls: asNamespace ... tryCatch -> tryCatchList -> tryCatchOne
-> <Anonymous>
    > | Execution halted

Yes, that example should not have assumed a recommended package
to be available unconditionally.

I've changed it, thank you!


    > I hope this helps!

    > Best regards,
    > Peter



    > [1] http://nixos.org/

    > ______________________________________________
    > R-devel at r-project.org mailing list
    > https://stat.ethz.ch/mailman/listinfo/r-devel

>>>>> Martin Maechler <maechler at stat.math.ethz.ch>
>>>>>     on Mon, 27 Feb 2017 16:08:40 +0100 writes:
>>>>> Peter Simons <simons at nospf.cryp.to>
>>>>>     on Mon, 27 Feb 2017 10:30:39 +0100 writes:
    >> Hi, I tried compiling the latest pre-release for R 3.3.3
    >> for the NixOS Linux distribution [1], but the build fails
    >> during the "make check" phase because of the following 2
    >> issues:

    >> 1) The "tools" test in "tests/Examples"
requires network
    >> access, which it doesn't have in our build
    >> environment. 

    > One may argue that the 'make check' (or even 'make
check-all')
    > tests could / should be allowed more resources than the pure
    > build environment.

    >> Therefore, it fails as follows according to
    >> "tools-Ex.Rout.fail":

    >> | [...]
    >> | > set.seed(11)
    >> | > ## End(Don't show)
    >> | > pdb <- CRAN_package_db()
    >> | Warning in url(sprintf("%s/%s", cran, path), open =
"rb") :
    >> |   URL
'http://CRAN.R-project.org/web/packages/packages.rds': status was
'Couldn't resolve host name'
    >> | Error in url(sprintf("%s/%s", cran, path), open =
"rb") :
    >> |   cannot open the connection to
'http://CRAN.R-project.org/web/packages/packages.rds'
    >> | Calls: CRAN_package_db -> as.data.frame -> read_CRAN_object
-> gzcon -> url
    >> | Execution halted

    >> I'm wondering whether it would be possible to extend the test
suite
    >> with a configure-time flag that disable tests which depend on
network
    >> access? My experience is that most modern Linux distributions run
    >> their builds in a restricted environment and therefore will run
into
    >> trouble if the suite assumes that it can access the Internet.

    > [see above] Is it necessary to also run the 'make check' part
in
    > that restricted environment?  Or could that ('checking") not
get
    > more priviledges?

    > Note that you can only run  "make check" if you don't
install
    > recommended packages, whereas more thorough testing would
    > include
    >      make check-devel
    > or even
    >      make check-all
    >
    > but these do have quite a bit more requirements including
    > recommended packages being present.

I have to correct myself:  The above paragraph may be misleading:

Much, if not all of
      make check-devel
and   make check-all
have worked well since R version 3.1.0  which had in its NEWS an entry

     ? More of 'make check' will work if recommended packages are not
       installed: but recommended packages remain needed for thorough
       checking of an R build.

Further, the 'R-admin' manual (on 'make check-all' etc) contains

 |  Note that these checks are only run completely 
 |  if the recommended packages are installed.

so their presence is not required  but much recommended for
thorough testing.

Martin

On 27/02/2017 09:30, Peter Simons wrote:
> Hi,
>
> I tried compiling the latest pre-release for R 3.3.3 for the NixOS Linux
> distribution [1], but the build fails during the "make check"
phase
> because of the following 2 issues:
Hmm, R-devel (your subject line) is not a pre-release of R 3.3.3: it is 
'R Under development' for what is planned as R 3.4.0.  Pre-release 
tarballs for 3.3.3 are things like R-rc_2017-02-26_r72260.tar.gz (and 
were previously labelled R-beta).

Your first point only occurs in R-devel, and was something already 
reported and under investigation.  Note that test does not actually 
depend on network access: it depends on having an accessible CRAN 
mirror.  The latter could be local (and is on the CRAN check farm, for 
example).  So we have been working on a more sophisticated condition to 
run that test.

There are lots of possible check environments: it is very time-consuming 
to check them all and reasonable coverage is only attempted once a 
pre-release reaches 'alpha' status -- 3.4.0 has not done so but is 
planned to be in late March.

I have just re-checked on Linux, and R 3.3.3 RC passed its checks 
without Internet access (which is checked for where needed).

I also re-checked 3.3.3 RC without recommended packages and got the 
expected messages about incomplete testing but no failures except in 
'make check-recommended' (expected!).

Our posting guide does ask you to include the output of sessionInfo(): 
that would have avoided the version confusion (and might even have 
alerted you to it).

> 1) The "tools" test in "tests/Examples" requires
network access, which
>    it doesn't have in our build environment. Therefore, it fails as
>    follows according to "tools-Ex.Rout.fail":
>
>    | [...]
>    | > set.seed(11)
>    | > ## End(Don't show)
>    | > pdb <- CRAN_package_db()
>    | Warning in url(sprintf("%s/%s", cran, path), open =
"rb") :
>    |   URL 'http://CRAN.R-project.org/web/packages/packages.rds':
status was 'Couldn't resolve host name'
>    | Error in url(sprintf("%s/%s", cran, path), open =
"rb") :
>    |   cannot open the connection to
'http://CRAN.R-project.org/web/packages/packages.rds'
>    | Calls: CRAN_package_db -> as.data.frame -> read_CRAN_object
-> gzcon -> url
>    | Execution halted
>
>    I'm wondering whether it would be possible to extend the test suite
>    with a configure-time flag that disable tests which depend on network
>    access? My experience is that most modern Linux distributions run
>    their builds in a restricted environment and therefore will run into
>    trouble if the suite assumes that it can access the Internet.
>
> 2) When R is compiled with the --without-recommended-packages flag
>    (which is our preferred configuration), the "base" test in
>    "tests/Examples" fails, apparently because it depends on MASS.
I'm
>    citing from "base-Ex.Rout.fail":
>
>    | >  ## The string "foo" and the symbol 'foo' can
be used interchangably here:
>    | >  stopifnot( identical(isNamespaceLoaded(  "foo"   ),
FALSE),
>    | +             identical(isNamespaceLoaded(quote(foo)), FALSE),
>    | +             identical(isNamespaceLoaded(quote(stats)), statL))
>    | >
>    | > hasM <- isNamespaceLoaded("MASS") # (to restore if
needed)
>    | > Mns <- asNamespace("MASS") # loads it if not already
>    | Error in loadNamespace(name) : there is no package called
'MASS'
>    | Calls: asNamespace ... tryCatch -> tryCatchList -> tryCatchOne
-> <Anonymous>
>    | Execution halted
>
> I hope this helps!
>
> Best regards,
> Peter
>
>
>
> [1] http://nixos.org/
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford

Hi Brian,

 > R-devel (your subject line) is not a pre-release of R 3.3.3: it is 'R
 > Under development' for what is planned as R 3.4.0. Pre-release
 > tarballs for 3.3.3 are things like R-rc_2017-02-26_r72260.tar.gz (and
 > were previously labelled R-beta).

I did not realize this distinction exists. I am sorry about the
misleading report. I followed the link for "3.3.3 release candidates"
and didn't expect that directory to contain tarballs for 3.4.0, too. In
hindsight, that should have been obvious from the different file names,
I suppose. Thank you for the clarification.

I re-ran the build and test suite with the correct tarball,
R-rc_2017-02-28_r72286.tar.gz, and the result looks different indeed.
With that version, I get a test suite error in reg-tests-1c from
"tests". I'm citing the relevant bit from
"reg-tests-1c.Rout.fail":

 | > [...]
 | > ## format.POSIXlt() of Jan.1 if  1941 or '42 is involved:
 | > tJan1 <- function(n1, n2)
 | +     strptime(paste0(n1:n2,"/01/01"), "%Y/%m/%d",
tz="CET")
 | > wDSTJan1 <- function(n1, n2)
 | +     which("CEST" == sub(".* ", '',
format(tJan1(n1,n2), usetz=TRUE)))
 | > (w8 <- wDSTJan1(1801, 2300))
 | integer(0)
 | > (w9 <- wDSTJan1(1901, 2300))
 | integer(0)
 | > stopifnot(identical(w8, 141:142),# exactly 1941:1942 had CEST on Jan.1
 | +           identical(w9,  41: 42))
 | Error: identical(w8, 141:142) is not TRUE
 | Execution halted

This happens in the --without-recommended-packages style build. Not sure
whether this is relevant.

The sessionInfo() output looks as follows:

 | R version 3.3.3 RC (2017-02-28 r72286)
 | Platform: x86_64-pc-linux-gnu (64-bit)
 | 
 | locale:
 | [1] C
 | 
 | attached base packages:
 | [1] stats     graphics  grDevices utils     datasets  base     

Best regards,
Peter

Hi Martin,

 > Is it necessary to also run the 'make check' part in that
restricted
 > environment? Or could that ('checking") not get more priviledges?

in NixOS, there is no way to run any kind of build process in an
environment that has network access. On one hand, that is a security
consideration, but the even more compelling reason for that choice is
that NixOS goes to great lengths to guarantee deterministic builds, and
a build process that connects to servers on the Internet and uses data
they provides is as non-deterministic as it can be, unfortunately.

What we can do for the 3.4.0 release is to extend our build environment
for R to ensure that all required resources are downloaded and made
available locally before the build starts.

 > Note that you can only run  "make check" if you don't
install
 > recommended packages, whereas more thorough testing would
 > include
 >      make check-devel
 > or even
 >      make check-all
 >
 > but these do have quite a bit more requirements including recommended
 > packages being present.

Thank you for pointing that out. We do offer our users the choice of
whether they want the recommended packages included or not (the default
choice being to not include them), so we should in fact parameterize our
build to run the extended test suite if the recommended packages are
built.

Another option would be to always build the recommended packages, but to
not install them if the user doesn't want them included. I'm not sure,
though, whether the R build system supports such a type of build easily?

Best regards,
Peter