R devel - Mar 2019 - Development version of R fails tests and is not installed

G'day all,

I have daily scripts running to install the patched version of the
current R version and the development version of R on my linux box
(Ubuntu 18.04.2 LTS).

The last development version that was successfully compiled and
installed was "R Under development (unstable) (2019-02-25 r76159)" on
26 February.  Since then the script always fails as a regression test
seems to fail.  Specifically, in the tests/ subdirectory of my build
directory I have a file reg-tests-1d.Rout.fail which ends with:
> ## checking ar.yw.default() multivariate case
> estd <- ar(unclass(y) , aic = FALSE, order.max = 2) ## Estimate VAR(2)
> es.d <- ar(unclass(y.), aic = FALSE, order.max = 2, na.action=na.pass)
> stopifnot(exprs = {
+     all.equal(est$ar[1,,], diag(0.8, 2), tol = 0.08)# seen 0.0038
+     all.equal(est[1:6], es.[1:6], tol = 5e-3)
+     all.equal(estd$x.mean, es.d$x.mean, tol = 0.01) # seen 0.0023
+     all.equal(estd[c(1:3,5:6)],
+               es.d[c(1:3,5:6)], tol = 1e-3)## seen {1,3,8}e-4
+     all.equal(lapply(estd[1:6],unname),
+               lapply(est [1:6],unname), tol = 2e-12)# almost identical
+     all.equal(lapply(es.d[1:6],unname),
+               lapply(es. [1:6],unname), tol = 2e-12)
+ })
Error: lapply(es.d[1:6], unname) and lapply(es.[1:6], unname) are not equal:
  Component "aic": Mean relative difference: 3.297178e-12
Execution halted

Would it be possible to make this tolerance more lenient?  In case it
matters, I am configuring R to be compiled using Openblas and this test
fails for the 64 bit installation, the 32 bit installation seems to
pass all tests.

Happy to provide any more information/context that might be needed.

Cheers,

	Berwin

>>>>> Berwin A Turlach 
>>>>>     on Tue, 5 Mar 2019 16:54:09 +0800 writes:
    > G'day all,
    > I have daily scripts running to install the patched version of the
    > current R version and the development version of R on my linux box
    > (Ubuntu 18.04.2 LTS).

    > The last development version that was successfully compiled and
    > installed was "R Under development (unstable) (2019-02-25
r76159)" on
    > 26 February.  Since then the script always fails as a regression test
    > seems to fail.  Specifically, in the tests/ subdirectory of my build
    > directory I have a file reg-tests-1d.Rout.fail which ends with:

    >> ## checking ar.yw.default() multivariate case
    >> estd <- ar(unclass(y) , aic = FALSE, order.max = 2) ## Estimate
VAR(2)
    >> es.d <- ar(unclass(y.), aic = FALSE, order.max = 2,
na.action=na.pass)
    >> stopifnot(exprs = {
    > +     all.equal(est$ar[1,,], diag(0.8, 2), tol = 0.08)# seen 0.0038
    > +     all.equal(est[1:6], es.[1:6], tol = 5e-3)
    > +     all.equal(estd$x.mean, es.d$x.mean, tol = 0.01) # seen 0.0023
    > +     all.equal(estd[c(1:3,5:6)],
    > +               es.d[c(1:3,5:6)], tol = 1e-3)## seen {1,3,8}e-4
    > +     all.equal(lapply(estd[1:6],unname),
    > +               lapply(est [1:6],unname), tol = 2e-12)# almost
identical
    > +     all.equal(lapply(es.d[1:6],unname),
    > +               lapply(es. [1:6],unname), tol = 2e-12)
    > + })
    > Error: lapply(es.d[1:6], unname) and lapply(es.[1:6], unname) are not
equal:
    > Component "aic": Mean relative difference: 3.297178e-12
    > Execution halted

    > Would it be possible to make this tolerance more lenient?  In case it
    > matters, I am configuring R to be compiled using Openblas and this test
    > fails for the 64 bit installation, the 32 bit installation seems to
    > pass all tests.

Interesting. But we had another report (on this mailing list!) by
Er. Ho., also using OpenBLAS, but then in combination with Cygwin.
After Cygwin related problems were solved, on March 1 he said
> I have an error at reg-1d.  It stops the process.  However, the mean
difference
> (as per the file) is 2.0e-12. 
and when I asked him to be more precise.. he wasn't but did
"solve" the problem by changing the test file.
Now we know from you Berwin (thank you for providing the details!)
what it has been.

==> I could investigate and -- lo and behold! --  the solution
is probably the fact that the RNGkind(sample.kind = *) was
introduced *and* the default was to differ from previous
versions of R.

    > Happy to provide any more information/context that might be needed.

Can you please try adding

    suppressWarnings(RNGversion("3.5.0"))

e.g. at the very beginning of the   tests/reg-tests-1d.R
file or just a few lines above the code you show above,
replace in line 1470

	set.seed(42)
by
	set.seed(42); suppressWarnings(RNGversion("3.5.0"))

It will use a different missingness pattern for that
multivariate AR example, i.e., different data.

Best, Martin

On 05/03/2019 08:54, Berwin A Turlach wrote:
> G'day all,
> 
> I have daily scripts running to install the patched version of the
> current R version and the development version of R on my linux box
> (Ubuntu 18.04.2 LTS).
> 
> The last development version that was successfully compiled and
> installed was "R Under development (unstable) (2019-02-25
r76159)" on
> 26 February.  Since then the script always fails as a regression test
> seems to fail.  Specifically, in the tests/ subdirectory of my build
> directory I have a file reg-tests-1d.Rout.fail which ends with:
> 
>> ## checking ar.yw.default() multivariate case
>> estd <- ar(unclass(y) , aic = FALSE, order.max = 2) ## Estimate
VAR(2)
>> es.d <- ar(unclass(y.), aic = FALSE, order.max = 2,
na.action=na.pass)
>> stopifnot(exprs = {
> +     all.equal(est$ar[1,,], diag(0.8, 2), tol = 0.08)# seen 0.0038
> +     all.equal(est[1:6], es.[1:6], tol = 5e-3)
> +     all.equal(estd$x.mean, es.d$x.mean, tol = 0.01) # seen 0.0023
> +     all.equal(estd[c(1:3,5:6)],
> +               es.d[c(1:3,5:6)], tol = 1e-3)## seen {1,3,8}e-4
> +     all.equal(lapply(estd[1:6],unname),
> +               lapply(est [1:6],unname), tol = 2e-12)# almost identical
> +     all.equal(lapply(es.d[1:6],unname),
> +               lapply(es. [1:6],unname), tol = 2e-12)
> + })
> Error: lapply(es.d[1:6], unname) and lapply(es.[1:6], unname) are not
equal:
>    Component "aic": Mean relative difference: 3.297178e-12
> Execution halted
> 
> Would it be possible to make this tolerance more lenient?  In case it
> matters, I am configuring R to be compiled using Openblas and this test
> fails for the 64 bit installation, the 32 bit installation seems to
> pass all tests.
> 
> Happy to provide any more information/context that might be needed.
The version of OpenBLAS is always helpful (a couple of bugs were fixed 
recently that impacted the checks in R packages).

I have a x86_64 Linux build with OpenBLAS 0.3.5 that I use for the 
'Additional issues' in package checking, and that passes its checks (and
has ca weekly for years, even with the buggy versions of OpenBLAS).

Some aspects of the RNG were changed in r76160: to isolate change that 
you can (with current R-devel) use

setenv _R_RNG_VERSION_ 3.5.0

and re-check.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford

>>>>> Martin Maechler 
>>>>>     on Tue, 5 Mar 2019 12:45:36 +0100 writes:
>>>>> Berwin A Turlach 
>>>>>     on Tue, 5 Mar 2019 16:54:09 +0800 writes:
    >> G'day all,
    >> I have daily scripts running to install the patched version of the
    >> current R version and the development version of R on my linux box
    >> (Ubuntu 18.04.2 LTS).

    >> The last development version that was successfully compiled and
    >> installed was "R Under development (unstable) (2019-02-25
r76159)" on
    >> 26 February.  Since then the script always fails as a regression
test
    >> seems to fail.  Specifically, in the tests/ subdirectory of my
build
    >> directory I have a file reg-tests-1d.Rout.fail which ends with:

    >>> ## checking ar.yw.default() multivariate case
    >>> estd <- ar(unclass(y) , aic = FALSE, order.max = 2) ##
Estimate VAR(2)
    >>> es.d <- ar(unclass(y.), aic = FALSE, order.max = 2,
na.action=na.pass)
    >>> stopifnot(exprs = {
    >> +     all.equal(est$ar[1,,], diag(0.8, 2), tol = 0.08)# seen 0.0038
    >> +     all.equal(est[1:6], es.[1:6], tol = 5e-3)
    >> +     all.equal(estd$x.mean, es.d$x.mean, tol = 0.01) # seen 0.0023
    >> +     all.equal(estd[c(1:3,5:6)],
    >> +               es.d[c(1:3,5:6)], tol = 1e-3)## seen {1,3,8}e-4
    >> +     all.equal(lapply(estd[1:6],unname),
    >> +               lapply(est [1:6],unname), tol = 2e-12)# almost
identical
    >> +     all.equal(lapply(es.d[1:6],unname),
    >> +               lapply(es. [1:6],unname), tol = 2e-12)
    >> + })
    >> Error: lapply(es.d[1:6], unname) and lapply(es.[1:6], unname) are
not equal:
    >> Component "aic": Mean relative difference: 3.297178e-12
    >> Execution halted

    >> Would it be possible to make this tolerance more lenient?  In case
it
    >> matters, I am configuring R to be compiled using Openblas and this
test
    >> fails for the 64 bit installation, the 32 bit installation seems to
    >> pass all tests.

    > Interesting. But we had another report (on this mailing list!) by
    > Er. Ho., also using OpenBLAS, but then in combination with Cygwin.
    > After Cygwin related problems were solved, on March 1 he said

    >> I have an error at reg-1d.  It stops the process.  However, the
mean difference
    >> (as per the file) is 2.0e-12. 

    > and when I asked him to be more precise.. he wasn't but did
    > "solve" the problem by changing the test file.
    > Now we know from you Berwin (thank you for providing the details!)
    > what it has been.

    > ==> I could investigate and -- lo and behold! --  the solution
    > is probably the fact that the RNGkind(sample.kind = *) was
    > introduced *and* the default was to differ from previous
    > versions of R.

    >> Happy to provide any more information/context that might be needed.

    > Can you please try adding

    > suppressWarnings(RNGversion("3.5.0"))

    > e.g. at the very beginning of the   tests/reg-tests-1d.R
    > file or just a few lines above the code you show above,
    > replace in line 1470

    > set.seed(42)
    > by
    > set.seed(42); suppressWarnings(RNGversion("3.5.0"))

oops!  That was a thinko (enticed by my "esthetics"):   Of
course you must *first* set the RNG version and *then* the seed,
so *only* this is correct

   suppressWarnings(RNGversion("3.5.0")); set.seed(42)

where I really would go for

   suppressWarnings(RNGversion("3.5.0")) # back compatibility for now
   set.seed(42)

Note that Brian's advice with an environment variable currently
works "more thoroughly", but it has been labeled "temporary"
as it
is not functional / language-explicit [i.e. not visible from
your own R code] ...
and you really should only use it when checking R, *not* otherwise.

Martin