R devel - Nov 2016 - Running package tests and not stop on first fail

>>>>> Oliver Keyes <ironholds at gmail.com>
>>>>>     on Fri, 4 Nov 2016 12:42:54 -0400 writes:
    > On Friday, 4 November 2016, Martin Maechler
    > <maechler at stat.math.ethz.ch> wrote:

    >> >>>>> Dirk Eddelbuettel <edd at debian.org
<javascript:;>>
    >> >>>>> on Fri, 4 Nov 2016 10:36:52 -0500 writes:
    >> 
    >> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
    >> My > proposed name '--no-stop-on-error' was a quick
shot;
    >> if | > somebody has a more concise or better "English
    >> style" > wording | (which is somewhat compatible with all
    >> the other > options you see | from 'R CMD check
--help'),
    >> | please > speak up.
    >> 
    >> > Why not keep it simple?  The similar feature this most
    >> > resembles is 'make -k' and its help page has
    >> 
    >> > -k, --keep-going
    >> 
    >> > Continue as much as possible after an > error.  While
    >> the target that failed, and those that > depend on it,
    >> cannot be remade, the other dependencies of > these
    >> targets can be processed all the same.
    >> 
    >> Yes, that would be quite a bit simpler and nice in my
    >> view.  One may think it to be too vague,

    > Mmn, I would agree on vagueness (and it breaks the pattern
    > set by other flags of human-readability). Deep familiarity
    > with make is probably not something we should ask of
    > everyone who needs to test a package, too.

    > I quite like stop-on-error=true (exactly the same as the
    > previous suggestion but shaves off some characters by
    > inverting the Boolean)

Thank you, Brian, Dirk and Oliver for these (and some offline)
thoughts and suggestions!

My current summary:

1) I really don't want a  --<option-key>=value
   but rather stay with logical/binary variables that "express
   themselves"... in the same way I strongly prefer

       if (A_is_special)       ....
   to
       if (A_special == TRUE)  ....

   for a logical variable A_* .   Yes, this is mostly a matter
   of taste,.. but related to how R style itself "works"

2) Brian mentioned that this is only about ./tests/ tests which
   are continued, not about the Examples which are treated separately.
   That's why we had contemplated additionally using 'tests'
(because that's
   the directory name used for unit/regression/.. tests) in the option
   name.

Even though Brian is correct, ideally we *would* want to also influence the
examples' running to *not* stop on a first error..   However that would
need more work, reorganizing how the examples are run and that may not be
worth the pain.   However it should be considered a goal in the long run.

After all that, I tend to remain with the original proposed name. It is at
least easy to pronounce and spell correctly...

Martin

On 11/05/2016 01:53 PM, Martin Maechler wrote:
>>>>>> Oliver Keyes <ironholds at gmail.com>
>>>>>>     on Fri, 4 Nov 2016 12:42:54 -0400 writes:
>
>     > On Friday, 4 November 2016, Martin Maechler
>     > <maechler at stat.math.ethz.ch> wrote:
>
>     >> >>>>> Dirk Eddelbuettel <edd at debian.org
<javascript:;>>
>     >> >>>>> on Fri, 4 Nov 2016 10:36:52 -0500 writes:
>     >>
>     >> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
>     >> My > proposed name '--no-stop-on-error' was a quick
shot;
>     >> if | > somebody has a more concise or better "English
>     >> style" > wording | (which is somewhat compatible with
all
>     >> the other > options you see | from 'R CMD check
--help'),
>     >> | please > speak up.
>     >>
>     >> > Why not keep it simple?  The similar feature this most
>     >> > resembles is 'make -k' and its help page has
>     >>
>     >> > -k, --keep-going
>     >>
>     >> > Continue as much as possible after an > error.  While
>     >> the target that failed, and those that > depend on it,
>     >> cannot be remade, the other dependencies of > these
>     >> targets can be processed all the same.
>     >>
>     >> Yes, that would be quite a bit simpler and nice in my
>     >> view.  One may think it to be too vague,
>
>     > Mmn, I would agree on vagueness (and it breaks the pattern
>     > set by other flags of human-readability). Deep familiarity
>     > with make is probably not something we should ask of
>     > everyone who needs to test a package, too.
>
>     > I quite like stop-on-error=true (exactly the same as the
>     > previous suggestion but shaves off some characters by
>     > inverting the Boolean)
>
> Thank you, Brian, Dirk and Oliver for these (and some offline)
> thoughts and suggestions!
>
> My current summary:
>
> 1) I really don't want a  --<option-key>=value
>    but rather stay with logical/binary variables that "express
>    themselves"... in the same way I strongly prefer
>
>        if (A_is_special)       ....
>    to
>        if (A_special == TRUE)  ....
>
>    for a logical variable A_* .   Yes, this is mostly a matter
>    of taste,.. but related to how R style itself "works"
>
> 2) Brian mentioned that this is only about ./tests/ tests which
>    are continued, not about the Examples which are treated separately.
>    That's why we had contemplated additionally using 'tests'
(because that's
>    the directory name used for unit/regression/.. tests) in the option
>    name.
>
> Even though Brian is correct, ideally we *would* want to also influence the
> examples' running to *not* stop on a first error..   However that would
> need more work, reorganizing how the examples are run and that may not be
> worth the pain.   However it should be considered a goal in the long run.
My name is Herv?, and I was not suggesting that what happens with the
examples should be changed. I was just preaching consistency (again
sorry) between what happens with the examples and what happens with
the tests. Why not simply change the latter? Do we really need an option
to control this? The behavior was changed for the examples a couple of
years ago and nobody felt the need to introduce an option to control
this at the time.
>
> After all that, I tend to remain with the original proposed name. It is at
> least easy to pronounce and spell correctly...
Unless --no-stop-on-error controls both (i.e. examples and tests), and 
both behave the same way by default, this is a misnomer.

If this option controls the tests only, it should be more something like
--no-stop-on-test-error. If in the long run, another option is added to
control the examples, then could be --no-stop-on-example-error.

But again, maybe all this is not needed at all. Maybe changing what
happens with the tests would be enough.

Cheers,
H.
>
> Martin
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319

>>>>> Herv? Pag?s <hpages at fredhutch.org>
>>>>>     on Mon, 7 Nov 2016 14:37:15 -0800 writes:
    > On 11/05/2016 01:53 PM, Martin Maechler wrote:
    >>>>>>> Oliver Keyes <ironholds at gmail.com>
    >>>>>>> on Fri, 4 Nov 2016 12:42:54 -0400 writes:
    >> 
    >> > On Friday, 4 November 2016, Martin Maechler
    >> > <maechler at stat.math.ethz.ch> wrote:
    >> 
    >> >> >>>>> Dirk Eddelbuettel <edd at
debian.org <javascript:;>>
    >> >> >>>>> on Fri, 4 Nov 2016 10:36:52 -0500
writes:
    >> >>
    >> >> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
    >> >> My > proposed name '--no-stop-on-error' was a
quick shot;
    >> >> if | > somebody has a more concise or better
"English
    >> >> style" > wording | (which is somewhat compatible
with all
    >> >> the other > options you see | from 'R CMD check
--help'),
    >> >> | please > speak up.
    >> >>
    >> >> > Why not keep it simple?  The similar feature this
most
    >> >> > resembles is 'make -k' and its help page has
    >> >>
    >> >> > -k, --keep-going
    >> >>
    >> >> > Continue as much as possible after an > error. 
While
    >> >> the target that failed, and those that > depend on it,
    >> >> cannot be remade, the other dependencies of > these
    >> >> targets can be processed all the same.
    >> >>
    >> >> Yes, that would be quite a bit simpler and nice in my
    >> >> view.  One may think it to be too vague,
    >> 
    >> > Mmn, I would agree on vagueness (and it breaks the pattern
    >> > set by other flags of human-readability). Deep familiarity
    >> > with make is probably not something we should ask of
    >> > everyone who needs to test a package, too.
    >> 
    >> > I quite like stop-on-error=true (exactly the same as the
    >> > previous suggestion but shaves off some characters by
    >> > inverting the Boolean)
    >> 
    >> Thank you, Brian, Dirk and Oliver for these (and some offline)
    >> thoughts and suggestions!
    >> 
    >> My current summary:
    >> 
    >> 1) I really don't want a  --<option-key>=value
    >> but rather stay with logical/binary variables that "express
    >> themselves"... in the same way I strongly prefer
    >> 
    >> if (A_is_special)       ....
    >> to
    >> if (A_special == TRUE)  ....
    >> 
    >> for a logical variable A_* .   Yes, this is mostly a matter
    >> of taste,.. but related to how R style itself "works"
    >> 
    >> 2) Brian mentioned that this is only about ./tests/ tests which
    >> are continued, not about the Examples which are treated separately.
    >> That's why we had contemplated additionally using
'tests' (because that's
    >> the directory name used for unit/regression/.. tests) in the option
    >> name.
    >> 
    >> Even though Brian is correct, ideally we *would* want to also
influence the
    >> examples' running to *not* stop on a first error..   However
that would
    >> need more work, reorganizing how the examples are run and that may
not be
    >> worth the pain.   However it should be considered a goal in the
long run.

    > My name is Herv?, and I was not suggesting that what happens with the
    > examples should be changed. I was just preaching consistency (again
    > sorry) between what happens with the examples and what happens with
    > the tests. 

Thank you, Herv? and excuse me for not answering more focused on
what you said.
I think I do understand what you say (at least by now :-)) and
agree that consistency is something important and to be strived for,
also with these options.

    > Why not simply change the latter?
    > Do we really need an option to control this? 

Very good questions.  If the change could be made much better,
I'd agree we'd not need a new option because the change could be
considerided uniformly better than the current (R 3.3.2, say) behavior.
However the change as it is currently, is not good enough to be
the only option (see below). 

    > The behavior was changed for the examples a couple of
    > years ago and nobody felt the need to introduce an option
    > to control this at the time.

Yes, that change was made very nicely (not by me) and I'd say
the result *was* uniformly better than the previous behavior, so
there did not seem much of a reason to still provide the old behavior.

      >> After all that, I tend to remain with the original proposed name.
It is at
      >> least easy to pronounce and spell correctly...

    > Unless --no-stop-on-error controls both (i.e. examples and tests), and 
    > both behave the same way by default, this is a misnomer.

Well, if you choose the option, there *is* no stop on errors anymore
... because the examples nowadays never stop the (R CMD check Pkg) 
from running on an error as we've mentioned above.

[[--- Digression on  Examples' checking 

  However / on the other hand: Because of the way the examples
  are run --- efficiently from one single R source file --- it
  is not so easy there, to let them run further: The first error
  from all the examples stops running the example-checks, i.e.,
  the <pkg>-Ex.R script, but at least *does* continue to run the
  next 'R CMD check ...' steps.
 
  To change this without incurring considerable drawbacks, or
  involve a lot of changes does not seem easy:

  - If each example would be run in a separate R process, R has to
    be restarted once for every man/*.Rd file.. which easily adds
    one minute or rather several minutes to the total testing time
    (and a I think quite a bit of the checking code had to be re-written).

  - Alternatively, all examples are parsed (fail-safe!) first and put together
    into an R list (or environment), and then each run via tryCatch(.).
    This is nicer and more efficient conceptually, but needs even
    more code changes and tends to lose the transparent *-Ex.R -> *-Ex.Rout
    interface we have now.

 --- end of digression ---]]
 
    > If this option controls the tests only, it should be more something
like
    > --no-stop-on-test-error.  If in the long run, another option is added
to
    > control the examples, then could be --no-stop-on-example-error.

    > But again, maybe all this is not needed at all. Maybe changing what
    > happens with the tests would be enough.

I agree: it would be enough, if the change was better than what we currently
have
(in R-devel):  On
    https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176,
Jan mentions that the patch is simple but suffers a deficiency
(my wording) in that the error *reporting* is not correct:

If you have three  tests/*.R  test scripts producing an error,
still only the first of these is reported (and counted in the
final 'Status: ..' line), and the package checker has to
manually look at the   tests/*.Rout.fail files to see which of
the checks failed exactly.... and the reporting of the first
error is *after* running all the tests/*.R scripts which is also
far from ideal.


Consequently, for now, it does need an option there, and it
should not be default because of the "inconsistent" report
output.

I don't mind to  add  the extra  "-test" to the option string,
i.e., change
from
      --no-stop-on-error
to    --no-stop-on-test-error  

Others?


    > Cheers,
    > H.

    > -- 
    > Herv? Pag?s

    > Program in Computational Biology
    > Division of Public Health Sciences
    > Fred Hutchinson Cancer Research Center
    > 1100 Fairview Ave. N, M1-B514
    > P.O. Box 19024
    > Seattle, WA 98109-1024

    > E-mail: hpages at fredhutch.org
    > Phone:  (206) 667-5791
    > Fax:    (206) 667-1319