Hi Martin, Jan, On 11/03/2016 03:45 AM, Martin Maechler wrote:>>>>>> Jan Gorecki <J.Gorecki at wit.edu.pl> >>>>>> on Tue, 1 Nov 2016 22:51:28 +0000 writes: > > > Hello community/devs, Is there an option to run package > > tests during R CMD check and not stop on first error? I > > know that testing frameworks (testhat and others) can do > > that but asking about just R and base packages. Currently > > when package check runs test scripts in ./tests directory > > it will stop after first fail. Do you think it could be > > optionally available to continue to run tests after > > failures? Regards, Jan Gorecki > > I agree that this would be a useful option sometimes. > > So I would be supportive to get such an option, say, > > R CMD check --no-stop-on-error <pkg>A couple of years ago the behavior of 'R CMD check' was changed to continue checking (e.g. the examples) after many types of errors, and to output a summary count of errors at the end if any have occurred. So --no-stop-on-error could easily be interpreted as an option that controls this behavior (and would also suggest that the default has been reverted back to what it was prior to R 3.2.0), rather than an option that specifically controls what should happen while running the tests. Cheers, H.> > into R if someone provided (relatively small) patches to the R > sources (i.e. subversion repos at https://svn.r-project.org/R/trunk/ ). > The relevant source code should basically all be in > src/library/tools/R/testing.R > > Note that this may be complicated, also because "parallel" > checking is available in parts, via the TEST_MC_CORES > environment variable ((which is currently only quickly > documented in the 'R Administration ..' manual)) > > > Martin Maechler > ETH Zurich > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >-- Herv? Pag?s Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpages at fredhutch.org Phone: (206) 667-5791 Fax: (206) 667-1319
Martin, I submitted very simple patch on https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176 Herve, While I like your idea, I prefer to keep my patch simple, it is now exactly what Martin mentions. I think it is a good start that can eventually be extended later for what you are asking. Regards, Jan On 3 November 2016 at 17:25, Herv? Pag?s <hpages at fredhutch.org> wrote:> > Hi Martin, Jan, > > On 11/03/2016 03:45 AM, Martin Maechler wrote: >>>>>>> >>>>>>> Jan Gorecki <J.Gorecki at wit.edu.pl> >>>>>>> on Tue, 1 Nov 2016 22:51:28 +0000 writes: >> >> >> > Hello community/devs, Is there an option to run package >> > tests during R CMD check and not stop on first error? I >> > know that testing frameworks (testhat and others) can do >> > that but asking about just R and base packages. Currently >> > when package check runs test scripts in ./tests directory >> > it will stop after first fail. Do you think it could be >> > optionally available to continue to run tests after >> > failures? Regards, Jan Gorecki >> >> I agree that this would be a useful option sometimes. >> >> So I would be supportive to get such an option, say, >> >> R CMD check --no-stop-on-error <pkg> > > > A couple of years ago the behavior of 'R CMD check' was changed to > continue checking (e.g. the examples) after many types of errors, and > to output a summary count of errors at the end if any have occurred. > So --no-stop-on-error could easily be interpreted as an option that > controls this behavior (and would also suggest that the default has > been reverted back to what it was prior to R 3.2.0), rather than an > option that specifically controls what should happen while running > the tests. > > Cheers, > H. > >> >> into R if someone provided (relatively small) patches to the R >> sources (i.e. subversion repos at https://svn.r-project.org/R/trunk/ ). >> The relevant source code should basically all be in >> src/library/tools/R/testing.R >> >> Note that this may be complicated, also because "parallel" >> checking is available in parts, via the TEST_MC_CORES >> environment variable ((which is currently only quickly >> documented in the 'R Administration ..' manual)) >> >> >> Martin Maechler >> ETH Zurich >> >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel >> > > -- > Herv? Pag?s > > Program in Computational Biology > Division of Public Health Sciences > Fred Hutchinson Cancer Research Center > 1100 Fairview Ave. N, M1-B514 > P.O. Box 19024 > Seattle, WA 98109-1024 > > E-mail: hpages at fredhutch.org > Phone: (206) 667-5791 > Fax: (206) 667-1319
Martin Maechler
2016-Nov-04 15:24 UTC
[Rd] Running package tests and not stop on first fail
>>>>> Jan Gorecki <J.Gorecki at wit.edu.pl> >>>>> on Fri, 4 Nov 2016 11:20:37 +0000 writes:> Martin, I submitted very simple patch on > https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176 > Herve, While I like your idea, I prefer to keep my patch > simple, it is now exactly what Martin mentions. I think it > is a good start that can eventually be extended later for > what you are asking. I tend to agree; this seems indeed much easier than I anticipated. Thank you, Jan! I'm testing a version which uses the logical variable 'stop_on_error' rather than 'no_stop_on_error' (because !no_stop_on_error is hard to mentally parse quickly). My proposed name '--no-stop-on-error' was a quick shot; if somebody has a more concise or better "English style" wording (which is somewhat compatible with all the other options you see from 'R CMD check --help'), please speak up. Martin > Regards, Jan > On 3 November 2016 at 17:25, Herv? Pag?s > <hpages at fredhutch.org> wrote: >> >> Hi Martin, Jan, >> >> On 11/03/2016 03:45 AM, Martin Maechler wrote: >>>>>>>> >>>>>>>> Jan Gorecki <J.Gorecki at wit.edu.pl> on Tue, 1 Nov >>>>>>>> 2016 22:51:28 +0000 writes: >>> >>> >>> > Hello community/devs, Is there an option to run >>> package > tests during R CMD check and not stop on first >>> error? I > know that testing frameworks (testhat and >>> others) can do > that but asking about just R and base >>> packages. Currently > when package check runs test >>> scripts in ./tests directory > it will stop after first >>> fail. Do you think it could be > optionally available >>> to continue to run tests after > failures? Regards, Jan >>> Gorecki >>> >>> I agree that this would be a useful option sometimes. >>> >>> So I would be supportive to get such an option, say, >>> >>> R CMD check --no-stop-on-error <pkg> >> >> >> A couple of years ago the behavior of 'R CMD check' was >> changed to continue checking (e.g. the examples) after >> many types of errors, and to output a summary count of >> errors at the end if any have occurred. So >> --no-stop-on-error could easily be interpreted as an >> option that controls this behavior (and would also >> suggest that the default has been reverted back to what >> it was prior to R 3.2.0), rather than an option that >> specifically controls what should happen while running >> the tests. >> >> Cheers, H. >> >>> >>> into R if someone provided (relatively small) patches to >>> the R sources (i.e. subversion repos at >>> https://svn.r-project.org/R/trunk/ ). The relevant >>> source code should basically all be in >>> src/library/tools/R/testing.R >>> >>> Note that this may be complicated, also because >>> "parallel" checking is available in parts, via the >>> TEST_MC_CORES environment variable ((which is currently >>> only quickly documented in the 'R Administration ..' >>> manual)) >>> >>> >>> Martin Maechler ETH Zurich >>> >>> ______________________________________________ >>> R-devel at r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-devel >>> >> >> -- >> Herv? Pag?s >> >> Program in Computational Biology Division of Public >> Health Sciences Fred Hutchinson Cancer Research Center >> 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA >> 98109-1024 >> >> E-mail: hpages at fredhutch.org Phone: (206) 667-5791 Fax: >> (206) 667-1319