Hi, I recently provoked some discussion over on the BioConductor developer list about testing and R CMD check. In brief, the salient points that arose from the discussion are: [1] Because of the need for a nightly build of BioConductor, the tests (in the ./tests directory) of a package that run routinely as part of "R CMD check" must complete in five minutes. [2] Nontrivial regression testing of complex algorithms on real data can easily take longer than five minutes. [3] Maintaining and improving code that works on various kinds of "omics" data is greatly facilitated by the inclusion of nontrivial regression tests. Of course, points [1] and [3] are incompatible in the current setup. Both could, however, be accommodated by changing the way "R CMD check" runs test scripts. There are at least two ways this could be accomplished. One possibility would be to define a separate directory as part of the package structure to store the more complex, longer, "deep" tests. That is, one might have package/R package/man package/tests package/deeptests [or some better name] Then the default for "R CMD check" would be to run all the .R scripts in tests but none of the .R scripts in deep tests. However, one would also have to add an option, perhaps something like R CMD check --run-deeptests [package] to run the more elaborate test scripts. The second possibility would be to keep all the tests in the same (existing ./tests) directory but include a standard file in that directory to indicate which scripts are more complex. This would be a little trickier to design, since it would be nice to keep the existing behavior for someone who ignores the new structure. One, could however, allow for something like ---------------------------- ## CONTROLTESTS complex: test1.R, test3.R simple: test2.R, test4.R ----------------------------- The default (in the absence of A CONTROLTESTS file or for scripts in the directory that are not listed in the file) is to assume "simple", which would then get the existing behavior without changes to existing packages. How hard would it be to implement one of these approaches for R CMD check? -- Kevin
A combination of RUnit from CRAN and .Rbuildignore provide a workable solution for the 'xts' package I work on, though this obviously removes the tests from the distributed code. http://cran.r-project.org/web/packages/RUnit/index.html --and-- http://wiki.r-project.org/rwiki/doku.php?id=developers:runit Jeff On Wed, Jun 11, 2008 at 10:05 AM, Kevin R. Coombes <krcoombes at mdacc.tmc.edu> wrote:> Hi, > > I recently provoked some discussion over on the BioConductor developer list > about testing and R CMD check. In brief, the salient points that arose from > the discussion are: > > [1] Because of the need for a nightly build of BioConductor, the tests (in > the ./tests directory) of a package that run routinely as part of "R CMD > check" must complete in five minutes. > [2] Nontrivial regression testing of complex algorithms on real data can > easily take longer than five minutes. > [3] Maintaining and improving code that works on various kinds of "omics" > data is greatly facilitated by the inclusion of nontrivial regression tests. > > Of course, points [1] and [3] are incompatible in the current setup. Both > could, however, be accommodated by changing the way "R CMD check" runs test > scripts. There are at least two ways this could be accomplished. > > One possibility would be to define a separate directory as part of the > package structure to store the more complex, longer, "deep" tests. That is, > one might have > package/R > package/man > package/tests > package/deeptests [or some better name] > Then the default for "R CMD check" would be to run all the .R scripts in > tests but none of the .R scripts in deep tests. However, one would also have > to add an option, perhaps something like > R CMD check --run-deeptests [package] > to run the more elaborate test scripts. > > The second possibility would be to keep all the tests in the same (existing > ./tests) directory but include a standard file in that directory to indicate > which scripts are more complex. This would be a little trickier to design, > since it would be nice to keep the existing behavior for someone who ignores > the new structure. One, could however, allow for something like > > ---------------------------- > ## CONTROLTESTS > > complex: test1.R, test3.R > simple: test2.R, test4.R > ----------------------------- > > The default (in the absence of A CONTROLTESTS file or for scripts in the > directory that are not listed in the file) is to assume "simple", which > would then get the existing behavior without changes to existing packages. > > How hard would it be to implement one of these approaches for R CMD check? > > -- Kevin > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >-- There's a way to do it better - find it. Thomas A. Edison
On Wed, 11 Jun 2008, Kevin R. Coombes wrote:> [1] Because of the need for a nightly build of BioConductor, the tests > (in the ./tests directory) of a package that run routinely as part of "R > CMD check" must complete in five minutes. > [2] Nontrivial regression testing of complex algorithms on real data can > easily take longer than five minutes. > [3] Maintaining and improving code that works on various kinds of > "omics" data is greatly facilitated by the inclusion of nontrivial > regression tests. > > Of course, points [1] and [3] are incompatible in the current setup. > Both could, however, be accommodated by changing the way "R CMD check" > runs test scripts. There are at least two ways this could be accomplished.Another approach is to have check use the new-to-2.8.0 function setSessionTimeLimit() to let the checker put a time limit on the tests. In a time-constrained environment you could ask check to spend no more than 5 minutes running the tests but you could also set the limit to 3 hours or Inf or anything else. This would only be useful if the quicker tests were done first, so you'd have to alphabetize your test files. It might be nice if check could print the time it took to do each test. ---------------------------------------------------------------------------- Bill Dunlap Insightful Corporation bill at insightful dot com "All statements in this message represent the opinions of the author and do not necessarily reflect Insightful Corporation policy or position."