R devel - Mar 2012 - seeing feedback when R CMD check pkg runs unit tests.

good day here,

I'm maintaining a couple of R modules, both on r-forge.

tests for these modules are written making use of unit testing, and I
make use of the svUnit module, part of SciViews-R.

I also make use of examples in the .Rd files.

my question regards 'R CMD check pkg'.

if an _example_ is not run correctly, I get clear feedback on the
command line where I run 'R CMD check pkg'.  _unit tests_ on the other
hand may give failure or error, or may be marked as skipped.  all these
conditions are caught by the unit testing module that allows the unit
testing script to produce a complete report.

if the unit testing module manages to run the tests, the report is
generated and 'R CMD check pkg' will say "OK".

maybe you find here a more verbose description of the issue:
http://stackoverflow.com/questions/2737680/
https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=14702

very briefly: I'd like the check to include the possibility to issue a
NOTE or a WARNING message, respectively for skipped and failing tests.

as I understood it, I can write my own check-my.R routine and use 'R CMD
check-my pkg'.

but this way my modules would check-my (on my system) differently than
they would check, say, on r-forge.  r-forge would tell me my module is
all right for releasing even if it has failing unit tests.

let me exaggerate: doesn't this sound to you as if unit testing was
discouraged by R?

many thanks for all replies,
MF

Mario

I think of the RUnit and svUnit testing to be in addition to the 
standard  R package testing framework. You can get unit and more 
integrated testing results, with the behaviour I think you are looking 
for, simply by putting small test files in the tests/ directory of the 
package. These files should stop("error message") or
warning("message")
when your code does not get expected results. I'm not sure about
"note"
but you might consider avoiding those, since the R CMD check "notes" 
tend to become warnings or errors in a later version. R CMD check gives 
the same sorts of messages for these as it does for examples. Examples 
only need to run, but in tests/ you can do things like

if( 2*2 != 4 ) stop("arithmatic is messed up.")

and R CMD check will flag this problem.

CRAN disables some of this testing for time consideration, but your own 
R CMD check will run the tests, and I think R-forge does not 
automatically disable anything (yet). So, it may be necessary to arrange 
  skipping tests that you expect to fail on some platforms.

You may be able to get similar results from RUnit or svUnit, it would 
just be a question of passing the stop() and warnings() back to the top 
level in the script file in tests/. If you don't do that, as you 
observed, R CMD check thinks the unit testing worked fine, it did its 
job and found the errors.

(Happy to hear additional points of view on this, my understanding of 
RUnit and svUnit is limited.)

Paul

On 12-03-02 10:49 AM, Mario Frasca wrote:
> good day here,
>
> I'm maintaining a couple of R modules, both on r-forge.
>
> tests for these modules are written making use of unit testing, and I
> make use of the svUnit module, part of SciViews-R.
>
> I also make use of examples in the .Rd files.
>
> my question regards 'R CMD check pkg'.
>
> if an _example_ is not run correctly, I get clear feedback on the
> command line where I run 'R CMD check pkg'.  _unit tests_ on the
other
> hand may give failure or error, or may be marked as skipped.  all these
> conditions are caught by the unit testing module that allows the unit
> testing script to produce a complete report.
>
> if the unit testing module manages to run the tests, the report is
> generated and 'R CMD check pkg' will say "OK".
>
> maybe you find here a more verbose description of the issue:
> http://stackoverflow.com/questions/2737680/
> https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=14702
>
> very briefly: I'd like the check to include the possibility to issue a
> NOTE or a WARNING message, respectively for skipped and failing tests.
>
> as I understood it, I can write my own check-my.R routine and use 'R
CMD
> check-my pkg'.
>
> but this way my modules would check-my (on my system) differently than
> they would check, say, on r-forge.  r-forge would tell me my module is
> all right for releasing even if it has failing unit tests.
>
> let me exaggerate: doesn't this sound to you as if unit testing was
> discouraged by R?
>
> many thanks for all replies,
> MF
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel