R devel - Dec 2011 - warning for inefficiently compressed datasets

Hi,

Recently added to doc/NEWS.Rd:

   'R CMD check' now gives a warning rather than a note if it finds
   inefficiently compressed datasets.  With 'bzip2' and 'xz'
compression
   having been available since R 2.10.0, there is no excuse for not
   using them.

Why isn't a note enough for this?

Generally speaking, warnings are for things that are dangerous,
or unsafe, or unportable, or for anything that could potentially
cause trouble. I don't see how using gzip instead of bzip2 or xz
could fall into that category (and BTW gzip is the default for
save() and for 'R CMD build' resave-data feature).

The problem is that bzip2 and xz compressions are slower and also
require more memory than gzip. Bioconductor has big data packages
and sometimes it makes sense to use gzip and not bzip2 or xz. For
example, when loading Human chromosome 1 from disk, bzip2 and xz
are 7 and 3.4 times slower than gzip, respectively:

   > system.time(load("chr1-gzip.rda"))
      user  system elapsed
     1.210   0.180   1.384

   > system.time(load("chr1-bzip2.rda"))
      user  system elapsed
     9.500   0.160   9.674

   > system.time(load("chr1-xz.rda"))
      user  system elapsed
      4.46    0.20    4.69

hpages at latitude:~/testing$ ls -lhtr chr1-*.rda
-rw-r--r-- 1 hpages hpages 61M 2011-12-06 12:13 chr1-gzip.rda
-rw-r--r-- 1 hpages hpages 55M 2011-12-06 12:15 chr1-bzip2.rda
-rw-r--r-- 1 hpages hpages 49M 2011-12-06 12:25 chr1-xz.rda

This is with R-2.14.0 on a 64-bit Ubuntu laptop with 8GB of RAM.

The size on disk doesn't really matter and it doesn't matter either
that the source tarball for the full Human genome ends up being 20%
bigger when using gzip instead of xz: the 20% extra time it takes to
download it (which needs to be done only once) will largely be
compensated by the fact that most analyses will run faster e.g. in
40-45 sec. instead of more than 2 minutes (for many short analyses,
loading the chromosomes into memory is the bottleneck).

Is there a way to turn this warning off? If not, could an option be
added to 'R CMD check' to turn this warning off? Something along the
lines of the --no-resave-data option for 'R CMD build'.

Thanks,
H.

-- 
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fhcrc.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319

On 06.12.2011 23:28, Herv? Pag?s wrote:
> Hi,
>
> Recently added to doc/NEWS.Rd:
>
> 'R CMD check' now gives a warning rather than a note if it finds
> inefficiently compressed datasets. With 'bzip2' and 'xz'
compression
> having been available since R 2.10.0, there is no excuse for not
> using them.
>
> Why isn't a note enough for this?
>
> Generally speaking, warnings are for things that are dangerous,
> or unsafe, or unportable, or for anything that could potentially
> cause trouble. I don't see how using gzip instead of bzip2 or xz
> could fall into that category (and BTW gzip is the default for
> save() and for 'R CMD build' resave-data feature).
>
> The problem is that bzip2 and xz compressions are slower and also
> require more memory than gzip. Bioconductor has big data packages
> and sometimes it makes sense to use gzip and not bzip2 or xz. For
> example, when loading Human chromosome 1 from disk, bzip2 and xz
> are 7 and 3.4 times slower than gzip, respectively:
>
>  > system.time(load("chr1-gzip.rda"))
> user system elapsed
> 1.210 0.180 1.384
>
>  > system.time(load("chr1-bzip2.rda"))
> user system elapsed
> 9.500 0.160 9.674
>
>  > system.time(load("chr1-xz.rda"))
> user system elapsed
> 4.46 0.20 4.69
>
> hpages at latitude:~/testing$ ls -lhtr chr1-*.rda
> -rw-r--r-- 1 hpages hpages 61M 2011-12-06 12:13 chr1-gzip.rda
> -rw-r--r-- 1 hpages hpages 55M 2011-12-06 12:15 chr1-bzip2.rda
> -rw-r--r-- 1 hpages hpages 49M 2011-12-06 12:25 chr1-xz.rda
>
> This is with R-2.14.0 on a 64-bit Ubuntu laptop with 8GB of RAM.
>
> The size on disk doesn't really matter and it doesn't matter either
> that the source tarball for the full Human genome ends up being 20%
> bigger when using gzip instead of xz: the 20% extra time it takes to
> download it (which needs to be done only once) will largely be
> compensated by the fact that most analyses will run faster e.g. in
> 40-45 sec. instead of more than 2 minutes (for many short analyses,
> loading the chromosomes into memory is the bottleneck).

Oh, from a European side this 20% extra time may be an hour when 
downloading from the BioC master rather than a mirror.
And space and traffic is an issue for CRAN.


> Is there a way to turn this warning off? If not, could an option be
> added to 'R CMD check' to turn this warning off? Something along
the
> lines of the --no-resave-data option for 'R CMD build'.

The manual tells us:

"The following environment variables can be used to customize the 
operation of check: a convenient place to set these is the file 
?~/.R/check.Renviron?.

[...]

_R_CHECK_COMPACT_DATA2_

If true, check data for ascii and uncompressed saves, and also check if 
using bzip2 or xz compression would be significantly better. Implies 
_R_CHECK_COMPACT_DATA_ is true. Default: true."


Uwe


>
> Thanks,
> H.
>