My thanks to Drs. Armstrong, Bates, Harrell, Liaw, Lumley,
Prager, Schwartz, and Mr. Wang for their replies. I have
pasted my original message and their replies below.
After viewing http://www.itl.nist.gov/div898/strd/ as suggested
by Dr. Schwartz, it occurred to me that it might be educational
to search for some data repositories on google. I was able to find
some,though I'm sure many of the R listserv readers are already
aware of them:
http://kdd.ics.uci.edu/
http://www.ics.uci.edu/~mlearn/MLOther.html
http://www.ldeo.columbia.edu/datarep/
http://data.geocomm.com/
http://libraries.mit.edu/gis/data/repository.html
http://nssdc.gsfc.nasa.gov/
-david paul
-----Original Message-----
I am one of only 5 or 6 people in my organization making the
effort to include R/Splus as an analysis tool in everyday
work - the rest of my colleagues use SAS exclusively.
Today, one of them made the assertion that he believes the
numerical algorithms in SAS are superior to those in Splus
and R -- ie, optimization routines are faster in SAS, the
SAS Institute has teams of excellent numerical analysts that
ensure its superiority to anything freely available, PROC
NLMIXED is more flexible than nlme( ) in the sense that it
allows a much wider array of error structures than can be used
in R/Splus, &etc.
I obviously do not subscribe to these views and would like
to refute them, but I am not a numerical analyst and am still
a novice at R/Splus. Do there exist refereed papers comparing the
numerical capabilities of these platforms? If not, are there
other resources I might look up and pass along to my colleagues?
---------------------------
This link might give you some insight, but SAS is not one of the
packages benchmarked here.
http://www.sciviews.org/other/benchmark.htm
[Whit Armstrong]
---------------------------
I don't have papers comparing the numerical capabilities but I say
bunk to your colleagues. The last time I looked, SAS still relies
on the out of date Gauss-Jordan sweep operator in many key places,
in place of the QR decomposition that R and S-Plus use in regression.
And SAS being closed source makes it impossible to see how it
really does calculations in some cases.
See http://hesweb1.med.virginia.edu/biostat/s/doc/splus.pdf Section
1.6 for a comparison of S and SAS (though this doesn't address
numerical reliability). Overall, SAS is about 11 years behind R
and S-Plus in statistical capabilities (last year it was about 10
years behind) in my estimation.
Frank Harrell
SAS User, 1969-1991
---
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
---------------------------
Too bad your colleagues weren't at the "State of Statistical
Software" session at JSM. I was there. It was so packed that
people ran out of standing room. The three speakers are all R
advocates (Jan De Leeuw, Luke Tierney and Duncan Temple Lang).
The most interesting thing (to me) about the session is that the
discussant is a person from SAS (first name Wolfgang). I just
had to hear what he'd say.
The SAS person essentially said that the numerical accuracy of
R (probability functions, especially) is unmatched because the
routines were written by authority figures in the area. (That's
one advantage he said R has, but also said that the fact that
that code is open, even SAS is looking at the R source, and that,
to him, is a disadvantage. He obviously missed the point of
open source.) One of the criticisms he had for R, compared to
SAS, is that R may not have undergone extensive QA tests. He
said that SAS now probably has only a handful of PROC developers
(not exactly the "team" your colleague imagined), but 5-6 times
more software testers.
I think hearing from the horse's mouth beats reading articles
in the journal for this sort of things. There was a recent article
in American statistician bashing the numerical instability and bad
quality of RNG in JMP (a SAS product). SAS posted a "white paper"
on their web site refuting some those claims (but they did changed
the RNG to Mersenne Twister in JMP5), comparing JMP with Excel and
SAS. I must say that comparison isn't convincing, as neither Excel
nor SAS can really be trusted as gold standard.
Andy [Liaw]
---------------------------
In follow up to Frank's reply, allow me to point you to some
additional papers and articles on numerical accuracy issues. I
have not reviewed these in some time and they may be a bit dated
relative to current versions. These do not cover R specifically,
but do address S-Plus and SAS. This is not an exhaustive list by
any means, but many of the papers do have other references that
may be of value.
1. http://www.stat.uni-muenchen.de/~knuesel/elv/accuracy.html
2. http://www.amstat.org/publications/tas/mccull-1.pdf
3. http://www.amstat.org/publications/tas/mccull.pdf
4. http://www.npl.co.uk/ssfm/download/documents/cmsc06_00.pdf
Another option is that NIST has reference datasets available for
comparison at:
http://www.itl.nist.gov/div898/strd/
These would allow you to conduct your own comparisons if you desire.
HTH,
Marc Schwartz
(Also a former SAS user)
---------------------------
I can't say for the optimisation routines, but I have found this...
When I was doing my MSc thesis, using tree-based models and neural
networks for classifications, I discovered something interesting.
Using SAS Enterprise Miner (SAS EM), its Tree Node is far more efficient
than the rpart package. Using the same (or very similar at least) parameter
settings, SAS EM can produce a tree in about 1 minute while it would take
rpart 5 ~ 6 minutes (same data, same machine....). Having said that, I
still prefer rpart as it can draw a beautiful tree, whereas it is very
difficult to fit the graphical tree produced by SAS EM into one A4 page --
in the end I had to use the text tree.
However, the Neural Network node in SAS EM is less efficient than nnet.
The time it takes to fit a neural network in R using nnet is much
faster....
Cheers,
Kevin [Wang]
---------------------------
I suspect it will be difficult to find the answer to your colleagues'
assertions without doing your own studies. How important is it to you
to settle this disagreement?
One could always name the many leading statisticians who contribute to
R, but I don't think that name-dropping settles anything.
Nonetheless, even if SAS were faster, that would be only part of the
issue. As you know, R offers vastly better exploratory graphics, better
graphics overall, far more flexible programming, user extensibility, and
more natural programming access to the results of previous
computations. So even if your colleagues were right in their
assertions, they would be overlooking many capabilities of the S
language that are not readily available in SAS.
IMO, SAS shines in its ability to read files in almost any format, to
handle gigantic data sets without burping, and to produce formatted
cross-tabulations and other highly structured text reports. However, if
your colleagues work at all in data exploration, they are ignoring
important tools by not exploring R or S-Plus.
Michael Prager, Ph.D.
NOAA Center for Coastal Fisheries and Habitat Research Beaufort,
North Carolina 28516 http://shrimp.ccfhrb.noaa.gov/~mprager/
DISCLAIMER: Opinions expressed are personal, not official. No
government endorsement of any commercial product is made or implied.
---------------------------
Although they are out of date, there are some comparisons of accuracy in
McCullough, B. D. (1998), "Assessing the reliability of statistical
software: Part I", The American Statistician, 52, 358-366.
McCullough, B. D. (1999), "Assessing the reliability of statistical
software: Part II", The American Statistician, 53, 149-159.
Regarding PROC NLMIXED versus nlme, there are a lot of differences
between them. I don't think that PROC NLMIXED will handle nested
random effects while nlme does. However, nlme assumes the underlying
noise is Gaussian while PROC NLMIXED allows Gaussian or binomial or
Poisson. PROC NLMIXED uses adaptive Gaussian quadrature to evaluate the
marginal log-likelihood whereas nlme uses a less accurate evaluation but
better parameterizations of the variance of the random effects. I think
it would be difficult to declare one to be superior to the other.
[Douglas Bates]
---------------------------
While I don't subscribe to the general theory, they have a point about
PROC NLMIXED. It does more accurate calculations for generalised linear
mixed models than are currently available in R/S-PLUS, and for logistic
random effects models the difference can sometimes be large enought to
matter.
-Thomas [Lumley]
