Hey, Can anyone answer this question. I am working with really large datasets and most of the programs I have been running take quite some time. I heard that R may be faster in Unix. I sthis true and if so can anyone reccomend which system and requirements may allow things to go faster for? Thanks!! Elizabeth Lawson --------------------------------- [[alternative HTML version deleted]]
Please read the posting guide and repost. Your question is too vague to respond to sensibly. Speed in R depends on two things: How you program and what sort of system resources R has available (especially memory). OS's can certainly make a difference, but this is a 2nd order effect, I think. -- Bert Gunter Genentech Non-Clinical Statistics South San Francisco, CA "The business of the statistician is to catalyze the scientific learning process." - George E. P. Box> -----Original Message----- > From: r-help-bounces at stat.math.ethz.ch > [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of > Elizabeth Lawson > Sent: Tuesday, December 13, 2005 2:23 PM > To: r-help at stat.math.ethz.ch > Subject: [R] getting faster results > > Hey, > > Can anyone answer this question. I am working with really > large datasets and most of the programs I have been running > take quite some time. > > I heard that R may be faster in Unix. I sthis true and if > so can anyone reccomend which system and requirements may > allow things to go faster for? > > Thanks!! > > Elizabeth Lawson > > > --------------------------------- > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html >
I am running algorithms for Bayesian hierarchical models with 20 or more
variables or writing alogrithms that solve for partially linear models using
wavlelet estimation. Soemtimes just creating the predicted vaules from the
Bayesian analysis takes over a day to run!
I am just wondering if R runs faster on any particular systems or if it is
about the same regardless.
jim holtman <jholtman@gmail.com> wrote:
Need to know what you are trying to do with the data. Depending on what you
want to do and how you do it will determine the speed. I can take a dataframe
with 1,000,000 rows of transaction data and compute the average and plot a
histogram of the response time in less that a couple of seconds. Is that fast
enough?
On 12/13/05, Elizabeth Lawson <lizzylaws@yahoo.com> wrote: Hey,
Can anyone answer this question. I am working with really large datasets and
most of the programs I have been running take quite some time.
I heard that R may be faster in Unix. I sthis true and if so can anyone
reccomend which system and requirements may allow things to go faster for?
Thanks!!
Elizabeth Lawson
---------------------------------
[[alternative HTML version deleted]]
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
--
Jim Holtman
Cincinnati, OH
+1 513 247 0281
What the problem you are trying to solve?
---------------------------------
[[alternative HTML version deleted]]
What is almost certainly critical here are the algorithms you use (to do the integrations in the Bayesian models) and perhaps a fast BLAS. -- Bert Gunter Genentech Non-Clinical Statistics South San Francisco, CA "The business of the statistician is to catalyze the scientific learning process." - George E. P. Box> -----Original Message----- > From: r-help-bounces at stat.math.ethz.ch > [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of > Elizabeth Lawson > Sent: Wednesday, December 14, 2005 9:53 AM > To: jim holtman > Cc: r-help at stat.math.ethz.ch > Subject: Re: [R] getting faster results > > I am running algorithms for Bayesian hierarchical models with > 20 or more variables or writing alogrithms that solve for > partially linear models using wavlelet estimation. Soemtimes > just creating the predicted vaules from the Bayesian analysis > takes over a day to run! > I am just wondering if R runs faster on any particular > systems or if it is about the same regardless. > > jim holtman <jholtman at gmail.com> wrote: > Need to know what you are trying to do with the data. > Depending on what you want to do and how you do it will > determine the speed. I can take a dataframe with 1,000,000 > rows of transaction data and compute the average and plot a > histogram of the response time in less that a couple of > seconds. Is that fast enough? > > On 12/13/05, Elizabeth Lawson <lizzylaws at yahoo.com> wrote: Hey, > > Can anyone answer this question. I am working with really > large datasets and most of the programs I have been running > take quite some time. > > I heard that R may be faster in Unix. I sthis true and if so > can anyone reccomend which system and requirements may allow > things to go faster for? > > Thanks!! > > Elizabeth Lawson > > > --------------------------------- > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > > > > > -- > Jim Holtman > Cincinnati, OH > +1 513 247 0281 > > What the problem you are trying to solve? > > > > --------------------------------- > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html >
It can depend on a whole list of things why the computations were taking so long, and just saying what methods you are using to do the computation is far from sufficient, as there are efficient ways and inefficient ways to do any computations, and we don't know how you did any of them. With the little information you gave, you cannot honestly expect much help. All anyone can give you is generic advise. Mine is as follows:\ - R has profiling facility that tells you what part(s) of the computation is taking the most time. Use it. (See ?Rprof.) - If you see (or hear) lots of disk activity, but fairly low CPU usage, you might be short in physical memory, and R is using virtual memory, which will be _very_ slow. - Given the same hardware, there's little difference in performance between different OSes, although Linux seems to have an edge. Compilers used can also make a difference. - Hardware can make a large difference. On some analyses I did, running the same code on an Opteron box running 64-bit Linux (SLES) was over six times faster than on my Pentium M laptop. Andy From: Elizabeth Lawson> > I am running algorithms for Bayesian hierarchical models with > 20 or more variables or writing alogrithms that solve for > partially linear models using wavlelet estimation. Soemtimes > just creating the predicted vaules from the Bayesian analysis > takes over a day to run! > I am just wondering if R runs faster on any particular > systems or if it is about the same regardless. > > jim holtman <jholtman at gmail.com> wrote: > Need to know what you are trying to do with the data. > Depending on what you want to do and how you do it will > determine the speed. I can take a dataframe with 1,000,000 > rows of transaction data and compute the average and plot a > histogram of the response time in less that a couple of > seconds. Is that fast enough? > > On 12/13/05, Elizabeth Lawson <lizzylaws at yahoo.com> wrote: Hey, > > Can anyone answer this question. I am working with really > large datasets and most of the programs I have been running > take quite some time. > > I heard that R may be faster in Unix. I sthis true and if so > can anyone reccomend which system and requirements may allow > things to go faster for? > > Thanks!! > > Elizabeth Lawson > > > --------------------------------- > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > > > > > -- > Jim > Holtman > Cincinnati, OH > +1 513 247 0281 > > What the problem you are trying to solve? > > > > --------------------------------- > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > >
I am not sure...
But my employer would like to get me a new machine and considering that R is
the program I use most often, he was wondering what type of machine would be the
best to get form me.
jim holtman <jholtman@gmail.com> wrote:
If you job is CPU intensive (it should be unless it is reading in data all the
time, or does not have enough memory so it is paging). Look at the amount of
time you are paging. Otherwise, it is probably using only 1/4 of the available
processors if you system has 4 CPUs. Do you know how many it has?
On 12/14/05, Elizabeth Lawson <lizzylaws@yahoo.com> wrote: I am
running algorithms for Bayesian hierarchical models with 20 or more variables or
writing alogrithms that solve for partially linear models using wavlelet
estimation. Soemtimes just creating the predicted vaules from the Bayesian
analysis takes over a day to run!
I am just wondering if R runs faster on any particular systems or if it is
about the same regardless.
jim holtman < jholtman@gmail.com> wrote:
Need to know what you are trying to do with the data. Depending on what you
want to do and how you do it will determine the speed. I can take a dataframe
with 1,000,000 rows of transaction data and compute the average and plot a
histogram of the response time in less that a couple of seconds. Is that fast
enough?
On 12/13/05, Elizabeth Lawson <lizzylaws@yahoo.com > wrote: Hey,
Can anyone answer this question. I am working with really large datasets and
most of the programs I have been running take quite some time.
I heard that R may be faster in Unix. I sthis true and if so can anyone
reccomend which system and requirements may allow things to go faster for?
Thanks!!
Elizabeth Lawson
---------------------------------
[[alternative HTML version deleted]]
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
--
Jim Holtman
Cincinnati, OH
+1 513 247 0281
What the problem you are trying to solve?
---------------------------------
--
Jim Holtman
Cincinnati, OH
+1 513 247 0281
What the problem you are trying to solve?
---------------------------------
[[alternative HTML version deleted]]