R help - Aug 2006 - help: cannot allocate vector of length 828310236

Hi all,

I was trying a probit regression using polr() and got this message,

Error in model.matrix.default(Terms, m, contrasts) :
        cannot allocate vector of length 828310236

The data is about 20M (a few days ago I asked a question about large file,
thank you for responses, then I use MS Access to select those columns I
would use).

R is 2.3.1, Windows XP, 512M Ram.

I am going to read some help on memory use in R, but hope anybody can give
me some quick hints.

Is it because iphysical memory runs out, or some other things could be wrong
with data or polr()?
Does R use virtual memory? If so, what options can I set?
If not, can R deal with really huge data (except adding RAM according to
data size)? If this is the case, it is too bad that I have to tell my boss
to go back to SAS. Now it is not a speed issue yet.

Thank you.

	[[alternative HTML version deleted]]

Does it make any statistical sense to do polr or probit regression (not 
the same thing) on `really huge data'?  There are few regression-like 
problems in which model inadequacy does not swamp estimation uncertainty 
for as few as a 1000 cases.

If you want to do that sort of thing, by all means use SAS to do it.
But if you are not prepared to spend a few $$ on adequate RAM, don't 
expect free technical consultancy, especially not from those whose work 
you are using and not crediting.

- The uncredited author of polr().


On Mon, 14 Aug 2006, T Mu wrote:
> Hi all,
> 
> I was trying a probit regression using polr() and got this message,
polr is a strange choice of tool for 'probit regression' as the term is 
usually used.  It does 'ordered probit regression'.
> Error in model.matrix.default(Terms, m, contrasts) :
>         cannot allocate vector of length 828310236
> 
> The data is about 20M (a few days ago I asked a question about large file,
> thank you for responses, then I use MS Access to select those columns I
> would use).
> 
> R is 2.3.1, Windows XP, 512M Ram.
> 
> I am going to read some help on memory use in R, but hope anybody can give
> me some quick hints.
Quick hint: read and follow the posting guide BEFORE posting.
> Is it because iphysical memory runs out, or some other things could be
wrong
> with data or polr()?
> Does R use virtual memory? If so, what options can I set?
> If not, can R deal with really huge data (except adding RAM according to
> data size)? If this is the case, it is too bad that I have to tell my boss
> to go back to SAS. Now it is not a speed issue yet.
> 
> Thank you.
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595