R help - Aug 2009 - Panel Data Analysis (PLM) - Fixed Effects - "cannot allocate vector of length"

Hello to all on the list,
I'm trying to estimate a fixed effects model from a large (unbalanced) panel
data set.

I have no problems when using only an individual effect or only a time
effect, but I get an error message when I try for a "twoways" effect.
Here
is some of the code:

paneldata27 is the entire panel data set:
> dim(paneldata27)
[1] 1178831       8
>
paneldata27<-plm.data(paneldata27,index=c("Account","Day.of.Sample"))
> nlevels(paneldata27$Account)
[1] 1573
> nlevels(paneldata27$Day.of.Sample)
[1] 831

That is there will be 1573 individual effects and 831 time effects.
>
paneldata27.fe<-plm(LnMidpeak~DUM1,data=paneldata27,model="within",effect="twoways")

Error in model.matrix.default(~id - 1) :
  cannot allocate vector of length 1854301163

Is my panel data set just too large for R?

Thanks very much in advance,

Peter Steele-Mosey

	[[alternative HTML version deleted]]

Hello Peter. Sorry, I was on vacation. I hope my answer is still of some
use to you.

Strictly speaking, this is not a 'plm' problem. As you guessed,
you're
hitting memory limits in R. See 

??memory 

and more specifically 

?Memory
?Memory-limits

to see whether you can increase them.

Re 'plm' functions, it is possible that we used an inefficient solution
here, and that the code be made more efficient in the future. We'll keep
this on our list and see if we ever come up with a better memory
management for 2-way FEs. The problem actually never surfaced because
people seldom estimate models with so many FEs: actually, with such big
samples, the independent sampling assumptions underlying RE estimators
often look plausible. 

HTH,
Giovanni

Giovanni Millo
Research Dept.,
Assicurazioni Generali SpA
Via Machiavelli 4, 
34132 Trieste (Italy)
tel. +39 040 671184 
fax  +39 040 671160 

------------------------------

Original message:
Date: Fri, 21 Aug 2009 16:36:24 -0400
From: Peter Steele-Mosey <petersm at gmail.com>
Subject: [R] Panel Data Analysis (PLM) - Fixed Effects - "cannot
	allocate	vector of length"
To: r-help at r-project.org
Message-ID:
	<e68753d70908211336t25d2fac6k2cefe8aa0b38a69c at mail.gmail.com>
Content-Type: text/plain

Hello to all on the list,
I'm trying to estimate a fixed effects model from a large (unbalanced)
panel
data set.

I have no problems when using only an individual effect or only a time
effect, but I get an error message when I try for a "twoways" effect.
Here
is some of the code:

paneldata27 is the entire panel data set:
> dim(paneldata27)
[1] 1178831       8
>
paneldata27<-plm.data(paneldata27,index=c("Account","Day.of.Sample"))
> nlevels(paneldata27$Account)
[1] 1573
> nlevels(paneldata27$Day.of.Sample)
[1] 831

That is there will be 1573 individual effects and 831 time effects.
>
paneldata27.fe<-plm(LnMidpeak~DUM1,data=paneldata27,model="within",effec
t="twoways")

Error in model.matrix.default(~id - 1) :
  cannot allocate vector of length 1854301163

Is my panel data set just too large for R?

Thanks very much in advance,

Peter Steele-Mosey


Giovanni Millo
Research Dept.,
Assicurazioni Generali SpA
Via Machiavelli 4, 
34132 Trieste (Italy)
tel. +39 040 671184 
fax  +39 040 671160

Ai sensi del D.Lgs. 196/2003 si precisa che le informazi...{{dropped:13}}