R help - Jun 2017 - Prediction with two fixed-effects - large number of IDs

Dear all,

I am running a panel regression with time and location fixed effects:

###

reg1 <- lm(lny ~ factor(id) + factor(year) + x1+ I(x1)^2 + x2+ I(x2)^2 ,
 data=mydata, na.action="na.omit")
###

My goal is to use the estimation for prediction. However, I have 8,500 IDs,
which is resulting in very slow computation. Ideally, I would like to do
the following:

###
reg2 <- felm(lny ~ x1+ I(x1)^2 + x2+ I(x2)^2 | id + year , data=mydata,
na.action="na.omit")
###

However, predict does not work with felm. Is there a way to either make lm
faster or use predict with felm? Is parallelizing an option?

Any help will be appreciated. Thank you!

Sincerely,

Milu

	[[alternative HTML version deleted]]

I have no direct experience with such horrific models, but your formula is a
mess and Google suggests the biglm package with ffdf.

Specifically, you should convert your discrete variables to factors before you
build the model, particularly since you want to use predict after the fact, for
which you will need a new data set with the exact same levels in the factors.

Also, your use of I() is broken and redundant.  I think formulas

lny ~ id + year + x1 + I(x1^2) + x2 + I(x2^2)

or

lny ~ id + year + x1^2 + x2^2

would obtain the intended prediction results.

-- 
Sent from my phone. Please excuse my brevity.

On June 17, 2017 11:24:05 AM PDT, Miluji Sb <milujisb at gmail.com>
wrote:
>Dear all,
>
>I am running a panel regression with time and location fixed effects:
>
>###
>
>reg1 <- lm(lny ~ factor(id) + factor(year) + x1+ I(x1)^2 + x2+ I(x2)^2
>,
> data=mydata, na.action="na.omit")
>###
>
>My goal is to use the estimation for prediction. However, I have 8,500
>IDs,
>which is resulting in very slow computation. Ideally, I would like to
>do
>the following:
>
>###
>reg2 <- felm(lny ~ x1+ I(x1)^2 + x2+ I(x2)^2 | id + year , data=mydata,
>na.action="na.omit")
>###
>
>However, predict does not work with felm. Is there a way to either make
>lm
>faster or use predict with felm? Is parallelizing an option?
>
>Any help will be appreciated. Thank you!
>
>Sincerely,
>
>Milu
>
>	[[alternative HTML version deleted]]
>
>______________________________________________
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

Dear Jeff,

Thank you so much and apologies for the typo in I() - it was silly.

I will try the biglm package - thanks!

Sincerely,

Milu

On Sat, Jun 17, 2017 at 9:01 PM, Jeff Newmiller <jdnewmil at
dcn.davis.ca.us>
wrote:
> I have no direct experience with such horrific models, but your formula is
> a mess and Google suggests the biglm package with ffdf.
>
> Specifically, you should convert your discrete variables to factors before
> you build the model, particularly since you want to use predict after the
> fact, for which you will need a new data set with the exact same levels in
> the factors.
>
> Also, your use of I() is broken and redundant.  I think formulas
>
> lny ~ id + year + x1 + I(x1^2) + x2 + I(x2^2)
>
> or
>
> lny ~ id + year + x1^2 + x2^2
>
> would obtain the intended prediction results.
>
> --
> Sent from my phone. Please excuse my brevity.
>
> On June 17, 2017 11:24:05 AM PDT, Miluji Sb <milujisb at gmail.com>
wrote:
> >Dear all,
> >
> >I am running a panel regression with time and location fixed effects:
> >
> >###
> >
> >reg1 <- lm(lny ~ factor(id) + factor(year) + x1+ I(x1)^2 + x2+
I(x2)^2
> >,
> > data=mydata, na.action="na.omit")
> >###
> >
> >My goal is to use the estimation for prediction. However, I have 8,500
> >IDs,
> >which is resulting in very slow computation. Ideally, I would like to
> >do
> >the following:
> >
> >###
> >reg2 <- felm(lny ~ x1+ I(x1)^2 + x2+ I(x2)^2 | id + year ,
data=mydata,
> >na.action="na.omit")
> >###
> >
> >However, predict does not work with felm. Is there a way to either make
> >lm
> >faster or use predict with felm? Is parallelizing an option?
> >
> >Any help will be appreciated. Thank you!
> >
> >Sincerely,
> >
> >Milu
> >
> >       [[alternative HTML version deleted]]
> >
> >______________________________________________
> >R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> >https://stat.ethz.ch/mailman/listinfo/r-help
> >PLEASE do read the posting guide
> >http://www.R-project.org/posting-guide.html
> >and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]

> On Jun 17, 2017, at 12:01 PM, Jeff Newmiller <jdnewmil at
dcn.davis.ca.us> wrote:
> 
> I have no direct experience with such horrific models, but your formula is
a mess and Google suggests the biglm package with ffdf.
> 
> Specifically, you should convert your discrete variables to factors before
you build the model, particularly since you want to use predict after the fact,
for which you will need a new data set with the exact same levels in the
factors.
> 
> Also, your use of I() is broken and redundant.  I think formulas
> 
> lny ~ id + year + x1 + I(x1^2) + x2 + I(x2^2)
> 
> or
> 
> lny ~ id + year + x1^2 + x2^2
This was offered as a formula to `felm` (but with no data example), a package
with which I have no experience either, but if experience with `lm` and `glm` is
any guide, an inferentially safer approach might be:

   lny ~ id + year + poly(x1,2) + poly(x2,2)

-- 

David

> 
> would obtain the intended prediction results.
> 
> -- 
> Sent from my phone. Please excuse my brevity.
> 
> On June 17, 2017 11:24:05 AM PDT, Miluji Sb <milujisb at gmail.com>
wrote:
>> Dear all,
>> 
>> I am running a panel regression with time and location fixed effects:
>> 
>> ###
>> 
>> reg1 <- lm(lny ~ factor(id) + factor(year) + x1+ I(x1)^2 + x2+
I(x2)^2
>> ,
>> data=mydata, na.action="na.omit")
>> ###
>> 
>> My goal is to use the estimation for prediction. However, I have 8,500
>> IDs,
>> which is resulting in very slow computation. Ideally, I would like to
>> do
>> the following:
>> 
>> ###
>> reg2 <- felm(lny ~ x1+ I(x1)^2 + x2+ I(x2)^2 | id + year ,
data=mydata,
>> na.action="na.omit")
>> ###
>> 
>> However, predict does not work with felm. Is there a way to either make
>> lm
>> faster or use predict with felm? Is parallelizing an option?
>> 
>> Any help will be appreciated. Thank you!
>> 
>> Sincerely,
>> 
>> Milu
>> 
>> 	[[alternative HTML version deleted]]
>> 
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
> 
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
David Winsemius
Alameda, CA, USA