R help - Sep 2012 - Memory errors using lmer

Hello R List,

I'm trying to run a linear mixed effects analysis on fairly large datasets
with lmer (from the lme4 package) on a 32-bit Windows machine running XP with 3
GB of RAM. It's not working. (details below)

I've researched the ff and bigmemory packages, but it appears they won't
handle the mixed mode dataset I'm analyzing. It has some character fields
for the categorical variables. It's also not clear a linear mixed effect
regression can be run with those packages. Can anyone point me to a lme solution
on larger datasets that can address the 32-bit memory limitations?

The smallest dataframe I'm trying to run is about 120,000 observations and 7
variables, but I'd rather run a for loop script on 620K+ observations.
I've seen several recommendations (Kabacoff in 'R in Action' and
others) that when possible, run R in a 64-bit build. Problem is I'm on a
deadline, and procuring a new computer takes time, and approvals up the food
chain. Suggestions? Is 64-bit my only option?

Here is the information about what I'm running:

R version 2.14.0 (2011-10-31)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods
[7] base

other attached packages:
[1] lme4_0.999999-0 Matrix_1.0-5    lattice_0.20-10
[4] RODBC_1.3-6

loaded via a namespace (and not attached):
[1] grid_2.14.0   nlme_3.1-104  stats4_2.14.0 tools_2.14.0

I run this code:
> install.packages("lme4")
> library(lme4)
> math07g4 <- sqlQuery(conn, "select ssid, ss_chg, campus2,
district_id, pblack, pfreelnch, pmob
         FROM codemob0607ma WHERE grade2 = 4")
> attach(math07g4)
> fit07ma4 <- lmer(ss_chg ~ 1 + factor(campus2) + factor(district_id) +
pblack + pfreelnch + pmob +
   (1 | campus2) + (1 | district_id), data=math07g4)

And I get this:

Error: cannot allocate vector of size 2.5 Gb
In addition: Warning messages:
1: In model.matrix.default(mt, mf, contrasts) :
  Reached total allocation of 2187Mb: see help(memory.size)
2: In model.matrix.default(mt, mf, contrasts) :
  Reached total allocation of 2187Mb: see help(memory.size)
3: In model.matrix.default(mt, mf, contrasts) :
  Reached total allocation of 2187Mb: see help(memory.size)
4: In model.matrix.default(mt, mf, contrasts) :
  Reached total allocation of 2187Mb: see help(memory.size)


Thanks in advance,

Ken

Ken McCall | Database reporter
Dayton Daily News<http://www.daytondailynews.com/> | 1611 S. Main St.
Dayton, Ohio 45409
Ph: 937-000-000  Cell: 937-225-2393  Email:
ken.mccall@coxinc.com<mailto:ken.mccall@coxinc.com>
COMPLETE. IN-DEPTH. DEPENDABLE.


	[[alternative HTML version deleted]]

McCall, Ken (CMG-Dayton <Ken.McCall <at> coxinc.com> writes:
> I'm trying to run a linear mixed effects analysis on fairly large
> datasets with lmer (from the lme4 package) on a 32-bit Windows
> machine running XP with 3 GB of RAM. It's not working. (details
> below)
 
> I've researched the ff and bigmemory packages, but it appears they
> won't handle the mixed mode dataset I'm analyzing. It has some
> character fields for the categorical variables. It's also not clear
> a linear mixed effect regression can be run with those packages. Can
> anyone point me to a lme solution on larger datasets that can
> address the 32-bit memory limitations?
 
> The smallest dataframe I'm trying to run is about 120,000
> observations and 7 variables, but I'd rather run a for loop script
> on 620K+ observations. I've seen several recommendations (Kabacoff
> in 'R in Action' and others) that when possible, run R in a 64-bit
> build. Problem is I'm on a deadline, and procuring a new computer
> takes time, and approvals up the food chain. Suggestions? Is 64-bit
> my only option?
> > install.packages("lme4")
  (only needs to be done once)
> > library(lme4)
> > math07g4 <- sqlQuery(conn, "select ssid, ss_chg, 
> campus2, district_id, pblack, pfreelnch, pmob
>          FROM codemob0607ma WHERE grade2 = 4")
  Why are you attach()ing?  Probably unnecessary ...
> > attach(math07g4)
> > fit07ma4 <- lmer(ss_chg ~ 1 + factor(campus2) + factor(district_id)
+
>  pblack + pfreelnch + pmob +
>    (1 | campus2) + (1 | district_id), data=math07g4)
> 
> And I get this:
> Error: cannot allocate vector of size 2.5 Gb
> In addition: Warning messages:
> 1: In model.matrix.default(mt, mf, contrasts) :
>   Reached total allocation of 2187Mb: see help(memory.size)
  Once upon a time there may have been an option for sparse
model matrices, but not now (I think).

  Depending on whether you have any budget at all,
I wonder if you could use Amazon ... google "r amazon ec2 instance"
for more information ...

  If you need more info, I would suggest posting to 
r-sig-mixed-models <at> r-project.org (a specialty mailing 
list for mixed models).