Hi All,
I am currently conducting a mixed model. I have 7 repeated measures on a
simulated clinical trial. If I understand the model correctly, the
outcome is the measure (as a factor) the predictors are clinical group
and trial (1-7). The fixed factors are the measure and group. The random
factors are the intercept and id and group.
I tried using 2 functions to calculate mixed effects.
Following previous correspondence .
Dataset <- read.table("C:/Program
Files/R/rw2011/data/miss/model1a.dat",
header=TRUE, sep="\t", na.strings="NA", dec=".",
strip.white=TRUE)
attach(Dataset)
require (nlme)
with(Dataset, table(runnb, id, grp))
b.lvls <- table(Dataset$runnb)
nb <- length(b.lvls)
fit <- vector(mode="list", nb)
for(i in 1:nb)
fit[[i]]<- lme (trans1 ~ Index1 + grp,
random = ~ 1 | id / grp ,
data = Dataset,
na.action = "na.exclude")
This (above) worked OK only I am having memory problems.
I have a gig of RAM set at --sdi --max-mem-size=512M (complete version
below)
I am wondering if running the file as a database be slower / faster?
Then I read that lme4 does it quicker and more accurately
so I thought that I should re-run the code but from the for line:
> for (i in 1:nb)
+ fit[[i]] <- lmer(trans1 ~ Index1 + grp + (1|id:grp) + (1|id),
+ Dataset, na.action = na.exclude)
Producing
Error in lmer(trans1 ~ Index1 + grp + (1 | id:grp) + (1 | id), Dataset,
:
flist[[2]] must be a factor of length 200000
In addition: Warning messages:
1: numerical expression has 200000 elements: only the first used in:
id:grp
2: numerical expression has 200000 elements: only the first used in:
id:grp
Any advice on the source and handling of this error would be
appreciated?
version: _
platform i386-pc-mingw32
arch i386
os mingw32
system i386, mingw32
status
major 2
minor 1.1
year 2005
month 06
day 20
language R
Windows XP Pro.
Many thanks
Stephen
???? ?"? ???? ????
http://mail.nana.co.il
[[alternative HTML version deleted]]
On 6/26/05, Stephen <szlevine at nana.co.il> wrote:> > > > Hi All, > > > > I am currently conducting a mixed model. I have 7 repeated measures on a > simulated clinical trial. If I understand the model correctly, the > outcome is the measure (as a factor) the predictors are clinical group > and trial (1-7). The fixed factors are the measure and group. The random > factors are the intercept and id and group. > > > > I tried using 2 functions to calculate mixed effects. > > Following previous correspondence . > > > > Dataset <- read.table("C:/Program Files/R/rw2011/data/miss/model1a.dat", > header=TRUE, sep="\t", na.strings="NA", dec=".", strip.white=TRUE) > > attach(Dataset) > > > > require (nlme) > > with(Dataset, table(runnb, id, grp)) > > b.lvls <- table(Dataset$runnb) > > nb <- length(b.lvls) > > fit <- vector(mode="list", nb) > > > > for(i in 1:nb) > > fit[[i]]<- lme (trans1 ~ Index1 + grp, > > random = ~ 1 | id / grp , > > data = Dataset, > > na.action = "na.exclude") > > > > > > This (above) worked OK only I am having memory problems. > > I have a gig of RAM set at --sdi --max-mem-size=512M (complete version > below) > > I am wondering if running the file as a database be slower / faster? > > > > Then I read that lme4 does it quicker and more accurately > > so I thought that I should re-run the code but from the for line: > > > > > for (i in 1:nb) > > + fit[[i]] <- lmer(trans1 ~ Index1 + grp + (1|id:grp) + (1|id), > > + Dataset, na.action = na.exclude) > > > > Producing > > > > Error in lmer(trans1 ~ Index1 + grp + (1 | id:grp) + (1 | id), Dataset, > : > > flist[[2]] must be a factor of length 200000 > > In addition: Warning messages: > > 1: numerical expression has 200000 elements: only the first used in: > id:grp > > 2: numerical expression has 200000 elements: only the first used in: > id:grpCheck str(Dataset) and, if necessary, convert id to a factor with Dataset$id <- factor(Dataset$id) In is not surprising that you are running into memory problems. Look at the size of one of the fitted objects from lme or from lmer. They are very large because they contain a copy of the model frame (the parts of Dataset that are needed to evaluate the model) plus a lot of other information. You have a large Dataset and you are saving multiple copies of it although I must admit that I don't understand why the calls to lme or lmer are in a loop.
Hi Thank you for your comments. Yes you are correct its a very big data set. Perhaps I am best splitting it up and then importing to R. The reason for the loop is that I am conducting the equivalent of Split file in SPSS. Specifically, I am conducting the analysis for each value of on the grouping variable 'runnb'. If there is a less memory intensive way of doing this I'd appreciate knowing about it. Many Thanks and comments appreciated Regards Stephen ________________________________ From: Douglas Bates [mailto:dmbates@gmail.com] Sent: Sun 26/06/2005 17:01 To: Stephen Cc: r-help@stat.math.ethz.ch Subject: Re: [R] Mixed model On 6/26/05, Stephen <szlevine@nana.co.il> wrote:> > > > Hi All, > > > > I am currently conducting a mixed model. I have 7 repeated measures ona> simulated clinical trial. If I understand the model correctly, the > outcome is the measure (as a factor) the predictors are clinical group > and trial (1-7). The fixed factors are the measure and group. Therandom> factors are the intercept and id and group. > > > > I tried using 2 functions to calculate mixed effects. > > Following previous correspondence . > > > > Dataset <- read.table("C:/ProgramFiles/R/rw2011/data/miss/model1a.dat",> header=TRUE, sep="\t", na.strings="NA", dec=".", strip.white=TRUE) > > attach(Dataset) > > > > require (nlme) > > with(Dataset, table(runnb, id, grp)) > > b.lvls <- table(Dataset$runnb) > > nb <- length(b.lvls) > > fit <- vector(mode="list", nb) > > > > for(i in 1:nb) > > fit[[i]]<- lme (trans1 ~ Index1 + grp, > > random = ~ 1 | id / grp , > > data = Dataset, > > na.action = "na.exclude") > > > > > > This (above) worked OK only I am having memory problems. > > I have a gig of RAM set at --sdi --max-mem-size=512M (complete version > below) > > I am wondering if running the file as a database be slower / faster? > > > > Then I read that lme4 does it quicker and more accurately > > so I thought that I should re-run the code but from the for line: > > > > > for (i in 1:nb) > > + fit[[i]] <- lmer(trans1 ~ Index1 + grp + (1|id:grp) + (1|id), > > + Dataset, na.action = na.exclude) > > > > Producing > > > > Error in lmer(trans1 ~ Index1 + grp + (1 | id:grp) + (1 | id),Dataset,> : > > flist[[2]] must be a factor of length 200000 > > In addition: Warning messages: > > 1: numerical expression has 200000 elements: only the first used in: > id:grp > > 2: numerical expression has 200000 elements: only the first used in: > id:grpCheck str(Dataset) and, if necessary, convert id to a factor with Dataset$id <- factor(Dataset$id) In is not surprising that you are running into memory problems. Look at the size of one of the fitted objects from lme or from lmer. They are very large because they contain a copy of the model frame (the parts of Dataset that are needed to evaluate the model) plus a lot of other information. You have a large Dataset and you are saving multiple copies of it although I must admit that I don't understand why the calls to lme or lmer are in a loop. ???? ?"? ???? ???? http://mail.nana.co.il [[alternative HTML version deleted]]
Thanks....
Most thoughtful...
Regards
S
________________________________
From: Spencer Graves [mailto:spencer.graves@pdf.com]
Sent: Mon 27/06/2005 19:52
To: Stephen
Cc: Douglas Bates; r-help@stat.math.ethz.ch
Subject: Re: [R] Mixed model
I often think carefully about what I want and store only that.
For
example, I might do something like the following:
b1 <- coef(lme(...))
kb <- length(b1)
B <- array(NA, dim=c(nb, kb))
for(i in 1:nb){
B[i, ] <- coef(lme(...))
}
With "fit[[i]] <- lme(...)", you store, as Doug said,
"a copy
of the
model frame (the parts of Dataset that are needed to evaluate the model)
plus a lot of other information)" for each pass through the loop. Since
you are doing a simulation, you probably only really care about a few
numbers for each "i". Identify those and only store those each time
through the loop.
spencer graves
p.s. Have you considered "simulate.lme"?
Stephen wrote:
> Hi
> Thank you for your comments.
> Yes you are correct its a very big data set.
> Perhaps I am best splitting it up and then importing to R.
> The reason for the loop is that I am conducting the equivalent of
Split> file in SPSS.
> Specifically, I am conducting the analysis for each value of on the
> grouping variable 'runnb'.
> If there is a less memory intensive way of doing this I'd appreciate
> knowing about it.
> Many Thanks and comments appreciated
> Regards
> Stephen
>
> ________________________________
>
> From: Douglas Bates [mailto:dmbates@gmail.com]
> Sent: Sun 26/06/2005 17:01
> To: Stephen
> Cc: r-help@stat.math.ethz.ch
> Subject: Re: [R] Mixed model
>
>
>
> On 6/26/05, Stephen <szlevine@nana.co.il> wrote:
>
>>
>>
>>Hi All,
>>
>>
>>
>>I am currently conducting a mixed model. I have 7 repeated measures on
>
> a
>
>>simulated clinical trial. If I understand the model correctly, the
>>outcome is the measure (as a factor) the predictors are clinical group
>>and trial (1-7). The fixed factors are the measure and group. The
>
> random
>
>>factors are the intercept and id and group.
>>
>>
>>
>>I tried using 2 functions to calculate mixed effects.
>>
>>Following previous correspondence .
>>
>>
>>
>>Dataset <- read.table("C:/Program
>
> Files/R/rw2011/data/miss/model1a.dat",
>
>>header=TRUE, sep="\t", na.strings="NA",
dec=".", strip.white=TRUE)
>>
>>attach(Dataset)
>>
>>
>>
>>require (nlme)
>>
>>with(Dataset, table(runnb, id, grp))
>>
>>b.lvls <- table(Dataset$runnb)
>>
>>nb <- length(b.lvls)
>>
>>fit <- vector(mode="list", nb)
>>
>>
>>
>>for(i in 1:nb)
>>
>> fit[[i]]<- lme (trans1 ~ Index1 + grp,
>>
>> random = ~ 1 | id / grp ,
>>
>> data = Dataset,
>>
>> na.action = "na.exclude")
>>
>>
>>
>>
>>
>>This (above) worked OK only I am having memory problems.
>>
>>I have a gig of RAM set at --sdi --max-mem-size=512M (complete version
>>below)
>>
>>I am wondering if running the file as a database be slower / faster?
>>
>>
>>
>>Then I read that lme4 does it quicker and more accurately
>>
>>so I thought that I should re-run the code but from the for line:
>>
>>
>>
>>
>>>for (i in 1:nb)
>>
>>+ fit[[i]] <- lmer(trans1 ~ Index1 + grp + (1|id:grp) + (1|id),
>>
>>+ Dataset, na.action = na.exclude)
>>
>>
>>
>>Producing
>>
>>
>>
>>Error in lmer(trans1 ~ Index1 + grp + (1 | id:grp) + (1 | id),
>
> Dataset,
>
>>:
>>
>> flist[[2]] must be a factor of length 200000
>>
>>In addition: Warning messages:
>>
>>1: numerical expression has 200000 elements: only the first used in:
>>id:grp
>>
>>2: numerical expression has 200000 elements: only the first used in:
>>id:grp
>
>
> Check
>
> str(Dataset)
>
> and, if necessary, convert id to a factor with
>
> Dataset$id <- factor(Dataset$id)
>
>
> In is not surprising that you are running into memory problems. Look
> at the size of one of the fitted objects from lme or from lmer. They
> are very large because they contain a copy of the model frame (the
> parts of Dataset that are needed to evaluate the model) plus a lot of
> other information. You have a large Dataset and you are saving
> multiple copies of it although I must admit that I don't understand
> why the calls to lme or lmer are in a loop.
>
>
>
> ???? ?"? ???? ????
> http://mail.nana.co.il
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
--
Spencer Graves, PhD
Senior Development Engineer
PDF Solutions, Inc.
333 West San Carlos Street Suite 700
San Jose, CA 95110, USA
spencer.graves@pdf.com
www.pdf.com <http://www.pdf.com>
Tel: 408-938-4420
Fax: 408-280-7915
???? ?"? ???? ????
http://mail.nana.co.il
[[alternative HTML version deleted]]
Hi,
Your last email was excellent and I got it working using the following.
It did the job extremely quickly, more so a commercial piece of software
my university uses!
#This works fine -
Dataset <- read.table("C:/Program
Files/R/rw2011/data/miss/model1.dat",
header=TRUE, sep="\t", na.strings="NA",
dec=".",strip.white=TRUE)
Dataset$id <- factor(Dataset$id)
attach(Dataset)
names (Dataset)
require (nlme)
fit <- vector(mode="list", 100)
i = 0
for (i in 1:100)
{ Dataset2 <- subset(Dataset, subset= i == runnb)
summary (fit[[i]]<- lme (trans1 ~ Index1 + grp,
random = ~ 1 | id / grp ,
data = Dataset2,
na.action = "na.exclude")
)
i =i +1
}
Strangely enough I was less lucky with LMER.
> for (i in 1:100)
+ { Dataset2 <- subset(Dataset, subset= i == runnb)
+ fit[[i]] <- lmer(trans1 ~ Index1 + grp + (1|id:grp) + (1|id),
Dataset2, na.action = na.exclude)
+ i =i +1
+ }
Error in lmer(trans1 ~ Index1 + grp + (1 | id:grp) + (1 | id), Dataset2,
:
flist[[2]] must be a factor of length 4000
In addition: Warning messages:
1: numerical expression has 4000 elements: only the first used in:
id:grp
2: numerical expression has 4000 elements: only the first used in:
id:grp
>
Many thanks to you all for your assistance on this issue.
Regards
Stephen
________________________________
From: Douglas Bates [mailto:dmbates@gmail.com]
Sent: Sun 26/06/2005 17:01
To: Stephen
Cc: r-help@stat.math.ethz.ch
Subject: Re: [R] Mixed model
On 6/26/05, Stephen <szlevine@nana.co.il> wrote:>
>
>
> Hi All,
>
>
>
> I am currently conducting a mixed model. I have 7 repeated measures on
a> simulated clinical trial. If I understand the model correctly, the
> outcome is the measure (as a factor) the predictors are clinical group
> and trial (1-7). The fixed factors are the measure and group. The
random> factors are the intercept and id and group.
>
>
>
> I tried using 2 functions to calculate mixed effects.
>
> Following previous correspondence .
>
>
>
> Dataset <- read.table("C:/Program
Files/R/rw2011/data/miss/model1a.dat",> header=TRUE, sep="\t", na.strings="NA",
dec=".", strip.white=TRUE)
>
> attach(Dataset)
>
>
>
> require (nlme)
>
> with(Dataset, table(runnb, id, grp))
>
> b.lvls <- table(Dataset$runnb)
>
> nb <- length(b.lvls)
>
> fit <- vector(mode="list", nb)
>
>
>
> for(i in 1:nb)
>
> fit[[i]]<- lme (trans1 ~ Index1 + grp,
>
> random = ~ 1 | id / grp ,
>
> data = Dataset,
>
> na.action = "na.exclude")
>
>
>
>
>
> This (above) worked OK only I am having memory problems.
>
> I have a gig of RAM set at --sdi --max-mem-size=512M (complete version
> below)
>
> I am wondering if running the file as a database be slower / faster?
>
>
>
> Then I read that lme4 does it quicker and more accurately
>
> so I thought that I should re-run the code but from the for line:
>
>
>
> > for (i in 1:nb)
>
> + fit[[i]] <- lmer(trans1 ~ Index1 + grp + (1|id:grp) + (1|id),
>
> + Dataset, na.action = na.exclude)
>
>
>
> Producing
>
>
>
> Error in lmer(trans1 ~ Index1 + grp + (1 | id:grp) + (1 | id),
Dataset,> :
>
> flist[[2]] must be a factor of length 200000
>
> In addition: Warning messages:
>
> 1: numerical expression has 200000 elements: only the first used in:
> id:grp
>
> 2: numerical expression has 200000 elements: only the first used in:
> id:grp
Check
str(Dataset)
and, if necessary, convert id to a factor with
Dataset$id <- factor(Dataset$id)
In is not surprising that you are running into memory problems. Look
at the size of one of the fitted objects from lme or from lmer. They
are very large because they contain a copy of the model frame (the
parts of Dataset that are needed to evaluate the model) plus a lot of
other information. You have a large Dataset and you are saving
multiple copies of it although I must admit that I don't understand
why the calls to lme or lmer are in a loop.
???? ?"? ???? ????
http://mail.nana.co.il
[[alternative HTML version deleted]]