R help - Apr 2005 - lme4: apparently different results between 0.8-2 and 0.95-6

I've been using lme4 to fit Poisson GLMMs with crossed random effects. The
data are counts(y) sampled at 55 sites over 4 (n=12) or 5 (n=43) years. Most
models use three fixed effects: x1 is a two level factor; x2 and x3 are
continuous. We are including random intercepts for YEAR and SITE. On
subject-matter considerations, we are also including a random coefficient
for x3 within YEAR.

Neglecting the log link, the model is

	y_{i,j} = x'_i \beta + \eta_i + z'_i \phi_j + \epsilon_{i,j}

where
	i indexes SITE and j indexes YEAR,
      \beta is the vector of fixed effects
	\eta_i in the random intercept for SITE
and
	\phi_j are the random intercept and coefficient for YEAR.

I have written x'_i because the covariates are assumed (reasonably) to be
constant over the 5 years. Thus, obviously, the z'_i = (1, x3_i) are
constant over j as well.


Using lme4 0.8-2 and R 1.9.0 (under Windows), the call

	GLMM(y~x1 + x2 + x3,random = list(YEAR=~1+x3, SITE=~1), data=foo,
family=poisson, offset=log(reps))

seemed to work correctly, so far as I can tell. The fixed effects were
more-or-less consistent with those estimated by an ordinary GLM, and the
random YEAR effects had signs, magnitudes and correlation appeared to be
sensible and consistent with my expectations.

Earlier today, we updated to lme4 0.95-6 and R 2.1.0. When we try to use
lmer to fit the same model, it complains bitterly:

	lmer(y ~ x1 + x2 + x3 + (1 + x3 | YEAR) + (1 | SITE), data=foo,
family=poisson, offset=log(reps))
	Error: Unable to invert singular factor of downdated X'X

Simpler models still work (or at least return):

	lmer(y ~ x1 + x2 + x3 + (1 + x3 | YEAR), ...)

	lmer(y ~ x1 + x2 + x3 + (1 | YEAR) + (1 | SITE), ...)

As I mentioned, the design is unbalanced. But, we get same "invert singular
factor" Error using the balanced subset.

Can anybody advise? Are we using lmer incorrectly? Or is the new error
perhaps telling us that GLMM in 0.8-2 wasn't actually working in some sense?

Best regards

Steve Cumming
Boreal Ecosystems Research Ltd.
780.432.1589

Steve Cumming wrote:
> I've been using lme4 to fit Poisson GLMMs with crossed random effects.
The
> data are counts(y) sampled at 55 sites over 4 (n=12) or 5 (n=43) years.
Most
> models use three fixed effects: x1 is a two level factor; x2 and x3 are
> continuous. We are including random intercepts for YEAR and SITE. On
> subject-matter considerations, we are also including a random coefficient
> for x3 within YEAR.
> 
> Neglecting the log link, the model is
> 
> 	y_{i,j} = x'_i \beta + \eta_i + z'_i \phi_j + \epsilon_{i,j}
> 
> where
> 	i indexes SITE and j indexes YEAR,
>       \beta is the vector of fixed effects
> 	\eta_i in the random intercept for SITE
> and
> 	\phi_j are the random intercept and coefficient for YEAR.
> 
> I have written x'_i because the covariates are assumed (reasonably) to
be
> constant over the 5 years. Thus, obviously, the z'_i = (1, x3_i) are
> constant over j as well.
> 
> 
> Using lme4 0.8-2 and R 1.9.0 (under Windows), the call
> 
> 	GLMM(y~x1 + x2 + x3,random = list(YEAR=~1+x3, SITE=~1), data=foo,
> family=poisson, offset=log(reps))
> 
> seemed to work correctly, so far as I can tell. The fixed effects were
> more-or-less consistent with those estimated by an ordinary GLM, and the
> random YEAR effects had signs, magnitudes and correlation appeared to be
> sensible and consistent with my expectations.
> 
> Earlier today, we updated to lme4 0.95-6 and R 2.1.0. When we try to use
> lmer to fit the same model, it complains bitterly:
> 
> 	lmer(y ~ x1 + x2 + x3 + (1 + x3 | YEAR) + (1 | SITE), data=foo,
> family=poisson, offset=log(reps))
> 	Error: Unable to invert singular factor of downdated X'X
> 
> Simpler models still work (or at least return):
> 
> 	lmer(y ~ x1 + x2 + x3 + (1 + x3 | YEAR), ...)
> 
> 	lmer(y ~ x1 + x2 + x3 + (1 | YEAR) + (1 | SITE), ...)
> 
> As I mentioned, the design is unbalanced. But, we get same "invert
singular
> factor" Error using the balanced subset.
> 
> Can anybody advise? Are we using lmer incorrectly? Or is the new error
> perhaps telling us that GLMM in 0.8-2 wasn't actually working in some
sense?
I think you are using lmer correctly and that deep in the code there is 
a glitch related to exactly the circumstances you outlined - a model 
matrix with more than one column for one grouping factor (i.e. 
(x3|YEAR)) and a model matrix with a different number of columns for 
another grouping factor (1|SITE)).  Because, as you mentioned, YEAR and 
SITE are crossed, the internal representation gets more complicated than 
  it would be if you had nested grouping factors.

The fact that you can fit the simpler models (thanks for checking that) 
is a strong indication where the problem is.

Is it possible for you to share the data with Deepayan Sarkar and me? 
If so it would help us track down the bug.  Please contact us off list 
if you can do so.  In the meantime I suggest using GLMM for this 
particular problem if you still have access to the old copy of the lme4 
package.  If not, contact us off list and we will provide you with a 
replacement package that can be used in conjunction with the current lme4.

This is an exemplary bug report.  You did a great job of isolating the 
problem.