R help - Sep 2008 - Difficulty understanding sem errors / failed confirmatory factor analysis

Hello,

 	I'm trying to fit a pretty simple confirmatory factor analysis using
the sem package. There's a CFA example in the examples, which is helpful,
but the output for my (failing) model is hard to understand. I'd be
interested in any other ways to do a CFA in R, if this proves troublesome.

 	The CFA is replicating a 5 uncorrelated-factor structure (for those
interested, it is a structure of word usage patterns in weblogs) in a
special population. The model looks like model.txt (attached as many people
hate long emails); the correlation matrix cors.txt as well.

 	I'm setting no overlap between factors, no correlation between
factors, and estimating a separate variance for each observed variable
(which should be everything on the right-hand side of the -> arrows), but
setting the factor variances equal to 1...pretty standard. I've ensured that
everything is typed correctly to the best I am able.

 	The problem:

library(sem)
model.kr <- specify.model(file="model.txt") # printing it checks
out ok
correl <- read.csv("cors.csv", header=TRUE) # printing it checks
out ok
kr.sem <- sem(ram=model.kr,S=correl,N=3034)
...about 10 seconds pass...
Warning message:
In sem.default(ram = ram, S = S, N = N, param.names = pars, var.names = vars,  :
   Could not compute QR decomposition of Hessian.
Optimization probably did not converge.

(running qr on correl works fine; randomly-generated correl matrices fail in
the same way; I do not know how to further troubleshoot this)

...and then the model itself (which is produced, as the above was just a
warning):

summary(kr.sem)
Error in data.frame(object$coeff, se, z, 2 * (1 - pnorm(abs(z))), par.code) :
   arguments imply differing number of rows: 47, 0

...both of these error messages are beyond my ability to troubleshoot. Any
help would be greatly appreciated. Because I am unsure what exactly the
problem with this analysis is, I can't create a simpler example for testing
purposes...but I think my model and correlation matrix are fairly simple.
> unlist(R.Version())
                       platform                           arch
     "x86_64-unknown-linux-gnu"                      
"x86_64"
                             os                         system
                    "linux-gnu"            "x86_64,
linux-gnu"
                         status                          major
                             ""                           
"2"
                          minor                           year
                          "7.2"                        
"2008"
                          month                            day
                           "08"                          
"25"
                        svn rev                       language
                        "46428"                           
"R"
                 version.string 
"R version 2.7.2 (2008-08-25)"

...sem installed via install.packages("sem") which I assume is
current.

Cordially,
Adam Kramer

Dear Adam,
> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at
r-project.org]
On
> Behalf Of Adam D. I. Kramer
> Sent: September-18-08 1:02 PM
> To: r-help at r-project.org
> Subject: [R] Difficulty understanding sem errors / failed confirmatory
factor
> analysis
> 
> Hello,
> 
>  	I'm trying to fit a pretty simple confirmatory factor analysis using
> the sem package. There's a CFA example in the examples, which is
helpful,
> but the output for my (failing) model is hard to understand. I'd be
> interested in any other ways to do a CFA in R, if this proves troublesome.
> 
>  	The CFA is replicating a 5 uncorrelated-factor structure (for those
> interested, it is a structure of word usage patterns in weblogs) in a
> special population. The model looks like model.txt (attached as many
people
> hate long emails); the correlation matrix cors.txt as well.
As far as I can see, the attachments aren't there. If you like, you can send
them to me privately. Without the input covariance matrix and your model,
it's very hard to tell what the source of the problem is, but one guess
(assuming that you've specified the model correctly) is that the assumption
of uncorrelated factors is too far off. Also see below.
> 
>  	I'm setting no overlap between factors, no correlation between
> factors, and estimating a separate variance for each observed variable
> (which should be everything on the right-hand side of the -> arrows),
but
> setting the factor variances equal to 1...pretty standard. I've ensured
that
> everything is typed correctly to the best I am able.
> 
>  	The problem:
> 
> library(sem)
> model.kr <- specify.model(file="model.txt") # printing it
checks out ok
> correl <- read.csv("cors.csv", header=TRUE) # printing it
checks out ok
> kr.sem <- sem(ram=model.kr,S=correl,N=3034)
> ...about 10 seconds pass...
> Warning message:
> In sem.default(ram = ram, S = S, N = N, param.names = pars, var.names vars,
> :
>    Could not compute QR decomposition of Hessian.
> Optimization probably did not converge.
> 
> (running qr on correl works fine; randomly-generated correl matrices fail
in
> the same way; I do not know how to further troubleshoot this)
Doing a QR decomposition on the correlation matrix of the data is
essentially irrelevant. The issue is the Hessian. (The scaled inverse
Hessian is the covariance matrix of the parameter estimates, not of the
data.) That you observe similar problems for randomly generated covariance
matrices may or may not be troublesome, depending upon how you generated
them.
> 
> ...and then the model itself (which is produced, as the above was just a
> warning):
> 
> summary(kr.sem)
> Error in data.frame(object$coeff, se, z, 2 * (1 - pnorm(abs(z))),
par.code) :
>    arguments imply differing number of rows: 47, 0
If the Hessian isn't positive-definite, it won't be possible to get
estimated coefficient standard errors. I suspect that this is the source of
this error message. If so, it would be better for summary.sem() to provide a
more informative error message.

Regards,
 John
> 
> ...both of these error messages are beyond my ability to troubleshoot. Any
> help would be greatly appreciated. Because I am unsure what exactly the
> problem with this analysis is, I can't create a simpler example for
testing
> purposes...but I think my model and correlation matrix are fairly simple.
> 
> > unlist(R.Version())
>                        platform                           arch
>      "x86_64-unknown-linux-gnu"                      
"x86_64"
>                              os                         system
>                     "linux-gnu"            "x86_64,
linux-gnu"
>                          status                          major
>                              ""                           
"2"
>                           minor                           year
>                           "7.2"                        
"2008"
>                           month                            day
>                            "08"                          
"25"
>                         svn rev                       language
>                         "46428"                           
"R"
>                  version.string
> "R version 2.7.2 (2008-08-25)"
> 
> ...sem installed via install.packages("sem") which I assume is
current.
> 
> Cordially,
> Adam Kramer
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

No new info, but the model and correlation table are pasted at the end of
this message.

--Adam

On Thu, 18 Sep 2008, Adam D. I. Kramer wrote:
> Hello,
>
> 	I'm trying to fit a pretty simple confirmatory factor analysis using
> the sem package. There's a CFA example in the examples, which is
helpful,
> but the output for my (failing) model is hard to understand. I'd be
> interested in any other ways to do a CFA in R, if this proves troublesome.
>
> 	The CFA is replicating a 5 uncorrelated-factor structure (for those
> interested, it is a structure of word usage patterns in weblogs) in a
> special population. The model looks like model.txt (attached as many people
> hate long emails); the correlation matrix cors.txt as well.
>
> 	I'm setting no overlap between factors, no correlation between
> factors, and estimating a separate variance for each observed variable
> (which should be everything on the right-hand side of the -> arrows),
but
> setting the factor variances equal to 1...pretty standard. I've ensured
that
> everything is typed correctly to the best I am able.
>
> 	The problem:
>
> library(sem)
> model.kr <- specify.model(file="model.txt") # printing it
checks out ok
> correl <- read.csv("cors.csv", header=TRUE) # printing it
checks out ok
> kr.sem <- sem(ram=model.kr,S=correl,N=3034)
> ...about 10 seconds pass...
> Warning message:
> In sem.default(ram = ram, S = S, N = N, param.names = pars, var.names =
vars,
> :
>  Could not compute QR decomposition of Hessian.
> Optimization probably did not converge.
>
> (running qr on correl works fine; randomly-generated correl matrices fail
in
> the same way; I do not know how to further troubleshoot this)
>
> ...and then the model itself (which is produced, as the above was just a
> warning):
>
> summary(kr.sem)
> Error in data.frame(object$coeff, se, z, 2 * (1 - pnorm(abs(z))), par.code)
:
>  arguments imply differing number of rows: 47, 0
>
> ...both of these error messages are beyond my ability to troubleshoot. Any
> help would be greatly appreciated. Because I am unsure what exactly the
> problem with this analysis is, I can't create a simpler example for
testing
> purposes...but I think my model and correlation matrix are fairly simple.
>
>> unlist(R.Version())
>                      platform                           arch
>    "x86_64-unknown-linux-gnu"                      
"x86_64"
>                            os                         system
>                   "linux-gnu"            "x86_64,
linux-gnu"
>                        status                          major
>                            ""                           
"2"
>                         minor                           year
>                         "7.2"                        
"2008"
>                         month                            day
>                          "08"                          
"25"
>                       svn rev                       language
>                       "46428"                           
"R"
>                version.string "R version 2.7.2 (2008-08-25)"
>
> ...sem installed via install.packages("sem") which I assume is
current.
>
> Cordially,
> Adam Kramer
>
model.kr <- specify.model()
Melancholy -> Affect, mel.aff, NA
Melancholy -> Negemo, mel.neg, NA
Melancholy -> Sad,  mel.sad, NA
Melancholy -> Physcal, mel.phys, NA
Melancholy -> Body, mel.phys, NA
Melancholy -> Eating, mel.eat, NA
Melancholy -> Groom, NA, 1
Social  -> numlines, soc.nrow, NA
Social  -> Leisure, soc.leis, NA
Social  -> Home, soc.home, NA
Social  -> Sports, soc.sports, NA
Social  -> TV,  soc.tv, NA
Social  -> Music, soc.mus, NA
Social  -> Money, NA, 1
Rant  -> Swear, rant.swear, NA
Rant  -> Sexual, rant.sex, NA
Rant  -> Anger, rant.anger, NA
Rant  -> I,  NA, 1
Metaphysical -> Metaph, met.met, NA
Metaphysical -> Relig, met.relig, NA
Metaphysical -> Death, NA, 1
Work  -> Occup, work.occ, NA
Work  -> School, work.school, NA
Work  -> Job,  NA, 1
Affect <-> Affect,Affect.var,NA
Negemo <-> Negemo,Negemo.var,NA
Sad <-> Sad,Sad.var,NA
Physcal <-> Physcal,Physcal.var,NA
Body <-> Body,Body.var,NA
Eating <-> Eating,Eating.var,NA
Groom <-> Groom,Groom.var,NA
numlines <-> numlines,numlines.var,NA
Leisure <-> Leisure,Leisure.var,NA
Home <-> Home,Home.var,NA
Sports <-> Sports,Sports.var,NA
TV <-> TV,TV.var,NA
Music <-> Music,Music.var,NA
Money <-> Money,Money.var,NA
Swear <-> Swear,Swear.var,NA
Sexual <-> Sexual,Sexual.var,NA
Anger <-> Anger,Anger.var,NA
I <-> I,I.var,NA
Metaph <-> Metaph,Metaph.var,NA
Relig <-> Relig,Relig.var,NA
Death <-> Death,Death.var,NA
Occup <-> Occup,Occup.var,NA
School <-> School,School.var,NA
Job <-> Job,Job.var,NA
Melancholy <-> Melancholy, NA, 1
Social <-> Social, NA, 1
Rant <-> Rant, NA, 1
Work <-> Work, NA, 1
Metaphysical <-> Metaphysical, NA, 1

correl <- matrix(0,nrow=24,ncol=24)
correl[lower.tri(correl,diag=TRUE)] <- c(1, 0.940530496413442,
0.765560263936915, 0.705665939921134,
0.659038546655712, 0.282665099938120, 0.234892888297051, 0.0554321360979252,
0.671137592040541, 0.54382418910777, 0.463922205203901, 0.353418190097785, 
0.414864918025334, 0.436177075274485, 0.401019650838241, 0.370116202378091,
0.777297151879925, 0.676782523496444, 0.384244495774914, 0.233569710080454,
0.381213315694389, 0.792870007525815, 0.50437548997674, 0.700522809676332, 
1, 0.789549528191674, 0.706300687178796, 0.677210560231246,
0.260692661416488, 0.229468929161093, 0.0715008916402315, 0.575592411881254,
0.495811001574585, 0.387005398515213, 0.286945225104112, 0.332994303548624,
0.380017238311269, 0.430036391324925, 0.331754269605511, 0.841233992941869, 
0.648407420058087, 0.383101710409617, 0.226088422466275, 0.38890278815819, 
0.694900967907394, 0.445767105915179, 0.611121179516979, 1,
0.554261042721096, 0.527433237517158, 0.195884095023418, 0.180722683807294,
0.0233909540224985, 0.516740406669213, 0.435018065577118, 0.357120136043065,
0.256978759119895, 0.310082859183584, 0.346636599600033, 0.286753730820772,
0.260480159024979, 0.549021337624957, 0.48888735524692, 0.310472252813801, 
0.162629025914762, 0.341951482059891, 0.604151525748714, 0.404455472396666,
0.513232045563814, 1, 0.926617967581845, 0.496550925591883,
0.348467198716147, 0.061642221232731, 0.520273451850585, 0.454688691566968,
0.361179853804786, 0.251729488407341, 0.276070295108034, 0.301418864648534, 
0.416591776932984, 0.444945330539674, 0.559869838931842, 0.544386905545782, 
0.27126759338533, 0.133247574277541, 0.310271891666118, 0.521537483862256,
0.311771883417551, 0.471080247067523, 1, 0.296375815811106,
0.245934996387214, 0.0955552805445385, 0.452337692426552, 0.363463874290988,
0.339558959480213, 0.244564602900861, 0.253767733666974, 0.271411427718931,
0.391479465009918, 0.232142608383647, 0.518262010456117, 0.491601506679358,
0.267644288407907, 0.125392653581526, 0.314025855170014, 0.481718318983948,
0.28021913467524, 0.425176692670869, 1, 0.183300375385584, 
0.033752849675936, 0.251307861560496, 0.273349357055731, 0.154752778367666,
0.0726227557418435, 0.0810855443617927, 0.169413646643786, 0.14876182751722,
0.090027807464796, 0.21707073520028, 0.227245091073637, 0.0648985279708279,
0.0184155556622148, 0.0917329040389644, 0.224131495587368,
0.141524409848304, 0.225250463174961, 1, 0.0422205901067624,
0.335813173017425, 0.462314781501668, 0.137068724813020, 0.0723839601971734,
0.0727494211937804, 0.112428911506304, 0.09425456138948, 0.0599407261374292,
0.190826379314435, 0.211704707365072, 0.0599306146160989,
0.0296829331196419, 0.0682294826412284, 0.188274984783352, 
0.139422045025596, 0.161353066485886, 1, 0.05257676892534,
0.0280480438748768, 0.00910257508927212, 0.0871857793285287,
0.0160729045542020, 0.0569498365312614, -0.0278792722216208,
-0.064821779008319, 0.0611534438874241, -0.0649834249949666,
0.0355509895650055, 0.0135445926170511, 0.0457566918082741,
0.0604365669560489, 0.0585041719421536, 0.0453745903935696, 1,
0.761816109289408, 0.589874357357904, 0.649504824987434, 0.658362125608313,
0.342115375404423, 0.236861363583780, 0.248095510733153, 0.443279264097818,
0.492471838105872, 0.248701650770021, 0.128552959879838, 0.276153894635528,
0.607556471316203, 0.449181155519957, 0.510375756616414, 1,
0.247939328658872, 0.239982254184636, 0.241566313660839, 0.287799155446014, 
0.209294226000604, 0.159562148116037, 0.379361578341484, 0.497577130930334,
0.218564559998989, 0.09910467906514, 0.260723180575602, 0.467924257159533,
0.326606718662009, 0.418920180615036, 1, 0.279500783890266,
0.315054486693380, 0.222378322027149, 0.175009967321649, 0.171392982263514,
0.303685681481775, 0.277189930927861, 0.112262899972587, 0.0486869353393796,
0.136799339182926, 0.439080141905746, 0.335750129078362, 0.357071201278175,
1, 0.354042555500904, 0.186671191628747, 0.0988194936238798,
0.161389983251376, 0.225124570709853, 0.166070213591245, 0.121050680740651,
0.0555233084604175, 0.143574719627254, 0.314332705080321, 0.233696321192899,
0.250453706966593, 1, 0.194173439107269, 0.128637091334456,
0.188288180271912, 0.249261729284129, 0.294002924006152, 0.180881281305379,
0.130825345306321, 0.152316871694807, 0.400492580214257, 0.313557675560858,
0.314387144865789, 1, 0.184856884327958, 0.147800700051061,
0.314348653487276, 0.223052727856440, 0.160836428547767, 0.0788465159084819,
0.184166318589758, 0.464779622013576, 0.296714228371417, 0.509475424226691,
1, 0.243783177939616, 0.433294149172899,
0.385435708448960,0.165204287365281, 0.122878967887846, 0.134705298271121,
0.292692353692189,
0.175825474932413, 0.302111138660379, 1, 0.325702026715680,
0.265660847381981, 0.144919463251553, 0.105698215649586, 0.120886293524908,
0.266994210674339, 0.158210355462592, 0.256685716586081, 1,
0.50128293661388, 0.376612773737787, 0.200216055489841, 0.410973414330935, 
0.544902004754666, 0.340576879662518, 0.495895604908182, 1,
0.210485715826384, 0.162841078754691, 0.163460255178148, 0.530490890488822,
0.327408998238198, 0.501112169084101, 1, 0.834421294834807,
0.697570273879324, 0.284190669063073, 0.165736914780320, 0.243184491397893,
1, 0.187175885786238, 0.178409162228837, 0.107264487137595,
0.157214632976541, 1, 0.274591668601347, 0.155954924109372,
0.229057302047291, 1, 0.824656000910955, 0.82923430112462, 1,
0.476782357161847, 1)