similar to: generalized inverse using matinv (Design)

Here's a skeletal example. Embellish as needed: p <- 5 n <- 300 set.seed(1) dat <- cbind(rnorm(n), matrix(runif(n * p), n, p)) write.table(dat, file="c:/temp/big.txt", row=FALSE, col=FALSE) xtx <- matrix(0, p + 1, p + 1) xty <- numeric(p + 1) f <- file("c:/temp/big.txt", open="r") for (i in 1:3) { x <- matrix(scan(f, nlines=100), 100,

I'd like to fit linear models on very large datasets. My data frames are about 2000000 rows x 200 columns of doubles and I am using an 64 bit build of R. I've googled about this extensively and went over the "R Data Import/Export" guide. My primary issue is although my data represented in ascii form is 4Gb in size (therefore much smaller considered in binary), R consumes about

Dear all, My double for loop as follows, but it is little efficient, I hope all friends can give me a "vectorized" program to replace my code. thanks x: is a matrix 202*263, that is 202 samples, and 263 independent variables num.compd<-nrow(x); # number of compounds diss.all<-0 for( i in 1:num.compd) for (j in 1:num.compd) if (i!=j) { S1<-sum(x[i,]*x[j,])

Hi all, I am using R trying to get a inverse matrix of (X^T)X , but I keep getting the error message like: no b argument and no default value for sprintf(gettext(fmt, domain = domain), ...) . -------------------------------------------------------------------------------------------- # my code X<-Matrix(rep(1,500),100,5) X[lower.tri(X)]<-1-10^-7 XtX<- t(X)%*% X XtXu<-lu(XtX)

I am cleaning up the rms package to not export functions not to be called directly by users. rms uses generic functions defined in other packages. For example there is a latex method in the Hmisc package, and rms has a latex method for objects of class "anova.rms" so there are anova.rms and latex.anova.rms functions in rms. I use:

Dear All, After playing with lmer for couple of days, I have to say that I am amazed! I've been using quite some multilevel/mixed modeling packages, lme4 is a strong candidate for the overall winner, especially for multilevel generzlized linear models. Now go back to my two-level poisson model with cross-classified model. I've been testing various different model specificatios for the

dear R experts: ?I am writing my own little newey-west standard error function, with heteroskedasticity and arbitrary x period autocorrelation corrections. ?including my function in this post here may help others searching for something similar. it is working quite well, except on occasion, it complains that Error in solve.default(crossprod(x.na.omitted, x.na.omitted)) : system is

Hi, is there a function in R that will give me the variances of a predicted values obtained using predict.lm(). If no function is available I would need to calculate them myself - which involves taking the inverse of X'X (' indicating transpose) where X is my model matrix. I know that calculating an inverse directly is not a good idea in general - could anybody suggest a way around

Dear R users, I'm wondering about the singular.ok option in lm. If singular.ok is set to TRUE does it mean that we allow the generalized inverse of (X'X)^-1 (where X are the independent variables and ' denotes the transpose)? Or is the singularity handled in some other way? Thank you! Best regards, Martin. ======================================== Martin Eklund PhD Student

Suppose I have the following hat matrix: H=X(X'X)^{-1}X' X is a n by p matrix, where n >= p and X_{i,1} = 1 I'm wondering why H1 = 1. (Here, 1 is column vector, whose each element is the number 1) Thank you!

The lm() function in R seems to handle the inversion of singular X'X matrices (where there is collinearity between regression inputs) in a way where one of the inputs is dropped and this also seems to be the default behavior in SAS (please let me know if i'm wrong about this). In some other packages (i.e. octave ols() function) the pseudo inverse is computed where singular values less

I have the following data set > data A B X1 X2 Y 1 A1 B1 1.1 2.9 1.2 2 A1 B2 1.0 3.2 2.3 3 A2 B1 1.0 3.3 1.6 4 A2 B2 0.5 2.6 3.1 > sapply(data, class) A B X1 X2 Y "factor" "factor" "numeric" "numeric" "numeric" I'd like to set a specific type of contrasts to all the categorical factors

Hello: I am trying to understand the method 'hatvalues(...)', which returns something similar to the diagonals of the plain vanilla hat matrix [X(X'X)^(-1)X'], but not quite.  A Fortran programmer I am not, but tracing through the code it looks like perhaps some sort of correction based on the notion of 'leave-one-out' variance is being applied. Whatever the

It was a real surprise, but a student in my class found that the function eigen is buggy. He traced to the problem from his inability of getting principal component analysis to work on his data. Chong Gu Here is a matrix I generated through X'X, where X is 2x3. > jj [,1] [,2] [,3] [1,] 0.8288469 -1.269783 -0.7533517 [2,] -1.2697829 2.162132 2.0262917 [3,]

Please forgive a statistics question. I know that a simple bivariate linear regression, y=f(x) or in R parlance lm(y~x) can be solved using the variance-covariance matrix: beta(x)=covariance(x,y)/variance(x). I also know that a linear regression with multiple independent variables, for example y=f(x,z) can also be solved using the variance-covariance matrix, but I don't know how to do this.

thank you, achim. I will try chol2inv. sandwich is a very nice package, but let me make some short suggestions. I am not a good econometrician, so I do not know what prewhitening is, and the vignette did not explain it. "?coeftest" did not work after I loaded the library. automatic bandwidth selection can be a good thing, but is not always. as to my own little function, I like the

Dear All, I have conducted an experiment in order to examine predation pressure in the surroundings of potential wildlife road-crossing structures. I have documented predation occurrence (binary?) in these structures and calculated several possible explanatory variables describing the spatial heterogeneity in several scales. At the landscape scale I have calculated the percentage of different

Here's my little discussion example for a quadratic regression: http://pj.freefaculty.org/R/WorkingExamples/regression-quadratic-1.R Students press me to know the benefits of poly() over the more obvious regression formulas. I think I understand the theory on why poly() should be more numerically stable, but I'm having trouble writing down an example that proves the benefit of this. I

Full_Name: Mike Danilov Version: 2.9.0 OS: Fedora Core 9 Submission from: (NULL) (142.103.121.198) When checking for the symmetry of a matrix, function isSymmetric.matrix() gets confused by the discrepancy of colnames/rownames if its argument. See the code snippet below. Perhaps it's a problem of the matrix product which copies colnames of the first argument but not the rownames of the

Dear List: I was able to run the following in nlme successfully, but the same model and code (same dataset) failed to run in lme4 and gave me the error message below. Any thoughts? lme(math~year, data=egsingle, random=~year|schoolid/childid) Error in lme(formula = math ~ year, data = egsingle, random = structure(list( : Unable to invert singular factor of downdated X'X