Dear List: I'm working to construct a very large sparse matrix and have found relief using the SparseM package. I have encountered an issue that is confusing to me and wonder if anyone may be able to suggest a smarter solution. The matrix I'm creating is a covariance matrix for a larger research problem that is subsequently used in a simulation. Below is the latex form of the matrix if anyone wants to see the pattern I am trying to create. The core of my problem seems to localize to the last line of the following portion of code. n<-sample.size*4 k<-n/4 vl.mat <- as.matrix.csr(0, n, n) block <- 1:k #each submatrix size for(i in 1:3) vl.mat[i *k + block, i*k + block] <- LE When the variable LE is 0, the matrix is easily created. For example, when sample.size = 10,000 this matrix was created on my machine in about 1 second. Here is the object size.> object.size(vl.mat)[1] 160692 However, when LE is any number other than 0, the code generates an error. For example, when I try LE <- 2 I get Error: cannot allocate vector of size 781250 Kb In addition: Warning message: Reached total allocation of 1024Mb: see help(memory.size) Error in as.matrix.coo(as.matrix.csr(value, nrow = length(rw), ncol length(cl))) : Unable to find the argument "x" in selecting a method for function "as.matrix.coo" I'm guessing that single digit integers should occupy the same amount of memory. So, I'm thinking that the matrix is "less sparse" and the problem is related to the introduction of a non-zero element (seems obvious). However, the matrix still retains a very large proportion of zeros. In fact, there are still more zeros than non-zero elements. Can anyone suggest a reason why I am not able to create this matrix? I'm at the limit of my experience and could use a pointer if anyone is able to provide one. Many thanks, Harold P.S. The matrix above is added to another matrix to create the covariance matrix below. The code above is designed to create the portion of the matrix \sigma^2_{vle}\bm{J} . \begin{equation} \label{vert:cov} \bm{\Phi} = var \left [ \begin{array}{c} Y^*_{1}\\ Y^*_{2}\\ Y^*_{3}\\ Y^*_{4}\\ \end{array} \right ] \left [ \begin{array}{cccc} \sigma^2_{\epsilon}\bm{I}& \sigma^2_{\epsilon}\rho\bm{I} & \bm{0} & \bm{0}\\ \sigma^2_{\epsilon}\rho\bm{I} & \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J} & \sigma^2_{\epsilon}\rho^2\bm{I} & \bm{0}\\ \bm{0} & \sigma^2_{\epsilon}\rho^2\bm{I} & \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J}& \sigma^2_{\epsilon}\rho^3\bm{I}\\ \bm{0} & \bm{0} & \sigma^2_{\epsilon}\rho^3\bm{I}& \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J} \\ \end{array} \right] \end{equation} where $\bm{I}$ is the identity matrix, $\bm{J}$ is the unity matrix, and $\rho$ is the autocorrelation. [[alternative HTML version deleted]]
Your statements for sample.size=10,000 try to construct a matrix with 3 dense 10,000 x 10,000 blocks. That's approximately 10*10*8 MB each and very likely explains your error message. Since you have a simple formula for the matrix, why not define a function to implement multiplication by this matrix (and whatever else you want to do with it), rather than use general sparse matrix representations? Reid Huntsinger -----Original Message----- From: r-help-bounces at stat.math.ethz.ch [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Doran, Harold Sent: Monday, April 18, 2005 4:55 PM To: r-help at stat.math.ethz.ch Subject: [R] Construction of a large sparse matrix Dear List: I'm working to construct a very large sparse matrix and have found relief using the SparseM package. I have encountered an issue that is confusing to me and wonder if anyone may be able to suggest a smarter solution. The matrix I'm creating is a covariance matrix for a larger research problem that is subsequently used in a simulation. Below is the latex form of the matrix if anyone wants to see the pattern I am trying to create. The core of my problem seems to localize to the last line of the following portion of code. n<-sample.size*4 k<-n/4 vl.mat <- as.matrix.csr(0, n, n) block <- 1:k #each submatrix size for(i in 1:3) vl.mat[i *k + block, i*k + block] <- LE When the variable LE is 0, the matrix is easily created. For example, when sample.size = 10,000 this matrix was created on my machine in about 1 second. Here is the object size.> object.size(vl.mat)[1] 160692 However, when LE is any number other than 0, the code generates an error. For example, when I try LE <- 2 I get Error: cannot allocate vector of size 781250 Kb In addition: Warning message: Reached total allocation of 1024Mb: see help(memory.size) Error in as.matrix.coo(as.matrix.csr(value, nrow = length(rw), ncol length(cl))) : Unable to find the argument "x" in selecting a method for function "as.matrix.coo" I'm guessing that single digit integers should occupy the same amount of memory. So, I'm thinking that the matrix is "less sparse" and the problem is related to the introduction of a non-zero element (seems obvious). However, the matrix still retains a very large proportion of zeros. In fact, there are still more zeros than non-zero elements. Can anyone suggest a reason why I am not able to create this matrix? I'm at the limit of my experience and could use a pointer if anyone is able to provide one. Many thanks, Harold P.S. The matrix above is added to another matrix to create the covariance matrix below. The code above is designed to create the portion of the matrix \sigma^2_{vle}\bm{J} . \begin{equation} \label{vert:cov} \bm{\Phi} = var \left [ \begin{array}{c} Y^*_{1}\\ Y^*_{2}\\ Y^*_{3}\\ Y^*_{4}\\ \end{array} \right ] \left [ \begin{array}{cccc} \sigma^2_{\epsilon}\bm{I}& \sigma^2_{\epsilon}\rho\bm{I} & \bm{0} & \bm{0}\\ \sigma^2_{\epsilon}\rho\bm{I} & \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J} & \sigma^2_{\epsilon}\rho^2\bm{I} & \bm{0}\\ \bm{0} & \sigma^2_{\epsilon}\rho^2\bm{I} & \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J}& \sigma^2_{\epsilon}\rho^3\bm{I}\\ \bm{0} & \bm{0} & \sigma^2_{\epsilon}\rho^3\bm{I}& \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J} \\ \end{array} \right] \end{equation} where $\bm{I}$ is the identity matrix, $\bm{J}$ is the unity matrix, and $\rho$ is the autocorrelation. [[alternative HTML version deleted]] ______________________________________________ R-help at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
The dense blocks are too big as Reid has already written -- for smaller instances of this sort of thing I would suggest that the the kronecker product %x% operator in SparseM, would be more convenient. url: www.econ.uiuc.edu/~roger Roger Koenker email rkoenker at uiuc.edu Department of Economics vox: 217-333-4558 University of Illinois fax: 217-244-6678 Champaign, IL 61820 On Apr 18, 2005, at 3:54 PM, Doran, Harold wrote:> Dear List: > > I'm working to construct a very large sparse matrix and have found > relief using the SparseM package. I have encountered an issue that is > confusing to me and wonder if anyone may be able to suggest a smarter > solution. The matrix I'm creating is a covariance matrix for a larger > research problem that is subsequently used in a simulation. Below is > the > latex form of the matrix if anyone wants to see the pattern I am trying > to create. > > The core of my problem seems to localize to the last line of the > following portion of code. > > n<-sample.size*4 > k<-n/4 > vl.mat <- as.matrix.csr(0, n, n) > block <- 1:k #each submatrix size > for(i in 1:3) vl.mat[i *k + block, i*k + block] <- LE > > When the variable LE is 0, the matrix is easily created. For example, > when sample.size = 10,000 this matrix was created on my machine in > about > 1 second. Here is the object size. > >> object.size(vl.mat) > [1] 160692 > > However, when LE is any number other than 0, the code generates an > error. For example, when I try LE <- 2 I get > > Error: cannot allocate vector of size 781250 Kb > In addition: Warning message: > Reached total allocation of 1024Mb: see help(memory.size) > Error in as.matrix.coo(as.matrix.csr(value, nrow = length(rw), ncol > length(cl))) : > Unable to find the argument "x" in selecting a method for > function "as.matrix.coo" > > I'm guessing that single digit integers should occupy the same amount > of > memory. So, I'm thinking that the matrix is "less sparse" and the > problem is related to the introduction of a non-zero element (seems > obvious). However, the matrix still retains a very large proportion of > zeros. In fact, there are still more zeros than non-zero elements. > > Can anyone suggest a reason why I am not able to create this matrix? > I'm > at the limit of my experience and could use a pointer if anyone is able > to provide one. > > Many thanks, > Harold > > > P.S. The matrix above is added to another matrix to create the > covariance matrix below. The code above is designed to create the > portion of the matrix \sigma^2_{vle}\bm{J} . > > > \begin{equation} > \label{vert:cov} > \bm{\Phi} = var > \left [ > \begin{array}{c} > Y^*_{1}\\ > Y^*_{2}\\ > Y^*_{3}\\ > Y^*_{4}\\ > \end{array} > \right ] > > \left [ > \begin{array}{cccc} > \sigma^2_{\epsilon}\bm{I}& \sigma^2_{\epsilon}\rho\bm{I} & \bm{0} & > \bm{0}\\ > \sigma^2_{\epsilon}\rho\bm{I} & > \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J} & > \sigma^2_{\epsilon}\rho^2\bm{I} & \bm{0}\\ > \bm{0} & \sigma^2_{\epsilon}\rho^2\bm{I} & > \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J}& > \sigma^2_{\epsilon}\rho^3\bm{I}\\ > \bm{0} & \bm{0} & \sigma^2_{\epsilon}\rho^3\bm{I}& > \sigma^2_{\epsilon}\bm{I}+\sigma^2_{vle}\bm{J} \\ > \end{array} > \right] > \end{equation} > > where $\bm{I}$ is the identity matrix, $\bm{J}$ is the unity matrix, > and > $\rho$ is the autocorrelation. > > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html