similar to: Res: gc() and memory efficiency

I have a program which reads in a very large data set, performs some analyses, and then repeats this process with another data set. As soon as the first set of analyses are complete, I remove the very large object and clean up to try and make memory available in order to run the second set of analyses. The process looks something like this: 1) read in data set 1 and perform analyses rm(list=ls())

I am new to R and would like to get some practice analyzing multilevel data. I wonder if anyone can point me to a sample data set and command lines that I might replicate for a sample session. I would then compare my output with HLM output. Any help is appreciated. ------ Harold C. Doran Director of Research and Evaluation New American Schools 675 N. Washington Street, Suite 220 Alexandria,

Dear List: A few weeks ago I posted some questions regarding data simulation and received some very helpful comments, thank you. I have modified my code accordingly and have made some progress. However, I now am facing a new challenge along similar lines. I am attempting to simulate 250 datasets and then run the data through a linear model. I use rm() and gc() as I move along to clean up the

I'm finalizing development of a package that will include a vignette. Without the vignette, the package builds fine with no warnings and is ready for distribution. Now, I am following the directions for developing vignettes "Sweave, Part II: Package Vignettes" by Friedrich Leisch. I am using a windows XP machine (other session info below). Here is what I have done. 1) I add the

First off, Marc Schwartz posted this link earlier today, read it. http://cran.r-project.org/doc/FAQ/R-FAQ.html#Why-are-p_002dvalues-not-di splayed-when-using-lmer_0028_0029_003f Second, your email is not really descriptive enough. I have no idea what OR is, so I have no reaction. Third, you're comparing estimates from different methods of estimation. lmer will give standard errors that

Dear List: When I click on the link to download a reference manual for a package on cran, I get an error message that the file is damaged and could not be repaired. I randomly chose various packages and the same error message appears. Are the links actually broken? I have also restarted my machine and closed and re-opened acrobat. I am using Windows XP, Acrobat Professional 6.0.0.5, and

Previously, I posed the question pasted down below to the list and received some very helpful responses. While the code suggestions provided in response indeed work, they seem to only work with *very* small data sets and so I wanted to follow up and see if anyone had ideas for better efficiency. I was quite embarrased on this as our SAS programmers cranked out programs that did this in the blink

Dear List: I am running into a memory issue that I haven't noticed before. I am running a simulation with all of the code used below. I have increased my memory to 712mb and have a total of 1 gb on my machine. What appears to be happening is I run a simulation where I create 1,000 datasets with a sample size of 100. I then run each dataset through a gls and obtain some estimates. This works

Dear List: When trying to open 1.9.0 this morning, I have the following error: "Fatal Error: Unable to restore saved data in .Rdata" I am using Windows 2000. The program then quits. Do I need to reinstall? Harold C. Doran One Massachusetts Avenue, NW · Suite 700 Washington, DC 20001-1431 202.336.7075 [[alternative HTML version deleted]]

Dear List: I have a question regarding an MDS procedure that I am accustomed to using. I have searched around the archives a bit and the help doc and still need a little assistance. The package isoMDS is what I need to perform the non-metric scaling, but I am working with similarity matrices, not dissimilarities. The question may end up being resolved simply. Here is a bit of substantive

Dear List: Earlier this week I posted a question and received no response, and I continue to struggle with my model. My original question is pasted below. I am using lme and want to fix the variance of the within group residual at 1 (e~n(0,1). I think the varFixed function should be used to accomplish this, but I am struggling to figure out how to do this. Can anyone offer suggestions on how

Dear List: It appears that simulating data where all dataframes are stored as a list will only work for relatively small analyses. Instead, it appears that creating N individual dataframes, saving them, and loading them when needed is the best way to save memory and make this a feasible task. As such, I now have a new(er) question with respect to dealing with individual files within a loop. To

Dear List I apologize for the multiple postings. After being in the weeds on this problem for a while I think my original post may have been a little cryptic. I think I can be clearer. Essentially, I need the following a <- c(2,3) b <- c(4,5,6) (2*4) + (2*5) + (2*6) + (3*4) + (3*5) +(3*6) But I do not know of a built in function that would do this. Any suggestions? -----Original

Dear List: I am encountering an error that I can't resolve. I'm looping through rows of a dataframe to generate individual tex files using Sweave. At random points along the way, I encounter the following error Error in file() : cannot find unused tempfile name At which point Sweave halts. There isn't a logical pattern that I can identify in terms of why the program stops at

I'm trying to read in some data from a .csv format and have come across the following issue. Here is a simple example for replication # A sample .csv format schid,sch_name 331-802-7081,School One 464-551-7357,School Two 388-517-7627,School Three \& Four 388-517-4394,School Five Note the third line includes the \ character. However, when I read the data in I get >

Dear R: Am I having trouble downloading the LME4 library. I am using Windows and am using ver 1.7 I have tried the following: 1) Install package from CRAN, but LME4 is not listed 2) Downloaded LME4 from http://cran.us.r-project.org/, however, I cannot open the file when I try install from local drive. I get the following error: Error in file(file, "r") : unable to open connection

Dear List: I am having to build a block-diagonal matrix (vl) and am currently using the following code. I<-diag(sample.size) vl<-kronecker(I,vl.mat) This code works fine, but for large N, it is a huge memory hog. Is there a more efficient method for constructing vl? Thanks, Harold [[alternative HTML version deleted]]

Hi all: I'm having difficulty with something I believe is very simple, but I'm stuck. I have a large data frame that took days to clean and prepare. All I now need to do is concatenate three variables into a single column. For example, I have tenn$up, tenn$down, and tenn$stable which all have values of 1 or 0. I simply want to put all three columns together to create a pattern (e.g.,

I would like to fit a hierarchical regression model from Witte et al. (1994; see reference below). It's a logistic regression of a health outcome on quntities of food intake; the linear predictor has the form, X*beta + W*gamma, where X is a matrix of consumption of 82 foods (i.e., the rows of X represent people in the study, the columns represent different foods, and X_ij is the amount of

I have some matrices stored as elements in a list that I am working with. On example is provided below as TP[[18]] > TP[[18]] level2 level1 1 2 3 4 1 79 0 0 0 2 0 0 0 0 3 0 0 0 0 4 0 0 0 0 Now, using prop.table on this gives > prop.table(TP[[18]],1) level2 level1 1 2 3 4 1 1 0 0 0 2 3