similar to: speed?

dear R wizards: X is factor with 20,000*20=800,000 observations of 20,000 factors. I.e., each factor has 20 observations. y is 800,000 normally distributed data points. I want to see how much R^2 the X factors can provide. Easy, right? > lm ( y ~ X) and > aov( y ~ X) Error: cannot allocate vector of size 3125000 Kb is this computationally infeasible? (I am not an expert, but

dear R wizards: I am trying to determine how to determine (no further recursion) where a built-in function is defined. In particular, I have decided I am going to add sd() to the existing basic summary function, rather than try to rewrite my own summary() function from scratch. So, I just installed R-2.2.1 (via gentoo; eventually I will figure out how to get atlas/sse/sse2 working on amd64,

dear R experts---now I have a case where I want to estimate very large regression models with many fixed effects---not just the mean type, but cross-fixed effects---years, months, locations, firms. Many millions of observations, a few thousand variables (most of these variables are interaction fixed effects). could someone please point me to packages, if any, that would help me estimate such

Dear R wizards: is there a clean way to assign to elements in a list? what I would like to do, in pseudo R+perl notation is f <- function(a,b) list(a+b,a-b) (c,d) <- f(1,2) and have c be assigned 1+2 and d be assigned 1-2. right now, I use the clunky x <- f(1,2) c <- x[[1]] d <- x[[2]] rm(x) which seems awful. is there a nicer syntax? regards, /iaw ---- Ivo Welch

dear R experts: has someone written a function that returns the results of by() as a data frame? ??of course, this can work only if the output of the function that is an argument to by() is a numerical vector. presumably, what is now names(byobject) would become a column in the data frame, and the by object's list elements would become columns. it's a little bit like flattening the by()

Dear R Wizards---is it possible to get R to show its current call stack (sys.calls()) upon an error abort? I don't use ESS for execution, and it is often not obvious how to locate how I triggered an error in an R internal function. Seeing the call stack would make this easier. (right now, I sprinkle "cat" statements everywhere, just to locate the line where the error appears.) Of

dear R wizards: an operation I execute often is the deletion of all observations (in a matrix or data set) that have at least one NA. (I now need this operation for kde2d, because its internal quantile call complains; could this be considered a buglet?) usually, my data sets are small enough for speed not to matter, and there I do not care whether my method is pretty inefficient (ok, I

dear R wizards: I have a large data frame, a million rows, 40 columns. In this data frame, there are some (about 100,000) rows which I want to recompute (update), while I want to leave others just as is. this is based on a condition that I need to compute, based on what is in a few of the columns. what is the right R way to do this? I could subset out the rows that I want to recompute into a

dear R wizards---more ignorance on my part, exacerbated by too few examples in the function documentations. > d <- data.frame( id=rep(1:3,3), x=rnorm(9), y=rnorm(9)) Question 1: how do I work with the output of "by"? for example, > b <- by( d, d$id, function(x) coef(lm( y ~ x, data=x ) )) > b d$id: 1 (Intercept) x 0.2303 0.3618

dear R wizards: here is the strange question for the day. It seems to me that nrow() is very slow. Let me explain what I mean: ds= data.frame( NA, x=rnorm(10000) ) ## a sample data set > system.time( { for (i in 1:10000) NA } ) ## doing nothing takes virtually no time user system elapsed 0.000 0.000 0.001 ## this is something that should take time; we need to add 10,000

dear R wizards:? I am wrestling with reshape.? I have a long data set that I want to convert into a wide data set, in which rows are firms and columns are years. > summary(rin) firm fyear sim1 Min. :1004.00 Min. :1964.0 Min. : -1.00000 1st Qu.:1010.00 1st Qu.:1979.0 1st Qu.: -0.14334 Median :1016.00 Median :1986.0 Median : 0.00116 Mean

Dear R wizards: This must have an obvious solution, but I am stumped. I can run a linear regression giving a matrix as the independent set of variables, but if I give a data frame (which I would like to give, because it should tell the linear model the names of the variables), R does not like it. An example is: N=20; y= rnorm(N); x.m <- (matrix( nrow=N, ncol=2 )); x.m[,1]=rnorm(N);

Dear R wizards: I decided to take the advice in the R data import/export manual and want to learn how to work with SQL for large data sets. I am trying SQLite with the DBI and RSQLite database interfaces. Speed is nice. Alas, I am struggling to find a tutorial that is geared for the kind of standard operations that I would want in R. Simple things: * how to determine the number of rows in a

Dear R wizards: I am playing (and struggling) with the example in the constrOptim function. simple example. let's say I want to constrain my variables to be within -1 and 1. I believe I want a whole lot of constraints where ci is -1 and ui is either -1 or 1. That is, I have 2*N constraints. Should the following work? N=10 x= rep(1:N) ci= rep(-1, 2*N) ui= c(rep(1, N), rep(-1, N))

dear wizards: my .Rprofile has just one command for testing, loadhistory("~/.Rhistory") but this gives me an error on R startup: Error: couldn't find function "loadhistory" Invoking loadhistory() as the first interactive command works fine; incidentally, I believe loadhistory() in the .Rprofile worked in earlier or other platform R releases, too. Is the .Rprofile

This email is intended for R users that are not that familiar with R internals and are searching google about how to speed up R. Despite common misperception, R is not slow when it comes to iterative access. R is fast when it comes to matrices. R is very slow when it comes to iterative access into data frames. Such access occurs when a user uses "data$varname[index]", which is a very

dear readers---I struggled with how to do nice fixed-effects regressions in large economic samples for a while. Eventually, I realized that nlme is not really what I needed (too complex), and all I really wanted is the plm package. so, I thought I would share a quick example. ################ sample code to show fixed-effects models? in R # create a sample panel data set with firms and years

Dear R wizards---I read Brian Ripley's post from 2004 which said that it was not possible to print to STDERR. Alas, I have more modest needs. I was wondering if it was possible to just send a string to STDERR with cat() while in CMD BATCH mode. Is it not possible to open STDERR in R? (Or does R use STDERR for itself and redirect it into the output stream?) This would be on a standard Unix

Dear R wizards--- I have a wrapper on mclapply() that makes it a little easier for me to do multiprocessing. (Posting this may make life easier for other googlers.) I pass a data frame, a vector that tells me what rows should be recomputed, and the function; and I get back a vector or matrix of answers. d <- data.frame( id=1:6, val=11:16 ) loc <- c(TRUE,TRUE,FALSE,TRUE,FALSE,TRUE)

I think I found a bug in the Cairo library, plus weird behavior in both the Cairo and the normal pdf device. The baseline of the spades symbol seems to be off. This is easier to show than it is to explain. The problem does not appear in the normal pdf device, which is why I am guessing this is a Cairo bug. moreover, I cannot figure out why three of the card symbols seem to be transparent, but