similar to: Skipping columns to save memory

I am reading in a very large file with names in it and R is truncating the number of rows it reads in. The separator in this file is a pipe '|' and so I use dat <- read.delim('pathToMyFile', header= TRUE, sep='|') It turns out that it is reading up to row 61145 and stopping and I think I see why, but am not sure of the best solution to this problem. I see the name of

Quite possibly, and I?ll look into that. Aside from the work I was doing, however, I wonder if there is a way such that sapply could avoid the overhead of having to call the identical function to determine the conditional path. From: William Dunlap [mailto:wdunlap at tibco.com] Sent: Tuesday, March 13, 2018 12:14 PM To: Doran, Harold <HDoran at air.org> Cc: Martin Morgan <martin.morgan

FYI, in R devel (to become 3.5.0), there's isFALSE() which will cut some corners compared to identical(): > microbenchmark::microbenchmark(identical(FALSE, FALSE), isFALSE(FALSE)) Unit: nanoseconds expr min lq mean median uq max neval identical(FALSE, FALSE) 984 1138 1694.13 1218.0 1337.5 13584 100 isFALSE(FALSE) 713 761 1133.53 809.5 871.5

Could your code use vapply instead of sapply? vapply forces you to declare the type and dimensions of FUN's output and stops if any call to FUN does not match the declaration. It can use much less memory and time than sapply because it fills in the output array as it goes instead of calling lapply() and seeing how it could be simplified. Bill Dunlap TIBCO Software wdunlap tibco.com On Tue,

Martin In terms of context of the actual problem, sapply is called millions of times because the work involves scoring individual students who took a test. A score for student A is generated and then student B and such and there are millions of students. The psychometric process of scoring students is complex and our code makes use of sapply many times for each student. The toy example used

You?re right, it sure does. My suggestion causes it to fail when simplify = ?array? From: William Dunlap [mailto:wdunlap at tibco.com] Sent: Tuesday, March 13, 2018 12:11 PM To: Doran, Harold <HDoran at air.org> Cc: r-help at r-project.org Subject: Re: [R] Possible Improvement to sapply Wouldn't that change how simplify='array' is handled? > str(sapply(1:3,

>>>>> Henrik Bengtsson <henrik.bengtsson at gmail.com> >>>>> on Tue, 13 Mar 2018 10:12:55 -0700 writes: > FYI, in R devel (to become 3.5.0), there's isFALSE() which will cut > some corners compared to identical(): > > microbenchmark::microbenchmark(identical(FALSE, FALSE), isFALSE(FALSE)) > Unit: nanoseconds > expr

Dear List: When I click on the link to download a reference manual for a package on cran, I get an error message that the file is damaged and could not be repaired. I randomly chose various packages and the same error message appears. Are the links actually broken? I have also restarted my machine and closed and re-opened acrobat. I am using Windows XP, Acrobat Professional 6.0.0.5, and

Dear List I apologize for the multiple postings. After being in the weeds on this problem for a while I think my original post may have been a little cryptic. I think I can be clearer. Essentially, I need the following a <- c(2,3) b <- c(4,5,6) (2*4) + (2*5) + (2*6) + (3*4) + (3*5) +(3*6) But I do not know of a built in function that would do this. Any suggestions? -----Original

Wouldn't that change how simplify='array' is handled? > str(sapply(1:3, function(x)diag(x,5,2), simplify="array")) int [1:5, 1:2, 1:3] 1 0 0 0 0 0 1 0 0 0 ... > str(sapply(1:3, function(x)diag(x,5,2), simplify=TRUE)) int [1:10, 1:3] 1 0 0 0 0 0 1 0 0 0 ... > str(sapply(1:3, function(x)diag(x,5,2), simplify=FALSE)) List of 3 $ : int [1:5, 1:2] 1 0 0 0 0 0 1 0 0

Hi all: I'm having difficulty with something I believe is very simple, but I'm stuck. I have a large data frame that took days to clean and prepare. All I now need to do is concatenate three variables into a single column. For example, I have tenn$up, tenn$down, and tenn$stable which all have values of 1 or 0. I simply want to put all three columns together to create a pattern (e.g.,

Dear R: Am I having trouble downloading the LME4 library. I am using Windows and am using ver 1.7 I have tried the following: 1) Install package from CRAN, but LME4 is not listed 2) Downloaded LME4 from http://cran.us.r-project.org/, however, I cannot open the file when I try install from local drive. I get the following error: Error in file(file, "r") : unable to open connection

Just out of curiosity, has anyone ever written a function that sends an email to you when an R process has finished? For instance, I often work with very large data sets and certain tasks (e.g., merging records, lmer runs) can take a long time and I find myself constantly looking over to see if R is done. But, it would just be neat if there was a way R could send me an email alerting me that a

While working with sapply, the documentation states that the simplify argument will yield a vector, matrix etc "when possible". I was curious how the code actually defined "as possible" and see this within the function if (!identical(simplify, FALSE) && length(answer)) This seems superfluous to me, in particular this part: !identical(simplify, FALSE) The preceding

I have some matrices stored as elements in a list that I am working with. On example is provided below as TP[[18]] > TP[[18]] level2 level1 1 2 3 4 1 79 0 0 0 2 0 0 0 0 3 0 0 0 0 4 0 0 0 0 Now, using prop.table on this gives > prop.table(TP[[18]],1) level2 level1 1 2 3 4 1 1 0 0 0 2 3

First off, Marc Schwartz posted this link earlier today, read it. http://cran.r-project.org/doc/FAQ/R-FAQ.html#Why-are-p_002dvalues-not-di splayed-when-using-lmer_0028_0029_003f Second, your email is not really descriptive enough. I have no idea what OR is, so I have no reaction. Third, you're comparing estimates from different methods of estimation. lmer will give standard errors that

I need to wrap a loop inside a function and am having a small bit of difficulty getting the results I need. Below is a replicable example. # define functions pcm <- function(theta,d,score){ exp(rowSums(outer(theta,d[1:score],'-')))/ apply(exp(apply(outer(theta,d, '-'), 1, cumsum)), 2, sum) } foo <- function(theta,items, score){ like.mat <-

This is great. Thank you for your help, but let me make sure I fully understand. Here is the looping file I use to subset the data frame, create a tex file, and Sweave it. This results in N number of tex files where N is equal to the number of rows in the data frame. list <- unique(wide$stuid) master = "master.tex" for (i in list){ tmp1 <- subset(wide, stuid==i) tmp2

Dear List: I am trying to find the area between two ECDFs. I am examining the gap in performance between two groups, males and females on a student achievement test in math, which is a continuous metric. I start by creating a subset of the dataframe male<-subset(datafile, female="Male") female<-subset(datafile, female="Female") I then plot the two CDFs via

I have matrices stored within a list like something as follows: a <- list(matrix(rnorm(50), ncol=5), matrix(rnorm(50), ncol=5)) b <- list(matrix(rnorm(50), nrow=5), matrix(rnorm(50), nrow=5)) I don't recall how to perform matrix multiplication on each list element such that the result is a new list result <- list(a[[1]]%*%b[[1]], a[[2]]%*%b[[2]]) I think I'm close with