search for: biglm

Hello folks, I am very excited to have discovered R and have been exploring its capabilities. R's regression models are of great interest to me as my company is in the business of running thousands of linear regressions on large datasets. I am using biglm to run linear regressions on datasets that are as large as several GB's. I have been pleasantly surprised that biglm runs the regressions extremely fast (one regression may take minutes in SPSS vs seconds in R). I have been trying to wrap my head around biglm and have a couple of questions. 1...

Hi, I'm trying to compile the package biglm, but when I build it with R CMD build biglm, it failed : C:\LOCAL\c-dutang\code\R\biglm2>R CMD build biglm * checking for file 'biglm/DESCRIPTION' ... OK * preparing 'biglm': * checking DESCRIPTION meta-information ...C:/DOCUME~1/c-dutang/Local: Can't op n C:/DOCUME~1/c-...

Dear all, I am working on a large dataset and need to use biglm() to perform OLS regressions. I have detected significant ARCH effects which I try to account for using the Newey-West correction. So far, I have worked with NeweyWest() in the sandwich package. NeweyWest() however seems to be unable to handle an object of class "biglm". Looking into t...

I am trying to figure out why 'biglm' can handle large data set... According to the R document - "biglm creates a linear model object that uses only p^2 memory for p variables. It can be updated with more data using update. This allows linear regression on data sets larger than memory." After reading the source code...

Hello colleagues, I have tried to use the package biglm. I want to specify a multivariate regression with a weight. I have imported a large dataset with the library(bigmemory). I load the library (biglm) and specified a regression with a weight. But I get everytime a error message like ?object not found? or ?`weights' must be a formula? or "er...

Hi Everyone, I often use the 'safe prediction' feature available through glm(). Now, I'm at a situation where I must use biglm:::bigglm. ## begin example library(splines) library(biglm) ff <- log(Volume)~ns(log(Girth), df=5) fit.glm <- glm(ff, data=trees) fit.biglm <- bigglm(ff, data=trees) predict(fit.glm, newdata=data.frame(Girth=2:5)) ## -1.3161465 -0.2975659 0.4251285 0.9856938 predict(fit.biglm, newdata...

I've been working with a rather large data set (~10M rows), and while biglm works beautifully for generating coefficients, it does not report an r-squared. It does report RSS. Any idea on how one could coax an R-squared out of biglm? Thanks in advance for any help with this! Bryan Lim Lecturer Department of Finance University of Melbourne [[alternative HTML ve...

Hello. I had been struggling with running linear regression using lm() primarily because my data has a few categorical variables with at least a thousand levels. I tried the biglm() function and it worked. My problem now is that i don't know how to get the R2 results. Could someone help? Thanks, sean

...rchive/browse_thread/thread/d9b6f87ce06a9fb7/e9be30a4688f239c?lnk=gst&q=dobomode#e9be30a4688f239c I am working on developing an open-source automated system for running batch-regressions on very large datasets. In my previous post, I posed the question of obtaining VIF's from the output of BIGLM. With a lot of help from Assoc. Professor, Biostatistics Thomas Lumley at University of Washington, I was able to make significant progress, but ultimately got stuck. The following post describes the steps and reasoning I undertook in trying to accomplish this task. Please note that I am not a stat...

Hello colleagues, I have tried to use the package bigmemory, biganalytics and biglm. I want to specify a multivariate regression with a weight. I have imported a large dataset with the library(bigmemory). I load the library (biglm) and specified a regression with a weight. But I get everytime an error message like "object not found" or "`weights' must be a for...

...o dear R mailing list members. I have recently became curious of the possibility applying model selection algorithms (even as simple as AIC) to regressions of large datasets. I searched as best as I could, but couldn't find any reference or wrapper for using step or stepAIC to packages such as biglm. Any ideas or directions of how to implement such a concept ? Best, Tal -- ---------------------------------------------- My contact information: Tal Galili Phone number: 972-50-3373767 FaceBook: Tal Galili My Blogs: www.talgalili.com www.biostatistics.co.il

Hi R experts, I used biglm to construct a model (which has categorical variables). When I run predict on the model output on a new data (for testing) or on the same data, I get only NA's. I'm able to run predict with some other models constructed with biglm. One reason I suspect is that the model itself has a few un...

Hi everyone, I have been looking at the Bigglm (Basically does Generalised Linear Models for big data under the Biglm package) command and I have done some profiling on this code and found that to do a GLM on a 100mb file (9 million rows by 5 columns matrix(most of the numbers were either a 0,1 or 2 randomly generated)) it took about 2 minutes on a linux machine with 8gb of RAM and 4 cores. Ideally I want to run t...

If a package defined an S3 generic and an S4 generic for the same function (so as to add methods for S4 classes to the existing code), how do I set up the namespace to have them exported? With import(stats) exportMethods(bigglm) importClassesFrom(DBI) useDynLib(biglm) export(biglm) export(bigglm) in NAMESPACE, the S3 generic is not exported. > methods("bigglm") [1] bigglm.RODBC* bigglm.data.frame* bigglm.function* Non-visible functions are asterisked Warning messages: 1: In findGeneric(generic.function, parent.frame()) : 'bigglm...

New versions of leaps and biglm are percolating through CRAN. The new version of biglm fixes a bug in sandwich standard errors with weights, and adds predict(), deviance() and AIC() methods [based on code from Christophe Dutang]. The new version of leaps adds a regsubsets() method for biglm objects, so that the subset selection...

Hi all, first of all, I'm not an expert on R, I'm still learning, so sorry if this is a stupid question... I have a large dataset that is to big for my computer memory, and I found quite useful the package biglm. Now everything is working perfectly. But if I want the residuals, how I can do it? Let's say that we are running the example: > data(trees)> ff<-log(Volume)~log(Girth)+log(Height)> > chunk1<-trees[1:10,]> chunk2<-trees[11:20,]> chunk3<-trees[21:31,]> > a &...

New versions of leaps and biglm are percolating through CRAN. The new version of biglm fixes a bug in sandwich standard errors with weights, and adds predict(), deviance() and AIC() methods [based on code from Christophe Dutang]. The new version of leaps adds a regsubsets() method for biglm objects, so that the subset selection...

(Sorry if this is a repost, I got a bounce reply from the r-help server) Hi, I’m using the biglm() function to create some linear models for a very large data set than lm() can’t fit due to memory issues (the problem is with the number of interactions, I can fit the main effects model) I need to determine if the 2-way interactions are necessary or not. Ideally I’d like to use anova() to get...

This is to submit a commented example function for use in the data argument to the bigglm(biglm) function, when you want to read the data from a file (instead of a URL), or rescale or modify the data before fitting the model. In the hope that this may be of help to someone out there. make.data <- function (filename, chunksize, ...) { conn<-NULL; function (reset=FALSE) { if (...

...roblems if you can give some hints: Problem-1: The model summary and residual deviance matched (in the mail below) but I didn't understand why AIC is still different. > AIC(m1) [1] 532965 > AIC(m1big_longer) [1] 101442.9 Problem-2: chunksize argument is there in bigglm but not in biglm, consequently, udate.biglm is there, but not update.bigglm Is my observation correct? If yes, why is this difference? Regards Utkarsh / / From: Thomas Lumley <tlumley_at_u.washington.edu <mailto:tlumley_at_u.washington.edu?Subject=Re:%20%5BR%5D%20bigglm%28%29%20results%20different%20fro...