Displaying 20 results from an estimated 4000 matches similar to: "R-Squared with biglm?"
2009 Feb 19
1
Questions about biglm
Hello folks,
I am very excited to have discovered R and have been exploring its
capabilities. R's regression models are of great interest to me as my
company is in the business of running thousands of linear regressions
on large datasets.
I am using biglm to run linear regressions on datasets that are as
large as several GB's. I have been pleasantly surprised that biglm
runs the
2011 Jul 25
1
biglm() and NeweyWest()
Dear all,
I am working on a large dataset and need to use biglm() to perform OLS
regressions. I have detected significant ARCH effects which I try to account
for using the Newey-West correction.
So far, I have worked with NeweyWest() in the sandwich package. NeweyWest()
however seems to be unable to handle an object of class "biglm".
Looking into the code, I figured out that
2010 Jun 15
1
help biglm.big.matrix; problem with weights
Hello colleagues,
I have tried to use the package biglm. I want to specify a
multivariate regression with a weight.
I have imported a large dataset with the library(bigmemory). I load
the library (biglm) and specified a regression with a weight. But I
get everytime a error message like ?object not found? or ?`weights'
must be a formula? or "error in eval(expr, envir, enclos)". I
2010 Oct 31
1
biglm: how it handles large data set?
I am trying to figure out why 'biglm' can handle large data set...
According to the R document - "biglm creates a linear model object that uses
only p^2 memory for p variables. It can be updated with more data using
update. This allows linear regression on data sets larger than memory."
After reading the source code below? I still could not figure out how
'update'
2009 Mar 20
1
Using predict on a biglm object returns NA
Hi R experts,
I used biglm to construct a model (which has categorical variables).
When I run predict on the model output on a new data (for testing) or on the
same data, I get only NA's. I'm able to run predict with some other models
constructed with biglm. One reason I suspect is that the model itself has a
few undefined terms (NA's). I'm wondering if there's any way to
2011 Nov 15
1
getting R2 (goodness of fit) result after using biglm()
Hello. I had been struggling with running linear regression using
lm() primarily because my data has a few categorical variables with at
least a thousand levels.
I tried the biglm() function and it worked.
My problem now is that i don't know how to get the R2 results. Could
someone help?
Thanks,
sean
2009 Feb 21
1
variable/model selction (step/stepAIC) for biglm ?
Hello dear R mailing list members.
I have recently became curious of the possibility applying model
selection algorithms (even as simple as AIC) to regressions of large
datasets. I searched as best as I could, but couldn't find any
reference or wrapper for using step or stepAIC to packages such as
biglm.
Any ideas or directions of how to implement such a concept ?
Best,
Tal
--
2007 Jan 22
1
Example function for bigglm (biglm) data input from file
This is to submit a commented example function for use in the data
argument to the bigglm(biglm) function, when you want to read the data
from a file (instead of a URL), or rescale or modify the data before
fitting the model. In the hope that this may be of help to someone out
there.
make.data <- function (filename, chunksize, ...) {
conn<-NULL;
function (reset=FALSE) {
if
2009 Feb 25
0
leaps and biglm
New versions of leaps and biglm are percolating through CRAN.
The new version of biglm fixes a bug in sandwich standard errors with weights, and adds predict(), deviance() and AIC() methods [based on code from Christophe Dutang].
The new version of leaps adds a regsubsets() method for biglm objects, so that the subset selection algorithms can be run efficiently on large data sets.
-thomas
2009 Feb 25
0
leaps and biglm
New versions of leaps and biglm are percolating through CRAN.
The new version of biglm fixes a bug in sandwich standard errors with weights, and adds predict(), deviance() and AIC() methods [based on code from Christophe Dutang].
The new version of leaps adds a regsubsets() method for biglm objects, so that the subset selection algorithms can be run efficiently on large data sets.
-thomas
2011 Jan 10
1
debug biglm response error on bigglm model
G'morning
What does the error message "Error in x %*% coef(object) : non-
conformable arguments" indicate when calculating the response values
for
newdata with a model from bigglm (in package biglm), and how can I
debug it? I am attempting to do Monte Carlo simulations, which may
explain the loop in the code that follows. After the code I
have included the output, which shows that
2009 Mar 11
1
regsubsets() [leaps package] - please share some good examples of use
Hello dear R-help members,
I recently became interested in using biglm with leaps, and found myself
somewhat confused as to how to use the two together, in different settings.
I couldn't find any example codes for the leaps() package (except for in the
help file, and the examples there are not as rich as they could be). That
is why I turn to you in case you could share some good tips and
2008 Aug 17
1
package building problem on windows
Hi,
I'm trying to compile the package biglm, but when I build it with R
CMD build biglm, it failed :
C:\LOCAL\c-dutang\code\R\biglm2>R CMD build biglm
* checking for file 'biglm/DESCRIPTION' ... OK
* preparing 'biglm':
* checking DESCRIPTION meta-information ...C:/DOCUME~1/c-dutang/Local:
Can't op
n C:/DOCUME~1/c-dutang/Local: No such file or directory
2007 Feb 12
0
predict on biglm class
Hi Everyone,
I often use the 'safe prediction' feature available through glm().
Now, I'm at a situation where I must use biglm:::bigglm.
## begin example
library(splines)
library(biglm)
ff <- log(Volume)~ns(log(Girth), df=5)
fit.glm <- glm(ff, data=trees)
fit.biglm <- bigglm(ff, data=trees)
predict(fit.glm, newdata=data.frame(Girth=2:5))
## -1.3161465 -0.2975659
2010 Jun 16
0
biglm.big.matrix: Problem with weighting
Hello colleagues,
I have tried to use the package bigmemory, biganalytics and biglm. I
want to specify a multivariate regression with a weight.
I have imported a large dataset with the library(bigmemory). I load the
library (biglm) and specified a regression with a weight. But I get
everytime an error message like "object not found" or "`weights' must be
a
2012 Jan 03
0
Biglm source code alternatives (E.g. Call to Fortran)
Hi everyone,
I have been looking at the Bigglm (Basically does Generalised Linear Models
for big data under the Biglm package) command and I have done some profiling
on this code and found that to do a GLM on a 100mb file (9 million rows by 5
columns matrix(most of the numbers were either a 0,1 or 2 randomly
generated)) it took about 2 minutes on a linux machine with 8gb of RAM and 4
cores.
2009 Apr 27
0
VIF's in R using BIGLM
Dear R-help
This is a follow-up to my previous post here:
http://groups.google.com/group/r-help-archive/browse_thread/thread/d9b6f87ce06a9fb7/e9be30a4688f239c?lnk=gst&q=dobomode#e9be30a4688f239c
I am working on developing an open-source automated system for running
batch-regressions on very large datasets. In my previous post, I posed
the question of obtaining VIF's from the output of
2011 Nov 03
0
anova or liklihood ratio test from biglm output
(Sorry if this is a repost, I got a bounce reply from the r-help server)
Hi,
I’m using the biglm() function to create some linear models for a very
large data set than lm() can’t fit due to memory issues (the problem is
with the number of interactions, I can fit the main effects model)
I need to determine if the 2-way interactions are necessary or not. Ideally
I’d like to use anova() to
2007 Oct 23
0
Residuals from biglm package
Hi all,
first of all, I'm not an expert on R, I'm still learning, so sorry if this is a stupid question...
I have a large dataset that is to big for my computer memory, and I found quite useful the package biglm. Now everything is working perfectly. But if I want the residuals, how I can do it?
Let's say that we are running the example:
> data(trees)>
2010 Dec 26
1
Calculation of BIC done by leaps-package
Hi Folks,
I've got a question concerning the calculation of the Schwarz-Criterion
(BIC) done by summary.regsubsets() of the leaps-package:
Using regsubsets() to perform subset-selection I receive an regsubsets
object that can be summarized by summary.regsubsets(). After this
operation the resulting summary contains a vector of BIC-values
representing models of size i=1,...,K.
My problem