search for: bigglm

Dear all, I have a large data set and would like to fit a logistic regression model using the bigglm function. I need to include an offset in the model but when I do this the bigglm function seems to ignore it. For example, running the two models below produces the same model and the offset is ignored bigglm(y~x,offset=z,data=Test,family=binomial(link = "logit")) bigglm(y~x,data=Tes...

If a package defined an S3 generic and an S4 generic for the same function (so as to add methods for S4 classes to the existing code), how do I set up the namespace to have them exported? With import(stats) exportMethods(bigglm) importClassesFrom(DBI) useDynLib(biglm) export(biglm) export(bigglm) in NAMESPACE, the S3 generic is not exported. > methods("bigglm") [1] bigglm.RODBC* bigglm.data.frame* bigglm.function* Non-visible functions are asterisked Warning messages: 1: In findGeneric(generic.funct...

Hi, Until now, I thought that the results of glm() and bigglm() would coincide. Probably a naive assumption? Anyways, I've been using bigglm() on some datasets I have available. One of the sets has >15M observations. I have 3 continuous predictors (A, B, C) and a binary outcome (Y). And tried the following: m1 <- bigglm(Y~A+B+C, family=bino...

...I am facing few problems if you can give some hints: Problem-1: The model summary and residual deviance matched (in the mail below) but I didn't understand why AIC is still different. > AIC(m1) [1] 532965 > AIC(m1big_longer) [1] 101442.9 Problem-2: chunksize argument is there in bigglm but not in biglm, consequently, udate.biglm is there, but not update.bigglm Is my observation correct? If yes, why is this difference? Regards Utkarsh / / From: Thomas Lumley <tlumley_at_u.washington.edu <mailto:tlumley_at_u.washington.edu?Subject=Re:%20%5BR%5D%20bigglm%28%29%20results%...

Dear all, I am using the bigglm package to fit a few GLM's to a large dataset (3 million rows, 6 columns). While trying to fit a Poisson GLM I noticed that the coefficient estimates were very different from what I obtained when estimating the model on a smaller dataset using glm(), I wrote a very basic toy example to com...

Hi, since bigglm doesn't have update, I was wondering how to achieve something like (similar to the example in ff package manual using biglm): first <- TRUE ffrowapply ({ if (first) { first <- FALSE fit <- bigglm(eqn, as.data.frame(bigdata[i1:i2,,drop=FALSE]), chunksize = 10000, family = bin...

Hi, there Since glm cannot handle factors very well. I try to use bigglm like this: logit_model <- bigglm(responser~var1+var2+var3, data, chunksize=1000, family=binomial(), weights=~trial, sandwich=FALSE) fitted <- predict(logit_model, data) only var2 is factor, var1 and var3 are numeric. I expect fitted should be a vector of value falls in (0,1) However, I g...

Hi all, I'm somewhat of a novice in terms of programming, so I thought I'd come here to seek some help with an issue I'm having. I'm trying to model a glm using bigglm, but in spite of my best efforts, I cannot get it to work! Here is the particular line of code that is giving me trouble: >mod = bigglm(Pres/wt ~ Xdes, data=dat, family=poisson(), weights = ~wt, maxit=100, tol=1.e-10, chunksize=(dim(Xdes)[1] + 1)) When I attempt to run this, it gives me the f...

This is to submit a commented example function for use in the data argument to the bigglm(biglm) function, when you want to read the data from a file (instead of a URL), or rescale or modify the data before fitting the model. In the hope that this may be of help to someone out there. make.data <- function (filename, chunksize, ...) { conn<-NULL; function (reset=FALSE) {...

...ic",5),rep("integer",2)), col.names = c("id","dt", "promdt","term", "termfac", "commintr","commbal","issuebal","intr","ri","loctime","resp") ) library(biglm); bigglm (formula = resp ~ poly(relage,2,raw=TRUE)+termfac+ri , data = a, family = binomial(link='logit')); ### output: > bigglm (formula = resp ~ poly(relage,2,raw=TRUE)+termfac+ri , + data = a, family = binomial(link='logit')); Error in is(object, Class) : trying to get slo...

Hello list, I am trying to avoid "Rifying" some of my SAS code to generate ROC plots, and the logistic.display() and lroc() functions in the epicalc package do what I want. However, I must generate my logistic model with bigglm because I have 1) limited hardware, 2) ~2.5 million rows, and 4 categorical and 2 continuous independent variables. When I attempt to invoke epicalc's "lroc" function on the bigglm model R returns the error "Error in logistic.display(mod.glm) : Model not from logistic regression...

G'morning What does the error message "Error in x %*% coef(object) : non- conformable arguments" indicate when calculating the response values for newdata with a model from bigglm (in package biglm), and how can I debug it? I am attempting to do Monte Carlo simulations, which may explain the loop in the code that follows. After the code I have included the output, which shows that the simulations are changing the response and input values, and that there are not any atypic...

...da el error de abajo, he intentado eliminar las filas con valor NA por si esta función no lo admite, pero sigue dando el mismo error. ¿Alguien sabe porqué ocurre? (probé previamente a usar la función glm pero obtenía out of memory) library(XLConnect) library(biglm) library(data.table) GLM <- bigglm(In.hospital_death ~ GCS + BUN, data = DatosGLM, family = binomial(logit)) Obtengo el error: In bigglm.function(formula = formula, data = datafun, ...) : ran out of iterations and failed to converge Un saludo Gracias MªLuz [[alternative HTML version deleted]]

Gracias! El 15 de junio de 2015, 16:54, Freddy Omar López Quintero < freddy.vate01 en gmail.com> escribió: > ?Holap.? > > ran out of iterations and failed to converge > > > ?Prueba aumentando el número de iteraciones, con el argumento maxit: > > ?GLM <- bigglm(In.hospital_death ~ GCS + BUN, data = DatosGLM, family = >> binomial(logit), maxit=1000)? > > > ?Salud.? > > -- > «No soy aquellas sombras tutelares > que honré con versos que no olvida el tiempo.» > > JL Borges > [[alternative HTML version deleted]]

We are wanting to use R instead of/in addition to our existing stats package because of it's huge assortment of stat functions. But, we routinely need to fit GLM models to files that are approximately 2-4GB (as SQL tables, un-indexed, w/tinyint-sized fields except for the response & weight variables). Is this feasible, does anybody know, given sufficient hardware, using R? It appears to

Hi Everyone, I often use the 'safe prediction' feature available through glm(). Now, I'm at a situation where I must use biglm:::bigglm. ## begin example library(splines) library(biglm) ff <- log(Volume)~ns(log(Girth), df=5) fit.glm <- glm(ff, data=trees) fit.biglm <- bigglm(ff, data=trees) predict(fit.glm, newdata=data.frame(Girth=2:5)) ## -1.3161465 -0.2975659 0.4251285 0.9856938 predict(fit.biglm, newdata=data.fra...

Hi R-experts, Does anyone have experience using R for handling large scale data (millions of rows, hundreds or thousands of features)? What is the largest size of data that anyone has used with glm? Also, is there a library to read data in sparse data format (like SVMlight format)? Thanks Pradheep [[alternative HTML version deleted]]

Please, tell what is the best way to create an R glm object with parameters etc estimated in SAS? I have a large dataset and bigglm fails to converge, so estimation is done is SAS. However, there are a lot of predictions that are much more easily done in R and I would like to use glm.predict and the flexibility of R. bigglm ends with an error, so no object is created. Thanks everybody. Stephen [[alternative HTML version del...

...mo se construye una columna nueva que de, para cada cliente y cada evaluación, de el número de días que han pasado desde la primera evaluación del cliente? Qué función se utiliza para verifica el consumo de memoria en las operaciones? Pregunta 2: Cómo se corre un modelo de regresión logística (`bigglm.ffdf`) para el rating =5 (`top box` del rating) en términos de `ClientID` (como factor) y la variable que se cree en el paso anterior (quizá transformada por `log`)? Intenta primero con un subconjunto (dentro de ff) para ver si funciona. Saludos, Tania [[alternative HTML version deleted]]

Greetings useRs, Can anyone provide an example how to use ff to feed a very large data frame to glm? The data.frame cannot be loaded in R using conventional read.csv as it is too big. glm(...,data=ff.file) ?? Thank you Stephen B